Continuous Gossip Warnings (Failed determining organization) causing disk space pressure after old certificates expire (new certificates have been updated)

[sky030b]

Description

Hi everyone,

may related issue: #5111

We are experiencing an issue with our peers and could really use some expert guidance.

After the old certificates of other peers in our channel expired (note: these peers had already been successfully updated with new certificates), we started seeing the following warning in our logs:

WARN [gossip.gossip] func3 -> Unable to determine org of message tag:EMPTY alive_msg:<membership:<endpoint:...

🔺 The first peer whose credentials expired (log starts half an hour after expiration).

Shortly after, a second type of warning started appearing continuously and in massive volumes, which is now causing severe disk space pressure on our machines:

WARN [gossip.gossip] func3 -> Failed determining organization of d5bfxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx5aab5ef8b
WARN [gossip.gossip] func3 -> Failed determining organization of b419xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7f5ac3ad4

🔺 Later, the second warning logs start showing.

Because this second warning storm only triggered after the first peer's old certificate officially expired, we suspect these two events are connected, but we were unable to reproduce the issue there, so we aren't sure this two log are related.🤔

Has anyone encountered this specific behavior where the gossip filter continues to spam these hashes even after a container restart? Any guidance on how to properly clear these ghost identities or stop the log spam would be greatly appreciated!

As can be seen from the graph, it continuously generates repeat log.

Steps to reproduce

Our Environment & Troubleshooting Steps:

Versions Affected: We checked our hyperledger/fabric-peer images. We have two separate environments running v2.5.10 and v2.5.13, and both are experiencing this exact same log spam.

Container Restarts: We have tried restarting the peer containers to clear the memory. We are aware that the v2.5.11 release notes mention a fix regarding "Gossip handling of expired certificates". However, our understanding is that even without that fix (e.g., on v2.5.10), simply restarting the container should clear the idMapper memory and stop the use of the old cache. Despite the restarts, the warnings persist continuously.

Failed Reproduction: To isolate the issue, we built a completely fresh environment from scratch using the v2.5.10 peer image and re-do certificates updated and expired process, but we were unable to reproduce the issue there.

We are currently stuck and aren't entirely sure if our issue is directly related to the v2.5.11 fix, given that v2.5.13 is also affected and container restarts aren't mitigating the log spam.

[yacovm]

However, our understanding is that even without that fix (e.g., on v2.5.10), simply restarting the container should clear the idMapper memory and stop the use of the old cache. Despite the restarts, the warnings persist continuously.

Have you restarted them at the same time? Maybe try to shutdown all containers, and then start them up after they're all shut down?

[sky030b]

Thank you for your reply. We are currently trying this method. Our system architecture has approximately 2x peers. We are currently collecting logs before and after each peer's restart, as well as the image version of each peer. Most are running version 2.5.10, only our peer is running 2.5.13.

Observing the logs, we found that almost every peer had previously seen some 'Failed determining organization' logs, and these same logs did not reappear within ten minutes of the restart. We will then observe whether these warning messages reappear.

However, there is a more challenging issue. Because our organization's peer is one of the main peers responsible for endorsement, it receives a higher number of 'Failed determining organization' messages. Even after restarting other peers (not all), our restart does not show any reduction in the number of warnings; they continue to cycle in groups of roughly the same number. We are unsure if this phenomenon can be improved after all peers have restarted. 🫣

Additional information: We are using a restart instead of a shut-down and then-up, and we are not shutting down all peers simultaneously, but rather allowing each peer to restart individually. I scheduled the restart myself, while other peers were still running. I wonder if these differences will have any impact.

[yacovm]

if you're not shutting down all peers simultaneously it could be that the identity gets replicated from nodes that are not yet restarted to nodes that have already been restarted and therefore survives the purge.

[sky030b]

Thank you for your reply.

Oh no, for our business and maintenance scenarios, it might be difficult for us to simultaneously shut down all peers and then restart them individually. I will discuss this with our team in more detail. Thank you.

But I would also like to ask, if we cannot shut down them simultaneously, can we solve this problem by having each peer update to the latest peer version 2.5.x individually (but not at the same time)?
This approach might be more feasible for us.

[yacovm]

Yeah you should try upgrading all peers to the latest version.

[sky030b]

Okay, thank you for your reply. I will pass this information on to our team. Thank you.

[sky030b]

Hi, thank you for your previous suggestions.

We noticed that the warning messages continuously popping up on our end seem slightly different from the ones mentioned in #5111. Since we haven't found similar reports from others, we'd like to ask: do you think these two warning messages are related under the hood?

Additionally, we observed the logs of peers across different orgs after restarting them. From the moment of restart up until more than 48 hours later, we haven't observed the Failed determining organization warning again on most peers. However, even after restarting all peers, our endorsing peer (which was also restarted) continues to generate this exact warning spam, and the amount of one group per loop hasn't decreased at all.

You mentioned earlier that shutting down all peers simultaneously or upgrading to the latest version could potentially resolve this. To help us build a strong case for our management team to approve the version upgrade, could you elaborate on the root causes behind why these two methods work? Specifically, which code changes or PRs in the newer versions explicitly address and fix this issue? Any concrete technical context would be greatly appreciated!

Thanks again for your time and help.

[yacovm]

It could be that you have a peer that hasn't joined a channel but it is designated to be an anchor peer in the channel config of some channel it hasn't joined to.

Do you have peers that don't have these messages? Or do all peers have these messages all the time?

Maybe you can attach the logs of the peers?

[sky030b]

Last Friday, we tried stopping all the peer containers under our org first (there is another peer1 under our org that isn't exposed externally but is still running normally) and then starting them all, and we found that the Failed determining organization logs stopped being generated. However, I'm not entirely sure why our previous restart didn't solve the problem(peer1 had already restarted at that time, and peer1 originally had no related warning logs); perhaps restart wasn't enough.😅

Thank you for your assistance.