Synapse 1.44.0 hangs up every few minutes

[Whodiduexpect]

Description

Ever since updating to 1.44.0 from 1.43.0, I've experienced frequent hanging from Synapse that lasts 10 seconds to 3 minutes. During this "hanging" period all requests (including client api, federation, static web page, etc) time out, with the rare request getting a response after a few seconds. It does not starve server resources at all, though it causes large Synapse CPU usage spikes up to 50% (average usage is around 5%). While this issue is not taking place, the homeserver performs well even in larger federated rooms.

Steps to reproduce

• Let a device in a few large federated rooms do an initial sync or wait a few minutes
• Homeserver freezes up for a little bit

I expect the server to not freeze up and perform normally like it does when this issue does not happen. The occasional errors I get when this happens are just a byproduct of a connection to a client timing out unexpectedly. Looking at Grafana, it all correlates to large spikes in the federation transmission loop.

Version information

• Homeserver: jepcraft.ddns.net:443

• Version: Synapse 1.44.0
• Install method: latest matrixdotorg/synapse Docker build
• No separate Synapse workers
• Uses both IPv4 and IPv6

• Platform:

• Debian GNU/Linux 10 (buster) x86_64
• Physical server that runs applications in Docker

[Iamsomeguy]

I have noticed similar problems on my server since the upgrade to 1.44.0 and I think I have managed to isolate the cause to dns request storms sent by the matrix server. I could match timeout errors in the nginx proxy log to thousands of dns request in our dns server query log. During these events the dns server can receive around 4000 queries per minute.

I have mitigated the problem somewhat by installing a nameservice cache daemon on the matrix server but it still happens from time to time. Specifically when clients start to connect in the morning.

Let me know if I can provide any more info that can be of use.

[babolivier]

Grafana screenshots shared by @Whodiduexpect in Synapse Admins yesterday:

With extra info, also from Synapse Admins yesterday:

this spike pattern matches up with when it all breaks

What are the name of the background jobs that's spiking here? (i.e. what's the label associated with the purple spikes)

federation transmission loop

https://matrix.to/#/!ehXvUhWNASUkSLvAGP:matrix.org/$A_cYK1J2Qlxplek3vbgPDB-JOGDy-cM1Cate7x2FTDw?via=matrix.org&via=matrix.breakpointingbad.com&via=libera.chat

[erikjohnston]

Thanks for the graphs, @Whodiduexpect: it would be useful to see the graphs under "Per-block metrics" and "Federation", which should hopefully give us a better idea of exactly what it's doing.

Based on the symptoms reported by @Iamsomeguy my guess would be that it's related to sending out presence updates to all the servers, which is known to be very heavyweight, but I can't see why that would start being an issue in 1.44.0.

[Whodiduexpect]

Time period is the same as the original screenshots

[Whodiduexpect]

After setting up proper client and federation workers, this issue just vanished (I hope I'm not speaking too soon but it's been quite a while and it hasn't happened). So it seems whatever was happening it's an issue with the main Synapse process only and does not affect any of the workers.

After that larger spike a bit past 18:40 is when the workers got fully setup to take all supported client and federation requests and from there, there are only much smaller infrequent CPU spikes that don't effect the server, towards the end I was trying to deliberately cause the issue to happen.

[Iamsomeguy]

A small update regarding my situation. Yesterday I turned off presence:

presence:
       enabled: false

This has reduced the number of dns queries to about half (~2000/min for two minutes), but I still see upstream timeouts in the nginx log:

2021/10/13 08:11:15 [error] 11304#11304: *992382 no live upstreams while connecting to upstream, client: 1.2.3.4, server: matrix.synap.se, request: "OPTIONS /_matrix/client/r0/joined_groups HTTP/1.1", upstream: "http://localhost/_matrix/client/r0/joined_groups", host: "matrix.synap.se"
2021/10/13 08:11:16 [error] 11304#11304: *992379 no live upstreams while connecting to upstream, client: 2.3.4.5, server: matrix.synap.se, request: "OPTIONS /_matrix/client/r0/sync?filter=2&timeout=0&since=s1025296_101588561_2162_2117272_722109_53_39010_1805136_2 HTTP/1.1", upstream: "http://localhost/_matrix/client/r0/sync?filter=2&timeout=0&since=s1025296_101588561_2162_2117272_722109_53_39010_1805136_2", host: "matrix.synap.se"

There were 61 such messages during 90 seconds this morning.
In the homeserver log, at the same time, I see around 1500 messages like this:

2021-10-13 08:11:38,412 - synapse.logging.context - 70 - WARNING - federation_transaction_transmission_loop-2733 - Re-starting finished log context federation_transaction_transmission_loop-2733
2021-10-13 08:11:38,412 - synapse.logging.context - 70 - WARNING - federation_transaction_transmission_loop-2733 - Re-starting finished log context federation_transaction_transmission_loop-2733

Nothing else that looks suspicious.
I haven't set up workers on this host.

[DMRobertson]

@Whodiduexpect Thank you for the graphs. I'm glad to hear workers made things better. Unfortunately I don't think there are any smoking guns there. @Iamsomeguy thank you for the update too.

We're not aware of anything in between 1.43 and 1.44 that would cause this, so this is a bit of a thorny problem to track down. Would either of you be willing to roll back synapse to 1.43 and confirm that you don't see these symptoms any more? (There shouldn't be any problems doing so, e.g. we didn't have any database migrations between those two versions.)