[Meta] List of flaky Synapse tests

[anoadragon453]

We'd like to compile a central list of flaky tests, so that they can be disabled/fixed as necessary in order to get CI back into a reliable state. This list should be edited by the team as flakes are found.

• trial (3.9, sqlite, all)
  • twisted.protocols.amp.TooLong (1, 2, 3) ⇒ Work around twisted.protocols.amp.TooLong error by reducing logging in some tests. #18736
  • tests.replication.test_sharded_event_persister.EventPersisterShardTestCase.test_basic (1)
• trial (3.10, postgres, 14, all)
  • tests.rest.admin.test_user.UserRedactionBackgroundTaskTestCase.test_redact_messages_all_rooms (1)
• sytest (bullseye)
  • After /purge_history users still get pushed for new messages (1)
• sytest (bullseye, postgres)
  • After /purge_history users still get pushed for new messages (1, 2, 3, 4 5)
  • Newly joined room includes presence in incremental sync (1)
• sytest (bullseye, multi-postgres, workers)
  • many tests
• sytest (bullseye, asyncio)
  • After /purge_history users still get pushed for new messages (1)
• sytest (bullseye, multi-postgres, workers, asyncio)
  • many tests (1, 2, 3, 4, 5, 6, 7, 8, 9)
• Complement (workers, postgres)
  • TestFederationRoomsInvite (/Parallel/Remote_invited_user_can_reject_invite_when_homeserver_is_already_participating_in_the_room) (1, 2. 3, 4, 5, 6, 7)
  • TestFederationRoomsInvite/Parallel/Invited_user_can_reject_invite_over_federation_several_times (1)
  • TestFederationRoomsInvite (/Parallel/Inviter_user_can_rescind_invite_over_federation) (1, 2, 3)
  • TestPartialStateJoin/Outgoing_device_list_updates/Device_list_updates_reach_newly_joined_servers_in_partial_state_rooms (1, 2, 3, 4, 5)
  • TestPartialStateJoin/Outgoing_device_list_updates/Device_list_updates_reach_all_servers_in_partial_state_rooms (1)
  • TestDelayedEvents/delayed_state_events_are_kept_on_server_restart) (1, 2) -> Fix TestDelayedEvents/delayed_state_events_are_kept_on_server_restart to account for slow servers (de-flake) matrix-org/complement#830
  • TestThreadedReceipts (1, 2, 3, 4, 5)
  • TestThreadReceiptsInSyncMSC4102 (1 2, 3, 4, 5, 6, 7, 8, 9, 10) -> Complement TestThreadReceiptsInSyncMSC4102 is flaky #19171
  • TestAsyncUpload/Cannot_upload_to_a_media_ID_that_has_already_been_uploaded_to (1) -> Don't assert human error message in TestAsyncUpload (de-flake) matrix-org/complement#839
  • TestTxnIdempotencyScopedToDevice (1)
  • TestDeviceListUpdates (1, 2)
• Complement (monolith, SQLite, 2)
  • TestPartialStateJoin (TestPartialStateJoin/Leave_during_resync/works_after_a_second_partial_join)
  • TestPartialStateJoin/Outgoing_device_list_updates/Device_list_updates_reach_incorrectly_absent_servers_once_partial_state_join_completes (1, 2)
  • TestPartialStateJoin/Outgoing_device_list_updates/Device_list_updates_reach_incorrectly_kicked_servers_once_partial_state_join_completes (1, 2)
  • TestOutboundFederationSend (1)
• Complement (monolith, Postgres)
  • TestMSC4289PrivilegedRoomCreators/admin_with_>PL100_sorts_after_the_room_creator_for_state_resolution (1)
  • TestPartialStateJoin/Outgoing_device_list_updates/Device_list_updates_reach_newly_joined_servers_in_partial_state_rooms (1)

[realtyem]

Noticed for sytest (bullseye, multi-postgres, workers, asyncio) (run #4 above) that there is a time out that is occurring on "server-0"'s worker event_persister2. You can search for Timed out waiting for repl stream and it will show similar to:

2025-06-30 10:10:46,341 - synapse.replication.tcp.client - 346 - INFO - POST-22 - Waiting for repl stream 'device_lists' to reach 71 (master); currently at: 1
2025-06-30 10:10:51,341 - synapse.replication.tcp.client - 356 - WARNING - POST-22 - Timed out waiting for repl stream 'device_lists' to reach 71 (master); currently at: 1
... 
2025-06-30 10:37:18,705 - synapse.replication.tcp.client - 346 - INFO - POST-1952 - Waiting for repl stream 'device_lists' to reach 1096 (master); currently at: 1
2025-06-30 10:37:23,714 - synapse.replication.tcp.client - 356 - WARNING - POST-1952 - Timed out waiting for repl stream 'device_lists' to reach 1096 (master); currently at: 1

• account_data, presence, receipts, un_partial_stated_room and device_list streams from advancing beyond "1"
• caches seems to time out as well, but at least it advances sometimes(weirdly, it sometimes went backwards? Not sure what to make of that)
• typing from advancing beyond "0"

There were 1_681 lines and there may have been more, but I gave up after checking about 100 from the top and bottom.

Near the beginning of the log while subscribing to the REPLICATE channel, it does throw an exception

2025-06-30 10:10:28,522 - synapse.replication.tcp.redis - 292 - INFO - sentinel - Connecting to redis server localhost:6379
...
2025-06-30 10:10:28,618 - synapse.replication.tcp.redis - 126 - INFO - sentinel - Connected to redis
2025-06-30 10:10:28,618 - synapse.replication.tcp.redis - 138 - INFO - subscribe-replication-0 - Sending redis SUBSCRIBE for ['127.0.0.1:8839']
2025-06-30 10:10:28,619 - synapse.replication.tcp.redis - 141 - INFO - subscribe-replication-0 - Successfully subscribed to redis stream, sending REPLICATE command
2025-06-30 10:10:28,619 - synapse.metrics.background_process_metrics - 253 - ERROR - subscribe-replication-0 - Background process 'subscribe-replication' threw an exception
Traceback (most recent call last):
  File "/synapse/synapse/metrics/background_process_metrics.py", line 251, in run
    return await func(*args, **kwargs)
  File "/synapse/synapse/replication/tcp/redis.py", line 145, in _send_subscribe
    await self._async_send_command(ReplicateCommand())
  File "/synapse/synapse/replication/tcp/redis.py", line 239, in _async_send_command
    await make_deferred_yieldable(
  File "/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1195, in __iter__
    result.raiseException()
  File "/venv/lib/python3.9/site-packages/twisted/python/failure.py", line 455, in raiseException
    raise self.value.with_traceback(self.tb)
  File "/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1857, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "/synapse/synapse/util/patch_inline_callbacks.py", line 207, in check_yield_points_inner
    raise Exception(err)
Exception: getConnection yielded with context subscribe-replication-0 rather than sentinel, yielded on line 2357 in /venv/lib/python3.9/site-packages/txredisapi.py

Just for this one single run, this exception happened early in the log for server-0 on:

• event_persister1(where it repeated itself 4 times, slightly different logging context after that first one: send-cmd-x)
• event_persister2
• event_creator

and on server-1:

• event_persister2
• frontend_proxy

For the same category, but this time for run #3 it appears to have stricken "server-1" instead. Which makes sense as those test failures feel more 'federation' related. Saw it on workers:

• appservice
• event_persister1
• media_repository
• synchrotron - (presence, presence_federation, and account_data streams did not advance here)
• user_dir

It seems rather random which worker gets hit. And I'm reasonably sure the only reason we see the streams not advancing is only because those particular workers care enough to 'wait' on that stream. This may be more prevalent but has managed to luckily only show when all the cards fall in the right place.