Inbound federation fails after upgrading to synapse 1.140

[TheArcaneBrony]

Description

Local users no longer receive events over federation, outbound federation remains functional

Steps to reproduce

• Upgrade to 1.140
• Send a message
• Note that it never arrives on other servers unless the server gets downgraded

Homeserver

the-apothecary.club

Synapse Version

v1.140

Installation Method

Other (please mention below)

Database

Single PG, ported years ago, restored months back

Workers

Single process

Platform

NixOS/unstable, using nixos module, i7-7700, 64GB of ram, no container

Configuration

Presence enabled
Using Synapse-HTTP-Antispam
Very large caches

experimental_features:
msc2815_enabled: true

Relevant log output

Unable to deduce which parts of the log *are* the cause.
Lots of instances of "Error attempting to resove state at missing prev_events", "Request failed: GET matrix-federation://", "ERROR - sync_partial_state_room-0 - Background process 'sync_partial_state_room' threw an exception" followed by stacktraces, "synapse.api.errors.FederationError: ERROR 403: We can't get valid state history.", etc...

I have a 2.3GB log file available I could share privately (Last 10M log lines through journalctl, most likely includes logs from before the attempt to upgrade to 1.140).

Anything else that would be useful to know?

No response

[heftig]

Same here (archlinux.org), using antispam with Draupnir. Frequent errors in the logs like this one:

synapse_http_antispam: [PUT-762951-!…] Error in async callback (POST http://localhost:8585/api/1/spam_check/check_event_for_spam)
Traceback (most recent call last):
  File "/var/lib/synapse/venv/lib/python3.13/site-packages/synapse_http_antispam.py", line 73, in _catch_errors
    await task
  File "/var/lib/synapse/venv/lib/python3.13/site-packages/synapse/http/client.py", line 564, in post_json_get_json
    response = await self.request(
               ^^^^^^^^^^^^^^^^^^^
        "POST", uri, headers=Headers(actual_headers), data=json_str
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/synapse/venv/lib/python3.13/site-packages/synapse/http/client.py", line 448, in request
    response = await make_deferred_yieldable(request_deferred)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/synapse/venv/lib/python3.13/site-packages/twisted/internet/defer.py", line 1187, in __iter__
    yield self
twisted.web._newclient.RequestGenerationFailed: [<twisted.python.failure.Failure twisted.internet.defer.CancelledError: >]

The module config is:

  - module: synapse_http_antispam.HTTPAntispam
    config:
      base_url: http://localhost:8585/api/1/spam_check
      authorization: "…"
      enabled_callbacks:
        - check_event_for_spam
        - user_may_invite
        - user_may_join_room
      fail_open:
        check_event_for_spam: true
        user_may_invite: true
        user_may_join_room: true
      async:
        check_event_for_spam: true

[Gnuxie]

for context it seems that disabling check_event_for_spam seems to be a workaround. Not good that safety critical infrastructure has been effected by internal synapse changes though.

[Gnuxie]

Apparently that just fixes events. Invitations will still fail with that.

[Gnuxie]

:(

[devonh]

Is this still broken?
Synapse v1.140 brought in this change for policy servers: #18934
If things were failing due to that you should see the following logs:
ask_policy_server_to_sign_event: server %s responded with error: %s
or maybe failed to verify event using public key in policy server event: %s
or policy server {policy_server} failed to sign event correctly

Looks like meowlnir added support for policy server event signing here: maunium/meowlnir#44
I wonder if that has something to do with it.

If not, then it's something else.

[TheArcaneBrony]

This was a failure unrelated to policy servers, though I forgot about the issue and it appears to be working on Synapse 1.143.