Declined invites remain in Element Web until cache clear

[ricardo-duarte-av]

Description

It was noted that invites coming from any server into our homeservers, once declined remain in Element Web until it gets it's cache cleared.

The refusal is received by the inviter.

This behaviour has been confirmed by users of multiple other homeserver, yet it seems that it does not affect Matrix.org (if we receive a invite on a matrix.org account using app.element.io and refuse it, it gets removed from EleWeb.)

Steps to reproduce

1. Use any server to invite a user from your own server running Element Web
2. Decline the invite in Element Web
3. Notice the Invite stuck while inviter recieves decline.

Homeserver

aguiarvieira.pt, daedric.net (inviter is matrix.org)

Synapse Version

1.125rc1, 1.124 respectively.

Installation Method

Docker (matrixdotorg/synapse)

Database

Postgres, single server/db ; Server was not ported ; Server has been restored from backup when updating major versions of pgsql

Workers

Multiple workers

Platform

Aguiarvieira.pt:
Linux ns3167742 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64
Intel(R) Xeon(R) E-2274G CPU @ 4.00GHz

Daedric.net:
Ubuntu 22.04.3 LTS (GNU/Linux 6.8.0-1018-oracle aarch64)
Arm Free Oracle

Configuration

Presence enabled in both servers, retention disabled in both.

Aguiarvieira.pt experimental_features

Modules:
shared_secret_authenticator.SharedSecretAuthProvider

experimental_features:
  msc1767_enabled: True #MSC1767 and friends: Extensible Events
  msc2246_enabled: True #MSC2246: Asynchronous media uploads
  #msc2409_enabled: True #MSC2409: Proposal to send EDUs to appservices
  #msc2409_to_device_messages_enabled: True
  msc2654_enabled: True #MSC2654: Unread counts
  msc2716_enabled: True #MSC2716: Incrementally importing history into existing rooms
  msc2815_enabled: True #MSC2815: Proposal to allow room moderators to view redacted event content
  msc2940_enabled: True #????
  msc3026_enabled: False #MSC3026: "busy" presence state
  msc3030_enabled: True #MSC3030: Jump to date API endpoint
  msc3202_enabled: True #MSC3202: Encrypted appservices
  msc3202_device_masquerading_enabled: True
  msc3202_transaction_extensions: True
  msc3244_enabled: True #MSC3244 room version capabilities
  msc3266_enabled: True #MSC3266: Room summary AP
  msc3381_polls_enabled: True
  msc3391_enabled: True #MSC3391: API to delete account data
  msc3575_enabled: True #MSC3575: Sliding Sync (aka Sync v3)
  msc3664_enabled: True #MSC3664: Pushrules to match on related events
  msc3757_enabled: True #MSC3757: Restricting who can overwrite a state event
  msc3720_enabled: True #MSC3720: (Account status endpoint)
  msc3771_enabled: True #MSC3771: Read receipts for threads
  msc3773_enabled: True #MSC3773: Notifications for threads
  msc3814_enabled: True #MSC3814: Dehydrated devices with SSSS
  msc3823_account_suspension: True #MSC3823 Account Suspension
  msc3827_enabled: True #MSC3827: Filtering of `/publicRooms` by room type
  msc3848_enabled: True #MSC3848: Introduce errcodes for specific event sending failures
  msc3852_enabled: True #MSC3852: Expose last seen user agent field on /_matrix/client/v3/devices.
  msc3874_enabled: True #MSC3874: Filtering threads from the /messages endpoint
  msc3881_enabled: True #MSC3881: Remotely toggle push notifications for another client
  msc3886_enabled: True #MSC3886: Simple client rendezvous capability
  msc3890_enabled: True #MSC3890: Remotely silence local notifications
  msc3912_enabled: True #MSC3912: Redaction of related events
  msc3946_enabled: True #MSC3946: Dynamic room predecessor
  msc3967_enabled: True #MSC3967: Do not require UIA when first uploading cross signing keys
  msc3981_enabled: True #MSC3981: `/relations` recursion
  msc3981_recurse_relations: True
  msc3983_appservice_otk_claims: True #MSC3983: Proxying OTK claim requests to exclusive ASes.
  msc3984_appservice_key_query: True # MSC3984: Proxying key queries to exclusive ASes.
  msc4028_push_encrypted_events: True  #MSC4028
  msc4057_enabled: True #MSC4057: Static Room Directory
  msc4076_enabled: true #MSC4076: Let E2EE clients calculate app badge counts themselves (disable_badge_count)
  msc4133_enabled: True #MSC4133: Extending User Profile API with Key:Value Pairs
  msc4151_enabled: True #MSC4151: Report room API (Client-Server API)
  msc4156_enabled: True #MSC4156: Migrate server_name to via
  msc4171_enabled: True #MSC4171: Service members
  msc4210_enabled: True #MSC4210: Remove legacy mentions
  msc4222_enabled: True #MSC4222: state_after to sync v2

  msc3814_enabled: True #MSC3814 (dehydrated devices with SSSS)
  msc2697_enabled: False #Needs to be disabled for 3814

# Enable delayed events (msc4140)
max_event_delay_duration: 24h

daedric.net experimental_features

experimental_features:
  msc1767_enabled: True #MSC1767 and friends: Extensible Events
  msc2246_enabled: True #MSC2246: Asynchronous media uploads
  #msc2409_enabled: True #MSC2409: Proposal to send EDUs to appservices
  #msc2409_to_device_messages_enabled: True
  msc2654_enabled: True #MSC2654: Unread counts
  msc2716_enabled: True #MSC2716: Incrementally importing history into existing rooms
  msc2815_enabled: True #MSC2815: Proposal to allow room moderators to view redacted event content
  msc2940_enabled: True #????
  msc3026_enabled: True #MSC3026: "busy" presence state
  msc3030_enabled: True #MSC3030: Jump to date API endpoint
  msc3202_enabled: True #MSC3202: Encrypted appservices
  msc3202_device_masquerading_enabled: True
  msc3202_transaction_extensions: True
  msc3244_enabled: True #MSC3244 room version capabilities
  msc3266_enabled: True #MSC3266: Room summary AP
  msc3381_polls_enabled: True
  msc3391_enabled: True #MSC3391: API to delete account data
  msc3575_enabled: True #MSC3575: Sliding Sync (aka Sync v3)
  msc3664_enabled: True #MSC3664: Pushrules to match on related events
  msc3757_enabled: True #MSC3757: Restricting who can overwrite a state event
  msc3720_enabled: True #MSC3720: (Account status endpoint)
  msc3771_enabled: True #MSC3771: Read receipts for threads
  msc3773_enabled: True #MSC3773: Notifications for threads
  msc3814_enabled: True #MSC3814: Dehydrated devices with SSSS
  msc3823_account_suspension: True #MSC3823 Account Suspension
  msc3827_enabled: True #MSC3827: Filtering of `/publicRooms` by room type
  msc3848_enabled: True #MSC3848: Introduce errcodes for specific event sending failures
  msc3852_enabled: True #MSC3852: Expose last seen user agent field on /_matrix/client/v3/devices.
  msc3874_enabled: True #MSC3874: Filtering threads from the /messages endpoint
  msc3881_enabled: True #MSC3881: Remotely toggle push notifications for another client
  msc3886_enabled: True #MSC3886: Simple client rendezvous capability
  msc3890_enabled: True #MSC3890: Remotely silence local notifications
  msc3912_enabled: True #MSC3912: Redaction of related events
  msc3946_enabled: True #MSC3946: Dynamic room predecessor
  msc3967_enabled: True #MSC3967: Do not require UIA when first uploading cross signing keys
  msc3981_enabled: True #MSC3981: `/relations` recursion
  msc3981_recurse_relations: True
  msc3983_appservice_otk_claims: True #MSC3983: Proxying OTK claim requests to exclusive ASes.
  msc3984_appservice_key_query: True # MSC3984: Proxying key queries to exclusive ASes.
  msc4028_push_encrypted_events: True  #MSC4028
  msc4057_enabled: True #MSC4057: Static Room Directory
  msc4076_enabled: true #MSC4076: Let E2EE clients calculate app badge counts themselves (disable_badge_count)
  msc4133_enabled: True #MSC4133: Extending User Profile API with Key:Value Pairs
  msc4151_enabled: True #MSC4151: Report room API (Client-Server API)
  msc4156_enabled: True #MSC4156: Migrate server_name to via
  msc4171_enabled: True #MSC4171: Service members
  msc4210_enabled: True #MSC4210: Remove legacy mentions
  msc4222_enabled: True #MSC4222: state_after to sync v2

  #Must disable one to enable the other.
  msc3814_enabled: True #MSC3814 (dehydrated devices with SSSS)
  msc2697_enabled: False #Needs to be disabled for 3814

  msc4108_enabled: true #MSC4108: Mechanism to allow OIDC sign in and E2EE set up via QR code
  msc3861:
    enabled: true

    # Synapse will call `{issuer}/.well-known/openid-configuration` to get the OIDC configuration
    issuer: https://mas.daedric.net/

    # Matches the `client_id` in the auth service config
    client_id: 0000000000000000000SYNAPSE
    # Matches the `client_auth_method` in the auth service config
    client_auth_method: client_secret_basic
    # Matches the `client_secret` in the auth service config
    client_secret: "xxxxxx"

    # Matches the `matrix.secret` in the auth service config
    admin_token: "xxxxxxx"

    # URL to advertise to clients where users can self-manage their account
    account_management_url: "https://mas.daedric.net/account"

# Enable delayed events (msc4140)
max_event_delay_duration: 24h

Relevant log output

Synapse logs from daedric.net

daedricnetsynapse             | 2025-02-21 10:38:50,537 - synapse.access.http.9093 - 508 - INFO - POST-190127 - 172.20.0.1 - 9093 - {None} Processed request: 0.005sec/0.000sec (0.002sec, 0.000sec) (0.002sec/0.001sec/2) 172B 200 "POST /_synapse/replication/fed_send_edu/m.device_list_update/nxhaWLpufJ HTTP/1.1" "Synapse/1.124.0" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:38:53,279 - synapse.access.http.8008 - 508 - INFO - GET-190128 - 96.244.139.2 - 8008 - {None} Processed request: 0.000sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 482B 200 "GET /_matrix/key/v2/server HTTP/1.1" "Synapse/1.124.0" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:38:53,850 - synapse.access.http.9093 - 508 - INFO - POST-190129 - 172.20.0.1 - 9093 - {None} Processed request: 0.006sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.004sec/3) 172B 200 "POST /_synapse/replication/store_room_on_outlier_membership/%21FUEaEqTMFXRFfNbJdO%3Amatrix.org/AqXVxdUXTG HTTP/1.1" "Synapse/1.124.0" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:38:53,979 - synapse.push.httppusher - 241 - INFO - httppush.process-6547 - Processing 1 unprocessed push actions for @daedric:daedric.net/im.vector.app.android/eVEewl4jQOCKJofJ8zqlQO:APA91bEU1klSaPU1YctbINFpXitF3AbBz1p2gHRX2tqNCwt2dmdDJ69eaffBa-VxJzFz84l3F3C1dW8MoPCE0XfcuSvtMWWPxsjzTtj12o9kbznPCtc-u9vd3RxKgijwkEnS4EVpkNqE starting at stream_ordering 4481624
daedricnetsynapse             | 2025-02-21 10:38:53,984 - synapse.push.httppusher - 241 - INFO - httppush.process-6544 - Processing 1 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upHs7r4BwmLUjz?up=1 starting at stream_ordering 4481624
daedricnetsynapse             | 2025-02-21 10:38:53,985 - synapse.push.httppusher - 241 - INFO - httppush.process-6546 - Processing 1 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upfcdXLJe3dLxB?up=1 starting at stream_ordering 4481624
daedricnetsynapse             | 2025-02-21 10:38:53,989 - synapse.push.httppusher - 241 - INFO - httppush.process-6549 - Processing 1 unprocessed push actions for @daedric:daedric.net/im.vector.app.android/f9lg1IzCT6uno6741J1nwC:APA91bElzcwCgq82u9swsfBvN7bLHyiiaXH8fMj1LqkUykoMZefHaZSLOpQI26BhUn8u0P9F0Ut57QAMVuL43XB2OQWgJWgzIr9HDHFOswFx78gZ6PDrekbTp4mgzPU6NYYmji8bhHiS starting at stream_ordering 4481624
daedricnetsynapse             | 2025-02-21 10:38:53,995 - synapse.push.httppusher - 241 - INFO - httppush.process-6545 - Processing 1 unprocessed push actions for @daedric:daedric.net/net.daedric.element/edLKAwuvSHe08HKIP2TlPK:APA91bEO2Ip5x7O9uVfm94qSiY4hWR-5IM-je6ajxhFtohMHlINieawVL_wmH1xFAtGKROjH7gLbETG6qAYxBPv7la2Ynq6hypgptkyIDqEyVCV77Dg5-zVuA6VKM_qieosoxSX4ikQB starting at stream_ordering 4481624
daedricnetsynapse             | 2025-02-21 10:38:53,996 - synapse.push.httppusher - 241 - INFO - httppush.process-6550 - Processing 1 unprocessed push actions for @daedric:daedric.net/net.daedric.element/cGAzXUrGSTag1qwpwHnvPA:APA91bG61B_9Opfrm5ERFaBqQzDZuHkWVIMaXCQMyNeY8DKfrxWfkmarOwvNXRjua5yxO3srFhQjY2s_JEj8ZWTGbQWoU5FTE3c5M3nW_V-EJGC4G2ScEKs starting at stream_ordering 4481624
daedricnetsynapse             | 2025-02-21 10:38:53,998 - synapse.push.httppusher - 241 - INFO - httppush.process-6548 - Processing 1 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upIdQxmte9GHyW?up=1 starting at stream_ordering 4481624
daedricnetsynapse             | 2025-02-21 10:38:53,998 - synapse.push.httppusher - 241 - INFO - httppush.process-6543 - Processing 1 unprocessed push actions for @daedric:daedric.net/net.daedric.element/et3iiavYQcmmNf8dGqFoh4:APA91bGsC1v1WBmo6yoEXdOwVZvlFCMqmgFoAn759HhfClzHw1_E0kDAPn4QAM-soDI_5fAgMrKtbYNQL1gAMqLslJI8O_w3WGgBBwe8zDKXLqNPkgibtIw starting at stream_ordering 4481624
daedricnetsynapse             | 2025-02-21 10:38:54,044 - synapse.http.client - 442 - INFO - httppush.process-6548 - Received response to POST https://ntfy.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:54,075 - synapse.http.client - 442 - INFO - httppush.process-6546 - Received response to POST https://ntfy.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:54,087 - synapse.http.client - 442 - INFO - httppush.process-6544 - Received response to POST https://ntfy.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:54,114 - synapse.http.client - 442 - INFO - httppush.process-6543 - Received response to POST https://sygnal.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:54,121 - synapse.http.client - 442 - INFO - httppush.process-6547 - Received response to POST https://matrix.org/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:54,123 - synapse.http.client - 442 - INFO - httppush.process-6545 - Received response to POST https://sygnal.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:54,161 - synapse.http.client - 442 - INFO - httppush.process-6550 - Received response to POST https://sygnal.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:54,190 - synapse.http.client - 442 - INFO - httppush.process-6549 - Received response to POST https://matrix.org/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:55,163 - synapse.replication.tcp.resource - 214 - INFO - replication_notifier-101083 - Streaming: caches -> 16162398 (limited: False, updates: 14, max token: 16162398)
daedricnetsynapse             | 2025-02-21 10:38:55,176 - synapse.federation.federation_server - 1046 - INFO - POST-190130 - Claimed one-time-keys: signed_curve25519:AAAAJQ for @daedric:daedric.net:mZwB8t4XdW,signed_curve25519:AAAALA for @daedric:daedric.net:TGMg7OHQJQ,signed_curve25519:AAAAAAAAACE for @daedric:daedric.net:pUk3JDcfLyq/7etm3xeqiEwy3xWtZWiVsBtIml05NXE,signed_curve25519:AAAAAAAAAFc for @daedric:daedric.net:LEhZxucoId,signed_curve25519:AAAAAAAAAB0 for @daedric:daedric.net:dHjmqDxOwQ,signed_curve25519:AAAAAAAAADU for @daedric:daedric.net:z7JhnSrrdC26W1sIw1SMS5TEQVP6wrd+D08VQ+SuSDM,signed_curve25519:AAAAAAAAABE for @daedric:daedric.net:d3yoEMvs3dw7GjiC/K4zYqmE8kbsHbzdiBv7JuHnBGs,signed_curve25519:AAAAAAAAACc for @daedric:daedric.net:rKClLGVt9C/ZL1oWD0lr4m0k3pjG5x0HmjGniix6vkE,signed_curve25519:AAAAAAAAACI for @daedric:daedric.net:8AFc8JBxRW,signed_curve25519:AAAAAAAAAC8 for @daedric:daedric.net:rw4d3JSLjW,signed_curve25519:AAAAAAAAACQ for @daedric:daedric.net:ZkULFSUR5RGyYmQIRCVxlOKW3EUBuZDuZhZROhDGp0Y,signed_curve25519:AAAAAAAAABY for @daedric:daedric.net:YDcbLAYu3F,signed_curve25519:AAAAAAAAACI for @daedric:daedric.net:Bc9F7VvRIl,signed_curve25519:AAAAAAAAACY for @daedric:daedric.net:IrXGFleiRxuli07QJf43DJ/FjvsTd9wod2ovrQbmX0A,signed_curve25519:AAAAAAAAAAA for @daedric:daedric.net:LPA38u/Asz84sv3gMASSabCrWhRyNv+ONe3sozz9Ylg
daedricnetsynapse             | 2025-02-21 10:38:55,177 - synapse.access.http.8008 - 508 - INFO - POST-190130 - 172.20.0.1 - 8008 - {matrix.org} Processed request: 0.057sec/0.001sec (0.003sec, 0.000sec) (0.001sec/0.049sec/2) 4277B 200 "POST /_matrix/federation/v1/user/keys/claim HTTP/1.1" "Synapse/1.125.0rc1 (b=matrix-org-hotfixes,db9a75c46b)" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:38:55,178 - synapse.replication.tcp.resource - 214 - INFO - replication_notifier-101084 - Streaming: caches -> 16162399 (limited: False, updates: 1, max token: 16162399)
daedricnetsynapse             | 2025-02-21 10:38:55,492 - synapse.http.client - 442 - INFO - POST-190131 - Received response to POST https://mas.daedric.net/oauth2/introspect: 200
daedricnetsynapse             | 2025-02-21 10:38:55,498 - synapse.handlers.e2e_keys - 925 - INFO - POST-190131 - Adding one_time_keys dict_keys(['signed_curve25519:AAAARA']) for device 'TGMg7OHQJQ' for user '@daedric:daedric.net' at 1740134335495
daedricnetsynapse             | 2025-02-21 10:38:55,507 - synapse.replication.tcp.resource - 214 - INFO - replication_notifier-101085 - Streaming: caches -> 16162400 (limited: False, updates: 1, max token: 16162400)
daedricnetsynapse             | 2025-02-21 10:38:55,513 - synapse.access.http.8008 - 508 - INFO - POST-190131 - 130.61.129.53 - 8008 - {@daedric:daedric.net} Processed request: 0.067sec/0.000sec (0.005sec, 0.001sec) (0.002sec/0.013sec/7) 48B 200 "POST /_matrix/client/v3/keys/upload HTTP/1.1" "gomuks/0.4.0+dev.db709896 mautrix-go/v0.23.1 go/1.24.0" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:38:55,650 - synapse.http.client - 442 - INFO - POST-190133 - Received response to POST https://mas.daedric.net/oauth2/introspect: 200
daedricnetsynapse             | 2025-02-21 10:38:55,660 - synapse.handlers.e2e_keys - 925 - INFO - POST-190133 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAIw']) for device 'LEhZxucoId' for user '@daedric:daedric.net' at 1740134335656
daedricnetsynapse             | 2025-02-21 10:38:55,672 - synapse.replication.tcp.resource - 214 - INFO - replication_notifier-101086 - Streaming: caches -> 16162401 (limited: False, updates: 1, max token: 16162401)
daedricnetsynapse             | 2025-02-21 10:38:55,712 - synapse.access.http.8008 - 508 - INFO - POST-190133 - 163.116.245.45 - 8008 - {@daedric:daedric.net} Processed request: 0.100sec/0.016sec (0.003sec, 0.002sec) (0.006sec/0.026sec/7) 48B 200 "POST /_matrix/client/v3/keys/upload HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:38:55,990 - synapse.util.caches.lrucache - 218 - INFO - LruCache._expire_old_entries-26910 - Dropped 2 items from caches
daedricnetsynapse             | 2025-02-21 10:38:56,000 - synapse.http.matrixfederationclient - 767 - INFO - _maybe_retry_device_resync-77599 - {GET-O-67798} [rauten.co.za] Got response headers: 404 Not Found
daedricnetsynapse             | 2025-02-21 10:38:56,000 - synapse.http.matrixfederationclient - 857 - WARNING - _maybe_retry_device_resync-77599 - {GET-O-67798} [rauten.co.za] Request failed: GET matrix-federation://rauten.co.za/_matrix/federation/v1/user/devices/%40jans%3Arauten.co.za: HttpResponseException('404: Not Found')
daedricnetsynapse             | 2025-02-21 10:38:56,000 - synapse.handlers.device - 1536 - WARNING - _maybe_retry_device_resync-77599 - Failed to handle device list update for @jans:rauten.co.za: 404: Not Found
daedricnetsynapse             | 2025-02-21 10:38:56,660 - synapse.push.httppusher - 241 - INFO - httppush.process-6554 - Processing 1 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upfcdXLJe3dLxB?up=1 starting at stream_ordering 4481644
daedricnetsynapse             | 2025-02-21 10:38:56,662 - synapse.push.httppusher - 241 - INFO - httppush.process-6551 - Processing 1 unprocessed push actions for @daedric:daedric.net/net.daedric.element/et3iiavYQcmmNf8dGqFoh4:APA91bGsC1v1WBmo6yoEXdOwVZvlFCMqmgFoAn759HhfClzHw1_E0kDAPn4QAM-soDI_5fAgMrKtbYNQL1gAMqLslJI8O_w3WGgBBwe8zDKXLqNPkgibtIw starting at stream_ordering 4481644
daedricnetsynapse             | 2025-02-21 10:38:56,664 - synapse.push.httppusher - 241 - INFO - httppush.process-6556 - Processing 1 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upIdQxmte9GHyW?up=1 starting at stream_ordering 4481644
daedricnetsynapse             | 2025-02-21 10:38:56,664 - synapse.push.httppusher - 241 - INFO - httppush.process-6557 - Processing 1 unprocessed push actions for @daedric:daedric.net/im.vector.app.android/f9lg1IzCT6uno6741J1nwC:APA91bElzcwCgq82u9swsfBvN7bLHyiiaXH8fMj1LqkUykoMZefHaZSLOpQI26BhUn8u0P9F0Ut57QAMVuL43XB2OQWgJWgzIr9HDHFOswFx78gZ6PDrekbTp4mgzPU6NYYmji8bhHiS starting at stream_ordering 4481644
daedricnetsynapse             | 2025-02-21 10:38:56,666 - synapse.push.httppusher - 241 - INFO - httppush.process-6553 - Processing 1 unprocessed push actions for @daedric:daedric.net/net.daedric.element/edLKAwuvSHe08HKIP2TlPK:APA91bEO2Ip5x7O9uVfm94qSiY4hWR-5IM-je6ajxhFtohMHlINieawVL_wmH1xFAtGKROjH7gLbETG6qAYxBPv7la2Ynq6hypgptkyIDqEyVCV77Dg5-zVuA6VKM_qieosoxSX4ikQB starting at stream_ordering 4481644
daedricnetsynapse             | 2025-02-21 10:38:56,670 - synapse.push.httppusher - 241 - INFO - httppush.process-6555 - Processing 1 unprocessed push actions for @daedric:daedric.net/im.vector.app.android/eVEewl4jQOCKJofJ8zqlQO:APA91bEU1klSaPU1YctbINFpXitF3AbBz1p2gHRX2tqNCwt2dmdDJ69eaffBa-VxJzFz84l3F3C1dW8MoPCE0XfcuSvtMWWPxsjzTtj12o9kbznPCtc-u9vd3RxKgijwkEnS4EVpkNqE starting at stream_ordering 4481644
daedricnetsynapse             | 2025-02-21 10:38:56,670 - synapse.push.httppusher - 241 - INFO - httppush.process-6552 - Processing 1 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upHs7r4BwmLUjz?up=1 starting at stream_ordering 4481644
daedricnetsynapse             | 2025-02-21 10:38:56,676 - synapse.push.httppusher - 241 - INFO - httppush.process-6558 - Processing 1 unprocessed push actions for @daedric:daedric.net/net.daedric.element/cGAzXUrGSTag1qwpwHnvPA:APA91bG61B_9Opfrm5ERFaBqQzDZuHkWVIMaXCQMyNeY8DKfrxWfkmarOwvNXRjua5yxO3srFhQjY2s_JEj8ZWTGbQWoU5FTE3c5M3nW_V-EJGC4G2ScEKs starting at stream_ordering 4481644
daedricnetsynapse             | 2025-02-21 10:38:56,725 - synapse.http.client - 442 - INFO - httppush.process-6556 - Received response to POST https://ntfy.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:56,754 - synapse.http.client - 442 - INFO - httppush.process-6554 - Received response to POST https://ntfy.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:56,762 - synapse.http.client - 442 - INFO - httppush.process-6552 - Received response to POST https://ntfy.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:56,771 - synapse.http.client - 442 - INFO - httppush.process-6558 - Received response to POST https://sygnal.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:56,788 - synapse.http.client - 442 - INFO - httppush.process-6553 - Received response to POST https://sygnal.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:56,793 - synapse.http.client - 442 - INFO - httppush.process-6555 - Received response to POST https://matrix.org/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:56,806 - synapse.http.client - 442 - INFO - httppush.process-6557 - Received response to POST https://matrix.org/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:56,829 - synapse.http.client - 442 - INFO - httppush.process-6551 - Received response to POST https://sygnal.daedric.net/_matrix/push/v1/notify: 200
daedricnetsynapse             | 2025-02-21 10:38:57,913 - synapse.http.client - 442 - INFO - GET-190135 - Received response to POST https://mas.daedric.net/oauth2/introspect: 200
daedricnetsynapse             | 2025-02-21 10:38:57,919 - synapse.access.http.8008 - 508 - INFO - GET-190135 - 163.116.245.45 - 8008 - {@daedric:daedric.net} Processed request: 0.054sec/0.000sec (0.003sec, 0.001sec) (0.001sec/0.003sec/2) 2B 200 "GET /_matrix/client/v3/thirdparty/user/im.vector.protocol.sip_virtual?native_mxid=%40echo-bot-1337%3Amatrix.org HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:38:59,537 - synapse.replication.http.federation - 198 - INFO - POST-190137 - Got 'm.device_list_update' edu from matrix.org
daedricnetsynapse             | 2025-02-21 10:38:59,540 - synapse.handlers.device - 1312 - WARNING - POST-190137 - Got device list update edu for '@kirikaza:matrix.org'/'TOHFYXZHYD', but don't share a room
daedricnetsynapse             | 2025-02-21 10:38:59,540 - synapse.access.http.9093 - 508 - INFO - POST-190137 - 172.20.0.1 - 9093 - {None} Processed request: 0.003sec/0.000sec (0.002sec, 0.000sec) (0.000sec/0.001sec/2) 172B 200 "POST /_synapse/replication/fed_send_edu/m.device_list_update/bJcPCERAFb HTTP/1.1" "Synapse/1.124.0" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:38:59,984 - synapse.http.matrixfederationclient - 364 - INFO - POST-190138 - {GET-O-67799} [matrix.org] Completed request: 200 OK in 0.11 secs, got 483 bytes - GET matrix-federation://matrix.org/_matrix/federation/v1/make_leave/%21FUEaEqTMFXRFfNbJdO%3Amatrix.org/%40daedric%3Adaedric.net
daedricnetsynapse             | 2025-02-21 10:39:00,262 - synapse.http.matrixfederationclient - 364 - INFO - POST-190138 - {PUT-O-67800} [matrix.org] Completed request: 200 OK in 0.28 secs, got 2 bytes - PUT matrix-federation://matrix.org/_matrix/federation/v2/send_leave/%21FUEaEqTMFXRFfNbJdO%3Amatrix.org/%24GdqdB0JhDKW-R0rA16zYJGIU711qKLlCfGJlZQfRk10
daedricnetsynapse             | 2025-02-21 10:39:00,301 - synapse.http.client - 952 - INFO - POST-190138 - Received response to POST synapse-replication://daedricnet-event-worker/_synapse/replication/fed_send_events/DEipiabZxC: 200
daedricnetsynapse             | 2025-02-21 10:39:00,301 - synapse.replication.tcp.client - 346 - INFO - POST-190138 - Waiting for repl stream 'events' to reach 4481646 (daedricnet-event-worker); currently at: 4481645
daedricnetsynapse             | 2025-02-21 10:39:00,314 - synapse.replication.tcp.client - 366 - INFO - POST-190138 - Finished waiting for repl stream 'events' to reach 4481646 (daedricnet-event-worker)
daedricnetsynapse             | 2025-02-21 10:39:00,318 - synapse.access.http.9093 - 508 - INFO - POST-190138 - 172.20.0.1 - 9093 - {@daedric:daedric.net} Processed request: 0.444sec/0.003sec (0.004sec, 0.001sec) (0.000sec/0.000sec/0) 250B 200 "POST /_synapse/replication/remote_reject_invite/%241eNCYrC4NpvR4GPMm83hdb8DaK_-woAPPzhg2LGRNag/oBzOANVmzO HTTP/1.1" "Synapse/1.124.0" [0 dbevts]
daedricnetsynapse             | 2025-02-21 10:39:00,365 - synapse.push.httppusher - 241 - INFO - httppush.process-6562 - Processing 0 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upfcdXLJe3dLxB?up=1 starting at stream_ordering 4481645
daedricnetsynapse             | 2025-02-21 10:39:00,367 - synapse.push.httppusher - 241 - INFO - httppush.process-6566 - Processing 0 unprocessed push actions for @daedric:daedric.net/net.daedric.element/cGAzXUrGSTag1qwpwHnvPA:APA91bG61B_9Opfrm5ERFaBqQzDZuHkWVIMaXCQMyNeY8DKfrxWfkmarOwvNXRjua5yxO3srFhQjY2s_JEj8ZWTGbQWoU5FTE3c5M3nW_V-EJGC4G2ScEKs starting at stream_ordering 4481645
daedricnetsynapse             | 2025-02-21 10:39:00,368 - synapse.push.httppusher - 241 - INFO - httppush.process-6564 - Processing 0 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upIdQxmte9GHyW?up=1 starting at stream_ordering 4481645
daedricnetsynapse             | 2025-02-21 10:39:00,368 - synapse.push.httppusher - 241 - INFO - httppush.process-6560 - Processing 0 unprocessed push actions for @daedric:daedric.net/de.spiritcroc.riotx/https://ntfy.daedric.net/upHs7r4BwmLUjz?up=1 starting at stream_ordering 4481645
daedricnetsynapse             | 2025-02-21 10:39:00,368 - synapse.push.httppusher - 241 - INFO - httppush.process-6563 - Processing 0 unprocessed push actions for @daedric:daedric.net/im.vector.app.android/eVEewl4jQOCKJofJ8zqlQO:APA91bEU1klSaPU1YctbINFpXitF3AbBz1p2gHRX2tqNCwt2dmdDJ69eaffBa-VxJzFz84l3F3C1dW8MoPCE0XfcuSvtMWWPxsjzTtj12o9kbznPCtc-u9vd3RxKgijwkEnS4EVpkNqE starting at stream_ordering 4481645
daedricnetsynapse             | 2025-02-21 10:39:00,369 - synapse.push.httppusher - 241 - INFO - httppush.process-6565 - Processing 0 unprocessed push actions for @daedric:daedric.net/im.vector.app.android/f9lg1IzCT6uno6741J1nwC:APA91bElzcwCgq82u9swsfBvN7bLHyiiaXH8fMj1LqkUykoMZefHaZSLOpQI26BhUn8u0P9F0Ut57QAMVuL43XB2OQWgJWgzIr9HDHFOswFx78gZ6PDrekbTp4mgzPU6NYYmji8bhHiS starting at stream_ordering 4481645
daedricnetsynapse             | 2025-02-21 10:39:00,374 - synapse.push.httppusher - 241 - INFO - httppush.process-6561 - Processing 0 unprocessed push actions for @daedric:daedric.net/net.daedric.element/edLKAwuvSHe08HKIP2TlPK:APA91bEO2Ip5x7O9uVfm94qSiY4hWR-5IM-je6ajxhFtohMHlINieawVL_wmH1xFAtGKROjH7gLbETG6qAYxBPv7la2Ynq6hypgptkyIDqEyVCV77Dg5-zVuA6VKM_qieosoxSX4ikQB starting at stream_ordering 4481645
daedricnetsynapse             | 2025-02-21 10:39:00,377 - synapse.push.httppusher - 241 - INFO - httppush.process-6559 - Processing 0 unprocessed push actions for @daedric:daedric.net/net.daedric.element/et3iiavYQcmmNf8dGqFoh4:APA91bGsC1v1WBmo6yoEXdOwVZvlFCMqmgFoAn759HhfClzHw1_E0kDAPn4QAM-soDI_5fAgMrKtbYNQL1gAMqLslJI8O_w3WGgBBwe8zDKXLqNPkgibtIw starting at stream_ordering 4481645
daedricnetsynapse             | 2025-02-21 10:39:04,918 - synapse.access.http.8008 - 508 - INFO - GET-190139 - 96.244.139.2 - 8008 - {None} Processed request: 0.000sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 482B 200 "GET /_matrix/key/v2/server HTTP/1.1" "Synapse/1.124.0" [0 dbevts]
daedricnet-inbound-worker     | 2025-02-21 10:39:10,951 - synapse.crypto.keyring - 632 - WARNING - PerspectivesKeyFetcher-24030 - Key lookup failed from 'keys.daedric.net': Failed to send request: HttpResponseException: 500: Internal Server Error
daedricnet-inbound-worker     | 2025-02-21 10:39:10,990 - synapse.storage.txn - 787 - WARNING - PerspectivesKeyFetcher-24030 - [TXN OPERROR] {store_server_keys_response-7d9908} could not serialize access due to concurrent update
daedricnet-inbound-worker     |  0/5
daedricnet-sender-worker-3    | 2025-02-21 10:39:10,993 - synapse.federation.sender.per_destination_queue - 412 - WARNING - federation_transaction_transmission_loop-477460 - TX [teh.entar.net] Failed to send transaction: Failed to send request: DNSLookupError: DNS lookup failed: no results for hostname lookup: teh.entar.net.
daedricnet-inbound-worker     | 2025-02-21 10:39:10,993 - synapse.storage.txn - 787 - WARNING - PerspectivesKeyFetcher-24030 - [TXN OPERROR] {store_server_keys_response-7d990a} could not serialize access due to concurrent update
daedricnet-inbound-worker     |  0/5
daedricnet-inbound-worker     | 2025-02-21 10:39:11,010 - synapse.storage.txn - 787 - WARNING - PerspectivesKeyFetcher-24030 - [TXN OPERROR] {store_server_keys_response-7d990f} could not serialize access due to concurrent update
daedricnet-inbound-worker     |  0/5
daedricnet-inbound-worker     | 2025-02-21 10:39:11,010 - synapse.storage.txn - 787 - WARNING - PerspectivesKeyFetcher-24030 - [TXN OPERROR] {store_server_keys_response-7d990d} could not serialize access due to concurrent update
daedricnet-inbound-worker     |  0/5
daedricnet-inbound-worker     | 2025-02-21 10:39:11,023 - synapse.storage.txn - 787 - WARNING - PerspectivesKeyFetcher-24030 - [TXN OPERROR] {store_server_keys_response-7d990d} could not serialize access due to concurrent update
daedricnet-inbound-worker     |  1/5

Anything else that would be useful to know?

I can be reached via @daedric:aguiarvieira.pt or @daedric:daedric.net in Matrix.

[anoadragon453]

Hi @ricardo-duarte-av, apologies for the long lead time in responding.

This case is something Synapse has specific code to handle:

synapse/synapse/handlers/room_member.py

Lines 1975 to 2010 in 3b5b6f6

	async def remote_reject_invite(
	self,
	invite_event_id: str,
	txn_id: Optional[str],
	requester: Requester,
	content: JsonDict,
	) -> Tuple[str, int]:
	"""
	Rejects an out-of-band invite received from a remote user
	Implements RoomMemberHandler.remote_reject_invite
	"""
	invite_event = await self.store.get_event(invite_event_id)
	room_id = invite_event.room_id
	target_user = invite_event.state_key
	# first of all, try doing a rejection via the inviting server
	fed_handler = self.federation_handler
	try:
	inviter_id = UserID.from_string(invite_event.sender)
	event, stream_id = await fed_handler.do_remotely_reject_invite(
	[inviter_id.domain], room_id, target_user, content=content
	)
	return event.event_id, stream_id
	except Exception as e:
	# if we were unable to reject the invite, we will generate our own
	# leave event.
	#
	# The 'except' clause is very broad, but we need to
	# capture everything from DNS failures upwards
	#
	logger.warning("Failed to reject invite: %s", e)
	return await self._generate_local_out_of_band_leave(
	invite_event, txn_id, requester, content
	)

I'm surprised that "Failed to reject invite" isn't in your logs however.

The fact that matrix.org invites can be rejected, but (I think from your description) aguiarvieira.pt invites cannot be suggests an issue with federation. I also notice:

daedricnet-inbound-worker     | 2025-02-21 10:39:10,951 - synapse.crypto.keyring - 632 - WARNING - PerspectivesKeyFetcher-24030 - Key lookup failed from 'keys.daedric.net': Failed to send request: HttpResponseException: 500: Internal Server Error

in the logs, which should not be happening.

Is this something you and your users are still experiencing?

[ricardo-duarte-av]

Hello @anoadragon453 , no need to apologize, this is issue is really not high prio.

Regarding the error about keys.daedric.net, that's a Matrix Key Server that i have around so it's bound to produce errors.

As for the decline invitation, i'm really not sure what i can offer to help debug. Indeed app.matrix.org using a matrix.org acc can refuse the invite and it vanished. Even using app.element.io with my server has the same consequence:

Is there anything i should monitor in synapse logs to catch the flaw ?

[anoadragon453]

To be clear, I understand the issue are:

• User A on aguiarvieira.pt sends an invite to User D on daedric.net
• User D rejects the invite
• User A sees the rejection, but User D has the invite stuck in their Element Web client

and yet:

• if User M sends an invite to User D on daedric.net
• and User D rejects the invite, the invite disappears from their Element Web client correctly

@ricardo-duarte-av are you able to reproduce this reliably even today?

If so, have a search for "Failed to reject invite" and any "Internal Server Errors" near to the time where an invite fails to be rejected. Turning on DEBUG logging will also help (no need to turn on DEBUG logging for the storage layer/sql statements - those can contain sensitive information).

If you're able to reproduce and can post logs (ideally from both aguiarvieira.pt and daedric.net) then that would be helpful.

[ricardo-duarte-av]

If anyone, from any server, invites @daedric:aguiarvieira.pt, and it refuses the invite using Element Web (either my selfhosted one or app.element.io logged in to aguiarvieira.pt) the room is stuck.

I did my best to get the logs you need.

@daedric7:matrix.org invited @daedric:aguiarvieira.pt to room !vrDtvncIKYRaPdapfY:matrix.org.

Logs attached

join.log

[anoadragon453]

Thanks for the logs. Here are the important ones I pulled out:

av-client-events-1              | 2025-08-05 18:27:44,435 - synapse.handlers.room_member - 1137 - INFO - POST-106 - @daedric:aguiarvieira.pt rejects invite to !vrDtvncIKYRaPdapfY:matrix.org from @daedric7:matrix.org
av-client-events-1              | 2025-08-05 18:27:44,435 - synapse.http.client - 537 - DEBUG - POST-106 - HTTP POST b'{"_INT_STREAM_POS":{"instance_name":"av-client-events-1","streams":{"caches":61303171}},"content":{},"requester":{"access_token_id":591183,"app_server_id":null,"authenticated_entity":"@daedric:aguiarvieira.pt","device_id":"IEUNJCFAYF","is_guest":0,"scope":[],"shadow_banned":false,"user_id":"@daedric:aguiarvieira.pt"},"txn_id":null}' -> synapse-replication://master/_synapse/replication/remote_reject_invite/%24egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE/MEDVXBZPxo
av-client-events-1              | 2025-08-05 18:27:44,435 - synapse.http.client - 896 - DEBUG - POST-106 - Sending request POST synapse-replication://master/_synapse/replication/remote_reject_invite/%24egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE/MEDVXBZPxo
matrix-synapse-1                | 2025-08-05 18:27:44,436 - synapse.access.http.unix - 463 - DEBUG - POST-15746 - /var/sockets/av-synapse-replication.sock - unix - Received request: POST /_synapse/replication/remote_reject_invite/%24egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE/MEDVXBZPxo
matrix-synapse-1                | 2025-08-05 18:27:44,440 - synapse.http.matrixfederationclient - 688 - DEBUG - POST-15746 - {GET-O-226} [matrix.org] Sending request: GET matrix-federation://matrix.org/_matrix/federation/v1/make_leave/%21vrDtvncIKYRaPdapfY%3Amatrix.org/%40daedric%3Aaguiarvieira.pt; timeout 20.000000s
matrix-synapse-1                | 2025-08-05 18:27:44,735 - synapse.http.matrixfederationclient - 739 - DEBUG - POST-15746 - {GET-O-226} [matrix.org] Got response headers: 200 OK
matrix-synapse-1                | 2025-08-05 18:27:44,736 - synapse.http.matrixfederationclient - 364 - INFO - POST-15746 - {GET-O-226} [matrix.org] Completed request: 200 OK in 0.30 secs, got 469 bytes - GET matrix-federation://matrix.org/_matrix/federation/v1/make_leave/%21vrDtvncIKYRaPdapfY%3Amatrix.org/%40daedric%3Aaguiarvieira.pt
matrix-synapse-1                | 2025-08-05 18:27:44,736 - synapse.federation.federation_client - 1062 - DEBUG - POST-15746 - Got response to make_leave: {'auth_events': ['$G_V4QUF9JIX6gXYqn6t8hsnA44wCpYo_974x2zJwz4M', '$egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE', '$339ZYVndFu9MtXHMrBYwxCV0EwIF3sCw92SBlQ6_jP4'], 'prev_events': ['$egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE'], 'type': 'm.room.member', 'room_id': '!vrDtvncIKYRaPdapfY:matrix.org', 'sender': '@daedric:aguiarvieira.pt', 'content': {'membership': 'leave'}, 'depth': 11, 'state_key': '@daedric:aguiarvieira.pt', 'origin_server_ts': 1754418464674}
matrix-synapse-1                | 2025-08-05 18:27:44,736 - synapse.crypto.event_signing - 164 - DEBUG - POST-15746 - Signed event: b'{"auth_events":["$G_V4QUF9JIX6gXYqn6t8hsnA44wCpYo_974x2zJwz4M","$egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE","$339ZYVndFu9MtXHMrBYwxCV0EwIF3sCw92SBlQ6_jP4"],"content":{"membership":"leave"},"depth":11,"hashes":{"sha256":"eNp1n6c9H7KJ6hMhBXenJ+rxvpg7LAiPB/Sd2bp6eks"},"origin_server_ts":1754418464674,"prev_events":["$egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE"],"room_id":"!vrDtvncIKYRaPdapfY:matrix.org","sender":"@daedric:aguiarvieira.pt","signatures":{"aguiarvieira.pt":{"ed25519:a_Yofy":"8mA0MWzLYmfzx4KB8VisBfvt9SfgYOF04LyVixS6ZsZ5qYVBt/bPs3V5oXMzIPQ5ydOSsCNL7vpLkYNj/fEtCA"}},"state_key":"@daedric:aguiarvieira.pt","type":"m.room.member"}'
matrix-synapse-1                | 2025-08-05 18:27:44,737 - synapse.http.matrixfederationclient - 688 - DEBUG - POST-15746 - {PUT-O-227} [matrix.org] Sending request: PUT matrix-federation://matrix.org/_matrix/federation/v2/send_leave/%21vrDtvncIKYRaPdapfY%3Amatrix.org/%24IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA; timeout 10.000000s
matrix-synapse-1                | 2025-08-05 18:27:45,239 - synapse.http.matrixfederationclient - 739 - DEBUG - POST-15746 - {PUT-O-227} [matrix.org] Got response headers: 200 OK
matrix-synapse-1                | 2025-08-05 18:27:45,240 - synapse.http.matrixfederationclient - 364 - INFO - POST-15746 - {PUT-O-227} [matrix.org] Completed request: 200 OK in 0.50 secs, got 2 bytes - PUT matrix-federation://matrix.org/_matrix/federation/v2/send_leave/%21vrDtvncIKYRaPdapfY%3Amatrix.org/%24IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA
matrix-synapse-1                | 2025-08-05 18:27:45,240 - synapse.http.client - 537 - DEBUG - POST-15746 - HTTP POST b'{"_INT_STREAM_POS":{"instance_name":"master","streams":{"caches":61310464,"device_lists":32953766,"federation":0,"pushers":3629,"un_partial_stated_event":1356,"un_partial_stated_room":821}},"backfilled":false,"events":[{"context":{"app_service_id":null,"partial_state":false,"rejected":null,"state_delta_due_to_event":null,"state_group":null,"state_group_before_event":null,"state_group_deltas":[]},"event":{"auth_events":["$G_V4QUF9JIX6gXYqn6t8hsnA44wCpYo_974x2zJwz4M","$egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE","$339ZYVndFu9MtXHMrBYwxCV0EwIF3sCw92SBlQ6_jP4"],"content":{"membership":"leave"},"depth":11,"hashes":{"sha256":"eNp1n6c9H7KJ6hMhBXenJ+rxvpg7LAiPB/Sd2bp6eks"},"origin_server_ts":1754418464674,"prev_events":["$egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE"],"room_id":"!vrDtvncIKYRaPdapfY:matrix.org","sender":"@daedric:aguiarvieira.pt","signatures":{"aguiarvieira.pt":{"ed25519:a_Yofy":"8mA0MWzLYmfzx4KB8VisBfvt9SfgYOF04LyVixS6ZsZ5qYVBt/bPs3V5oXMzIPQ5ydOSsCNL7vpLkYNj/fEtCA"}},"state_key":"@daedric:aguiarvieira.pt","type":"m.room.member","unsigned":{"age_ts":1754418464736}},"event_format_version":3,"internal_metadata":{"out_of_band_membership":true},"outlier":true,"rejected_reason":null,"room_version":"10"}],"room_id":"!vrDtvncIKYRaPdapfY:matrix.org"}' -> synapse-replication://av-event-persister-2/_synapse/replication/fed_send_events/wAcvPBMESo
matrix-synapse-1                | 2025-08-05 18:27:45,240 - synapse.http.client - 896 - DEBUG - POST-15746 - Sending request POST synapse-replication://av-event-persister-2/_synapse/replication/fed_send_events/wAcvPBMESo
av-event-persister-2            | 2025-08-05 18:27:45,241 - synapse.replication.http.federation - 149 - INFO - POST-91 - Got batch of 1 events to persist to room !vrDtvncIKYRaPdapfY:matrix.org
av-event-persister-2            | 2025-08-05 18:27:45,266 - synapse.storage.databases.main.events - 2417 - DEBUG - persist_events-19 - _update_outliers_txn: events=['$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA'] have_persisted={}
av-event-persister-2            | 2025-08-05 18:27:45,266 - synapse.storage.databases.main.events - 2426 - DEBUG - persist_events-19 - _update_outliers_txn: event=$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA outlier=True outlier_persisted=None
matrix-synapse-1                | 2025-08-05 18:27:45,286 - synapse.replication.tcp.client - 346 - INFO - POST-15746 - Waiting for repl stream 'events' to reach 8063747 (av-event-persister-2); currently at: 8063746
av-sync-worker-2                | 2025-08-05 18:27:45,289 - synapse.replication.tcp.handler - 592 - DEBUG - process-replication-data-13165 - Received rdata events (av-event-persister-2) -> 8063747
av-sync-worker-1                | 2025-08-05 18:27:45,291 - synapse.replication.tcp.handler - 602 - DEBUG - replication_command_handler-139824941851920 - Handling 'POSITION events av-event-persister-1 8063746 8063747'
av-sync-worker-1                | 2025-08-05 18:27:45,291 - synapse.replication.tcp.handler - 610 - DEBUG - replication_command_handler-139824941851920 - Discarding redundant POSITION av-event-persister-1/events 8063746 8063747
matrix-synapse-1                | 2025-08-05 18:27:45,292 - synapse.access.http.unix - 508 - INFO - POST-15746 - /var/sockets/av-synapse-replication.sock - unix - {@daedric:aguiarvieira.pt} Processed request: 0.854sec/0.002sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 231B 200 "POST /_synapse/replication/remote_reject_invite/%24egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE/MEDVXBZPxo HTTP/1.1" "Synapse/1.134.0rc1" [1 dbevts]
av-client-events-1              | 2025-08-05 18:27:45,293 - synapse.http.client - 952 - INFO - POST-106 - Received response to POST synapse-replication://master/_synapse/replication/remote_reject_invite/%24egBiXirWVYCCJuokGMicmxPZyyBVJ4EdsIJdusfIlFE/MEDVXBZPxo: 200
av-sync-worker-1                | 2025-08-05 18:27:45,295 - synapse.storage.databases.main.events_worker - 1500 - DEBUG - GET-710 - Loading 1 events: {'$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA'}
av-sync-worker-1                | 2025-08-05 18:27:45,299 - synapse.handlers.sync - 2119 - INFO - GET-710 - User membership change between getting rooms and current token: @daedric:aguiarvieira.pt leave !vrDtvncIKYRaPdapfY:matrix.org
av-sync-worker-1                | 2025-08-05 18:27:45,299 - synapse.handlers.sync - 1832 - DEBUG - GET-710 - Calculating sync response for @daedric:aguiarvieira.pt between StreamToken(room: RoomStreamToken(stream: 8063746, topological: None, instances: {}), presence: 222909305, typing: 34, receipt: MultiWriterStreamToken(stream: 13836059, instances: {}), account_data: 1337070, push_rules: 907, to_device: 109897, device_list: 32953766, groups: 0, un_partial_stated_rooms: 821) and StreamToken(room: RoomStreamToken(stream: 8063747, topological: None, instances: {}), presence: 222909305, typing: 34, receipt: MultiWriterStreamToken(stream: 13836059, instances: {}), account_data: 1337070, push_rules: 907, to_device: 109897, device_list: 32953766, groups: 0, un_partial_stated_rooms: 821)
av-sync-worker-1                | 2025-08-05 18:27:45,300 - synapse.handlers.sync - 2558 - DEBUG - GET-710 - Membership changes in !vrDtvncIKYRaPdapfY:matrix.org: [$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA (leave)]
av-sync-worker-1                | 2025-08-05 18:27:45,301 - synapse.handlers.sync - 2119 - INFO - GET-707 - User membership change between getting rooms and current token: @daedric:aguiarvieira.pt leave !vrDtvncIKYRaPdapfY:matrix.org
av-sync-worker-1                | 2025-08-05 18:27:45,302 - synapse.handlers.sync - 1832 - DEBUG - GET-707 - Calculating sync response for @daedric:aguiarvieira.pt between StreamToken(room: RoomStreamToken(stream: 8063746, topological: None, instances: {}), presence: 222909305, typing: 34, receipt: MultiWriterStreamToken(stream: 13836059, instances: {}), account_data: 1337070, push_rules: 907, to_device: 109897, device_list: 32953766, groups: 0, un_partial_stated_rooms: 821) and StreamToken(room: RoomStreamToken(stream: 8063747, topological: None, instances: {}), presence: 222909305, typing: 34, receipt: MultiWriterStreamToken(stream: 13836059, instances: {}), account_data: 1337070, push_rules: 907, to_device: 109897, device_list: 32953766, groups: 0, un_partial_stated_rooms: 821)
av-sync-worker-1                | 2025-08-05 18:27:45,303 - synapse.storage.controllers.state - 470 - INFO - GET-710 - Failed to find any events in room !vrDtvncIKYRaPdapfY:matrix.org at RoomStreamToken(stream: 8063746, topological: None, instances: {})
av-sync-worker-1                | 2025-08-05 18:27:45,385 - synapse.visibility - 380 - DEBUG - GET-710 - _check_client_allowed_to_see_event(event=$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA): Returning out-of-band-membership event <FrozenEventV3 event_id=$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA, type=m.room.member, state_key=@daedric:aguiarvieira.pt, membership=leave, outlier=True>
av-sync-worker-1                | 2025-08-05 18:27:45,388 - synapse.visibility - 380 - DEBUG - GET-707 - _check_client_allowed_to_see_event(event=$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA): Returning out-of-band-membership event <FrozenEventV3 event_id=$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA, type=m.room.member, state_key=@daedric:aguiarvieira.pt, membership=leave, outlier=True>

Note that there were two simultaneous /sync requests here; gomuks (GET-707) and element web (GET-710).

The most important bit is this line from the sync worker:

av-sync-worker-1                | 2025-08-05 18:27:45,303 - synapse.storage.controllers.state - 470 - INFO - GET-710 - Failed to find any events in room !vrDtvncIKYRaPdapfY:matrix.org at RoomStreamToken(stream: 8063746, topological: None, instances: {})

The "Returning out-of-band-membership event" is actually coming from the push notification code - so unrelated to /sync.

"Failed to find any events in room" is logged here:

synapse/synapse/storage/controllers/state.py

Lines 444 to 475 in 8306cee

	# FIXME: This gets the state at the latest event before the stream ordering,
	# which might not be the same as the "current state" of the room at the time
	# of the stream token if there were multiple forward extremities at the time.
	last_event_id = (
	await self.stores.main.get_last_event_id_in_room_before_stream_ordering(
	room_id,
	end_token=stream_position.room_key,
	)
	)
	# FIXME: This will return incorrect results when there are timeline gaps. For
	# example, when you try to get a point in the room we haven't backfilled before.
	if last_event_id:
	state = await self.get_state_after_event(
	last_event_id,
	state_filter=state_filter,
	await_full_state=await_full_state,
	)
	else:
	# no events in this room - so presumably no state
	state = {}
	# (erikj) This should be rarely hit, but we've had some reports that
	# we get more state down gappy syncs than we should, so let's add
	# some logging.
	logger.info(
	"Failed to find any events in room %s at %s",
	room_id,
	stream_position.room_key,
	)

when the return value of get_last_event_id_in_room_before_stream_ordering is None. That function specifically ignores outlier events (such as your remote invite rejection). Hence why the event does not end up down /sync:

synapse/synapse/storage/databases/main/stream.py

Lines 1639 to 1651 in 8306cee

	AND NOT outlier
	AND rejections.event_id IS NULL
	ORDER BY stream_ordering DESC
	) AS a
	UNION ALL
	SELECT * FROM (
	SELECT instance_name, stream_ordering, topological_ordering, event_id
	FROM events
	LEFT JOIN rejections USING (event_id)
	WHERE room_id = ?
	%s
	AND stream_ordering <= ?
	AND NOT outlier

The question is why we ignore outliers in get_last_event_pos_in_room_before_stream_ordering, and whether that's the right thing to do. Perhaps there are cases where we do and don't want them, and thus we should have a way to toggle the conditional?

[anoadragon453]

The AND NOT outlier clause was added way back in https://github.com/matrix-org/synapse/pull/536/files#diff-5dab08610e070df2b5e7cba9acc37c38914bacb9c8b3ba3ef2a765d8933b2ddcR215.

Definition of an "outlier" event: https://element-hq.github.io/synapse/latest/development/room-dag-concepts.html#outliers

That documentation explicitly calls out the case we're discussing:

Out-of-band membership events

A special case of outlier events are some membership events for federated rooms
that we aren't full members of. For example:

• invites received over federation, before we join the room
• rejections for said invites
• knock events for rooms that we would like to join but have not yet joined.

In all the above cases, we don't have the state for the room, which is why they
are treated as outliers. They are a bit special though, in that they are
proactively sent to clients via /sync.

But where do we do this special handling for outlier leave membership events?

We expect this to be hit:

synapse/synapse/storage/databases/main/events.py

Lines 3012 to 3037 in 2b73bb0

	if (
	self.is_mine_id(event.state_key)
	and not inhibit_local_membership_updates
	and event.internal_metadata.is_outlier()
	and event.internal_metadata.is_out_of_band_membership()
	):
	# The only sort of out-of-band-membership events we expect to see here
	# are remote invites/knocks and LEAVE events corresponding to
	# rejected/retracted invites and rescinded knocks.
	assert event.type == EventTypes.Member
	assert event.membership in (
	Membership.INVITE,
	Membership.KNOCK,
	Membership.LEAVE,
	)
	self.db_pool.simple_upsert_txn(
	txn,
	table="local_current_membership",
	keyvalues={"room_id": event.room_id, "user_id": event.state_key},
	values={
	"event_id": event.event_id,
	"event_stream_ordering": event.internal_metadata.stream_ordering,
	"membership": event.membership,
	},
	)

@ricardo-duarte-av Could you run the following queries for me on aguiarvieira.pt's DB?

SELECT * FROM local_current_membership WHERE event_id = '$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA';
SELECT * FROM room_memberships WHERE event_id = '$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA';
SELECT * FROM events WHERE event_id = '$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA';
SELECT * FROM current_state_events WHERE room_id = '!vrDtvncIKYRaPdapfY:matrix.org';

I also just realised that aguiarvieira.pt has a shared_secret_authenticator.SharedSecretAuthProvider module. I wonder if that's significant...

[ricardo-duarte-av]

Hello @anoadragon453

SELECT * FROM local_current_membership WHERE event_id = '$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA';
            room_id             |         user_id          |                   event_id                   | membership | event_stream_ordering 
--------------------------------+--------------------------+----------------------------------------------+------------+-----------------------
 !vrDtvncIKYRaPdapfY:matrix.org | @daedric:aguiarvieira.pt | $IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA | leave      |               8063747

 SELECT * FROM room_memberships WHERE event_id = '$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA';
                   event_id                   |         user_id          |          sender          |            room_id             | membership | forgotten | display_name | avatar_url | event_stream_ordering | participant 
----------------------------------------------+--------------------------+--------------------------+--------------------------------+------------+-----------+--------------+------------+-----------------------+-------------
 $IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA | @daedric:aguiarvieira.pt | @daedric:aguiarvieira.pt | !vrDtvncIKYRaPdapfY:matrix.org | leave      |         0 |              |            |               8063747 | f

 SELECT * FROM events WHERE event_id = '$IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA';
 topological_ordering |                   event_id                   |     type      |            room_id             | content | unrecognized_keys | processed | outlier | depth | origin_server_ts |  received_ts  |          sender          | contains_url |    instance_name     | stream_ordering |        state_key         | rejection_reason 
----------------------+----------------------------------------------+---------------+--------------------------------+---------+-------------------+-----------+---------+-------+------------------+---------------+--------------------------+--------------+----------------------+-----------------+--------------------------+------------------
                   11 | $IpF92L9Z1gxoCNYm7J29b_Bnk9uNDVD0whhvF5GokSA | m.room.member | !vrDtvncIKYRaPdapfY:matrix.org |         |                   | t         | t       |    11 |    1754418464674 | 1754418465267 | @daedric:aguiarvieira.pt | f            | av-event-persister-2 |         8063747 | @daedric:aguiarvieira.pt |

SELECT * FROM current_state_events WHERE room_id = '!vrDtvncIKYRaPdapfY:matrix.org';
 event_id | room_id | type | state_key | membership | event_stream_ordering 
----------+---------+------+-----------+------------+-----------------------
(0 rows)

As for the shared_secret_authenticator.SharedSecretAuthProvider i can remove it and see if it influences.