Synapse sometimes fails to deliver media

[neil-lcv-cs]

Description

Hello,

Our Matrix instance sometimes fails to deliver media to our Element web client (v1.12.0, though the bug happened prior to this version).

Media loads well most of the time. But several times a day, in a seemingly random fashion, Synapse would respond 404 to all existing medias, both remote and local. The issue seems to solve itself after a few minutes.

We did not test with other clients. Authenticated media feature is enabled (I suppose it may be related).

I don’t know if the culprit is Synapse or Element, but I would say Synapse since the query seems to be valid on the Element side.

Steps to reproduce

The bug is very inconsistent, thus quite hard to reproduce.

• Open an image from the Element web client.
  • The client will likely issue this kind of request: GET https://matrix.lacontrevoie.fr/_matrix/media/v3/download/lacontrevoie.fr/<MEDIA_ID>?allow_redirect=true
• If the bug triggers, the image just won’t display in Element. In the dev console, Synapse responds 404 − {"errcode":"M_NOT_FOUND","error":"Not found"} and will do so for all media during a few minutes.
• If you refresh the page during this time (or even clear the Element cache and reload), all users and room avatars will disappear, as well as the image thumbnails. Synapse will respond 404 for all media.
• Wait a few minutes.
• Try to open the same image again. The very same request and media ID will be queried by the client, though Synapse will respond 200 with the image.

Here’s a filtered view of Firefox’s development console, network tab:

Homeserver

matrix.lacontrevoie.fr

Synapse Version

1.138.2

Installation Method

Docker (matrixdotorg/synapse)

Database

Fresh install from PostgreSQL 17.6

Workers

Single process

Platform

Host: Debian Trixie, Linux 6.12.41, AMD64 architecture, dedicated server with 64GB RAM, Docker 28.4.0.

Configuration

We are using MAS and a few other stuff (EC RTC setup, appservices, sticker widget…).
The used room is not encrypted and uses the default media policy.
Presence is disabled.

Media retention uses the following policy:

retention:
  enabled: true
  default_policy:
      max_lifetime: 2y
  allowed_lifetime_min: 7d
  allowed_lifetime_max: 3y

media_retention:
  local_media_lifetime: 1y
  remote_media_lifetime: 180d

The following experimental features are enabled:

experimental_features:
  msc3266_enabled: true
  msc4222_enabled: true
  msc4140_enabled: true
  msc3861:
    enabled: true

The following module is enabled: synapse_http_antispam.HTTPAntispam

The following appservices are used and enabled:

• mautrix-discord
• draupnir

We are not using any storage provider such as S3. The media is directly stored on block storage, the classic way.

Relevant log output

Grep on the Synapse logs with the media ID. Censored some data.
There is a huge time gap between the 404 and the 200 response (30 minutes) because I was in the middle of writing this ticket, though the issue usually solves itself in about 5 minutes.

2025-09-28 15:22:58,549 - synapse.http.server - 133 - INFO - GET-118417 - <XForwardedForRequest at 0x<memory_addr> method='GET' uri='/_matrix/media/v3/thumbnail/lacontrevoie.fr/<media_id>?width=800&height=600&method=scale&allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found
2025-09-28 15:22:58,550 - synapse.access.http.8008 - 515 - INFO - GET-118417 - <client_ip> - 8008 - {None} Processed request: 0.003sec/0.001sec (0.001sec, 0.001sec) (0.000sec/0.001sec/1) 45B 404 "GET /_matrix/media/v3/thumbnail/lacontrevoie.fr/<media_id>?width=800&height=600&method=scale&allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
2025-09-28 15:22:58,649 - synapse.http.server - 133 - INFO - GET-118453 - <XForwardedForRequest at 0x<memory_addr> method='GET' uri='/_matrix/media/v3/download/lacontrevoie.fr/<media_id>?allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found
2025-09-28 15:22:58,649 - synapse.access.http.8008 - 515 - INFO - GET-118453 - <client_ip> - 8008 - {None} Processed request: 0.001sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.001sec/1) 45B 404 "GET /_matrix/media/v3/download/lacontrevoie.fr/<media_id>?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
2025-09-28 15:23:06,442 - synapse.http.server - 133 - INFO - GET-118458 - <XForwardedForRequest at 0x<memory_addr> method='GET' uri='/_matrix/media/v3/download/lacontrevoie.fr/<media_id>?allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found
2025-09-28 15:23:06,444 - synapse.access.http.8008 - 515 - INFO - GET-118458 - <client_ip> - 8008 - {None} Processed request: 0.003sec/0.001sec (0.001sec, 0.000sec) (0.000sec/0.001sec/1) 45B 404 "GET /_matrix/media/v3/download/lacontrevoie.fr/<media_id>?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
2025-09-28 15:23:12,419 - synapse.http.server - 133 - INFO - GET-118460 - <XForwardedForRequest at 0x<memory_addr> method='GET' uri='/_matrix/media/v3/download/lacontrevoie.fr/<media_id>?allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found
2025-09-28 15:23:12,420 - synapse.access.http.8008 - 515 - INFO - GET-118460 - <client_ip> - 8008 - {None} Processed request: 0.002sec/0.001sec (0.001sec, 0.000sec) (0.000sec/0.001sec/1) 45B 404 "GET /_matrix/media/v3/download/lacontrevoie.fr/<media_id>?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
2025-09-28 15:57:23,103 - synapse.access.http.8008 - 515 - INFO - GET-119245 - <client_ip> - 8008 - {@neil:lacontrevoie.fr} Processed request: 0.007sec/-0.000sec (0.001sec, 0.001sec) (0.000sec/0.001sec/1) 256088B 200 "GET /_matrix/client/v1/media/download/lacontrevoie.fr/<media_id>?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]

Anything else that would be useful to know?

• Default and tested room version : 12
• Authenticated media feature enabled

Reverse-proxy (Caddy) configuration is the following (irrelevant lines omitted):

matrix.lacontrevoie.fr {
    header Content-Security-Policy "default-src 'self' 'unsafe-inline';"

    route {
        # unrelated
        header /.well-known/* {
            ...
        }

        # forward authentication to MAS
        @matrix_auth path_regexp ^/_matrix/client/(.*)/(login|logout|refresh)
        reverse_proxy @matrix_auth <MAS_INTERNAL_IP>

        # forward reports to Draupnir
        @matrix_reports path_regexp re_reports ^/_matrix/client/(r0|v3)/rooms/(?<room_id>[^/]*)/report/(?<event_id>.*)$
        handle @matrix_reports {
            rewrite /api/1/report/{re.re_reports.room_id}/{re.re_reports.event_id}
            reverse_proxy {
                to <DRAUPNIR_INTERNAL_IP>
                header_down ...
            }
        }

        # Forward the rest to Synapse
        reverse_proxy <SYNAPSE_INTERNAL_IP>
    }
}

[anoadragon453]

I can't explain the timing, but if you upload media with enable_authenticated_media set to true then attempting to download that media using the old download/thumbnail endpoints (/_matrix/media/(r0|v3|v1)/(download|thumbnail)) as your logs show, will result in a 404.

Are you using an older client version which does not support the newer authenticated media endpoints (_matrix/client/v1/media/(download|thumbnail)?

[neil-lcv-cs]

Are you using an older client version which does not support the newer authenticated media endpoints (_matrix/client/v1/media/(download|thumbnail)?

As indicated, I am using Element Web in version 1.12.0, which is the latest available.

$ curl https://element.lacontrevoie.fr/version
v1.12.0

If necessary, the client configuration I am using is here:
https://element.lacontrevoie.fr/config.json

[neil-lcv-cs]

This bug is still a thing on Element Web v1.12.2 and Synapse v1.141.0.

[TommyTran732]

Same issue on Element Web v1.12.4 and Synapse v1.143.0

[neil-lcv-cs]

Hi,
Still a thing as of Synapse v1.150.0, Element 1.12.13 (both latest versions).

Sometimes it even takes more than 3 minutes before media are available again.

The queried endpoint on this screenshot is https://matrix.lacontrevoie.fr/_matrix/media/v3/thumbnail/lacontrevoie.fr/qAD......?width=20&height=20&method=crop&allow_redirect=true.

The returned JSON until the image becomes available again is {"errcode":"M_NOT_FOUND","error":"Not found"}.

This is what is going on server-side:

2026-04-04 15:48:45,142 - synapse.http.server - 131 - INFO - GET-906845 - <XForwardedForRequest at 0xXXXX method='GET' uri='/_matrix/media/v3/thumbnail/lacontrevoie.fr/qAD.......?width=20&height=20&method=crop&allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found

Until it just… works. I see no difference or specific log information that could indicate why the server stops serving all media for a few minutes then just works again. I don't know why it is just happening at random times of the day either. I guess I should try disabling authenticated media as it seems to be source of most bugs.

@anoadragon453 does it satisfy your need for information as indicated by the X-Needs-Info tag? Please feel free to ask if you need me to follow any further debugging steps.

[Lippiece]

Could it be, by chance, a result of using HTTP/1.1?

[neil-lcv-cs]

Could it be, by chance, a result of using HTTP/1.1?

I would say it seems unlikely, since the images ends up being sent over HTTP/1.1 when Synapse finally decides to send them.
Both the successful and failed (404) queries sent from the client are exactly the same.

On a side note, the server (Caddy) supports HTTP 1 to 3, and most of the data is exchanged with HTTP/3. For some reason, Firefox uses HTTP/1 when the query initiator is lazy-img, and HTTP/3 otherwise.

[neil-lcv-cs]

On a side note, I recently tried a few times to replicate this issue on a Chromium-based browser (Vivaldi) and couldn’t, which hints for a client or a session-related issue (I thought of element-hq/element-web#28370, though the media are only unavailable for a few minutes in our case, so it’s probably slightly different).

[ginkel]

Experienced this issue yesterday evening. Images weren't showing up on Element Web in a channel, but other clients (such as FluffyChat) were unaffected.

2026-04-14 21:55:44,432 - synapse.http.server - 131 - INFO - GET-469017 - <XForwardedForRequest at 0x71c4c11391d0 method='GET' uri='/_matrix/media/v3/thumbnail/***/ppd******XbAlzFnpjlUIBKZ?width=37&height=37&method=crop&allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found

The interesting thing is that a PDF sent at the same time to me in the channel could also not be decrypted by Element Web (unfortunately, I don't remember the exact error message, but the issue clearly was about decryption).

Both issues vanished within ~ 20 minutes.

Element Web v1.12.13, Synapse v1.150.0-ess.1

[neil-lcv-cs]

The interesting thing is that a PDF sent at the same time to me in the channel could also not be decrypted by Element Web

Yep, same issue. All media, including PDF, videos and images, stops being delivered, returning 404 for a little while, then magically reappearing.

@ginkel by any chance, are you using Firefox?

[ginkel]

@ginkel by any chance, are you using Firefox?

Nope, Chromium (currently Version 147.0.7727.55 (Official Build) Arch Linux (64-bit)).