`media_upload_limits`: If a user triggers this limit, Synapse does not clean up user-uploaded media, which can lead to junk files occupying disk space.

[YamatoRyou]

Title modified.

Description

Enabling media_upload_limits may generate junk files that can take up server disk space. The amount of junk files generated depends on whether users continue to upload after hitting the limit.
media_upload_limits: If a user triggers this limit, Synapse does not clean up user-uploaded media, which can lead to junk files consuming disk space. If this continues for a long time, the server's hard drive may become full of junk files.

Steps to reproduce

In order to highlight the problem, ensure that no files larger than 512 MB exist in the media repository of the current server instance before performing the following steps.

1. Configure Synapse to enable media_upload_limits:

media_upload_limits:
- time_period: 24h
- max_size: 1536M
# This configuration means that the server allows each user to upload a maximum of 1536 MB of media in a 24-hour period.

2. Continue configuring Synapse to allow large file uploads. Here, increase the limit to 512 MB:

max_upload_size: 512M

3. As a user, upload three 512 MB files in a room (using a non-encrypted room).
4. After the upload is complete, I will query the media uploaded by the user (I will bypass Synapse and execute SQL directly).

SELECT
  media_id,
  media_length,
  created_ts,
  user_id
FROM
  public.local_media_repository
WHERE
  user_id = '@someuser:example.com'
  AND
  media_length >= 524288000;

5. Count the media in the Synapse media repository that are 524288000 bytes or larger and note the number of media.
6. Upload a 512 MB file again (this upload will trigger the limit).
7. Repeat steps 4 & 5.

---

You will notice that the last uploaded media has no record in the database, but is fully stored in the media repository. Four 512 MB files appear at the file system level, but only three are returned in the Synapse database.
This means that the fourth uploaded file is not managed by Synapse, but still consumes disk space (indicates that it is an orphan file). If a user repeatedly uploads media even after reaching the limit during a period, More media will continue to occupy the disk (and this media remains unmanaged by Synapse).

For Synapse instances with relatively relaxed media upload policies, this phenomenon can be easily exploited by malicious users.

Homeserver

Self-hosted

Synapse Version

Synapse 1.138.0

Installation Method

Docker (matrixdotorg/synapse)

Database

Postgres 16, n/a, no, no

Workers

Single process

Platform

Synology DiskStation Manager 6.2.3 (Linux 3.10.105, x64)

Configuration

media_upload_limits:
- time_period: 24h
- max_size: 1536M

max_upload_size: 512M

Relevant log output

n/a

Anything else that would be useful to know?

Currently known solutions:
a) Strictly restrict user uploads;
b) Use crontab Such media is periodically deleted using other methods. This requires a "reverse match" approach: using a rule to convert the file path to an MXC ID and feed it into a SQL query. If the ID isn't found in the database, the corresponding file is considered orphaned and ultimately deleted.

My (perhaps unprofessional) view of this approach is:
Before the upload begins, only the file size is sent to the server for calculation. If this value triggers the limit, the upload is blocked. Ultimately, this prevents such media from taking up the server's hard drive. The current implementation is that as a user, I have to transmit data to the server again to know whether I am about to (or have already) triggered the limit.

---

I've written several scripts that work together to clean up these files and have iterated through several versions, but after testing, I've found their effectiveness to be mediocre:
The scripts are log-driven and therefore completely dependent on Synapse logs. If there's a gap in the logs the scripts rely on, or if hardware resources are limited, the script's workflow will become disrupted and fail to work as expected. I'd rather Synapse help me resolve this issue.

[anoadragon453]

Thanks for flagging this!

This is due to Synapse checking media_upload_limits after the file has been stored to disk:

synapse/synapse/media/media_repository.py

Lines 332 to 387 in 4367fb2

	fname = await self.media_storage.store_file(sha256reader.wrap(), file_info)
	sha256 = sha256reader.hexdigest()
	should_quarantine = await self.store.get_is_hash_quarantined(sha256)
	logger.info("Stored local media in file %r", fname)
	if should_quarantine:
	logger.warning(
	"Media has been automatically quarantined as it matched existing quarantined media"
	)
	# Check that the user has not exceeded any of the media upload limits.
	# Use limits from module API if provided
	media_upload_limits = (
	await self.media_repository_callbacks.get_media_upload_limits_for_user(
	auth_user.to_string()
	)
	)
	# Otherwise use the default limits from config
	if media_upload_limits is None:
	# Note: the media upload limits are sorted so larger time periods are
	# first.
	media_upload_limits = self.default_media_upload_limits
	# This is the total size of media uploaded by the user in the last
	# `time_period_ms` milliseconds, or None if we haven't checked yet.
	uploaded_media_size: Optional[int] = None
	for limit in media_upload_limits:
	# We only need to check the amount of media uploaded by the user in
	# this latest (smaller) time period if the amount of media uploaded
	# in a previous (larger) time period is below the limit.
	#
	# This optimization means that in the common case where the user
	# hasn't uploaded much media, we only need to query the database
	# once.
	if (
	uploaded_media_size is None
	or uploaded_media_size + content_length > limit.max_bytes
	):
	uploaded_media_size = await self.store.get_media_uploaded_size_for_user(
	user_id=auth_user.to_string(), time_period_ms=limit.time_period_ms
	)
	if uploaded_media_size + content_length > limit.max_bytes:
	await self.media_repository_callbacks.on_media_upload_limit_exceeded(
	user_id=auth_user.to_string(),
	limit=limit,
	sent_bytes=uploaded_media_size,
	attempted_bytes=content_length,
	)
	raise SynapseError(
	400, "Media upload limit exceeded", Codes.RESOURCE_LIMIT_EXCEEDED
	)

and not deleting the file afterwards. The impact of this is that the feature does not really prevent someone from uploading a bunch of media to your server (the same situation as if media_upload_limits were disabled).

A good path forwards would be to check the content_length against the media_upload_limits before storing the file (perhaps here).

[YamatoRyou]

Thanks for flagging this!

This is due to Synapse checking media_upload_limits after the file has been stored to disk:

synapse/synapse/media/media_repository.py

Lines 332 to 387 in 4367fb2

fname = await self.media_storage.store_file(sha256reader.wrap(), file_info)
sha256 = sha256reader.hexdigest()
should_quarantine = await self.store.get_is_hash_quarantined(sha256)

logger.info("Stored local media in file %r", fname)

if should_quarantine:
logger.warning(
"Media has been automatically quarantined as it matched existing quarantined media"
)

Check that the user has not exceeded any of the media upload limits.

Use limits from module API if provided

media_upload_limits = (
await self.media_repository_callbacks.get_media_upload_limits_for_user(
auth_user.to_string()
)
)

Otherwise use the default limits from config

if media_upload_limits is None:
# Note: the media upload limits are sorted so larger time periods are
# first.
media_upload_limits = self.default_media_upload_limits

This is the total size of media uploaded by the user in the last

time_period_ms milliseconds, or None if we haven't checked yet.

uploaded_media_size: Optional[int] = None

for limit in media_upload_limits:
# We only need to check the amount of media uploaded by the user in
# this latest (smaller) time period if the amount of media uploaded
# in a previous (larger) time period is below the limit.
#
# This optimization means that in the common case where the user
# hasn't uploaded much media, we only need to query the database
# once.
if (
uploaded_media_size is None
or uploaded_media_size + content_length > limit.max_bytes
):
uploaded_media_size = await self.store.get_media_uploaded_size_for_user(
user_id=auth_user.to_string(), time_period_ms=limit.time_period_ms
)

 if uploaded_media_size + content_length > limit.max_bytes: 
     await self.media_repository_callbacks.on_media_upload_limit_exceeded( 
         user_id=auth_user.to_string(), 
         limit=limit, 
         sent_bytes=uploaded_media_size, 
         attempted_bytes=content_length, 
     ) 
     raise SynapseError( 
         400, "Media upload limit exceeded", Codes.RESOURCE_LIMIT_EXCEEDED 
     ) 

and not deleting the file afterwards. The impact of this is that the feature does not really prevent someone from uploading a bunch of media to your server (the same situation as if media_upload_limits were disabled).

A good path forwards would be to check the content_length against the media_upload_limits before storing the file (perhaps here).

While I'm a bit confused by this code, I discovered that the media file path is reflected in the logs (an oversight on my part). It seems I can modify the script to make it more efficient. However, it still:

• doesn't prevent users from continuing to upload after hitting the limit;
• doesn't let users know if they're about to hit the limit before uploading their next file.

Since users can only tell if they've hit the limit after uploading data to the server, this also presents another problem: if users are using a data-based network, this could waste their data.

---

Another question: Can the HTTP status code 400 (which appears in the logs) be the sole indicator that a user has hit the limit by uploading too much media?

[anoadragon453]

Can the HTTP status code 400 (which appears in the logs) be the sole indicator that a user has hit the limit by uploading too much media?

@YamatoRyou I would key on 400 + RESOURCE_LIMIT_EXCEEDED, as a 400 can be raised for M_UNKNOWN errors as well (among other things).

[YamatoRyou]

Can the HTTP status code 400 (which appears in the logs) be the sole indicator that a user has hit the limit by uploading too much media?

@YamatoRyou I would key on 400 + RESOURCE_LIMIT_EXCEEDED, as a 400 can be raised for M_UNKNOWN errors as well (among other things).

I found the following entry in the log:

synapse.http.server - 133 - INFO - POST-3447190 - <XForwardedForRequest at 0x7f0b11b3d370 method='POST' uri='/_matrix/media/v3/upload?filename=********************' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - Media upload limit exceeded

However, it doesn't indicate who triggered the restriction. Another line is needed to help identify the person:

synapse.access.http.8008 - 515 - INFO - POST-3447190 - ********************** - 8008 - {@test8:example.com} Processed request: 9.644sec/0.002sec (3.078sec, 5.408sec) (0.001sec/0.002sec/2) 77B 400 "POST /_matrix/media/v3/upload?filename=******************** HTTP/1.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.112 Chrome/138.0.7204.224 Electron/37.3.0 Safari/537.36" [0 dbevts]

The previous line already contains the error message and file name. Here's a small suggestion: it would be much better if the user ID was included in the previous line.