merge in develop, and...#9
Merged
clokep merged 54 commits intoclokep:psycopg3from Apr 28, 2026
Merged
Conversation
…ng keys (element-hq#19649) The Rust port of `KNOWN_ROOM_VERSIONS` (element-hq#19589) made `__contains__` strict about key types, raising `TypeError` when called with `None` instead of returning `False` like a Python dict would. This broke `/sync` for rooms with a NULL `room_version` in the database. ``` File "/home/synapse/src/synapse/handlers/sync.py", line 2628, in _get_room_changes_for_initial_sync if event.room_version_id not in KNOWN_ROOM_VERSIONS: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: argument 'key': 'NoneType' object cannot be cast as 'str' ```
Synapse uses topological ordering for initial sync (first time a room is sent down `/sync`), https://github.com/element-hq/synapse/blob/2e9b8202f0a1a8ceba9f02bb5ec227498d51dcbd/synapse/handlers/sync.py#L768-L805
…portdb` (element-hq#19675) Part of element-hq#19671 Spawning from [discussion in `#synapse-dev:matrix.org`](https://matrix.to/#/!i5D5LLct_DYG-4hQprLzrxdbZ580U9UB6AEgFnk6rZQ/$Z3nqbH0Qy21FWC3qJOim6LSRCRpJ3pxV5DLXm98IA6I?via=element.io&via=matrix.org&via=beeper.com) with roots in element-hq#19558 (comment). As trialed/discovered by @turt2live alongside @reivilibre and @clokep ❤️ ### Why is this necessary? If you forget to add `_setup_sequence(...)`, you can run into the following error if there is 1 row in SQLite and then you use the `portdb` script to try to migrate to Postgres (as [explained](https://matrix.to/#/!i5D5LLct_DYG-4hQprLzrxdbZ580U9UB6AEgFnk6rZQ/$mHU6dcTNL7NMfKBCJUekCh7vDj1lr1GDjriZQl7oeeU?via=element.io&via=matrix.org&via=beeper.com) by @reivilibre) ``` Postgres sequence 'quarantined_media_id_seq' is inconsistent with associated stream position of 'quarantined_media' in the 'stream_positions' table. ```
…9558) Fixes element-hq#19352 (See issue for history of this feature and previous PRs) > First, a [naive implementation](element-hq#19268) of the endpoint was introduced, but it quickly ran into [performance issues on query](element-hq#19312) and [long startup times](element-hq#19349), leading to its [removal](element-hq#19351). It also didn't actually work, and would fail to expose media when it was "unquarantined", so a [partial fix](element-hq#19308) was attempted, where the suggested direction is to use a [stream](https://element-hq.github.io/synapse/latest/development/synapse_architecture/streams.html#cheatsheet-for-creating-a-new-stream) instead of a timestamp column. This PR re-introduces the API building on the previous feedback: * Adds a stream which tracks when media becomes (un)quarantined. * Runs a background update to capture already-quarantined media. * Adds a new admin API to return rows from the stream table. We track both quarantine and unquarantine actions in the stream to allow downstream consumers to process the records appropriately. Namely, to allow our Synapse exchange in HMA to remove hashes for unquarantined media (use case further explained in the [issue](element-hq#19352)). **Note**: This knowingly does not capture all cases of media being quarantined. Other call sites are lower priority for T&S, and can be addressed in a future PR. ~~An issue will be created after this PR is merged to track those sites.~~ element-hq#19672 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: turt2live <1190097+turt2live@users.noreply.github.com> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Eric Eastwood <erice@element.io>
…element-hq#19677) Following up on element-hq#19558 (comment) Changelog for this PR is intended to overlap with the above PR. `get_current_quarantined_media_stream_id` wasn't being used anywhere else, so we can replace it like we do in this PR. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
…lement-hq#19630) Incorrectly labeled in matrix-org/synapse#13535. `maybe_backfill` already accurately describes `limit` (introduced in matrix-org/synapse#8349) Spotted in element-hq#19611 (comment)
The spec says `device_keys` may be omitted, but not set to `null`. This was temporarily allowed as a workaround for misbehaving clients (see element-hq#19023), which have since been fixed. Fixes element-hq#19030
This is to make it easier to port to Rust, as well as making things conceptually simpler. Two changes: 1. Remove the `__getitem__` interface on events 2. Remove `.user_id` as an alias of `.sender`.
This adds a way to re-sign all locally-created events with a new signing key, which is useful when rotating server signing keys. This doesn't trigger automatically, instead needs to be triggered when needed via the admin API. c.f. matrix-org/internal-config#1670 (comment) for internal discussion. --------- Co-authored-by: Kegan Dougall <kegan@element.io> Co-authored-by: Erik Johnston <erikj@element.io>
…19453) Follows: element-hq#19365 Part of: MSC4354 Sticky Events (experimental feature element-hq#19409) This PR introduces a `spam_checker_spammy` flag, analogous to `policy_server_spammy`, as an explicit flag that an event was decided to be spammy by a spam-checker module. The original Sticky Events PR (element-hq#18968) just reused `policy_server_spammy`, but it didn't sit right with me because we (at least appear to be experimenting with features that) allow users to opt-in to seeing `policy_server_spammy` events (presumably for moderation purposes). Keeping these flags separate felt best, therefore. As for why we need this flag: soon soft-failed status won't be permanent, at least for sticky events. The spam checker modules currently work by making events soft-failed. We want to prevent spammy events from getting reconsidered/un-soft-failed, so it seems like we need a flag to track spam-checker spamminess *separately* from soft-failed. Should be commit-by-commit friendly, but is also small. --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
…ed with `SQLITE_DBCONFIG_DEFENSIVE` by default, such as macOS. (element-hq#19690) Fixes: element-hq#19616 This caused 2+ people trouble now, so worth batting away with a low-effort change if we can. Only seen on macOS so far, but nothing stops SQLite being configured in defensive mode by default on other platforms, so it is not necessarily entirely specific to macOS. We *could* also do this for Python < 3.12 but it'd be more effort and I don't know if it's worth it. (For context @kegsay says the interpreter with this problem was installed through `pyenv install`.) --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
…-hq#19640) When we return events to clients we need to annotate them with the membership of the user at the time of the event, in the `unsigned` section. We already check the membership at the event during the visibility checks, and so we annotate events there. However, since this a per-user field we end up having to clone the event in question. Instead, let's add a `FilteredEvent` class that is returned by the visibility checks, which allows returning the membership without editing the event. This has three benefits: 1. Avoids the clones of the event. 2. Allows us to statically check that we have filtered events before returning them to clients. 3. We no longer edit `unsigned` data after event deserialization, this makes it easier to port the event class to Rust. The last benefit is why we're doing this *now*, however IMV it shouldn't affect whether we want this change or not. Reviewable commit-by-commit --------- Co-authored-by: Olivier 'reivilibre' <oliverw@element.io>
…9669) This moves the reference counting from PyO3 into standard Rust types, allowing the class to be used natively from Rust without needing a Python runtime.
…q#19694) Fixes element-hq#19692 Introduced by element-hq#19558 --------- Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
…hentication with Legacy Single Sign-On. (element-hq#19693) Closes: element-hq#19688 Part of: MSC4450 whose Experimental Feature tracking issue is element-hq#19691 Add an unstable, namespaced `idp_id` query parameter to `fallback/web` \ This allows clients to specify the identity provider they'd like to log in with for SSO when they have multiple upstream IdPs associated with their account. Previously, Synapse would just pick one arbitrarily. But this was undesirable as you may want to use a different one at that point in time. When logging in, the user is able to choose when IdP they use - during UIA (which uses fallback auth mechanism) they should be able to do the same. ----- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org> Co-authored-by: Andrew Morgan <andrew@amorgan.xyz> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
This implements [MSC4242: State DAGs](matrix-org/matrix-spec-proposals#4242), without support for federation. A general overview: - It adds a new room version and new event type. - It adds a new field `calculated_auth_event_ids` to internal metadata. - It stores the state DAG via new state DAG edges / forward extremities tables. - It adds new auth rules as per the MSC. - It uses the new `prev_state_events` field instead of `prev_event_ids()` when doing state resolution. Complement tests: matrix-org/complement#841 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Eric Eastwood <erice@element.io>
Adds [Admin API](https://element-hq.github.io/synapse/latest/usage/administration/admin_api/index.html) endpoints to list, fetch and delete user reports from the homeserver. Follows on from element-hq#18120, which added the endpoints to report users.
…9473) Fixes element-hq#13043 The usages of the table mostly already correctly handled if we don't have old entries, as that was needed when we first added the table. I arbitrarily set the prune time to 30 days. The only use for old entries is for sync streams that haven't synced since then, and we should very rarely see sync streams that haven't been used in 30 days. Reviewable commit-by-commit. --------- Co-authored-by: Olivier 'reivilibre' <oliverw@element.io> Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>
Spawning from the follow-up necessary when adding a new stream (element-hq#19694)
Follows on from element-hq#19473. We should be recording where we have deleted up to in the same transaction as we perform the delete, rather than at the end. This code only starts deleting rows after a month (and the original PR isn't in a release yet), so no server should have run into this problem yet. Also let's log more regularly, as the initial set of deletions will likely take a long time.
Both `__getitem__` and `.user_id` were removed in element-hq#19680 to simplify the event class. However, `EventBase` is exposed to modules who might also make use of those methods, so let's reinstate them (but otherwise not reinstate the usage of them in the code).
…nt and profile updates are disabled (element-hq#19398) Currently synapse returns `M_FORBIDDEN` when trying to use the account deactivation API, if the server admin disabled displayname changes. This is undesirable, since it prevents GDPR erasure without admin interaction. The admin API seems to work fine though. This also only seems to affect the deactivate API, when the erase flag is true. Relevant endpoint: https://spec.matrix.org/latest/client-server-api/#post_matrixclientv3accountdeactivate This change only removes the checked for condition that the displayname and profile avatar are allowed to be changed per the configuration setting. If a user is deleting themselves, why is that denied? There did not seem to be a basic test for this endpoint that checks the `erase` usage, so that was added as well as checking the above mentioned behavior.
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
…re new results to sync (element-hq#19714) This fixes the bug described in element-hq#19713 (and double-checked against the SDK integration test, which now passes with this change). A sync response must be returned immediately if a room subscription configuration change caused a new non-empty response (checked with `if response` in the code) to be produced. Fixes element-hq#19713. Fixes element-hq#18844. --------- Co-authored-by: Erik Johnston <erik@matrix.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…9720) MSC3266 is merged in v1.15, let's stabilize it as part of element-hq#18731 1. Add support for the stable `/_matrix/client/v1/room_summary/` endpoint, keeping both unstable endpoints for compat 2. Remove the experimental `msc3266_enabled` flag
…lement-hq#19710) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
When upgrading a room to v12, we accidentally ended up mutating the content of the old power level. Since we cache events, this meant any future usage of the old power level event would see the wrong content (until it dropped from the cache). This meant that the creator of the new room would not be able to perform admin actions on the old room. Any federation requests for the event would fail the hash checks, since the content had been changed. All in all, quite a nasty bug.
…lement-hq#19711) Known problems: element-hq#18117 As a follow-up, we should consider removing this config option altogether. It's "expensive" and claims to "prevent bugs" but actually introduces a whole new class of bugs. It could be re-introduced with a more holistic solution to the typing. Or a completely new approach (safe mode that blows up when someone mutates the event content, always make deep clones when handing out references, etc) The `use_frozen_dict` config option was there [since inception](element-hq@a7b65bd) but was only recently [documented](element-hq#18122) for completeness sake.
…hq#18475) Implements: [MSC4163: Make ACLs apply to EDUs](matrix-org/matrix-spec-proposals#4163) Part of element-hq#18118 to declare support for Matrix v1.13 Complement PR: ~~matrix-org/complement#783 -> matrix-org/complement#862 --------- Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: Quentin Gliech <quenting@element.io>
…hq#19722) Part of MSC4311: invites and knocks should contain the create event (stripped state for the client API) Part of element-hq#19414
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update poetry lock hash and deal with complement workflow changing to a reusable workflow file
The base
developbranch here will need a 'sync' please to make this easier to look at