Synapse: fix requests being routed to initial-synchrotron incorrectly#632
Merged
Synapse: fix requests being routed to initial-synchrotron incorrectly#632
Conversation
dyff of changes in rendered templates of CI manifestsFull contents of manifests and dyffs are available in https://github.com/element-hq/ess-helm/actions/runs/16598769229/artifacts/3639677811 pytest-synapse-values.yaml@@ ConfigMap/ess-ci/release-name-haproxy - data.haproxy.cfg @@
global
maxconn 40000
log stdout format raw local0 info
[151 lines unchanged)]
acl has_get_map path -m reg -M -f /synapse/path_map_file_get
http-request set-var(req.backend) path,map_reg(/synapse/path_map_file_get,main) if has_get_map METH_GET
http-request set-var(req.backend) path,map_reg(/synapse/path_map_file,main) unless { var(req.backend) -m found }
+
+ acl has_available_initial_syncs nbsrv('synapse-initial-synchrotron') ge 1
+
+ # Set to the initial-synchrotron backend if it is one of these magic paths AND we have workers in the initial-synchrotron backend
+ # This means that we don't update the backend from synchrotron if that's configured but there's no initial-synchrotron servers available
+ # And then can it fallback to main if there are no synchrotron servers either
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(api/v1|r0|v3)/initialSync$ }
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$ }
+ # https://spec.matrix.org/v1.14/client-server-api/#get_matrixclientv3sync
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(r0|v3)/sync$ } { urlp("full_state") -m str true }
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(r0|v3)/sync$ } !{ urlp("since") -m found }
+ # https://spec.matrix.org/latest/client-server-api/#get_matrixclientv3events
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(api/v1|r0|v3)/events$ } !{ urlp("from") -m found }
use_backend return_204_synapse if { method OPTIONS }
acl has_failover var(req.backend) -m str "event-persister"
+ acl has_failover var(req.backend) -m str "initial-synchrotron"
acl has_failover var(req.backend) -m str "sliding-sync"
+ acl has_failover var(req.backend) -m str "synchrotron"
acl backend_unavailable str(),concat('synapse-',req.backend),nbsrv lt 1
use_backend synapse-main-failover if has_failover backend_unavailable
[17 lines unchanged)]
# Use DNS SRV service discovery on the headless service
server-template main 1 _synapse-http._tcp.release-name-synapse-main.ess-ci.svc.cluster.local resolvers kubedns init-addr none check
+ backend synapse-initial-synchrotron
+ option httpchk
+ http-check connect port 8080
+ http-check send meth GET uri /health
+ # increase the server timeout, as it can take a long time to generate and
+ # return the initial sync.
+ timeout server 180s
+
+ # Balance on hash of access token
+ balance hdr(X-Access-Token)
+
+ # limit the number of concurrent requests to each synchrotron,
+ # to stop the reactor tick time rocketing
+ default-server maxconn 50
+ # Use DNS SRV service discovery on the headless service
+ server-template initial-sync 20 _synapse-http._tcp.release-name-synapse-initial-sync.ess-ci.svc.cluster.local resolvers kubedns init-addr none check
+
backend synapse-media-repository
option httpchk
http-check connect port 8080
http-check send meth GET uri /health
[27 lines unchanged)]
# requests quickly than let them queue up on the haproxy.
timeout queue 5s
# Use DNS SRV service discovery on the headless service
server-template sliding-sync 20 _synapse-http._tcp.release-name-synapse-sliding-sync.ess-ci.svc.cluster.local resolvers kubedns init-addr none check
+
+ backend synapse-synchrotron
+ option httpchk
+ http-check connect port 8080
+ http-check send meth GET uri /health
+ # Balance on the hash of the access token.
+ # When using stick tables, the stickiness only takes effect once the backend
+ # has responded at least once. If a user keeps timing out on their first
+ # incremental sync in a while, then they will keep 'bouncing' around
+ # different synchrotrons, preventing their sync from making progress.
+ #
+ # We can still use stick tables to ensure that once a client gets assigned
+ # to a Synchrotron it stays on that worker, allowing us to rebalance the
+ # pool without moving existing sessions.
+ #
+ # If the header doesn't exist it will round robin requests,
+ # though in that case they should all just be 4xx'd due
+ # to lack of an access token.
+ balance hdr(X-Access-Token)
+
+ # synchrotrons are long-polled, so we need to allow many
+ # concurrent connections.
+ default-server maxconn 2000
+
+ # if we *do* hit the limit, it's probably better we shed further
+ # requests quickly than let them queue up on the haproxy.
+ timeout queue 5s
+ # Use DNS SRV service discovery on the headless service
+ server-template synchrotron 20 _synapse-http._tcp.release-name-synapse-synchrotron.ess-ci.svc.cluster.local resolvers kubedns init-addr none check
# a backend which responds to everything with a 204 mirroring https://github.com/element-hq/synapse/blob/v1.124.0/synapse/http/server.py#L901-L932
backend return_204_synapse
http-request return status 204 hdr "Access-Control-Allow-Origin" "*" hdr "Access-Control-Allow-Methods" "GET, HEAD, POST, PUT, DELETE, OPTIONS" hdr "Access-Control-Allow-Headers" "Origin, X-Requested-With, Content-Type, Accept, Authorization, Date" hdr "Access-Control-Expose-Headers" "Synapse-Trace-Id, Server"
[two lines unchanged)]
# a fake backend which fonxes every request with a 500. Useful for
# handling overloads etc.
backend return_500
http-request deny deny_status 500
@@ ConfigMap/ess-ci/release-name-synapse-haproxy - data.path_map_file @@
# A map file that is used in haproxy config to map from matrix paths to the
# named backend. The format is: path_regexp backend_name
[eight lines unchanged)]
^/_synapse/admin/v1/quarantine_media/.*$ media-repository
^/_synapse/admin/v1/users/.*/media$ media-repository
# sliding-sync
^/_matrix/client/unstable/org.matrix.simplified_msc3575/.* sliding-sync
+ # synchrotron
+ ^/_matrix/client/(r0|v3)/sync$ synchrotron
+ ^/_matrix/client/(api/v1|r0|v3)/events$ synchrotron
+ ^/_matrix/client/(api/v1|r0|v3)/initialSync$ synchrotron
+ ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$ synchrotron
@@ ConfigMap/ess-ci/release-name-synapse - data @@
+ 05-initial-synchrotron.yaml: |
+
+ worker_app: synapse.app.generic_worker
+ worker_name: ${APPLICATION_NAME}
+
+ worker_listeners:
+ - port: 8008
+ tls: false
+ bind_addresses: ['0.0.0.0']
+ type: http
+ x_forwarded: true
+ resources:
+ - names:
+ - client
+ - federation
+ compress: false
+ - type: metrics
+ port: 9001
+ bind_addresses: ['0.0.0.0']
+ - port: 8080
+ tls: false
+ bind_addresses: ['0.0.0.0']
+ type: http
+ x_forwarded: false
+ resources:
+ - names: [health]
+ compress: false
+ # Stub out the media storage provider for processes not responsible for media
+ media_storage_providers:
+ - module: file_system
+ store_local: false
+ store_remote: false
+ store_synchronous: false
+ config:
+ directory: "/media/media_store"
+
+ 05-synchrotron.yaml: |
+
+ worker_app: synapse.app.generic_worker
+ worker_name: ${APPLICATION_NAME}
+
+ worker_listeners:
+ - port: 8008
+ tls: false
+ bind_addresses: ['0.0.0.0']
+ type: http
+ x_forwarded: true
+ resources:
+ - names:
+ - client
+ - federation
+ compress: false
+ - type: metrics
+ port: 9001
+ bind_addresses: ['0.0.0.0']
+ - port: 8080
+ tls: false
+ bind_addresses: ['0.0.0.0']
+ type: http
+ x_forwarded: false
+ resources:
+ - names: [health]
+ compress: false
+ # Stub out the media storage provider for processes not responsible for media
+ media_storage_providers:
+ - module: file_system
+ store_local: false
+ store_remote: false
+ store_synchronous: false
+ config:
+ directory: "/media/media_store"
@@ Deployment/ess-ci/release-name-haproxy - metadata.labels.k8s.element.io/shared-haproxy-config-hash @@
- cca11447d59aee9665c6501646c205cb1d02574d
+ 4b174803f851fbaa70a30443029c1c9b00676912
@@ Deployment/ess-ci/release-name-haproxy - metadata.labels.k8s.element.io/synapse-haproxy-config-hash @@
- 71c5591570780d8730dd76f60d22e4b89fa8c203
+ f4f8765d4c1356f8dd0851a574a297245a6d53bb
@@ Deployment/ess-ci/release-name-haproxy - spec.template.metadata.labels.k8s.element.io/shared-haproxy-config-hash @@
- cca11447d59aee9665c6501646c205cb1d02574d
+ 4b174803f851fbaa70a30443029c1c9b00676912
@@ Deployment/ess-ci/release-name-haproxy - spec.template.metadata.labels.k8s.element.io/synapse-haproxy-config-hash @@
- 71c5591570780d8730dd76f60d22e4b89fa8c203
+ f4f8765d4c1356f8dd0851a574a297245a6d53bb
@@ Service/ess-ci/release-name-synapse-initial-sync @@
+ ---
+ # Source: matrix-stack/templates/synapse/synapse_service.yaml
+ apiVersion: v1
+ kind: Service
+ metadata:
+ labels:
+ helm.sh/chart: "matrix-stack-25.7.1-dev"
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/part-of: matrix-stack
+ app.kubernetes.io/component: matrix-server
+ app.kubernetes.io/name: synapse-initial-synchrotron
+ app.kubernetes.io/instance: release-name-synapse-initial-synchrotron
+ app.kubernetes.io/version: "v1.134.0"
+ k8s.element.io/synapse-instance: release-name-synapse
+ name: release-name-synapse-initial-sync
+ namespace: ess-ci
+ spec:
+ clusterIP: None
+ ports:
+ - name: synapse-http
+ port: 8008
+ targetPort: synapse-http
+ - name: synapse-health
+ port: 8080
+ targetPort: synapse-health
+ - name: synapse-metrics
+ port: 9001
+ targetPort: synapse-metrics
+ selector:
+ app.kubernetes.io/instance: release-name-synapse-initial-synchrotron
@@ Service/ess-ci/release-name-synapse-synchrotron @@
+ ---
+ # Source: matrix-stack/templates/synapse/synapse_service.yaml
+ apiVersion: v1
+ kind: Service
+ metadata:
+ labels:
+ helm.sh/chart: "matrix-stack-25.7.1-dev"
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/part-of: matrix-stack
+ app.kubernetes.io/component: matrix-server
+ app.kubernetes.io/name: synapse-synchrotron
+ app.kubernetes.io/instance: release-name-synapse-synchrotron
+ app.kubernetes.io/version: "v1.134.0"
+ k8s.element.io/synapse-instance: release-name-synapse
+ name: release-name-synapse-synchrotron
+ namespace: ess-ci
+ spec:
+ clusterIP: None
+ ports:
+ - name: synapse-http
+ port: 8008
+ targetPort: synapse-http
+ - name: synapse-health
+ port: 8080
+ targetPort: synapse-health
+ - name: synapse-metrics
+ port: 9001
+ targetPort: synapse-metrics
+ selector:
+ app.kubernetes.io/instance: release-name-synapse-synchrotron
@@ StatefulSet/ess-ci/release-name-synapse-event-persist - metadata.labels.k8s.element.io/synapse-config-hash @@
- 6184a7175111af0640ee267b81afa9cb29ca8385
+ c449a85585b080c97bcbd11e5a5878389ce172f9
@@ StatefulSet/ess-ci/release-name-synapse-event-persist - spec.template.metadata.labels.k8s.element.io/synapse-config-hash @@
- 6184a7175111af0640ee267b81afa9cb29ca8385
+ c449a85585b080c97bcbd11e5a5878389ce172f9
@@ StatefulSet/ess-ci/release-name-synapse-initial-sync @@
+ ---
+ # Source: matrix-stack/templates/synapse/synapse_statefulset.yaml
+ apiVersion: apps/v1
+ kind: StatefulSet
+ metadata:
+ labels:
+ helm.sh/chart: "matrix-stack-25.7.1-dev"
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/part-of: matrix-stack
+ app.kubernetes.io/component: matrix-server
+ app.kubernetes.io/name: synapse-initial-synchrotron
+ app.kubernetes.io/instance: release-name-synapse-initial-synchrotron
+ app.kubernetes.io/version: "v1.134.0"
+ k8s.element.io/synapse-instance: release-name-synapse
+ k8s.element.io/synapse-config-hash: "c449a85585b080c97bcbd11e5a5878389ce172f9"
+ k8s.element.io/synapse-secret-hash: "b304a4524adc98db5cb677362ba64b7351ed7ab3"
+ k8s.element.io/postgres-password-synapse-hash: da9c3f72f44eda433c1be814c6e8531633da8a67
+ name: release-name-synapse-initial-sync
+ namespace: ess-ci
+ spec:
+ serviceName: release-name-synapse-initial-sync
+ replicas: 1
+ selector:
+ matchLabels:
+ app.kubernetes.io/instance: release-name-synapse-initial-synchrotron
+ updateStrategy:
+ type: RollingUpdate
+ # Without this CrashLoopBackoffs due to config failures block pod recreation
+ podManagementPolicy: Parallel
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/part-of: matrix-stack
+ app.kubernetes.io/component: matrix-server
+ app.kubernetes.io/name: synapse-initial-synchrotron
+ app.kubernetes.io/instance: release-name-synapse-initial-synchrotron
+ app.kubernetes.io/version: "v1.134.0"
+ k8s.element.io/synapse-instance: release-name-synapse
+ k8s.element.io/synapse-config-hash: "c449a85585b080c97bcbd11e5a5878389ce172f9"
+ k8s.element.io/synapse-secret-hash: "b304a4524adc98db5cb677362ba64b7351ed7ab3"
+ k8s.element.io/postgres-password-synapse-hash: da9c3f72f44eda433c1be814c6e8531633da8a67
+ spec:
+ automountServiceAccountToken: false
+ serviceAccountName: release-name-synapse
+ securityContext:
+ fsGroup: 10091
+ runAsGroup: 0
+ runAsNonRoot: true
+ runAsUser: 10091
+ seccompProfile:
+ type: RuntimeDefault
+ supplementalGroups: []
+ initContainers:
+ - name: render-config
+ image: "ghcr.io/element-hq/ess-helm/matrix-tools:0.5.4"
+ imagePullPolicy: Always
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ command:
+ - "/matrix-tools"
+ - render-config
+ - -output
+ - /conf/homeserver.yaml
+ - /config-templates/01-homeserver-underrides.yaml
+ - /secrets/release-name-synapse/user-00-userconfig.yaml
+ - /secrets/release-name-synapse-secrets/01-other-user-config.yaml
+ - /config-templates/04-homeserver-overrides.yaml
+ - /config-templates/05-initial-synchrotron.yaml
+ env:
+ - name: APPLICATION_NAME
+ value: '{{ hostname }}'
+ - name: DEBUG_RENDERING
+ value: "1"
+ - name: SYNAPSE_POSTGRES_PASSWORD
+ value: '{{ readfile "/secrets/release-name-generated/POSTGRES_SYNAPSE_PASSWORD" | quote }}'
+ resources:
+ limits:
+ memory: 4Gi
+ requests:
+ cpu: 100m
+ memory: 100Mi
+ volumeMounts:
+ - mountPath: /config-templates
+ name: plain-config
+ readOnly: true
+ - mountPath: /secrets/release-name-synapse
+ name: "secret-3778ad710292"
+ readOnly: true
+ - mountPath: /secrets/release-name-generated
+ name: "secret-f20f994b9a6a"
+ readOnly: true
+ - mountPath: /secrets/release-name-synapse-secrets
+ name: "secret-04c439f8a117"
+ readOnly: true
+ - mountPath: /conf
+ name: rendered-config
+ readOnly: false
+ - name: db-wait
+ image: "ghcr.io/element-hq/ess-helm/matrix-tools:0.5.4"
+ imagePullPolicy: Always
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ command:
+ - "/matrix-tools"
+ - tcpwait
+ - -address
+ - "release-name-postgres.ess-ci.svc.cluster.local:5432"
+ resources:
+ limits:
+ memory: 4Gi
+ requests:
+ cpu: 100m
+ memory: 100Mi
+ containers:
+ - name: synapse
+ image: "ghcr.io/element-hq/synapse:v1.134.0"
+ imagePullPolicy: Always
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ command:
+ - "python3"
+ - "-m"
+ - synapse.app.generic_worker
+ - "-c"
+ - /conf/homeserver.yaml
+ - "--no-secrets-in-config"
+ env:
+ - name: DEBUG_RENDERING
+ value: "1"
+ - name: LD_PRELOAD
+ value: libjemalloc.so.2
+ ports:
+ - containerPort: 8008
+ name: synapse-http
+ protocol: TCP
+ - containerPort: 8080
+ name: synapse-health
+ protocol: TCP
+ - containerPort: 9001
+ name: synapse-metrics
+ protocol: TCP
+ startupProbe:
+ failureThreshold: 21
+ periodSeconds: 2
+ successThreshold: 1
+ timeoutSeconds: 1
+ httpGet:
+ path: /health
+ port: synapse-health
+ livenessProbe:
+ failureThreshold: 3
+ periodSeconds: 6
+ successThreshold: 1
+ timeoutSeconds: 2
+ httpGet:
+ path: /health
+ port: synapse-health
+ readinessProbe:
+ failureThreshold: 3
+ periodSeconds: 2
+ successThreshold: 2
+ timeoutSeconds: 2
+ httpGet:
+ path: /health
+ port: synapse-health
+ resources:
+ limits:
+ memory: 4Gi
+ requests:
+ cpu: 100m
+ memory: 100Mi
+ volumeMounts:
+ - mountPath: "/conf/homeserver.yaml"
+ name: rendered-config
+ subPath: homeserver.yaml
+ readOnly: true
+ - mountPath: /secrets/release-name-synapse
+ name: "secret-3778ad710292"
+ readOnly: true
+ - mountPath: /secrets/release-name-generated
+ name: "secret-f20f994b9a6a"
+ readOnly: true
+ - mountPath: /secrets/release-name-synapse-secrets
+ name: "secret-04c439f8a117"
+ readOnly: true
+ - mountPath: /conf/log_config.yaml
+ name: plain-config
+ subPath: log_config.yaml
+ readOnly: false
+ - mountPath: /media
+ name: media
+ readOnly: false
+ - mountPath: /tmp
+ name: tmp
+ readOnly: false
+ volumes:
+ - configMap:
+ defaultMode: 420
+ name: release-name-synapse
+ name: plain-config
+ - secret:
+ secretName: release-name-synapse
+ name: secret-3778ad710292
+ - secret:
+ secretName: release-name-generated
+ name: secret-f20f994b9a6a
+ - secret:
+ secretName: release-name-synapse-secrets
+ name: secret-04c439f8a117
+ - emptyDir:
+ medium: Memory
+ name: "rendered-config"
+ - emptyDir:
+ medium: Memory
+ name: "media"
+ - emptyDir:
+ medium: Memory
+ name: "tmp"
@@ StatefulSet/ess-ci/release-name-synapse-main - metadata.labels.k8s.element.io/synapse-config-hash @@
- 6184a7175111af0640ee267b81afa9cb29ca8385
+ c449a85585b080c97bcbd11e5a5878389ce172f9
@@ StatefulSet/ess-ci/release-name-synapse-main - spec.template.metadata.labels.k8s.element.io/synapse-config-hash @@
- 6184a7175111af0640ee267b81afa9cb29ca8385
+ c449a85585b080c97bcbd11e5a5878389ce172f9
@@ StatefulSet/ess-ci/release-name-synapse-media-repo - metadata.labels.k8s.element.io/synapse-config-hash @@
- 6184a7175111af0640ee267b81afa9cb29ca8385
+ c449a85585b080c97bcbd11e5a5878389ce172f9
@@ StatefulSet/ess-ci/release-name-synapse-media-repo - spec.template.metadata.labels.k8s.element.io/synapse-config-hash @@
- 6184a7175111af0640ee267b81afa9cb29ca8385
+ c449a85585b080c97bcbd11e5a5878389ce172f9
@@ StatefulSet/ess-ci/release-name-synapse-sliding-sync - metadata.labels.k8s.element.io/synapse-config-hash @@
- 6184a7175111af0640ee267b81afa9cb29ca8385
+ c449a85585b080c97bcbd11e5a5878389ce172f9
@@ StatefulSet/ess-ci/release-name-synapse-sliding-sync - spec.template.metadata.labels.k8s.element.io/synapse-config-hash @@
- 6184a7175111af0640ee267b81afa9cb29ca8385
+ c449a85585b080c97bcbd11e5a5878389ce172f9
@@ StatefulSet/ess-ci/release-name-synapse-synchrotron @@
+ ---
+ # Source: matrix-stack/templates/synapse/synapse_statefulset.yaml
+ apiVersion: apps/v1
+ kind: StatefulSet
+ metadata:
+ labels:
+ helm.sh/chart: "matrix-stack-25.7.1-dev"
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/part-of: matrix-stack
+ app.kubernetes.io/component: matrix-server
+ app.kubernetes.io/name: synapse-synchrotron
+ app.kubernetes.io/instance: release-name-synapse-synchrotron
+ app.kubernetes.io/version: "v1.134.0"
+ k8s.element.io/synapse-instance: release-name-synapse
+ k8s.element.io/synapse-config-hash: "c449a85585b080c97bcbd11e5a5878389ce172f9"
+ k8s.element.io/synapse-secret-hash: "b304a4524adc98db5cb677362ba64b7351ed7ab3"
+ k8s.element.io/postgres-password-synapse-hash: da9c3f72f44eda433c1be814c6e8531633da8a67
+ name: release-name-synapse-synchrotron
+ namespace: ess-ci
+ spec:
+ serviceName: release-name-synapse-synchrotron
+ replicas: 1
+ selector:
+ matchLabels:
+ app.kubernetes.io/instance: release-name-synapse-synchrotron
+ updateStrategy:
+ type: RollingUpdate
+ # Without this CrashLoopBackoffs due to config failures block pod recreation
+ podManagementPolicy: Parallel
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/part-of: matrix-stack
+ app.kubernetes.io/component: matrix-server
+ app.kubernetes.io/name: synapse-synchrotron
+ app.kubernetes.io/instance: release-name-synapse-synchrotron
+ app.kubernetes.io/version: "v1.134.0"
+ k8s.element.io/synapse-instance: release-name-synapse
+ k8s.element.io/synapse-config-hash: "c449a85585b080c97bcbd11e5a5878389ce172f9"
+ k8s.element.io/synapse-secret-hash: "b304a4524adc98db5cb677362ba64b7351ed7ab3"
+ k8s.element.io/postgres-password-synapse-hash: da9c3f72f44eda433c1be814c6e8531633da8a67
+ spec:
+ automountServiceAccountToken: false
+ serviceAccountName: release-name-synapse
+ securityContext:
+ fsGroup: 10091
+ runAsGroup: 0
+ runAsNonRoot: true
+ runAsUser: 10091
+ seccompProfile:
+ type: RuntimeDefault
+ supplementalGroups: []
+ initContainers:
+ - name: render-config
+ image: "ghcr.io/element-hq/ess-helm/matrix-tools:0.5.4"
+ imagePullPolicy: Always
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ command:
+ - "/matrix-tools"
+ - render-config
+ - -output
+ - /conf/homeserver.yaml
+ - /config-templates/01-homeserver-underrides.yaml
+ - /secrets/release-name-synapse/user-00-userconfig.yaml
+ - /secrets/release-name-synapse-secrets/01-other-user-config.yaml
+ - /config-templates/04-homeserver-overrides.yaml
+ - /config-templates/05-synchrotron.yaml
+ env:
+ - name: APPLICATION_NAME
+ value: '{{ hostname }}'
+ - name: DEBUG_RENDERING
+ value: "1"
+ - name: SYNAPSE_POSTGRES_PASSWORD
+ value: '{{ readfile "/secrets/release-name-generated/POSTGRES_SYNAPSE_PASSWORD" | quote }}'
+ resources:
+ limits:
+ memory: 4Gi
+ requests:
+ cpu: 100m
+ memory: 100Mi
+ volumeMounts:
+ - mountPath: /config-templates
+ name: plain-config
+ readOnly: true
+ - mountPath: /secrets/release-name-synapse
+ name: "secret-3778ad710292"
+ readOnly: true
+ - mountPath: /secrets/release-name-generated
+ name: "secret-f20f994b9a6a"
+ readOnly: true
+ - mountPath: /secrets/release-name-synapse-secrets
+ name: "secret-04c439f8a117"
+ readOnly: true
+ - mountPath: /conf
+ name: rendered-config
+ readOnly: false
+ - name: db-wait
+ image: "ghcr.io/element-hq/ess-helm/matrix-tools:0.5.4"
+ imagePullPolicy: Always
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ command:
+ - "/matrix-tools"
+ - tcpwait
+ - -address
+ - "release-name-postgres.ess-ci.svc.cluster.local:5432"
+ resources:
+ limits:
+ memory: 4Gi
+ requests:
+ cpu: 100m
+ memory: 100Mi
+ containers:
+ - name: synapse
+ image: "ghcr.io/element-hq/synapse:v1.134.0"
+ imagePullPolicy: Always
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ command:
+ - "python3"
+ - "-m"
+ - synapse.app.generic_worker
+ - "-c"
+ - /conf/homeserver.yaml
+ - "--no-secrets-in-config"
+ env:
+ - name: DEBUG_RENDERING
+ value: "1"
+ - name: LD_PRELOAD
+ value: libjemalloc.so.2
+ ports:
+ - containerPort: 8008
+ name: synapse-http
+ protocol: TCP
+ - containerPort: 8080
+ name: synapse-health
+ protocol: TCP
+ - containerPort: 9001
+ name: synapse-metrics
+ protocol: TCP
+ startupProbe:
+ failureThreshold: 21
+ periodSeconds: 2
+ successThreshold: 1
+ timeoutSeconds: 1
+ httpGet:
+ path: /health
+ port: synapse-health
+ livenessProbe:
+ failureThreshold: 3
+ periodSeconds: 6
+ successThreshold: 1
+ timeoutSeconds: 2
+ httpGet:
+ path: /health
+ port: synapse-health
+ readinessProbe:
+ failureThreshold: 3
+ periodSeconds: 2
+ successThreshold: 2
+ timeoutSeconds: 2
+ httpGet:
+ path: /health
+ port: synapse-health
+ resources:
+ limits:
+ memory: 4Gi
+ requests:
+ cpu: 100m
+ memory: 100Mi
+ volumeMounts:
+ - mountPath: "/conf/homeserver.yaml"
+ name: rendered-config
+ subPath: homeserver.yaml
+ readOnly: true
+ - mountPath: /secrets/release-name-synapse
+ name: "secret-3778ad710292"
+ readOnly: true
+ - mountPath: /secrets/release-name-generated
+ name: "secret-f20f994b9a6a"
+ readOnly: true
+ - mountPath: /secrets/release-name-synapse-secrets
+ name: "secret-04c439f8a117"
+ readOnly: true
+ - mountPath: /conf/log_config.yaml
+ name: plain-config
+ subPath: log_config.yaml
+ readOnly: false
+ - mountPath: /media
+ name: media
+ readOnly: false
+ - mountPath: /tmp
+ name: tmp
+ readOnly: false
+ volumes:
+ - configMap:
+ defaultMode: 420
+ name: release-name-synapse
+ name: plain-config
+ - secret:
+ secretName: release-name-synapse
+ name: secret-3778ad710292
+ - secret:
+ secretName: release-name-generated
+ name: secret-f20f994b9a6a
+ - secret:
+ secretName: release-name-synapse-secrets
+ name: secret-04c439f8a117
+ - emptyDir:
+ medium: Memory
+ name: "rendered-config"
+ - emptyDir:
+ medium: Memory
+ name: "media"
+ - emptyDir:
+ medium: Memory
+ name: "tmp"
synapse-worker-example-values.yaml@@ ConfigMap/ess-ci/release-name-haproxy - data.haproxy.cfg @@
global
maxconn 40000
log stdout format raw local0 info
[153 lines unchanged)]
http-request set-var(req.backend) path,map_reg(/synapse/path_map_file_get,main) if has_get_map METH_GET
http-request set-var(req.backend) path,map_reg(/synapse/path_map_file,main) unless { var(req.backend) -m found }
- acl is_initial_sync path -m reg ^/_matrix/client/(api/v1|r0|v3)/initialSync$
- acl is_initial_sync path -m reg ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$
- # https://spec.matrix.org/v1.14/client-server-api/#get_matrixclientv3sync
- acl is_initial_sync path -m reg ^/_matrix/client/(r0|v3)/sync$ { urlp("full_state") -m str true }
- acl is_initial_sync path -m reg ^/_matrix/client/(r0|v3)/sync$ !{ urlp("since") -m found }
- # https://spec.matrix.org/latest/client-server-api/#get_matrixclientv3events
- acl is_initial_sync path -m reg ^/_matrix/client/(api/v1|r0|v3)/events$ !{ urlp("from") -m found }
+ acl has_available_initial_syncs nbsrv('synapse-initial-synchrotron') ge 1
# Set to the initial-synchrotron backend if it is one of these magic paths AND we have workers in the initial-synchrotron backend
# This means that we don't update the backend from synchrotron if that's configured but there's no initial-synchrotron servers available
# And then can it fallback to main if there are no synchrotron servers either
- http-request set-var(req.backend) str('initial-synchrotron') if is_initial_sync { nbsrv('synapse-initial-synchrotron') ge 1 }
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(api/v1|r0|v3)/initialSync$ }
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$ }
+ # https://spec.matrix.org/v1.14/client-server-api/#get_matrixclientv3sync
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(r0|v3)/sync$ } { urlp("full_state") -m str true }
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(r0|v3)/sync$ } !{ urlp("since") -m found }
+ # https://spec.matrix.org/latest/client-server-api/#get_matrixclientv3events
+ http-request set-var(req.backend) str('initial-synchrotron') if has_available_initial_syncs { path -m reg ^/_matrix/client/(api/v1|r0|v3)/events$ } !{ urlp("from") -m found }
use_backend return_204_synapse if { method OPTIONS }
[224 lines unchanged)]
# a fake backend which fonxes every request with a 500. Useful for
# handling overloads etc.
backend return_500
http-request deny deny_status 500
@@ Deployment/ess-ci/release-name-haproxy - metadata.labels.k8s.element.io/shared-haproxy-config-hash @@
- e443dca51617b5f5cf49242deca069678095d407
+ 61a0bf448cbc921171f507ec50646d131b380fcb
@@ Deployment/ess-ci/release-name-haproxy - spec.template.metadata.labels.k8s.element.io/shared-haproxy-config-hash @@
- e443dca51617b5f5cf49242deca069678095d407
+ 61a0bf448cbc921171f507ec50646d131b380fcb
|
Member
gaelgatelement
left a comment
gaelgatelement
left a comment
There was a problem hiding this comment.
As discussed, let's try to add a test in integration tests which reads haproxy logs when we send requests against synapse.
gaelgatelement
approved these changes
Jul 29, 2025
Member
Author
|
Test added. Without the fix in the 1st commit it fails with: |
gaelgatelement
approved these changes
Jul 29, 2025
benbz
force-pushed
the
bbz/fix-routing-to-initial-sync
branch
from
c99ba70 to
57e23c9
Compare
Member
Author
|
In the course of this, I've ended up reverting #531. Doing that appears to retry all 4xx/5xx responses regardless of whether they are in |
gaelgatelement
approved these changes
Jul 29, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #631. Regression from #508
HAProxy
acls don't appear to support multiple matches. It appears if multiple matches are specified (attempting to do ANDs), HAProxy silently accepts and just uses the final matcher.Replace the
is_initial_syncACLs with multiplehttp-request set-var(req.backend) str('initial-synchrotron')to resolve.Re-running the tests from #508 and adding a test based on #631 (logs and commands lightly edited for readability):
Results in