Refactor cache metrics to be homeserver-scoped

[MadLittleMods]

Refactor cache metrics to be homeserver-scoped (add server_name label to cache metrics).

Part of #18592

This can be reviewed commit by commit to skip over some of the bulk refactor but there are some fixes down the line and I'd prefer to keep the history than clean it all up in a rebase.

Testing strategy

See behavior of previous metrics listener

1. Add the metrics listener in your homeserver.yaml

   listeners:
     - port: 9323
       type: metrics
       bind_addresses: ['127.0.0.1']

2. Start the homeserver: poetry run synapse_homeserver --config-path homeserver.yaml
3. Fetch http://localhost:9323/metrics
4. Observe response includes the cache metrics (synapse_util_caches_cache_size, synapse_util_caches_cache_hits, synapse_util_caches_cache_evicted_size, etc)

See behavior of the http metrics resource

1. Add the metrics resource to a new or existing http listeners in your homeserver.yaml

   listeners:
     - port: 9322
       type: http
       bind_addresses: ['127.0.0.1']
       resources:
         - names: [metrics]
           compress: false

2. Start the homeserver: poetry run synapse_homeserver --config-path homeserver.yaml
3. Fetch http://localhost:9322/_synapse/metrics (it's just a GET request so you can even do in the browser)
4. Observe response includes the cache metrics (synapse_util_caches_cache_size, synapse_util_caches_cache_hits, synapse_util_caches_cache_evicted_size, etc): example, example from develop

Dev notes

LruCache/@cached, CacheMetric

register_cache(


ExpiringCache(

ResponseCache(

StreamChangeCache(

TTLCache(
	WellKnownResolver( -> MatrixFederationAgent(

LruCache(
	DeferredCache( -> DeferredCacheDescriptor( -> _CachedFunctionDescriptor( -> cached(
	AsyncLruCache(
	DictionaryCache(

Todo

• Update @cached
• Ensure scripts-dev/mypy_synapse_plugin.py works correctly with cached functions
  • This was more relevant when I was thinking I needed to change @cached more but should be fine with how we've done it.

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[MadLittleMods]

The metrics being refactored to be homeserver scoped are in this file.

The rest of the changes are to support that change and supply the server_name to the instance label.

[MadLittleMods]

This pattern is copied from Measure

synapse/synapse/util/metrics.py

Lines 95 to 96 in a2bee2f

	class HasClock(Protocol):
	clock: Clock

(The Measure pattern is also updated in #18601)

[anoadragon453]

This LGTM! Thank you for updating the docstrings of each function you touched to include all parameter names.

[anoadragon453]

absolute nit, readability:

Suggested change

	Normally, this would be set automatically by the Prometheus server scraping the data but
	since we support multiple instances of Synapse running in the same process and all
	Normally, this would be set automatically by the Prometheus server scraping the data. But
	since we support multiple instances of Synapse running in the same process and all

[MadLittleMods]

That is from an old version of the diff. This is the latest:

synapse/synapse/metrics/__init__.py

Lines 69 to 82 in fc10a5e

	SERVER_NAME_LABEL = "server_name"
	"""
	The `server_name` label is used to identify the homeserver that the metrics correspond
	to. Because we support multiple instances of Synapse running in the same process and all
	metrics are in a single global `REGISTRY`, we need to manually label any metrics.
	In the case of a Synapse homeserver, this should be set to the homeserver name
	(`hs.hostname`).
	We're purposely not using the `instance` label for this purpose as that should be "The
	<host>:<port> part of the target's URL that was scraped.". Also: "In Prometheus
	terms, an endpoint you can scrape is called an *instance*, usually corresponding to a
	single process." (source: https://prometheus.io/docs/concepts/jobs_instances/)
	"""

[MadLittleMods]

Thanks for the review @anoadragon453 🦜