Synapse tries to use `run_as_background_process` during homeserver shutdown process (which fails)

[MadLittleMods]

Synapse tries to use run_as_background_process(...) during homeserver shutdown process but since we prevent new background processes from starting after someone calls hs.shutdown(), those just end up failing and we log the failure:

ERROR - Error calling shutdown async handler: Cannot start background process. HomeServer has been shutdown

Dev notes

This was introduced in #18828

Example usage to register a new _async_shutdown_handlers (notice that it's using @wrap_as_background_process):

synapse/synapse/handlers/presence.py

Lines 533 to 544 in 408a05e

	hs.register_async_shutdown_handler(
	phase="before",
	eventType="shutdown",
	shutdown_func=self._on_shutdown,
	)
	@wrap_as_background_process("WorkerPresenceHandler._on_shutdown")
	async def _on_shutdown(self) -> None:
	if self._track_presence:
	self.hs.get_replication_command_handler().send_command(
	ClearUserSyncsCommand(self.instance_id)
	)

In hs.shutdown(), we try to call of the _async_shutdown_handlers:

synapse/synapse/server.py

Lines 490 to 496 in 408a05e

	for shutdown_handler in self._async_shutdown_handlers:
	try:
	self.get_reactor().removeSystemEventTrigger(shutdown_handler.trigger_id)
	defer.ensureDeferred(shutdown_handler.func(**shutdown_handler.kwargs))
	except Exception as e:
	logger.error("Error calling shutdown async handler: %s", e)
	self._async_shutdown_handlers.clear()

Reproduction

This is even reproducible in our tests but because we just log, the test doesn't fail.

SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.app.test_homeserver_shutdown.HomeserverCleanShutdownTestCase.test_clean_homeserver_shutdown

_trial_temp/test.log

ERROR - Error calling shutdown async handler: Cannot start background process. HomeServer has been shutdown