Skip to content

Fix 2-second systemd notify stalls in cloud-hypervisor #493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rzhikharevich wants to merge 3 commits into
base: main
Choose a base branch
from
+41 −4
Open

Fix 2-second systemd notify stalls in cloud-hypervisor #493

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2b85690
Add vsock-notify-proxy
rzhikharevich Mar 27, 2026
c4f9d8f
Revert back to single read
rzhikharevich Mar 27, 2026
0581c74
Improve comment
rzhikharevich Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 41 additions & 4 deletions lib/runners/cloud-hypervisor.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,45 @@ let
vulkan = true;
};

# A proxy that forwards systemd notifications from cloud-hypervisor's vsock socket. It
# intentionally calls read only once and closes the connection immediately, which unblocks the
# guest's blocking recv() (systemd does shutdown(SHUT_WR) + recv() as an
# end-of-message handshake in sd-daemon.c, but cloud-hypervisor does not propagate the half-close
# to the host-side Unix socket).
#
# Previously, this used socat with a 2-second timeout but this causes systemd to stall on every
# notification, causing extremely slow boot times.
#
# An important caveat is that the message could theoretically be truncated but this should not
# happen in practice unless the message is large (systemd notifications are small).
#
# TODO(rzhikharevich): Ideally, cloud-hypervisor should propagate the half-close, then the
Copy link
Copy Markdown
Member

@SuperSandro2000 SuperSandro2000 Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# TODO(rzhikharevich): Ideally, cloud-hypervisor should propagate the half-close, then the
# TODO: Ideally, cloud-hypervisor should propagate the half-close, then the

# theoretical truncation risk could be fixed by closing the connection after detecting it.
vsockNotifyProxy = pkgs.writers.writePython3Bin "vsock-notify-proxy" {} ''
import asyncio
import os
import socket

notify_socket = os.environ["NOTIFY_SOCKET"]
notify = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)


async def handle(reader, writer):
data = await reader.read(65536)
writer.close()
Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After calling writer.close(), it’s generally best to await writer.wait_closed() (or equivalent) to ensure the transport is actually closed and file descriptors are released promptly. This helps avoid accumulating open connections under load and prevents noisy ResourceWarnings in some Python/asyncio versions.

Suggested change
writer.close()
writer.close()
await writer.wait_closed()

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member

@astro astro Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 1st example in the Python asyncio documentation does this, too.

if data:
notify.sendto(data, notify_socket)


async def main():
path = os.environ["VSOCK_NOTIFY_PATH"]
server = await asyncio.start_unix_server(handle, path=path)
Comment on lines +160 to +161
Copy link
Copy Markdown
Member

@SuperSandro2000 SuperSandro2000 Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
path = os.environ["VSOCK_NOTIFY_PATH"]
server = await asyncio.start_unix_server(handle, path=path)
server = await asyncio.start_unix_server(handle, path=sys.argv[0])

We can make this kinda dirty here, as we control all invocations and this fails loudly when no arg is there, os.environ does IIRC return empty string.

await server.serve_forever()


asyncio.run(main())
'';

oemStringValues = platformOEMStrings ++ lib.optional supportsNotifySocket "io.systemd.credential:vmm.notify_socket=vsock-stream:2:8888";
oemStringOptions = lib.optional (oemStringValues != []) "oem_strings=[${lib.concatStringsSep "," oemStringValues}]";
platformExtracted = extractOptValues "--platform" extraArgs;
Expand Down Expand Up @@ -155,11 +194,9 @@ in {
# Ensure notify sockets are removed if cloud-hypervisor didn't exit cleanly the last time
rm -f ${vsockPath} ${vsockPath}_8888

# Start socat to forward systemd notify socket over vsock
# Forward systemd notify messages from guest (via vsock) to host
if [ -n "''${NOTIFY_SOCKET:-}" ]; then
# -T2 is required because cloud-hypervisor does not handle partial
# shutdown of the stream, like systemd v256+ does.
${pkgs.socat}/bin/socat -T2 UNIX-LISTEN:${vsockPath}_8888,fork UNIX-SENDTO:$NOTIFY_SOCKET &
VSOCK_NOTIFY_PATH=${vsockPath}_8888 ${vsockNotifyProxy}/bin/vsock-notify-proxy &
Copy link
Copy Markdown
Member

@SuperSandro2000 SuperSandro2000 Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
VSOCK_NOTIFY_PATH=${vsockPath}_8888 ${vsockNotifyProxy}/bin/vsock-notify-proxy &
${lib.getExe vsockNotifyProxy} ${vsockPath}_8888 &

fi
'' + lib.optionalString graphics.enable ''
rm -f ${graphics.socket}
Expand Down
Loading
⚡