dind-box: warn when the nested daemon runs on the vfs storage driver (issue #104)

Landing on the `vfs` storage driver was silent: a single `log` line named the
driver, but nothing flagged that vfs performs NO copy-on-write. vfs stores every
image layer as a full, independent copy, so a multi-GB image's on-disk footprint
becomes the SUM of all cumulative layer sizes (many times the image size), and
`docker pull`/`docker run` can fail with `failed to register layer: no space
left on device` on a disk far larger than the image — with no breadcrumb
pointing at the driver. Downstream this overflowed a disk with a >30 GB image
(link-assistant/hive-mind#1914).

The active driver ends up being vfs either pinned explicitly via
`DIND_STORAGE_DRIVER=vfs` (legitimate for overlay-on-overlay compatibility) or
reached as the last-resort auto-detect fallback. This is observability, not a
default change — vfs stays the safe fallback. `start_dockerd` now calls
`warn_if_vfs_storage_driver` right after the daemon becomes ready, emitting one
actionable warning whenever the active driver is vfs: it explains the
copy-on-write/disk implication and names the `DIND_STORAGE_DRIVER=fuse-overlayfs`
remediation (copy-on-write, works overlay-on-overlay, already shipped in the
image). The remediation adapts to whether `/dev/fuse` is present, pointing at
`--privileged` / `--device /dev/fuse` first when it is missing. The
`DIND_STORAGE_DRIVER` doc comment now spells out the vfs disk amplification too.

Covered by a new unit test (experiments/test-issue104-vfs-warning.sh) and a new
assertion in the CI-run tests/dind/example-storage-driver-vfs.sh; documented in
docs/dind/USAGE.md. Adds a patch changeset.

Author: konard

.changeset/issue-104-vfs-storage-driver-warning.md

@@ -0,0 +1,29 @@
+---
+bump: patch
+---
+
+dind-box: warn when the nested daemon runs on the `vfs` storage driver (issue
+#104).
+
+When the inner dockerd ends up on `vfs` — either pinned explicitly via
+`DIND_STORAGE_DRIVER=vfs` (e.g. for overlay-on-overlay compatibility) or reached
+as the last-resort auto-detect fallback — large images could fail to pull/run
+with a cryptic `failed to register layer: no space left on device` and **no
+hint** that the storage driver was the cause. `vfs` performs no copy-on-write: it
+stores every image layer as a full, independent copy, so a multi-GB image's
+on-disk footprint becomes the *sum* of all cumulative layer sizes (many times the
+image size), and a >30 GB image can overflow a disk with far more than 30 GB free
+(`link-assistant/hive-mind#1914`).
+
+This is observability, not a default change — `vfs` stays the safe fallback. The
+entrypoint now emits a single, actionable warning right after the daemon becomes
+ready whenever the active driver is `vfs`, explaining the copy-on-write/disk
+implication and naming the `DIND_STORAGE_DRIVER=fuse-overlayfs` remediation
+(copy-on-write, works overlay-on-overlay, already shipped in the image). The
+remediation line adapts to whether `/dev/fuse` is present, so when it is missing
+it points at `--privileged` / `--device /dev/fuse` first. The
+`DIND_STORAGE_DRIVER` doc comment now spells out the `vfs` disk amplification too.
+
+Covered by a new unit test (`experiments/test-issue104-vfs-warning.sh`) and a new
+assertion in the CI-run `tests/dind/example-storage-driver-vfs.sh`; documented in
+`docs/dind/USAGE.md`.

.gitkeep

@@ -352,6 +352,29 @@ docker run -d --privileged \
   konard/box-dind sleep infinity
 ```
 
+Because that trade-off is easy to hit by accident, the entrypoint emits a
+one-time warning whenever the **active** driver ends up being `vfs` — whether
+pinned explicitly or reached as the last-resort fallback (issue #104). `vfs`
+stores every image layer as a full, independent copy, so a multi-GB image's
+on-disk footprint becomes the *sum* of all cumulative layer sizes — many times
+the image size — and `docker pull`/`docker run` can fail with `failed to register
+layer: no space left on device` on a disk far larger than the image. The warning
+makes that failure traceable instead of looking like a generic "out of disk".
+
+If your host supports it, prefer `DIND_STORAGE_DRIVER=fuse-overlayfs`: it is
+copy-on-write **and** works overlay-on-overlay (the compatibility reason `vfs` is
+sometimes chosen), is already shipped in the image, and needs `/dev/fuse`
+(provided by `--privileged`). The warning's remediation line adapts to whether
+`/dev/fuse` is present, so when it is missing it tells you to add `--privileged`
+or `--device /dev/fuse` before switching.
+
+```bash
+docker run -d --privileged \
+  -e DIND_STORAGE_DRIVER=fuse-overlayfs \
+  --name box-dind-cow \
+  konard/box-dind sleep infinity
+```
+
 CI verifies the forced `vfs` path:
 
 ```bash

docs/dind/USAGE.md

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# Isolated unit test for the issue #104 vfs storage-driver warning in
+# dind-entrypoint.sh.
+#
+# Landing on the `vfs` storage driver used to be silent (only a `log` line named
+# the driver), so an operator hitting `failed to register layer: no space left on
+# device` had no breadcrumb pointing at the copy-on-write-less driver. The
+# entrypoint now emits a one-time `warn` whenever the *active* driver is `vfs`,
+# with a remediation hint whose wording depends on whether the fuse-overlayfs
+# device node is available.
+#
+# Building the full box-dind image requires overlay-backed nested Docker, which
+# this sandbox cannot provide, so — exactly like preload-unit-test.sh — we source
+# the real entrypoint (DIND_ENTRYPOINT_SOURCE_ONLY=1 returns before the
+# startup/handoff flow) to get `warn_if_vfs_storage_driver` verbatim and drive it
+# directly. The /dev/fuse probe is pointed at a temp path via DIND_FUSE_DEVICE so
+# both remediation branches are exercised deterministically without a real device
+# node.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENTRYPOINT="$SCRIPT_DIR/../ubuntu/24.04/dind/dind-entrypoint.sh"
+
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+
+# Source the real entrypoint for its functions only.
+# shellcheck disable=SC1090
+DIND_ENTRYPOINT_SOURCE_ONLY=1 . "$ENTRYPOINT"
+
+pass=0; fail=0
+check() { # check <description> <condition-cmd...>
+  desc="$1"; shift
+  if "$@"; then echo "  PASS: $desc"; pass=$((pass+1)); else echo "  FAIL: $desc"; fail=$((fail+1)); fi
+}
+
+ERR="$WORK/err.log"
+PRESENT_FUSE="$WORK/fuse-present"   # an existing path: stands in for /dev/fuse
+MISSING_FUSE="$WORK/fuse-missing"   # a path that does not exist
+: > "$PRESENT_FUSE"
+rm -f "$MISSING_FUSE"
+
+echo "== Case 1: vfs driver emits the copy-on-write warning =="
+: > "$ERR"
+DIND_FUSE_DEVICE="$PRESENT_FUSE" warn_if_vfs_storage_driver vfs 2>"$ERR"
+check "warns the driver is vfs"                     grep -q "'vfs' storage driver" "$ERR"
+check "calls out NO copy-on-write"                  grep -q "NO copy-on-write" "$ERR"
+check "names the disk failure mode"                 grep -q "no space left on device" "$ERR"
+check "names the fuse-overlayfs remediation"        grep -q "DIND_STORAGE_DRIVER=fuse-overlayfs" "$ERR"
+
+echo "== Case 2: overlay2 driver stays silent =="
+: > "$ERR"
+DIND_FUSE_DEVICE="$PRESENT_FUSE" warn_if_vfs_storage_driver overlay2 2>"$ERR"
+check "no warning for overlay2" bash -c '! test -s "$1"' _ "$ERR"
+
+echo "== Case 3: fuse-overlayfs driver stays silent =="
+: > "$ERR"
+DIND_FUSE_DEVICE="$PRESENT_FUSE" warn_if_vfs_storage_driver fuse-overlayfs 2>"$ERR"
+check "no warning for fuse-overlayfs" bash -c '! test -s "$1"' _ "$ERR"
+
+echo "== Case 4: empty/unknown driver stays silent =="
+: > "$ERR"
+DIND_FUSE_DEVICE="$PRESENT_FUSE" warn_if_vfs_storage_driver "" 2>"$ERR"
+check "no warning for empty driver" bash -c '! test -s "$1"' _ "$ERR"
+
+echo "== Case 5: /dev/fuse present -> 'set fuse-overlayfs' remediation =="
+: > "$ERR"
+DIND_FUSE_DEVICE="$PRESENT_FUSE" warn_if_vfs_storage_driver vfs 2>"$ERR"
+check "remediation says the device is present"   grep -q "is present" "$ERR"
+check "remediation does NOT claim it is missing" bash -c '! grep -q "is missing" "$1"' _ "$ERR"
+
+echo "== Case 6: /dev/fuse missing -> explains why fuse-overlayfs is unavailable =="
+: > "$ERR"
+DIND_FUSE_DEVICE="$MISSING_FUSE" warn_if_vfs_storage_driver vfs 2>"$ERR"
+check "remediation explains the device is missing"   grep -q "is missing" "$ERR"
+check "remediation suggests --device /dev/fuse"      grep -q -- "--device /dev/fuse" "$ERR"
+check "remediation suggests --privileged"            grep -q -- "--privileged" "$ERR"
+check "still names the fuse-overlayfs driver"        grep -q "DIND_STORAGE_DRIVER=fuse-overlayfs" "$ERR"
+
+echo "== Case 7: function returns success so the start_dockerd success branch is unaffected =="
+# warn_if_vfs_storage_driver runs immediately before `return 0`; under `set -e`
+# a non-zero return would abort startup. Assert exit status 0 for both vfs and
+# non-vfs drivers.
+check "returns 0 for vfs"      bash -c 'DIND_ENTRYPOINT_SOURCE_ONLY=1 . "$1"; DIND_FUSE_DEVICE="$2" warn_if_vfs_storage_driver vfs >/dev/null 2>&1' _ "$ENTRYPOINT" "$PRESENT_FUSE"
+check "returns 0 for overlay2" bash -c 'DIND_ENTRYPOINT_SOURCE_ONLY=1 . "$1"; warn_if_vfs_storage_driver overlay2 >/dev/null 2>&1' _ "$ENTRYPOINT"
+
+echo
+echo "RESULT: $pass passed, $fail failed"
+[ "$fail" -eq 0 ]

experiments/test-issue104-vfs-warning.sh

@@ -19,3 +19,16 @@ if [ "$driver" != "vfs" ]; then
 fi
 
 log "inner dockerd is using the vfs storage driver"
+
+# issue #104: landing on vfs must not be silent. The entrypoint (PID 1) emits a
+# copy-on-write warning to stderr, which docker captures in the container logs.
+log "verifying the vfs copy-on-write warning was emitted (issue #104)"
+logs="$(docker logs "$container" 2>&1 || true)"
+for needle in "'vfs' storage driver" "no space left on device" "DIND_STORAGE_DRIVER=fuse-overlayfs"; do
+  if ! printf '%s' "$logs" | grep -qF "$needle"; then
+    printf '%s\n' "$logs" >&2
+    fail "expected the vfs warning to mention \"${needle}\", but it was absent from the container logs"
+  fi
+done
+
+log "vfs copy-on-write warning is present in the container logs"

tests/dind/example-storage-driver-vfs.sh

@@ -16,7 +16,14 @@
 #   - Sysbox :  docker run --runtime=sysbox-runc konard/<base>-dind   (no --privileged)
 #
 # Environment overrides:
-#   DIND_STORAGE_DRIVER  Override storage driver (default: auto-detected: overlay2, fuse-overlayfs, vfs)
+#   DIND_STORAGE_DRIVER  Override storage driver (default: auto-detected: overlay2,
+#                        fuse-overlayfs, vfs). Note: vfs has NO copy-on-write — it
+#                        stores every image layer as a full, independent copy, so
+#                        large (multi-GB) images consume many times their size on
+#                        disk and 'docker pull'/'docker run' can fail with 'no
+#                        space left on device'. When the active driver ends up
+#                        being vfs the entrypoint emits a one-time warning naming
+#                        the fuse-overlayfs (copy-on-write) remediation. (issue #104)
 #   DIND_DATA_ROOT       Override --data-root for dockerd (default: /var/lib/docker)
 #   DIND_LOG_FILE        Where to write dockerd logs (default: /var/log/dockerd.log)
 #   DIND_WAIT_SECONDS    How long to wait for dockerd to come up (default: 30)
@@ -189,6 +196,40 @@ wait_for_dockerd_ready() {
   return 2
 }
 
+# Device node fuse-overlayfs needs for copy-on-write. Overridable so the unit
+# test can exercise both the "present" and "missing" remediation branches without
+# a real device node; in production it is always /dev/fuse.
+DIND_FUSE_DEVICE="${DIND_FUSE_DEVICE:-/dev/fuse}"
+
+# When the active storage driver is vfs, emit a one-time warning explaining the
+# copy-on-write footgun. vfs is a safe last-resort fallback (and a legitimate
+# explicit pin for overlay-on-overlay compatibility), so this is observability,
+# not a default change: it stores every image layer as a full, independent copy,
+# so a multi-GB image's on-disk footprint becomes the SUM of all cumulative layer
+# sizes — many times the image size — and 'docker pull'/'docker run' can fail with
+# 'failed to register layer: no space left on device' on a disk far larger than
+# the image. Without this breadcrumb the generic disk error is easily misdiagnosed
+# as "not enough disk" instead of "wrong driver wastes the disk"
+# (link-assistant/hive-mind#1914). The remediation depends on whether the
+# copy-on-write fuse-overlayfs driver's device node is available. (issue #104)
+warn_if_vfs_storage_driver() {
+  [ "$1" = "vfs" ] || return 0
+
+  warn "dockerd is using the 'vfs' storage driver, which has NO copy-on-write:"
+  warn "every image layer is stored as a full copy, so a multi-GB image's on-disk"
+  warn "footprint becomes the SUM of all cumulative layer sizes (many times the"
+  warn "image size). 'docker pull'/'docker run' can then fail with 'failed to"
+  warn "register layer: no space left on device' on a disk far larger than the image."
+  if [ -e "$DIND_FUSE_DEVICE" ]; then
+    warn "For copy-on-write here, set DIND_STORAGE_DRIVER=fuse-overlayfs (works"
+    warn "overlay-on-overlay; ${DIND_FUSE_DEVICE} is present)."
+  else
+    warn "fuse-overlayfs (copy-on-write, works overlay-on-overlay) is unavailable"
+    warn "because ${DIND_FUSE_DEVICE} is missing; run with --privileged or"
+    warn "--device /dev/fuse, then set DIND_STORAGE_DRIVER=fuse-overlayfs."
+  fi
+}
+
 start_dockerd() {
   if pgrep -x dockerd >/dev/null 2>&1; then
     log "dockerd already running (pid $(pgrep -x dockerd | head -n1))"
@@ -215,6 +256,7 @@ start_dockerd() {
     launch_dockerd "$DIND_STORAGE_DRIVER"
 
     if wait_for_dockerd_ready "$DIND_DOCKERD_PID" "$DIND_STORAGE_DRIVER"; then
+      warn_if_vfs_storage_driver "$DIND_STORAGE_DRIVER"
       return 0
     else
       result="$?"