-
Notifications
You must be signed in to change notification settings - Fork 5.8k
[BUG] Compose watch can recreate a depends_on service and leave it stuck in Created #13717
Copy link
Copy link
Open
Labels
Description
Description
Compose watch/reconcile appears to mishandle depends_on when both the watched service and its dependency have develop.watch configured.
I reduced this to a minimal public repro in https://github.com/jackmpcollins/compose-watch-backend-recreate-repro
Current behavior:
- I run
docker compose up --build --watch - I touch only
frontend/trigger.txt - Compose reports that it is rebuilding
frontend - Compose also rebuilds/reconciles
backend backendis recreated and then left stuck inCreatedfrontendstays up, but requests return502- frontend logs show
lookup backend on 127.0.0.11:53: no such host
Expected behavior:
- touching only the frontend trigger file should not strand
backend - either only
frontendshould be recreated, or ifbackendis touched by reconciliation it should come
back up cleanly
This seems related to:
- [BUG] compose watch combined with depends_on can lead to failure of dependencies #11695
- [BUG] docker compose watch <service> ignores depdends_on definitions #11813
- [BUG] compose watch stops services using
linksanddepends_onwithrestart: true#12814 - also (re)start dependent services after watch rebuilt image #12879
Steps To Reproduce
I reduced this to a minimal public repro in https://github.com/jackmpcollins/compose-watch-backend-recreate-repro
Setup
Minimal docker-compose.yml:
services:
backend:
build: ./backend
develop:
watch:
- action: restart
path: ./backend/Dockerfile
frontend:
depends_on:
- backend
build: ./frontend
ports:
- "3002:3000"
develop:
watch:
- action: rebuild
path: ./frontend/trigger.txtbackend/Dockerfile:
FROM hashicorp/http-echo:1.0.0
CMD ["-text=backend ok", "-listen=:8000"]frontend/Dockerfile:
FROM caddy:2.10.2-alpine
COPY Caddyfile /etc/caddy/Caddyfilefrontend/Caddyfile:
:3000 {
reverse_proxy backend:8000
}
frontend/trigger.txt:
touch me to trigger a frontend rebuild
Repro
Repro path 1:
docker compose down
docker compose up -d --build
docker compose watch frontend
Repro path 2:
docker compose down
docker compose up --build --watch
In another shell:
touch frontend/trigger.txt
docker compose ps
docker compose ps -a
curl -i http://localhost:3002/
docker compose logs --since=1m frontend
Observed watch output includes:
Rebuilding service(s) ["frontend"] after changes were detected...
Container <project>-backend-1 Recreate
Container <project>-backend-1 Recreated
Container <project>-frontend-1 Recreate
Container <project>-frontend-1 Recreated
With docker compose up --build --watch, I also see:
backend-1 has been recreated
backend-1 ... received interrupt, shutting down...
backend-1 exited with code 2
Observed broken state after that:
docker compose psno longer lists backenddocker compose ps -ashows backend in Createdcurl http://localhost:3002/returns 502- frontend logs show
lookup backend on 127.0.0.11:53: no such host
Ablation notes:
- removing
backend.develop.watchmakes the bug disappear - removing
frontend.depends_onmakes the bug disappear - removing
frontend.develop.watchmakes the bug disappear - changing backend to a pure
image: servicemakes the bug disappear - the backend watch action does not need to be
rebuild;restartis enough
Compose Version
Docker Compose version v5.1.1
Docker Compose version v5.1.1
Docker Environment
Client:
Version: 29.3.1
Context: desktop-linux
Debug Mode: false
Plugins:
agent: Docker AI Agent Runner (Docker Inc.)
Version: v1.34.0
Path: /Users/jack/.docker/cli-plugins/docker-agent
ai: Docker AI Agent - Ask Gordon (Docker Inc.)
Version: v1.20.1
Path: /Users/jack/.docker/cli-plugins/docker-ai
buildx: Docker Buildx (Docker Inc.)
Version: v0.32.1-desktop.1
Path: /Users/jack/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v5.1.1
Path: /Users/jack/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.47
Path: /Users/jack/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Docker Inc.)
Version: v0.3.0
Path: /Users/jack/.docker/cli-plugins/docker-desktop
dhi: CLI for managing Docker Hardened Images (Docker Inc.)
Version: v0.0.2
Path: /Users/jack/.docker/cli-plugins/docker-dhi
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.31
Path: /Users/jack/.docker/cli-plugins/docker-extension
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: /Users/jack/.docker/cli-plugins/docker-init
mcp: Docker MCP Plugin (Docker Inc.)
Version: v0.40.3
Path: /Users/jack/.docker/cli-plugins/docker-mcp
model: Docker Model Runner (Docker Inc.)
Version: v1.1.28
Path: /Users/jack/.docker/cli-plugins/docker-model
offload: Docker Offload (Docker Inc.)
Version: v0.5.77
Path: /Users/jack/.docker/cli-plugins/docker-offload
pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
Version: v0.0.24
Path: /Users/jack/.docker/cli-plugins/docker-pass
sandbox: Docker Sandbox (Docker Inc.)
Version: v0.12.0
Path: /Users/jack/.docker/cli-plugins/docker-sandbox
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/jack/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.20.3
Path: /Users/jack/.docker/cli-plugins/docker-scout
Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 37
Server Version: 29.3.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Discovered Devices:
cdi: docker.com/gpu=webgpu
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
runc version: v1.3.4-0-gd6d73eb8
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.12.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 12
Total Memory: 23.43GiB
Name: docker-desktop
ID: 48c0ae5a-cc67-41ed-9230-7e1bc099096d
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/jack/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
::1/128
127.0.0.0/8
Live Restore Enabled: false
Firewall Backend: iptables
Anything else?
No response
Reactions are currently unavailable