Describe the bug
We are observing that Emissary Ingress pods consistently terminate with exit code 1, even during expected and graceful shutdown scenarios (e.g., during pod eviction or rollout). This behavior incorrectly flags the shutdown as an error in monitoring systems and Kubernetes events.
To Reproduce
Steps to reproduce the behavior:
- Delete a emissary pod or scale down the number of pods
- Check the logs for the terminated pod to see that the above shut down with error is printed
Pods
- pods have a graceful termination period set.
- A defined lifecycle preStop hook
- have mesh(linkerd) enabled
Logs
Initial shutdown message when termination signal is received
{"msg":"goroutine \":signal_handler:0\" exited with error: received signal terminated (triggering graceful shutdown)","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:380","func":"github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1","level":"error","PID":1,"CMD":"entrypoint","time":"2025-04-03 10:48:17.4820","THREAD":":signal_handler:0"}
last container log msg
{"msg":"shut down with error error: received signal terminated (triggering graceful shutdown)","file":"/go/pkg/busy/busy.go:87","func":"github.com/emissary-ingress/emissary/v3/pkg/busy.Main","level":"error","PID":1,"CMD":"entrypoint","time":"2025-04-03 11:02:28.6516"}
This behavior happens even with mesh disabled or lifecycle hook or grace termination period removed.This can lead to confusion in observability tools (e.g., logs, dashboards) where it appears that the pod failed or crashed, even though it shut down as part of normal operations.
Expected behavior
When Emissary is gracefully shut down (e.g., due to SIGTERM), the process should exit with code 0 or another code that accurately reflects a clean shutdown.
Versions (please complete the following information):
- Ambassador: 3.9.1*
- Kubernetes environment : EKS
- Version: 1.32
Additional context
- Is Emissary incorrectly logging shutdown events as errors instead of info or warnings?
- This behavior may be related to how the entrypoint script or shutdown logic handles termination signals.
I’d appreciate any insights on whether this is just a logging issue or if it affects the graceful termination process.
Describe the bug
We are observing that Emissary Ingress pods consistently terminate with exit code 1, even during expected and graceful shutdown scenarios (e.g., during pod eviction or rollout). This behavior incorrectly flags the shutdown as an error in monitoring systems and Kubernetes events.
To Reproduce
Steps to reproduce the behavior:
Pods
Logs
Initial shutdown message when termination signal is received
{"msg":"goroutine \":signal_handler:0\" exited with error: received signal terminated (triggering graceful shutdown)","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:380","func":"github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1","level":"error","PID":1,"CMD":"entrypoint","time":"2025-04-03 10:48:17.4820","THREAD":":signal_handler:0"}last container log msg
{"msg":"shut down with error error: received signal terminated (triggering graceful shutdown)","file":"/go/pkg/busy/busy.go:87","func":"github.com/emissary-ingress/emissary/v3/pkg/busy.Main","level":"error","PID":1,"CMD":"entrypoint","time":"2025-04-03 11:02:28.6516"}This behavior happens even with mesh disabled or lifecycle hook or grace termination period removed.This can lead to confusion in observability tools (e.g., logs, dashboards) where it appears that the pod failed or crashed, even though it shut down as part of normal operations.
Expected behavior
When Emissary is gracefully shut down (e.g., due to SIGTERM), the process should exit with code 0 or another code that accurately reflects a clean shutdown.
Versions (please complete the following information):
Additional context
I’d appreciate any insights on whether this is just a logging issue or if it affects the graceful termination process.