Skip to content

Emissary pods always terminate with error #5834

@zendad

Description

@zendad

Describe the bug
We are observing that Emissary Ingress pods consistently terminate with exit code 1, even during expected and graceful shutdown scenarios (e.g., during pod eviction or rollout). This behavior incorrectly flags the shutdown as an error in monitoring systems and Kubernetes events.

To Reproduce
Steps to reproduce the behavior:

  • Delete a emissary pod or scale down the number of pods
  • Check the logs for the terminated pod to see that the above shut down with error is printed

Pods

  • pods have a graceful termination period set.
  • A defined lifecycle preStop hook
  • have mesh(linkerd) enabled

Logs

Initial shutdown message when termination signal is received
{"msg":"goroutine \":signal_handler:0\" exited with error: received signal terminated (triggering graceful shutdown)","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:380","func":"github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1","level":"error","PID":1,"CMD":"entrypoint","time":"2025-04-03 10:48:17.4820","THREAD":":signal_handler:0"}

last container log msg
{"msg":"shut down with error error: received signal terminated (triggering graceful shutdown)","file":"/go/pkg/busy/busy.go:87","func":"github.com/emissary-ingress/emissary/v3/pkg/busy.Main","level":"error","PID":1,"CMD":"entrypoint","time":"2025-04-03 11:02:28.6516"}

This behavior happens even with mesh disabled or lifecycle hook or grace termination period removed.This can lead to confusion in observability tools (e.g., logs, dashboards) where it appears that the pod failed or crashed, even though it shut down as part of normal operations.

Expected behavior
When Emissary is gracefully shut down (e.g., due to SIGTERM), the process should exit with code 0 or another code that accurately reflects a clean shutdown.

Versions (please complete the following information):

  • Ambassador: 3.9.1*
  • Kubernetes environment : EKS
  • Version: 1.32

Additional context

  • Is Emissary incorrectly logging shutdown events as errors instead of info or warnings?
  • This behavior may be related to how the entrypoint script or shutdown logic handles termination signals.

I’d appreciate any insights on whether this is just a logging issue or if it affects the graceful termination process.

Metadata

Metadata

Assignees

No one assigned

    Labels

    t:bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions