Always deregister from Consul

[far-blue]

I've hunted through the wiki documentation but can't find anywhere referring to how to shutdown fabio in a graceful manner that deregisters the service and associated health check from Consul. I know with other apps you can send a different sig (e.g. sigint) for a graceful shutdown. Does Fabio support such behaviour and, if not, what's the recommended approach for graceful shutdown that also cleans up Consul records?

[magiconair]

SIGQUIT or SIGTERM should do this. Let me check whether this is a regression. How are you shutting it down?

[far-blue]

I'm just running the docker container (in the foreground) via docker-compose and when I want to make changes I'm using ctl-c to shut it down and just running docker-compose up again.

[magiconair]

Looks like a shutdown race. I'll try to make it more robust and provide a patch. Can you work with that and build a fabio binary yourself or do you need a full Docker container to test this?

[far-blue]

What's the race? Is it that docker-compose isn't giving enough time for the shutdown? If that's the case I can tell docker-compose to wait longer before sending the kill signal.

If it's something internal then yes, I can prob. build my own docker image from the source or, if not, I can wait for a new image release. As long as I know the problem is being sorted I can cope with cleaning out the Consul data manually during testing :)

[magiconair]

Actually, I don't think there is a race. startListeners blocks until it gets a shutdown signal which is triggered by signal.Notify(sigchan, os.Interrupt, os.Kill, syscall.SIGTERM). That closes the quit channel which returns startListeners and then triggers registry.Default.Deregister() as the last call. And that one waits for completion, AFAICT.

You should have the following log output:

2016/07/26 16:50:53 [INFO] Updated config to
2016/07/26 16:50:53 [INFO] consul: Registered fabio with id "fabio-LM-AMS-00963965-9998"
2016/07/26 16:50:53 [INFO] consul: Registered fabio with address "10.249.126.25"
2016/07/26 16:50:53 [INFO] consul: Registered fabio with tags ""
2016/07/26 16:50:53 [INFO] consul: Registered fabio with health check to "http://[10.249.126.25]:9998/health"
2016/07/26 16:50:53 [INFO] consul: Health changed to #3360
2016/07/26 16:50:54 [INFO] consul: Health changed to #3361
2016/07/26 16:51:00 [INFO] Graceful shutdown over 0
2016/07/26 16:51:00 [INFO] Down
2016/07/26 16:51:00 [INFO] consul: Deregistering fabio
2016/07/26 16:51:00 [INFO] consul: Health changed to #3363

[far-blue]

Ah, right, I think I see the problem. My mistake. It must be that it only leaves the data in Consul if it dies for some reason while processing the config file or it fails to talk to Vault. I think I'd miss-counted the number of left-over healthchecks vs the number of times I'd started and stopped the container.

[far-blue]

It also doesn't help that the 'Down' line and all after it are not visible via docker-compose during the shutdown - although if you run docker-compose logs after the container has terminated you can then see the log output :)

Thanks for looking into things and sorry it was my error!

[magiconair]

I think by moving the deregistration into a defer block should take care of a stale registration if fabio fatals during startup but after registration. I'll have a look.

[far-blue]

ah, that would be great :) I guess leaving the health check in place to turn critical when/if Fabio fails is a good thing as long as Fabio has started up correctly and has been running for a few seconds.

As I run consul-alert I find out pretty quickly if health checks are left dangling!

[magiconair]

Yeah, that's the edge case. If fabio FATALs after initBackend it does not clean up after itself. defer won't help since FATAL calls os.Exit(1) I'll see how to improve that.