fabio service entries may stay in Consul on dirty exit

[pires]

Abstract

In #426, the register option was introduced in order to provide a way for Fabio to register itself in Consul under user-specified names, so that certain services become reachable through Fabio via Consul-provided A records.

Said functionality (code here) registers each service with:

• one Consul HTTP check that queries the Fabio health endpoint, and
• a Consul safeguard that will deregister the service if abovementioned check has failed and didn't recover within a pre-configured timeout.

It seems to me that, in theory, this meant to the author and reviewers of the feature that whenever a Fabio instance wasn't reported as healthy, all services pointing to it would be marked as unhealthy and it would be given a certain amount of time for that Fabio instance to recover before Consul deregistering said services. However, this is not truly the case.

Detailed problem description

At this point, it's important to understand each service ID registered by Fabio is computed from the Fabio's instance hostname and configured IP:port endpoint. This information will be relevant shortly.

The problem is two-fold:

• Consul may pass the HTTP check for service entries owned by a now defunct Fabio instance;
• Following-up, and since the services' checks pass, Consul reaper will not deregister service entries owned by defunct Fabio instances, effectively resulting in Consul piling up duplicated service and related check entries.

Here's one example of how this can happen. If one runs Fabio as a Docker container and maps ports to host network, and said container crash-loops, one will end up with duplicated service entries that only differ in the service ID, given the hostname used to build the service ID is unique per container instance.
Now, all duplicated services belonging to Fabio instances that shared the same Docker host will point to the same HTTP check endpoint, the Fabio's (Docker host) HTTP endpoint, effectively resulting in said duplicated entries to not be removed and pile up as more crashes happen.
This will eventually resolve itself if no more instances of Fabio container run on said host and the HTTP check fails and doesn't recover until the Consul reaper kicks-in (Fabio defaults to 90m).

Potential solution

So how can we have:

• Fabio understand the difference between a service it owns and a similar service that was owned by a now defunct instance of Fabio?
• Consul reaping duplicated, dead service and related check entries?

Consul supports the notion of multiple checks per service and even different types of checks. One type of check that seems to be fully deterministic for using here, is the TTL check.
One potential solution is for Fabio to add a TTL check to each service, with a check ID computed the same way as the related service ID, so that only it (this instance of Fabio) will know about the TTL check and explicitly reset its clock, periodically (before the TTL expires). This way, whenever a Fabio instance fails to reset a TTL check clock, the same check will be marked as critical, and assuming Fabio is configured with registry.consul.checksrequired = all, if the HTTP check passes but the TTL fails, Fabio will not serve this service. It will, however, serve the service that this instance owns and periodically refreshes the TTL check for.

Last but not least, since the seemingly defunct services will be marked as unhealthy, the Consul reaper will finally be able to do its job. One can tune Fabio's registry.consul.checkDeregisterCriticalServiceAfter configuration parameter, in order to have more control of when the service and related checks are deleted after the former is marked as unhealthy.

If we can agree on this solution, I am ready to open a PR that addresses it.

[aaronhurt]

I understand the problem and your proposed solution. However, I'm not sure it's the responsibility of fabio to ensure the health of another service.

I'm more concerned with your replication of the problem relying on fabio crashing and being constantly restarted. Why is that happening?

Am I missing something here?

[pires]

I understand your concern and my only argument in favor of introducing this is that I believe Fabio can be made more resilient when something nasty, like what I described, happens.

In the example I shared, there may be multiple reasons why the container would crash, such as Linux oom-killer kicking in, Docker daemon misbehaving, or just poor orchestration of the Docker container. In my case, this happened due to the first, since the Fabio version I'm running has some memory leaks which may have been fixed since then.

[pschultz]

I think if Fabio registers a service in Consul it is also responsible for setting up reliable health checks.

pires' suggestion sounds good, but it does assume that the Consul agent can connect to the upstream directly which may not be possible, depending on the network configuration (the agent may be on another host which can't reach the upstream, for example).

I feel the most reliable check is one that establishes a connection through fabio to the upstream, just like a client would that discovers the service in Consul.

For TCP proxies this seems as simple as setting up a TCP check that uses fabio's respective TCP port.

For HTTP(S) proxies Consul should send HTTP requests through fabio. The specifics for this request would have to be configurable, of course. HTTP checks can be quite complex, especially if TLS is involved. On the other hand, this would also consider whether fabio is able to serve a valid certificate. Obviously fabio shouldn't accept traffic for a route if the certificate source is broken, independent of the upstream's health.

I doubt that we could infer a sensible default HTTP check from the route config, but I haven't thought this through.

[pires]

pires' suggestion sounds good, but it does assume that the Consul agent can connect to the upstream directly which may not be possible

The solution I suggest, adding also one TTL check per register, doesn't require any connectivity other than Fabio to Consul agent, or am I missing something here?

[pschultz]

My brain must have replaced one or more instances of TTL in your opening post with TCP. Nevermind that comment.

Adding the TTL check seems like a simple solution for the specific problem you encountered. However, I'm concerned that the intended regular TTL update requests cause fabio to detect a change in the health status and reload the routing table. That would be a considerable performance issue. An alternative to the TTL check would be to make the hostname segment in the service ID configurable (or even make the whole ID a Go template).

We can talk about changing the checks to something generally more useful some other time. I shouldn't have hijacked this issue.

[pires]

@pschultz by all means, thank you for your feedback, appreciate it. I didn't think passing the TTL check would result on an update, unless it changes the check status, eg critical to passing. I'll be validating this and keep you posted.

[pires]

FYI, I've opened PR #664.

@pschultz during my debugging sessions I didn't find any instances of fabio being notified that a service had been updated after TTL was reset. I will continue to try and find it out but help is always appreciated.