Not deregistering when consul health status fails

[discobean]

Hi, first thanks for some great software.

I am having an issue with consul health checks not working as expected when more than one service is registered with the same urlprefix.

If I have one service registered on for example: urlprefix=/microservice1, and the HTTP health check in consul is returning a successful result I get a new route added which has 100% weight, this is correct.

Then I add on a new host another urlprefix=/microservice1 service, then I check fabio routes, and I see it has now 2 routes, 50% going to the first host, 50% going to the second, that is expected.

However, when a HTTP health check fails for the first host and is critical, the route for the failing host is not removed. Only when both HTTP health checks fail on both servers are both routes removed. This is not expected.

You can see in the logs that fabio tries to route to the host, but it fails, and yet the attached healthcheck is critical, and it is still routing :(
2016/03/30 13:27:10 http: proxy error: dial tcp 10.90.34.115:39775: connection refused
2016/03/30 13:27:11 http: proxy error: dial tcp 10.90.34.115:39775: connection refused

It is as if the healthchecks are linked to both hosts somehow, maybe we have implemented it wrong?

[discobean]

I am using python consul client to register our checks, everything appears ok in the Consul GUI.

I'm also running Fabio 1.0.8. I'm trying with 1.1.1 now

[discobean]

I checked and 1.1.1 has the same issue, here is the log:

When I start fabio, one service is running, but the other health check is failing in consul:

2016/03/30 14:08:24 [INFO] Updated config to
route add ErrorTestingService-40 api-40.prod/ErrorTestingService/v1 http://10.90.44.52:41114/ tags "urlprefix-api-40.prod/ErrorTestingService/v1"
route add ErrorTestingService-40 api-40.prod/ErrorTestingService/v1 http://10.90.34.115:40414/ tags "urlprefix-api-40.prod/ErrorTestingService/v1"
-- So the above 2 routes are wrong, because one health check is failing

2016/03/30 14:08:39 [INFO] consul: Health changed to #339283
-- This health status is now a change from critical -> OK for one service above

2016/03/30 14:09:08 [INFO] consul: Health changed to #339289
-- This health status is now a change from OK -> critical, but no route removed

-- At this point only 1 server is running, but 2 routes exist in fabio

-- Now below, I stopped the last single service, and it unregistered both the services
2016/03/30 14:11:32 [INFO] consul: Health changed to #339315
2016/03/30 14:11:32 [INFO] Unregistered timer errortestingservice-40.api-40_prod./errortestingservice/v1.10_90_44_52_41114
2016/03/30 14:11:32 [INFO] Unregistered timer errortestingservice-40.api-40_prod./errortestingservice/v1.10_90_34_115_40414

Not sure what is going on here :(

[discobean]

Could it be because the service name is the same for the 2 checks? Even though they are on different nodes? I would have thought that was ok.

[magiconair]

You need to register services with unique service ids in consul or otherwise, this won't work. I'll update the readme to make this clear since you're not the first one to run into this issue. However, this is how you should use consul. It's not fabio related.

[discobean]

Thanks mate, I will update and I'm sure it will fix it as it makes sense now.

[discobean]

Haha it worked :D Thanks so much once more.