All my dynamic routes suddenly vanished!

[far-blue]

I have no idea if this is a Fabio bug, a consul bug or some other weirdness but overnight last night, all ~ 400 dynamic routes from our Consul cluster disappeared from Fabio. Fabio isn't outputting and debug / trace logs, it can still read the static routes in consul's kv store, it can still talk to Vault but there are no routes. All the services are still reporting as healthy in Consul.

Anyone got any ideas?!

Obviously this is a huge headache for us and I've had to manually create proxy routes with nginx and hard-coded IP addresses at the moment - not good!

[far-blue]

more info:

I can see fabio is successfully requesting service and check info from consul via tcpdump.

I can also see fabio 'knows' about services because I can get it to list them by pushing metrics to stdout.

However, there are no dynamic routes listed in the UI - the request returns 'null' - and fabio doesn't open the ports configured in the config (except the admin port).

This is using the latest release of fabio.

I've a mix of consul 1.9.4, 1.9.6 and 1.10.0 agents on our network and it doesn't appear to matter which fabio is pointed at. However, in case it's important, the 'server' instances of consul are all 1.10.0

[far-blue]

Our consul setup does have ACLs enabled with default-deny but I'm currently testing with the master consul token with no difference in behaviour

[far-blue]

Ok, I've identified the problem and raised a bug ticket here #838 so I'll close this issue but people can still find it if they hit the same problem :)