Fabio 1.5.13 - no more "[INFO] Config updates" message in the logs

[samm-git]

Hi,

After updating from Fabio 1.5.11 to the Fabio 1.5.13 we found that [INFO] Config updates is completely gone. We are using this message to validate that Fabio configuration is correctly updated, so it is critical for us to have it. Also i did not found anything in the logs telling that it is removed.

I am using INFO error log, also tried with DEBUG one, with the same result. Fabio itself is updating routes, so looks like that issue affects only messaging.

This is how update was looking in 1.5.11:

 2020/02/04 14:58:03 [INFO] Config updates
 - fabio-heartbeat-nomad-5 fabio-heartbeat-nomad-5/ http://127.0.0.1:7/ tags "public,public:fabio-heartbeat-nomad-5/,private"
 - route add fabio-heartbeat-nomad-4 fabio-heartbeat-nomad-4/ http://127.0.0.1:7/ tags "public,public:fabio-heartbeat-nomad-4/,private"
 - route add

In 1.5.13 it is just empty. fabio-heartbeat-nomad-* is a special service which goes up and down every 5 minutes to validate that all routes are correctly distributed on all fabio nodes. However, it looks like that all route messages are missing entirely.

Please let me know if any other information is needed.

[aaronhurt]

This will be fixed in the next release. See #737 and #749 ... this was an unintended side affect of using a rolling buffer to build the route table.

[aaronhurt]

Closing this issue, feel free to re-open if this is not a duplicate of the above issues.

[aaronhurt]

Side note, would be curious about your nomad test. We're using Fabio with Nomad to deploy our applications as well.

[samm-git]

@leprechau thank you for the quick feedback! Test is very simple - in very rare case we have config updates stalled. It could be due to few issues:

1. Routing table is bad due to application bugs (should never be the case, but better to check)
2. Fabio is not reading updates from consul properly (i reported that already, no root cause found yet, happens not often).
3. Consul is not working

So using this heartbeat services we are checking if there are updates of the config in the last n minutes and if not - generating alert. We can also try to self-heal it by restarting fabio, but not doing it now as still hope to catch the root cause of the issue.

P.S. will just wait for the new release, thanks again.

[aaronhurt]

@samm-git Thank you. I'm hoping to roll a 1.5.14-beta at least this week. With all the changes lately I think a beta release may be the best to get some real-world testing done before moving forward with an actual release.

[codyja]

@leprechau thank you for the quick feedback! Test is very simple - in very rare case we have config updates stalled. It could be due to few issues:

1. Routing table is bad due to application bugs (should never be the case, but better to check)
2. Fabio is not reading updates from consul properly (i reported that already, no root cause found yet, happens not often).
3. Consul is not working

So using this heartbeat services we are checking if there are updates of the config in the last n minutes and if not - generating alert. We can also try to self-heal it by restarting fabio, but not doing it now as still hope to catch the root cause of the issue.

P.S. will just wait for the new release, thanks again.

Hi @samm-git , I was curious if you could share that hearbeat service you mentioned?