Ways to have log verbosity

[danlsgiga]

Hi, I couldn't find a way to control the log verbosity. Looks like it's using INFO by default and I'd like to have WARN as the base level but also the option to have ERROR or DEBUG would be awesome.

Is this in the roadmap or if it does exist where can I find its documentation?

[magiconair]

@danlsgiga Why do you want to dial it down? There is no DEBUG log and fabio shouldn't log that much. If it does then this usually points to something in your setup: a flapping health check or a check with volatile output like a timestamp for example. You'd want the INFO level since it logs how fabio is started including the runtime configuration.

[danlsgiga]

Hey @magiconair, thanks for the quick follow up.

So, I'm getting lots of log entries with:

2017/09/06 14:45:07 [INFO] consul: Health changed to #10372843

I understand fabio should not be logging that much but it is and as you mentioned, how can I understand the above log message without more details? These details would usually be provided by a lower log level like DEBUG or in the worst cases even TRACE. And the option to be able to silence those in case I need for could be available for greater control.

IMHO having the runtime configuration dumped into the logs are not great as well. A better approach imo would be to provide an authenticated api endpoint where we could grab that. In our case, since we have central logging, the runtime configuration is readable by a lot of people and that include IP addresses and even consul tokens. If that's not going to change, I think obfuscating those sensitive data should be something to think about.

[magiconair]

Hi @danlsgiga,

The message should probably read consul: Raft index changed to #10372843 which explains its true nature. The reason I'm logging those as INFO is that a high rate of these messages points a problem in your consul setup because you are generating lots of writes on the consul master. This can either be caused by flapping health checks or health checks which have volatile output like a timestamp. In any case it points to an issue within your setup which should be fixed. Providing you with a switch to hide this makes it more likely to just tune it out.

At eBay we had exactly that issue where my team had a stable log which wouldn't grow and the other team was asking for log rotation since their log grew somewhat unbounded. Ultimately, they found the issue in their consul integration framework which was causing excessive writes on consul. They fixed it and it was stable ever since.

Since the root cause for lots of these log messages is in almost all cases volatile output in the health check and the fix is usually trivial I'm still in favor of saying "please fix your setup".

However, I've learned that there are other log messages that do not provide much value but can appear a lot in the output. I'll have a look at those on whether we really need them. I might add a debug log level which can be enabled and disabled but I'd still leave the above messages at INFO since they point to an issue in your cluster setup that needs fixing.

As for the runtime config: I agree that the consul tokens should be obfuscated and I'll add that since it is a good idea. However, I'd argue that the API endpoint for the config should exist in addition to adding the runtime config to the log since that provides you with historical information. So you can go back in time to see whether the config has been changed which you can't do with just an API endpoint. I'm curious which IP addresses you'd like to obfuscate since I don't consider them to be sensitive but I'm open to arguments.

[danlsgiga]

Hi @magiconair, thanks for the great insight! I'll surely look into my consul environment and check what is causing so much flaping on the health checks.

Really appreciated for the effort on making fabio even more awesome!

So, if the runtime config is being dumped into the logs and this is being centralized in our environment... having the consul IP address and the token there is pretty much giving access to your whole consul infra. Hence my request to have log verbosity since it is the most common way of controlling what goes or not to the logs.

Having the default logging at INFO is totally fine and I encourage it actually, but at the same time having the ability to choose the log level would be great.

[magiconair]

If I obfuscate the token the consul ip becomes somewhat useless, doesn't it?

[danlsgiga]

IMHO, not in the scenario where someone uses consul in an acl_default_policy = "allow" setup.
My position towards this is that dumping any configuration (even the ones with no secrets) should be seen as bad practice and discouraged. Configuration should never be dumped into logs for the same reasons Stack Traces should never be exposed to users. Simply because it may or may not contain sensitive data.

[magiconair]

I've just merged #366 which allows you to control the log level.