Support multiple metrics libraries

[postwait]

Current the metrics subsystem doesn't support histogram reporting of request latencies. Can we make the metrics reporting interface such that the implementation can be changed out for "gometrics"-style recording that supports histograms?

I'd like to integrate circonus-gometrics, but layered above the go-metrics currently used, all good measurement data is destroyed.

[maier]

just discussed a shim approach which sounds like it will solve the issue. if there is a metric interface in fabio itself then swapping out backends which want/support full raw measurement values will be easier.

[magiconair]

@postwait and @maier I've moved the metrics library implementation behind an interface in the metrics/metricslib package where I've extracted the methods from the go-metrics library I'm currently using. This should make it possible to plug in any metrics library you want as long as it supports this interface. If the interface is not sufficient for what you are trying to do feel free to enhance it but keep it simple and generic if possible. Looking forward to the PRs :)

[magiconair]

I think I missed a spot. The current implementation distinguishes between two different namespaces for metrics: the service metrics and the general fabio metrics (like requests). Service metrics get purged when the service or an instance disappears. This was implemented by using two different registries but that abstraction got lost with this change. I need to add that back in or make the purge code aware of what is a service metric and what isn't. Just keep that in mind when implementing your change.

[maier]

Cool, thanks, I'm working off that branch to do an integration.

[magiconair]

@maier My previous approach was too complex. The whole notion of the metrics package providing two registries was somewhat broken which made the testing more complex than needed to be. I've dropped the metricslib and the gometrics package and rolled this all into the metrics package. There you can now create multiple registries each with a single name space. The main.go then sets up the two name spaces for global and service metrics. That makes it cleaner and simpler IMO but you need to change something in your code. Sorry about that.

Supporting multiple different providers could then be implemented by adding a MultiRegistry analogous to the io.MultiWriter