1.0 stabilization

[fabxc]

As discussed in the last SIG instrumentation meeting, we plan to do a first stable release of kube-state-metrics.
As we have been mostly adding functionality for a while, rather than changing existing one, there's nothing fundamental to change here.

• double-check all existing metrics for compliance with our guidelines
• double-check current functionality does not conflict with future plans of fetching partial metrics, i.e. only for pod metrics for certrain deployments
• load test kube-state-metrics to ensure it scales with large clusters and derive resource requirements (@piosz, can you help with that?)
• provide deployment manifest that scales with cluster size using pod nanny

[piosz]

@kubernetes/api-reviewers are there any guidelines/requirements for graduating a feature to GA which doesn't Kubernetes API?

cc @wojtek-t @gmarek re: scalability testing

[smarterclayton]

I think we'd probably recommend documenting what the compatibility expectations of that feature are going forward (in a doc in that repo), make sure there is a process for API changes reasonably consistent with the use of the goals, and then make sure the feature repo contains an issue to the graduated feature.

[piosz]

@loburm will help with scalability tests

[brancz]

I'm now back from vacation and can help coordinate. @loburm and @piosz let me know how/when you want to tackle this.

[loburm]

@brancz for the scalability testing we need to have some scenario to test against. I think we should concentrate only on testing metric related to nodes and pods mostly (I assume that other parts should consume significantly smaller amount of resources). How many nodes and pod should be present in the test scenario?

[brancz]

@loburm I'm completely new to the load tests, so I suggest to start with whatever seems reasonable to you. My thoughts are the same as yours, the number of pods metrics are expected to increase linearly with the number of other objects, so focusing on those and nodes sounds perfect for our load scenarios.

Testing with the recommended upper bound of recommended pods/nodes in a single cluster would be best to see if we can actually handle this, but I'm not sure that's reasonable given that we have never performed load tests before.

[piosz]

We had a chat offline and we will try to test the following scenarios:

1. 100 nodes cluster, 30pods/node
2. O(1000) nodes cluster, 30 pods/node

@loburm will verify:

1. what is the approximate resource usage in both cases
2. what is the average request latency
3. whether there any other obvious issues

[brancz]

Great thanks a lot @piosz and @loburm !

[matthiasr]

One issue (that we can only do so much about) is the size of `/metrics` and the time it takes Prometheus to scrape it. Putting some bound on that could inform future decisions on adding metrics.

…

On Thu, Jul 20, 2017, 11:51 Frederic Branczyk ***@***.***> wrote: Great thanks a lot @piosz <https://github.com/piosz> and @loburm <https://github.com/loburm> ! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#124 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAICBrPEar4LErT__nbok0UGNaJcxtfSks5sPyMdgaJpZM4M_5pt> .

[brancz]

Thanks for the heads up @matthiasr! Yes that's one of the bottle-necks I can see happening. We may have to start thinking of sharding strategies for kube-state-metrics.

[piosz]

Do you think it would be possible to have some kind of pagination support for /metrics handler?

[andyxning]

How about we split the scrape endpoints according to different collectors, i.e., pods state metrics is available on /metrics/pods.

Or, how about we support both /metrics to fetch all metrics and /metrics/* to every collectors.

Cons: this will make Prometheus configuration more complicated.

[brancz]

What @andyxning is suggesting is certainly possible, but is likely to just postpone the problem. @piosz I'm not aware of any precedence of that, but paging within the same instance of kube-state-metrics would also just postpone the problem, as I can imagine that the memory consumption is also very large in cases where response timeouts are hit.

[loburm]

@andyxning I think that will add unnecessary complexity and as I have understood common rule is to expose metrics on /metrics. We can achieve the same by creating multiple instances of kube-state-metrics and each one is responsible for one or multiple collectors we have a special flag for this.

And let me first perform some tests and once we have some real numbers, we can start thinking about possible issues and how they can be solved.

[brancz]

Completely agree with @loburm, measure first.