-
Notifications
You must be signed in to change notification settings - Fork 811
Override WARM_IP_TARGET, MINIMUM_IP_TARGET (et al) for nodes via labels or ENIConfig #3630
Description
What would you like to be added:
The ability to override WARM_IP_TARGET, MINIMUM_IP_TARGET, and family, at the node level via labels or ENIConfigs.
Why is this needed:
It has been difficult to run non-homogenous nodes through Karpenter (or even EKS managed node groups) due to the varying limitations of IPs per ENI based on instance type. Since WARM_IP_TARGET and MINIMUM_IP_TARGET are cluster wide settings, it makes balancing IP waste a difficult trade off and a moving target. With the default settings of WARM_ENI_TARGET=1 we experienced about 60% waste of IP address space and thus needed to pivot to WARM_IP_TARGET and MINIUM_IP_TARGET. We graphed the average waste per node and realized that when we tune MINIMUM_IP_TARGET down to a low number (to reduce waste on smaller nodes that run fairly static workloads) it would increase API rates on the AWS EC2 API. If we tune the minimum to be higher, we'd experience more waste. This made us realize that tuning this figure highly depends on the nature of both our workloads, and the instance types backing the cluster, making this a moving target that would technically need tuning if either API EC2 calls or wasted IPs got out of hand.
It would be nice to be able to override MINIMUM_IP_TARGET based on a label (or more realistically, based on the existing label to select an ENIConfig).
Possible Suggestion:
- Allow ENIConfig spec to define
MINIMUM_IP_TARGET,WARM_IP_TARGET,WARM_PREFIX_TARGET,WARM_ENI_TARGETet al. The order of precedence would be ENIConfig (if spec for these are not defined) falls back to the Env Vars of the DaemonSet. - Allow setting these vars as
k8s.amazonaws.com/<warm-setting-var>as node labels, which the aws-node pod would then need to be able to read. - This would allow for direct override and not tightly couple it to ENIConfig so you don't need many ENIConfigs for overrides