Skip to content

Override WARM_IP_TARGET, MINIMUM_IP_TARGET (et al) for nodes via labels or ENIConfig #3630

@endzyme

Description

@endzyme

What would you like to be added:
The ability to override WARM_IP_TARGET, MINIMUM_IP_TARGET, and family, at the node level via labels or ENIConfigs.

Why is this needed:
It has been difficult to run non-homogenous nodes through Karpenter (or even EKS managed node groups) due to the varying limitations of IPs per ENI based on instance type. Since WARM_IP_TARGET and MINIMUM_IP_TARGET are cluster wide settings, it makes balancing IP waste a difficult trade off and a moving target. With the default settings of WARM_ENI_TARGET=1 we experienced about 60% waste of IP address space and thus needed to pivot to WARM_IP_TARGET and MINIUM_IP_TARGET. We graphed the average waste per node and realized that when we tune MINIMUM_IP_TARGET down to a low number (to reduce waste on smaller nodes that run fairly static workloads) it would increase API rates on the AWS EC2 API. If we tune the minimum to be higher, we'd experience more waste. This made us realize that tuning this figure highly depends on the nature of both our workloads, and the instance types backing the cluster, making this a moving target that would technically need tuning if either API EC2 calls or wasted IPs got out of hand.

It would be nice to be able to override MINIMUM_IP_TARGET based on a label (or more realistically, based on the existing label to select an ENIConfig).

Possible Suggestion:

  1. Allow ENIConfig spec to define MINIMUM_IP_TARGET, WARM_IP_TARGET, WARM_PREFIX_TARGET, WARM_ENI_TARGET et al. The order of precedence would be ENIConfig (if spec for these are not defined) falls back to the Env Vars of the DaemonSet.
  2. Allow setting these vars as k8s.amazonaws.com/<warm-setting-var> as node labels, which the aws-node pod would then need to be able to read. - This would allow for direct override and not tightly couple it to ENIConfig so you don't need many ENIConfigs for overrides

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions