Multi AZ Subnet to be selected only if there is available IPs

[zakariais]

Version

Karpenter Version: v0.18.1
Kubernetes Version: v1.23

Expected Behavior

Karpenter should select a subnet with available IPs from all subnets available for EKS. We are facing a issue where subnet from one AZ is running out of IPs, we have one subnet for each AZs. I understand the subnet is chosen randomly, which is ok, but a given subnet might run out of IP, and it would be better if Karpenter selects subnet with (the most?) available IPs from all subnets available in a VPC.

Actual Behavior

If multiple subnets from different AZs are available, karpenter choose one randomly, without considering if the subnet has IPs available. And unfortunately, we have a situation where we are running out of IPs in specific availability zones and karpenter seems to be creating more instances on that zone without even considering other subnets in other AZs.

Steps to Reproduce the Problem

1. Have multiple subnet that matches the subnetSelector from your provisioner for different AZs. One of those subnet must have no free IPs.
2. Scale up a deployment to ensure Karpenter needs to create instance (I use inflate deployment from Getting started tutorial) and with a few tries, most probably you will end up with the provisioner to select this given subnet.

Resource Specs and Logs

Provisioner spec:

spec:
  kubeletConfiguration: {}
  labels:
    group: default
  limits: {}
  provider:
    apiVersion: extensions.karpenter.sh/v1alpha1
    instanceProfile: <our_instance_profile_name>
    kind: AWS
    launchTemplate: <our_launch_template_name>
    securityGroupSelector:
      karpenter.sh/cluster/clusterName: "owned"
    subnetSelector:
      karpenter.sh/cluster/clusterName: "owned"
  requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values:
    - spot
  - key: node.kubernetes.io/instance-type
    operator: In
    values:
    - m5.4xlarge
  - key: topology.kubernetes.io/zone
    operator: In
    values:
    - us-east-1a
    - us-east-1b
    - us-east-1c
  - key: kubernetes.io/arch
    operator: In
    values:
    - amd64
  ttlSecondsAfterEmpty: 30
  ttlSecondsUntilExpired: 86400

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[FernandoMiguel]

Part of your problem comes from using SPOT, which will choose the cheapest AZ
AZ prices can be dramatically different between them

the other part of the problem comes from
#2572

you can consider using CGNAT to extend your IP range. using 100.64/19 AZs would increase your IP pool to values that you would never have an issue anytime soon

but yes, Karpenter could do a better job at avoiding AZs without IPs
in the meantime you could use different provisioners , each with different AZs in them, and weight the provisioners, avoiding the more used AZs.

[liorfranko]

We wouldn't want to override Karpenter choosing the cheapest AZ.
We think that Karpenter should choose the cheapest AZ, but if the subnet is full -> fallback to the next AZ.

[tjhiggins]

I was just about to create a similar issue. Agreed with @liorfranko on picking the cheapest, but falling back to the next available. Also using spot and running out of ips. I have 3 subnets each in their own AZ and it mostly only allocates ips in the first AZ.

[FernandoMiguel]

Class C CIDRs are too small for EKS given each pod eats one IP.
Take a look https://aws.amazon.com/blogs/containers/addressing-ipv4-address-exhaustion-in-amazon-eks-clusters-using-private-nat-gateways/

[tjhiggins]

Class C CIDRs are too small for EKS given each pod eats one IP. Take a look https://aws.amazon.com/blogs/containers/addressing-ipv4-address-exhaustion-in-amazon-eks-clusters-using-private-nat-gateways/

Can you still expose services publicly if you use a private nat gateway? We use traefik behind an nlb on eks.

The Karpenter terraform install tutorial should probably be updated to use larger subnets then or use a private nat gateway.

[FernandoMiguel]

@tjhiggins "carrier grade NAT" is an industry name. It has nothing to do with AWS VPC Nat Gateways.
Nothing changes in that part of the infrastructure.

Karpenter is agnostic. It's up to practitioners to architect their infrastructure.
AWS Best Practices do recommend clients to use extended subnets so they don't run out of IPs

[FernandoMiguel]

given that Traefik is just another Pod running in the cluster, and the ALB/NLB will be talking to those Pods IPs, everything is perfectly reachable.
nothing changes, other than having a private subnet IP in the CIDR range 10/16, your Pods will be in 100.64/19, while your EC2 ENIs keep the 10.x/16 IP

[tjhiggins]

@tjhiggins "carrier grade NAT" is an industry name. It has nothing to do with AWS VPC Nat Gateways. Nothing changes in that part of the infrastructure.

Karpenter is agnostic. It's up to practitioners to architect their infrastructure. AWS Best Practices do recommend clients to use extended subnets so they don't run out of IPs

I understand that Karpenter eventually wants to be agnostic, but it currently only supports aws and has documentation for creating your VPC which could be updated to use a private nat:
https://karpenter.sh/v0.18.1/getting-started/getting-started-with-terraform/#create-a-cluster

Thank you for the suggestion on a private nat and I will give that a go.

[bwagner5]

Karpenter does sort IPs for subnets that are in the same AZ. But when a request is made for capacity that doesn't constrain the AZ, we defer to EC2 Fleet to make that decision. Since Fleet is unaware of pods requiring IP addresses, it could make a decision where there's enough IPs for the node but not for pods. We could assume that pods will need IP addresses and limit subnets if they don't have enough IPs for max-pods +1 (probably through instance type offerings). This is probably safe in most cases, although doesn't make sense for cases where you are using an overlay network CNI.

If you are able to add more subnets in the same AZ, Karpenter should select the one with the most IPs available.

[James-Quigley]

You can utilize Pod Topology Spread Contraints to help evenly distribute your workloads across AZs

[igoratencompass]

We have experienced the same issue with uneven distribution of nodes in AZ, You can see here Karpenter launching 10 nodes in the same 10.171.236.x subnet in the same AZ:

$ kc get nodes -l karpenter.sh/provisioner-name=default -o wide
NAME                                           STATUS   ROLES    AGE     VERSION                INTERNAL-IP      EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-171-228-10.eu-west-1.compute.internal    Ready    <none>   7d3h    v1.22.15-eks-fb459a0   10.171.228.10    <none>        Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   containerd://1.6.6
ip-10-171-230-123.eu-west-1.compute.internal   Ready    <none>   7h22m   v1.22.15-eks-fb459a0   10.171.230.123   <none>        Amazon Linux 2   5.4.219-126.411.amzn2.x86_64   containerd://1.6.6
ip-10-171-233-121.eu-west-1.compute.internal   Ready    <none>   31m     v1.22.15-eks-fb459a0   10.171.233.121   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-233-203.eu-west-1.compute.internal   Ready    <none>   31m     v1.22.15-eks-fb459a0   10.171.233.203   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-233-205.eu-west-1.compute.internal   Ready    <none>   31m     v1.22.15-eks-fb459a0   10.171.233.205   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-233-30.eu-west-1.compute.internal    Ready    <none>   31m     v1.22.15-eks-fb459a0   10.171.233.30    <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-233-61.eu-west-1.compute.internal    Ready    <none>   4h12m   v1.22.15-eks-fb459a0   10.171.233.61    <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-109.eu-west-1.compute.internal   Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.109   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-12.eu-west-1.compute.internal    Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.12    <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-126.eu-west-1.compute.internal   Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.126   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-157.eu-west-1.compute.internal   Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.157   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-191.eu-west-1.compute.internal   Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.191   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-200.eu-west-1.compute.internal   Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.200   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-243.eu-west-1.compute.internal   Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.243   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-245.eu-west-1.compute.internal   Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.245   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-64.eu-west-1.compute.internal    Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.64    <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-236-67.eu-west-1.compute.internal    Ready    <none>   30m     v1.22.15-eks-fb459a0   10.171.236.67    <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-239-179.eu-west-1.compute.internal   Ready    <none>   32m     v1.22.15-eks-fb459a0   10.171.239.179   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-239-196.eu-west-1.compute.internal   Ready    <none>   32m     v1.22.15-eks-fb459a0   10.171.239.196   <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-239-30.eu-west-1.compute.internal    Ready    <none>   33m     v1.22.15-eks-fb459a0   10.171.239.30    <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6
ip-10-171-239-65.eu-west-1.compute.internal    Ready    <none>   32m     v1.22.15-eks-fb459a0   10.171.239.65    <none>        Amazon Linux 2   5.4.226-129.415.amzn2.x86_64   containerd://1.6.6

It seems like Karpenter decided 10 nodes are needed to allocate the pending pods, chose an AZ and I assume randomly chose a subnet from that AZ and launched all 10 of them in that one. While I agree topology spread constraint can help here I would still expect Karpenter to implement better randomization logic while choosing the subnet i.e. select 5 subnets and launch 2 nodes per each for example.

[ellistarn]

Right now, the algorithm optimizes for cost.

We've heard this feedback a fair bit. One option is to inject an implicit zonal topology rule into each pod as part of scheduling, unless of course the user has defined a different topology rule. This will yield default spread behavior across workloads, resulting in rough capacity balance.