DaemonSets not being correctly calculated when choosing a node

[kfirsch]

Version

Karpenter Version: v0.24.0

Kubernetes Version: v1.21.0

Context

Due to the significant resource usage of certain Daemonsets, particularly when operating on larger machines, we have chosen to divide these Daemonsets based on affinity rules that use Karpenter's labels such as karpenter.k8s.aws/instance-cpu or karpenter.k8s.aws/instance-size.

Expected Behavior

When selecting a node for provisioning, Karpenter must only consider the appropriate Daemonsets that will run on that node.

Actual Behavior

It appears that Karpenter is wrongly including all of the split Daemonsets instead of only the appropriate one, which can result in poor instance selection when provisioning new nodes or inaccurate consolidation actions.

Steps to Reproduce the Problem

• Create a fresh cluster with Karpenter deployed and a default provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  providerRef:
    name: default
  requirements:
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["c6i"]
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["on-demand"]
  consolidation:
    enabled: true

• Duplicate one of your Daemonsets and split them into small/large machines using the following settings:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: karpenter.k8s.aws/instance-cpu
          operator: Lt
          values:
          - "31"

  resources:
    requests:
      cpu: 1

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values:
          - "30"

  resources:
    requests:
      cpu: 10

• Create a simple Pod with 1 CPU request, Karpenter Should provision a 2 or max 4 cpu Instance but will instead provision a large >10 cpu machine due wrongly include the bigger Daemonset in the 2\4\8 cpu evaluation.

• Same behavior when using karpenter.k8s.aws/instance-size or even podAntiAffinity rules in the Daemonset affinities.

Thank you for your help in addressing this issue.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[njtran]

Trying it out myself with a 1 CPU request and the same affinities and resource requests on the daemonsets, I was able to get a c5.4xlarge, but I saw that my logs were printing out the wrong resource requests.

karpenter-67c74b794c-rxplx controller 2023-03-23T01:38:04.125Z	INFO	controller.provisioner	launching machine with 1 pods requesting {"cpu":"12125m","pods":"5"} from types c4.4xlarge, c6i.4xlarge, m4.4xlarge, m5a.4xlarge, m6idn.4xlarge and 32 other(s)	{"commit": "17a08aa-dirty", "provisioner": "default"}

Once I removed the large DS with 10 cpu, I saw it change the resource requests.

karpenter-67c74b794c-rxplx controller 2023-03-23T01:39:12.402Z	INFO	controller.provisioner	launching machine with 1 pods requesting {"cpu":"2125m","pods":"4"} from types r6idn.2xlarge, r6a.2xlarge, m5d.xlarge, m5dn.2xlarge, c6in.xlarge and 110 other(s)	{"commit": "17a08aa-dirty", "provisioner": "default"}

This makes me think our daemonset overhead computation logic is including daemonsets that aren't compatible with the instance types that are being launched, which may be why the larger instance is being created, resulting in the behavior you see.

[njtran]

This definitely doesn't sound like the right behavior. I'll dig into this first thing tomorrow to see if this is the case.

[njtran]

Hey @kfirsch, tracked down the issue. So this is something that's not currently supported with the scheduling code. The scheduling logic calculates the resource requirements of non daemonset pods differently than daemonsets.

Karpenter optimistically includes all daemonsets that are compatible with a Provisioner's requirements during bin-packing. This means that Karpenter thinks the daemonset overhead for every instance type allowed by this Provisioner will be at least 11 vcpu, in this case, it'll think there's more overhead than there actually is for each instance type, which is why it tends to pick larger instance types.

To fix this would require a non-trivial amount of code changes to the scheduling logic, but it definitely is a bug.

In the meantime, if you're able to use multiple provisioners for each of these daemonsets to ensure that the bin-packing only considers one of them at a time, that should solve your issue.

[maxforasteiro]

I don't know if it's a similar issue but on my cluster I have a daemonset that stays in pending state due to lack of CPU in one of the nodes and Karpenter doesn't know how to deal with that. I tried to cordon the node, delete some of the pods running on the node, which makes karpenter spin up a new instance but as soon as I uncordon the old node, Karpenter removes one the nodes since it calculates as it is not needed anymore, causing the same daemonset to start being un-schedulable again.

    - key: karpenter.k8s.aws/instance-category
      operator: In
      values: ["c", "m", "r"]
    - key: karpenter.k8s.aws/instance-size
      operator: NotIn
      values: [nano, micro, small, medium, large]

this is my provider requirements.

should I open another issue or are these two problems related?

[ellistarn]

There are a number of reasons these calculations could be off. Can you take a look at the logs and see the line that describes our binpacking decision? If you have custom AMIs or custom system reserved resources that Karpenter isn't aware of, it can cause problem.

[maxforasteiro]

I see this in the logs:

2023-03-30T11:01:20.604Z INFO controller.provisioner found provisionable pod(s) {"commit": "dc3af1a", "pods": 1}
2023-03-30T11:01:20.604Z INFO controller.provisioner computed new node(s) to fit pod(s) {"commit": "dc3af1a", "nodes": 1, "pods": 1}
2023-03-30T11:01:20.604Z INFO controller.provisioner launching machine with 1 pods requesting {"cpu":"1005m","memory":"768Mi","pods":"7"} from types r6idn.16xlarge, m6id.2xlarge, c6in.xlarge, m5a.16xlarge, r6a.xlarge and 294 other(s) {"commit": "dc3af1a", "provisioner": "default-provisioner"}
2023-03-30T11:01:21.013Z DEBUG controller.provisioner.cloudprovider created launch template {"commit": "dc3af1a", "provisioner": "default-provisioner", "launch-template-name": "Karpenter-fleet-sandbox-13145882181278882901", "launch-template-id": "lt-07145405b3919a1b0"}
2023-03-30T11:01:23.863Z DEBUG controller.provisioner.cloudprovider removing offering from offerings {"commit": "dc3af1a", "provisioner": "default-provisioner", "reason": "InsufficientInstanceCapacity", "instance-type": "m2.2xlarge", "zone": "eu-west-1c", "capacity-type": "spot", "ttl": "3m0s"}
2023-03-30T11:01:24.066Z INFO controller.provisioner.cloudprovider launched new instance {"commit": "dc3af1a", "provisioner": "default-provisioner", "id": "i-02e939aee74f23a04", "hostname": "ip-yy-yy-yy-yy.eu-west-1.compute.internal", "instance-type": "c5.xlarge", "zone": "eu-west-1c", "capacity-type": "spot"}
2023-03-30T11:05:17.972Z INFO controller.deprovisioning deprovisioning via consolidation delete, terminating 1 nodes ip-xx-xx-xx-xx.eu-west-1.compute.internal/c5.xlarge/spot {"commit": "dc3af1a"}
2023-03-30T11:05:17.995Z INFO controller.termination cordoned node {"commit": "dc3af1a", "node": "ip-xx-xx-xx-xx.eu-west-1.compute.internal"}
2023-03-30T11:05:51.281Z INFO controller.termination cordoned node {"commit": "dc3af1a", "node": "ip-xx-xx-xx-xx.eu-west-1.compute.internal"}
2023-03-30T11:05:51.347Z INFO controller.termination cordoned node {"commit": "dc3af1a", "node": "ip-xx-xx-xx-xx.eu-west-1.compute.internal"}
2023-03-30T11:06:20.817Z INFO controller.termination deleted node {"commit": "dc3af1a", "node": "ip-xx-xx-xx-xx.eu-west-1.compute.internal"}
2023-03-30T11:11:47.715Z DEBUG controller.aws deleted launch template {"commit": "dc3af1a"}

When my daemonset was pending and un-schedulable, I didn't see any events being created on karpenter. These events only appeared when I cordoned and delete one of the pods running on the node, allowing the daemonset to be scheduled on the cordoned node and creating a new node for the deleted pod.
After some fiddling around I was able to make everything fit.

[maxforasteiro]

I found the answer over #731 and I agree with #731 comment.

[tzneal]

Karpenter currently calculates the applicable daemonsets at the provisioner level with label selectors/taints, etc. It does not look to see if there are requirements on the daemonsets that would exclude it from running on particular instances that the provisioner could or couldn't launch.

The workaround for now is to use multiple provisioners with taints/tolerations or label selectors to limit daemonsets to only nodes launched from specific provisioners.

[marksumm]

@tzneal I have a nodeSelector based on a custom label, which has been applied to the ebs-csi-node DaemonSet. The idea is that the CSI node pod should only run on machines that actually need dynamically provisioned EBS volumes. For some reason, these pods are not affected by this issue.

[tzneal]

Yes, it works with taints/tolerations and labels on the provisioner. It doesn't work for labels that need to be discovered from instance types that the provisioner might potentially launch.