Version

Karpenter Version: v0.24.0

Kubernetes Version: v1.21.0

Context

Due to the significant resource usage of certain Daemonsets, particularly when operating on larger machines, we have chosen to divide these Daemonsets based on affinity rules that use Karpenter's labels such as karpenter.k8s.aws/instance-cpu or karpenter.k8s.aws/instance-size.

Expected Behavior

When selecting a node for provisioning, Karpenter must only consider the appropriate Daemonsets that will run on that node.

Actual Behavior

It appears that Karpenter is wrongly including all of the split Daemonsets instead of only the appropriate one, which can result in poor instance selection when provisioning new nodes or inaccurate consolidation actions.

Steps to Reproduce the Problem

• Create a fresh cluster with Karpenter deployed and a default provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  providerRef:
    name: default
  requirements:
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["c6i"]
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["on-demand"]
  consolidation:
    enabled: true

• Duplicate one of your Daemonsets and split them into small/large machines using the following settings:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: karpenter.k8s.aws/instance-cpu
          operator: Lt
          values:
          - "31"

  resources:
    requests:
      cpu: 1

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values:
          - "30"

  resources:
    requests:
      cpu: 10

• Create a simple Pod with 1 CPU request, Karpenter Should provision a 2 or max 4 cpu Instance but will instead provision a large >10 cpu machine due wrongly include the bigger Daemonset in the 2\4\8 cpu evaluation.

• Same behavior when using karpenter.k8s.aws/instance-size or even podAntiAffinity rules in the Daemonset affinities.

Thank you for your help in addressing this issue.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment