feat: Add feature-gated pod deletion cost management controller

[nathangeology]

Pod Deletion Cost Management Controller

RFC: #2935
Status: Ready for review

What This PR Does

Adds a new feature-gated controller (pod.deletioncost) that manages controller.kubernetes.io/pod-deletion-cost annotations on pods running on Karpenter-managed nodes. This bridges the coordination gap between the ReplicaSet controller (which decides which pods to delete during scale-down) and Karpenter's consolidation controller (which decides which nodes to drain).

How It Works

PodCount ranking with three-tier drift partitioning:

1. Partition nodes into three tiers:

   • Tier 1 (lowest cost): Drifted nodes — ConditionTypeDrifted=True
   • Tier 2 (middle): Normal nodes — not drifted, no do-not-disrupt pods
   • Tier 3 (highest cost): Do-not-disrupt nodes — has at least one karpenter.sh/do-not-disrupt pod

2. Sort each tier by pod count ascending (fewest pods = lowest deletion cost = drained first), with deterministic tiebreak by node name.

3. Assign sequential ranks starting at -n (where n = total managed nodes), so the range is [-n, -1].

4. Annotate pods on the top 50 consolidation candidate nodes with their node's rank as the pod-deletion-cost value.

This aligns ReplicaSet scale-down with Karpenter's consolidation goals: scale-down events remove pods from nodes Karpenter wants to consolidate → those nodes empty faster → Karpenter consolidates them with less disruption.

Feature Gate

--feature-gates=PodDeletionCostManagement=true

Key Design Decisions

PodCount-only strategy: Ranks nodes by total pod count, matching Karpenter's own consolidation candidate sorting. No configurable strategies — simplicity over flexibility.

Three-tier drift priority: Drifted nodes get the lowest deletion costs so RS scale-down naturally drains them first, helping both consolidation and drift progress in a single action.

Bounded labeling (top 50 nodes): Only the top 50 consolidation candidate nodes are annotated per cycle. Nodes that drop out of the top 50 have their annotations cleaned up. This bounds API server write load for large clusters.

Third-party annotation conflict detection: Tracks the last value Karpenter set on each pod. If a third-party controller modifies a Karpenter-managed annotation, Karpenter detects the change, removes its sentinel annotation, and yields management of that pod to the third party.

Customer annotation protection: Pods with an existing pod-deletion-cost annotation but without Karpenter's sentinel are never modified.

Change detection: SHA-256 hash of cluster state skips annotation updates when nothing has changed (zero API writes in steady state).

What's Included

• ranking.go — PodCount ranking with three-tier drift partitioning
• annotation.go — Safe pod annotation updates with third-party conflict detection and bounded cleanup
• changedetector.go — Hash-based optimization to skip unchanged state
• controller.go — Orchestrates ranking, bounded labeling (top 50), and orphan cleanup
• events.go — Events for ranking completion, failures, conflict detection
• metrics.go — Prometheus metrics for nodes ranked, pods updated, ranking duration
• ranking_test.go, annotation_test.go, changedetector_test.go, controller_test.go — Unit tests
• Feature gate PodDeletionCostManagement (default: false)
• RBAC: pods get update and patch verbs

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nathangeology
Once this PR has been reviewed and has the lgtm label, please assign jmdeal for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @nathangeology. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.