Reasonable defaults with eviction and PodDisruptionBudget

[foxish]

We are moving to have kubectl drain use the eviction subresource (#34277), and honor PodDisruptionBudget.
From comment by @erictune which better captures this requirement.

There should be a default behavior for disruption budgets.
The default should optimize for these things:

1. Users should not need to create, view, or even know about with the PodDisruptionBudget resource, and still get "reasonable" default behaviors.
2. The default behavior should be reasonable for a wide variety of app use cases.
3. The default behavior could be expressed succinctly to users.
4. Cluster admins should be able to upgrade nodes quickly in clusters that have typical apps with default budgets, without interacting with app owners.

Some options:

1. All pods in the cluster that match no user-created disruption budget have a budget of 1 at a time.

• could be too conservative to allow for parallel node drains.

2. All pods in a namespace cluster that match no user-created disruption budget have a budget of 1 at a time.

• less conservative than 1
• probably not too conservative for systems with many namespaces relative to number of nodes.

3. Each set of pods with the same controller and that match no user-created disruption budget, have a disruption budget of 1 at a time.

• assuming spreading of collections by scheduler, parallel node drains likely to be possible
• the disruption controller already computes this grouping of pods, sort of.

4. Each set of pods with the same service, that match no user-created disruption budget, have a disruption budget of 1 at a time.

• matches a default spreading behavior by the scheduler, so unlikely to have two on the same machine from same service, so parallel node drains likely to be possible.

The actual implementation could be to have a controller that makes PDBs for things without them and then cleans them up when not needed anymore, or for the eviction logic to just compute these sets, but not expose them.

cc @erictune @davidopp @kow3ns @janetkuo @caesarxuchao @ymqytw @kubernetes/sig-apps

[davidopp]

I'm on the fence on this one. Would be interested to hear other folks' opinions. Here are my thoughts:

It seems an alternative to your proposal, which would have the same effect, would be to just add an admission controller that adds a PDB to every collection that you think should have the rule you want (e.g. max 1 eviction at a time). The downside of this would be a proliferation of PDBs.

If we implement your proposal, clients that want to get the behavior they get today in the absence of a PDB, instead of the new behavior you're introducing, could just use regular delete. (Maybe their logic could be: try eviction subresource first, if it is rejected, then just do regular delete. That way they wouldn't need to look up whether the pod is covered by a PDB first.) So it wouldn't be a big inconvenience.

The one case I am worried about with your proposal is pods that are managed by a controller with replicas=1. Such a pod would block removing the node, even in the absence of a PDB. Maybe that is the right thing to do, but it would be better if we knew the user's intent, and the only way to know that would be to require them to put a PDB on it if they really don't want to allow such a node to be removed.

I guess I am leaning slightly towards endorsing your suggestion but there may be other issues I didn't think of.

cc/ @mml

[smarterclayton]

What class of things run as rc=1, ps=1?

Important:
Databases
Leader services
The B of an A/B test

---

Not important:
Testing instances of production services
Random stuff you start under an rs (so it will survive node death) but for which you forget about.

In large clusters the latter are going to dwarf the former (many OpenShift clusters today are 95% the latter). If this change added the PR above, we'd probably revert it because it would break all of those very large, dense clusters (eviction would be impossible). On small clusters, it's much more likely you have the former - in which case this might be a reasonable default. But if you have a database or leader service and it doesn't have a PDB, you're just not using the system like you should.

The only downside today is that eviction can be surprising (if you are relying on emptydir), but you can't prevent that even with PDB, so I don't think it's unreasonable to ask users to use PDB to avoid disruption.

[smarterclayton]

A global config flag to opt in to this makes sense, but I don't think that rs=1 or ps=1 implies disruption budget

[davidopp]

(Maybe their logic could be: try eviction subresource first, if it is rejected, then just do regular delete. That way they wouldn't need to look up whether the pod is covered by a PDB first.) So it wouldn't be a big inconvenience.

Sorry, what I said here was wrong. If we implement your proposal, clients that want to always delete in the absence of a PDB but respect the PDB if it exists (which is the behavior they get today) would have to look up whether the pod is covered by a PDB first, and use that to decide where to use /eviction or regular delete.

A global config flag to opt in to this makes sense

I guess a config flag to choose the behavior would be one approach, though I always worry about the support implications of making core behaviors configurable...

[erictune]

There should be a default behavior for disruption budgets.

The default should optimize for these things:

1. Users should not need to create, view, or even know about with the PodDisruptionBudget resource, and still get "reasonable" default behaviors.
2. The default behavior should be reasonable for a wide variety of app use cases.
3. The default behavior could be expressed succinctly to users.
4. Cluster admins should be able to upgrade nodes quickly in clusters that have typical apps with default budgets, without interacting with app owners.

Some options:

1. All pods in the cluster that match no user-created disruption budget have a budget of 1 at a time.
   • could be too conservative to allow for parallel node drains.
2. All pods in a namespace cluster that match no user-created disruption budget have a budget of 1 at a time.
   • less conservative than 1
   • probably not too conservative for systems with many namespaces relative to number of nodes.
3. Each set of pods with the same controller and that match no user-created disruption budget, have a disruption budget of 1 at a time.
   • assuming spreading of collections by scheduler, parallel node drains likely to be possible
   • the disruption controller already computes this grouping of pods, sort of.
4. Each set of pods with the same service, that match no user-created disruption budget, have a disruption budget of 1 at a time.
   • matches a default spreading behavior by the scheduler, so unlikely to have two on the same machine from same service, so parallel node drains likely to be possible.

The actual implementation could be to have a controller that makes PDBs for things without them and then cleans them up when not needed anymore, or for the eviction logic to just compute these sets, but not expose them.

[davidopp]

I would add one more criteria: the intended behavior of the system should be as simple to understand as possible. In this vein, I think saying "absence of PDB means unlimited evictions" is less surprising than "absence of PDB is like presence of a PDB tied to " Thus we should make the behavior explicit by requiring a PDB whenever you want to limit evictions--at least from the standpoint of this criteria, which I think is an important one for debugability and for administrators to understand what's going on.

As for the options, it seems like (3) and (4) are clearly better than (1) and (2), or maybe I am missing something? I think maybe the right way to think about this feature in general is that one of the characteristics of a controller should be that it has a default policy for disruptions. Most controllers probably won't do anything; the default is "eviction is always allowed." Some, like PetSet, might want a policy (e.g. at most one down from the set at a time), and they can use an admission controller to apply it. The main issue I can see with this idea of requiring the default policies to be materialized, is that if the user wants their own policy, then they would presumably have to delete the PDB that was created by the admission controller before adding their own, which is annoying. The benefit of building a default policy into the PDB controller itself is that the user-supplied PDB can just override the default one that's built into the PDB, and the user doesn't have to delete anything to apply their own. But if you build the default policy into the PDB controller, you can't easily customize it on a per-kind basis (creator of a new kind would have to modify the PDB controller when they build a custom controller...ugh).

So both options (absence of PDB means unlimited evictions, and absence of PDB means some default policy is applied) seem to have downsides.

But because I think the default policy should probably depend on the type of controller, I think (3) is better than (4).

[davidopp]

A couple of other reasons why (3) seems better than (4)

• user may create service after creating controller, so there will be a period where the collection would be unprotected if you tie the PDB to the service (of course if the user wants to do it manually, that's fine, they are involved in the loop)
• a pod may be in more than one service, but never have more than one controller; today PDB controller can't handle a pod being in more than one PDB-protected collection (and I'm not sure we want to ever allow it; the rules get dicey from a usability standpoint... it's basically an "and" of whether you're allowed by each PDB you're in, but I think this is a recipe for confusion, and race conditions debiting multiple PDBs for one eviction due to no transactions, etc.).