[question] Clarification on ResourceSlice behavior discussed in google/dranet#97

[Elthis]

Hi,

While looking into DRANET’s ResourceSlice handling, I came across the discussion in the previous repository:

google/dranet#97

From that thread, my understanding is that DRANET currently publishes only devices visible in the root network namespace. When a device is allocated, the interface is moved into the pod's namespace, which effectively removes it from the ResourceSlice.

The issue also mentioned a possible alternative approach where all devices would remain listed in the ResourceSlice, with allocation state determined via ResourceClaims instead.

Since the project has moved to kubernetes-sigs/dranet and a year has passed, I wanted to check whether this behaviour is still the intended design, or if the thinking around this has evolved.

Thanks for any clarification!

/kind question

[k8s-ci-robot]

@Elthis: The label(s) kind/question cannot be applied, because the repository doesn't have them.

Details

In response to this:

Hi,

While looking into DRANET’s ResourceSlice handling, I came across the discussion in the previous repository:

google/dranet#97

From that thread, my understanding is that DRANET currently publishes only devices visible in the root network namespace. When a device is allocated, the interface is moved into the pod's namespace, which effectively removes it from the ResourceSlice.

The issue also mentioned a possible alternative approach where all devices would remain listed in the ResourceSlice, with allocation state determined via ResourceClaims instead.

Since the project has moved to kubernetes-sigs/dranet and a year has passed, I wanted to check whether this behaviour is still the intended design, or if the thinking around this has evolved.

Thanks for any clarification!

/kind question

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[gauravkghildiyal]

Thanks for bringing this up @Elthis

So we partly solved the devices getting removed from ResourceSlices as part of google/dranet#194. Essentially, if we can figure out the mapping of the network interface to a PCI device, then the ResourceSlice always contains the information that's available based on just inspecting the PCI device. When I say "partly solved", this approach still leaves a few gaps that I'm aware of:

• Only the attributes available from inspecting the PCI device remains permanent in the ResourceSlice. Attributes part of the network interface get removed when the network interface is assigned and moved to the pod's network namespace
• For devices in the ResourceSlice corresponding to a virtual network interface (think ipvlan, macvlan, veth, etc) where we commonly fail to determine the underlying PCI, these get removed altogether from the ResourceSlice upon pod assignment.

By my understanding, the limitations above may have been previously acceptable, but with more use cases popping up which require a more persistent version of ResourceSlices (e.g. I expect Pod Preemption to need this when DRA starts supporting it kubernetes/enhancements#5690, and similarly for the device taints mentioned in the earlier issue kubernetes/enhancements#5055), I think we'll want to look towards finding a good solution for these persistent ResourceSlices

(I have some thoughts on a possible approach that could leverage the local state we'll add as part of #89 to make ResourceSlices more persistent "in a good way", keeping in mind the concerns raised in google/dranet#97.)

[Elthis]

Thanks for the detailed explanation @gauravkghildiyal.

I agree that keeping all interfaces in the ResourceSlice - including those currently assigned - will likely become important given the upcoming features you mentioned.

One aspect I’m still trying to reason about is how we should track the health of a device/network interface while it is assigned to a workload.

Since the workload has control over the interface inside its namespace, it could potentially perform operations that leave the interface in an unusable state (for example deleting the interface). In principle this might be recoverable - e.g. recreating the interface on the host based on configuration stored in local state - which could allow some form of auto-healing.

Should we assume that if an interface is assigned to a workload, the expectation is that it should still be possible to recover it in a way that preserves device health once it is returned to the default network namespace?

Related to this, it might be useful to preserve as many device attributes as possible in the ResourceSlice (not just those derived from the PCI device). Otherwise, if attributes disappear while the interface is assigned, the scheduler may stop considering that device even though those attributes would reappear once the interface returns to the host namespace, making preemption a bit wonky.

Another concern I see with keeping devices in the ResourceSlice after assignment relates to the transition period when the interface is returned to the default namespace. In practice, the device may need some time to be reconfigured before it is actually usable again. During that window, the device might appear healthy/ready in the ResourceSlice even though it is not yet ready for another workload.

Previously, removing the device from the ResourceSlice implicitly avoided this situation because scheduling and PrepareResourceClaims would only occur once the device reappeared. In the future this may be addressed by Binding Conditions (kubernetes/enhancements#5007), which would allow the scheduler to reserve the resource while postponing PrepareResourceClaims until all conditions are satisfied.

In the meantime, would it make sense to verify device health during UnprepareResourceClaims, so that after release we confirm the interface is actually ready to be reused? If it is not, the device could be temporarily marked unhealthy (effectively hidden) until it becomes usable again.

Also I see that you've marked this issue as an feature, so If you have some issue title in mind that would fit better (than the current one), then feel free to change it! :)

[gauravkghildiyal]

Thanks, those are all excellent questions. We've had discussions around these topics you brought up, but haven't written down my thoughts anywhere, so collecting them here:

Broadly, I think we can condense the discussion to these two (related) questions:

1. How do we fixup a device if the user modifies it?

Current stance is that we may don't have all the necessary information to revert any and all modifications the user might have made. For the devices so far which DRANET manages (netdevice, rdma device), DRANET is not the owner/creator of these devices on the host. Current expectation is that whoever created the device on the host can reconfigure them back once a pod releases them from the pod's network namespace.

2. How do we determine the device is healthy?

This again is related to the first question and notoriously difficult to get right. The "true" measure of figuring out the device being healthy would be to actually try to use the device for whatever purpose it is meant. But that of course isn't feasible here. So the next best thing would be to try to check for criterion like, "is the interface link UP?", "does the interface have an IP?", and so on. I suppose there can be some minimal set of things that we can come up with to decide that the device is "healthy enough". Beyond that, users may have some very specific criteria of their own and the DRANET config API will need to allow that expression. This is where we'll need some more data and use cases to come up with the right API.

I do recognize this as an issue and ideally we don't want some end-user to complain about this until we solve this. This is just an initial thought, but to be able to offer some mitigation for a user who does run into this edge case, perhaps we can start with the simplest thing of allowing the user to set some wait time for the device to be healthy and reconfigured before making it available to a newer Pod. That can be combined together with http://kep.k8s.io/5055 (tainting the device) to make it unschedulable and then undoing the taint once the wait time is over. (Yes I believe http://kep.k8s.io/5007 is another alternative that can be utilized, although they work at different stages of the DRA lifecycle)

---

Independent of these two more long term plans, yes I'm on the same page with the idea of atleast having more stable configuration in ResourceSlices (for even the devices which have been already allocated). This requirement has shown up for multiple use cases now (autoscaling, device tainting, preemption) so we'll try to get that resolved sooner.