Graceful shutdown a Cache

[h2zh]

Pelican Service:

• Client
• Plugin
• Registry
• Director
• Origin
• Cache
• Other (please give the detail)

Is your feature request related to a problem? Please describe.
When a Cache server shuts down, all the ongoing file transfers from this Cache also fail, which contributes a large portion of job failures in OSDF. To ameliorate this problem, we want to put this Cache in downtime once it is about to shutdown, so it can finish the remaining transfer jobs while the Director will not give it new jobs.

Describe the solution you'd have

This solution is outdated and removed. Go to #2318 to view the latest.

@bbockelm Could we find a time to talk about this plan?

[bbockelm]

I would like to have a more "status" based approach.

The server ads should contain a status indicating whether they want to service requests. This should be separate from the downtime system as we may have several reasons why we want to have less load.

We can use the Prometheus metrics to determine the count of in-progress transfers. This will likely be heuristic-based and we should have a max drain time: there are some clients that hard code the name of caches and don't use the director.