Graceful shutdown an Origin or Cache

[h2zh]

Trigger the graceful shutdown once a Pelican Origin or Cache receives a SIGTERM signal (kill -TERM <pid>)

Upon receiving SIGTERM, the Origin or Cache process waits one minute for in-flight transfers to complete. If any transfers remain after the one-minute grace period, the server shuts down immediately.

Before the wait, the Origin or Cache sends a shutdown advertisement with the new Status field (="shutting down") to the Director, instructing it to stop routing any new transfer requests to this server. The Director then applies an indefinite downtime window to this server, so that offline servers still listed in its cached serverAds will be excluded from new requests. Whenever the server is back online, it will send a regular server ad to flush out the "shutting down" downtime.

cc @bbockelm Here is a formal up-to-date system design diagram. (Content in this PR is marked in red)

[h2zh]

Hey @jhiemstrawisc, could you review this before/after (whenever you think it's the good time) you work on status field? You mentioned you have plan to integrate shuttingdown, degraded to the existing healthStatus in the serverAds, so I think I might need to change this PR a lot to incorporate that.

[brianaydemir]

If any transfers remain after the one-minute grace period, the server shuts down immediately.

@jhiemstrawisc @h2zh Is this grace period configurable? Do we have an idea how long "immediately" actually takes?

I'm thinking ahead to origins and caches running on a Kubernetes cluster, and what it might take to configure them to shutdown gracefully.

[brianaydemir]

The Director then applies a short-term downtime window (=Director_AdvertisementTTL+Shutdown_Grace_period=15+1=16m by default) to this server, so that offline servers still listed in its cached serverAds are excluded from new requests.

So, imagine the case where an admin is performing a routine upgrade of a cache or origin. We gracefully shutdown the service, and very soon after start it back up. If all goes well, this only takes a few minutes.

In this scenario, what's the mechanism for removing the downtime? (Do we need the downtime in the first place?)

[h2zh]

Hey @brianaydemir, thanks for your suggestions!

1. I updated the code to make the shutdown timeout configurable. "Immediately" here is its literal meaning (definitely less than 1s).
2. I modified the logics in the code. Director applies an indefinite downtime window to the server when it receives the shutdown signal. Origin/Cache server would lost the StatusShuttingDown in its in-memory healthStatus once its process gets killled. Whenever the server is back online, it will send a regular server ad to flush out the old "shutting down" downtime.

I also updated the write-up and diagram on the top to reflect the latest workflow in the code.

[jhiemstrawisc]

A few comments/questions/suggestions. Otherwise I tested this locally and observed both the graceful shutdown and that restarting the Origin cleans out the old filter. Nice job!

[jhiemstrawisc]

One other note @h2zh -- I love that you often provide system diagrams with your PRs. You set a good example for the rest of us. Do you also store these in our internal documentation in the Pelican drive? If not, can you find a reasonable spot for them and put them there so we have some organized internal references to them?

[jhiemstrawisc]

Getting super close! I'm going to pre-approve so you can merge once the last grammar change is in. (I did not review the react changes)

Nice work with this one, I hope it's really helpful for our "0.1%" goal!

[h2zh]

One other note @h2zh -- I love that you often provide system diagrams with your PRs. You set a good example for the rest of us. Do you also store these in our internal documentation in the Pelican drive? If not, can you find a reasonable spot for them and put them there so we have some organized internal references to them?

@jhiemstrawisc Thanks! I really try to make my work neat, structured, and easy for everyone to understand, especially since I recently learned about the idea of "translational computing".

You bring up a good point about storing these diagrams in our internal documentation. I've been relying on GitHub PRs for feature write-ups because it offers some advantages:

• Single Source of Truth: I only have to maintain one version, keeping the PR write-up up-to-date and linking all relevant tickets and other PRs directly within it.
• Contextual Timeline: It's easier for anyone viewing the PR to grasp the entire timeline and how the feature evolved alongside the code changes.

However, I agree that having a central reference point in our Google Drive would be beneficial. What if we create a "symlink" (a file containing a link) in the "Design Docs" folder? I would name the file after the feature, and its content would simply be the direct link to the GitHub PR. This way, we get the best of both: the PR remains the living document, and our drive serves as an organized index.

What do you think of that approach?