SystemD: stop restarting Pelican if it exits too frequently

[matyasselmeci]

Currently (7.16.1), the SystemD .service files will restart Pelican after 20 seconds if it exited (without an explicit systemctl stop from the admin). We should disable that if Pelican is in a crash-loop, otherwise the logs will just get filled up with crash messages.

[matyasselmeci]

Unfortunately, the StartLimit* in SystemD do not do what we want for this use case (since they apply to manual restarts, not just automatic, and it's not obvious to the user how to clear them (systemctl daemon-reload)). I'll have to think of something else. Maybe I can mitigate the problem slightly by bumping the restart delay to 60 seconds. systemd 254 and new have separate RestartSec and RestartMaxDelaySec (so they can increase the delay between restarts), but that version of systemd is only available on Alma 10.

[matyasselmeci]

Some things to experiment with:

1. We can add a script as an ExecStopPost, which will have the $SERVICE_RESULT, $EXIT_CODE, and $EXIT_STATUS vars in the environment, which should let us distinguish between manual restart and auto-restart, and implement a delay backoff ourselves.

2. Perhaps we can have Pelican exit with a different code for 'your config is probably broken, take a look before you start again' -- then we can use RestartPreventExitStatus