Recipe for usage with an HPC job scheduling environment

[Andrew-S-Rosen]

💻

• [] Would you like to work on this feature?

Yes, but as a tester. I don't have enough knowledge of dask-jobqueue and Prefect to write it from scratch.

What problem are you trying to solve?

With prefect-dask it is possible to use Prefect with an HPC environment that uses a job scheduler like Slurm. However, there is effectively no documentation on how to use Prefect with these kinds of compute environments. It would be great to have some recipes to describe this scenario.

Describe the solution you'd like

An example of how to have an agent set up on an HPC environment that will continually poll for work and submit tasks as Slurm compute jobs when they come in.

Describe alternatives you've considered

Not using Prefect

Documentation, Adoption, Migration Strategy

No response