Description:
Currently, Dask workflows are supported only on a Kubernetes backend. However, some users prefer running their jobs on HTCondor or Slurm. To enhance flexibility and cater to different computing environments, we should extend support to these backends.
Proposed Solution:
We can achieve this by using the Dask JobQueue to integrate Dask into REANA for HTCondor and Slurm backends, similar to how we integrated Dask with the Dask Kubernetes Operator for Kubernetes backend. The official documentation for Dask JobQueue provides relevant details: Dask JobQueue.
Tasks
Description:
Currently, Dask workflows are supported only on a Kubernetes backend. However, some users prefer running their jobs on HTCondor or Slurm. To enhance flexibility and cater to different computing environments, we should extend support to these backends.
Proposed Solution:
We can achieve this by using the Dask JobQueue to integrate Dask into REANA for HTCondor and Slurm backends, similar to how we integrated Dask with the Dask Kubernetes Operator for Kubernetes backend. The official documentation for Dask JobQueue provides relevant details: Dask JobQueue.
Tasks
Investigate how Dask JobQueue manages HTCondor and Slurm backends.
Implement the necessary changes to integrate HTCondor and Slurm with Dask workflows.