Snellius mag by vdejager · Pull Request #867 · nf-core/configs

vdejager · 2025-03-18T08:51:28Z

name: New Config for Snellius HPC (Dutch National Supercomputer faclility
about: A new cluster config for both Snellius in general and one config for the 'mag' pipeline

Please follow these steps before submitting your PR:

If your PR is a work in progress, include [WIP] in its title
Your PR targets the master branch
You've included links to relevant issues, if any

Steps for adding a new config profile:

Add your custom config file to the conf/ directory
Add your documentation file to the docs/ directory
Add your custom profile to the nfcore_custom.config file in the top-level directory
Add your custom profile to the README.md file in the top-level directory
Add your profile name to the profile: scope in .github/workflows/main.yml
OPTIONAL: Add your custom profile path and GitHub user name to .github/CODEOWNERS (**/<custom-profile>** @<github-username>)

added ng-core config specific for pipeline 'mag'

pontus · 2025-03-18T08:58:21Z

+    time           = params.max_time
+    // job names need to be unique a
+    jobName = {
+                //n = "${task.name}" -> n.replaceAll('[\\\s:()]', '_')


Suggested change

//n = "${task.name}" -> n.replaceAll('[\\\s:()]', '_')

Seems like trial and error? (I'm not sure why you can't just do "${task.name}".replaceAll('[^a-zA-Z0-9]', '_').

I'll see if that works. The above was suggested by the snellius admins

pontus · 2025-03-18T09:04:04Z

+#SBATCH --error=/projects/0/<your project space>/jobs/error_%j.out
+#-----------------------------Required resources-----------------------
+# this defines where the nextflow process runs (as separate job)
+# 16 cores is the minimum partition with 28 GB memory


28 Gbyte for the nextflow monitoring process is really a lot. I haven't run mag but would be surprised if anywhere near that is needed.

no, its not needed perse, but this is the smallest job one can specify on the cluster

Ok, I read the comment as the smallest that gets 28 Gbyte of RAM. Maybe clarify that's the minimum job size?

But that is still confusing since there are smaller requests used, do they still work and are silently scaled up? Since the expected task attempt handling of throwing more resources won't work as expected if they still get the same, maybe they should be adjusted then?

pontus · 2025-03-18T09:04:47Z

+        errorStrategy  = { task.exitStatus in [1, 143, 137, 104, 134, 139, 140] ? 'retry' : 'finish' }
+        cpus   = { 16 * task.attempt }
+        memory = { 28.GB * task.attempt }
+        time   = 5.d


Maybe remove time when it's not changing the default?

Maybe. In between I've been pointed at the scheduler strategy, so maybe I should make the default short. The mag pipeline typically runs longer per task (and 5 days is sometimes too short, so restarting the pipeline is needed)

In general, setting anything other than the maximum allowed doesn't make sense - the occasional earlier scheduling by backfill doesn't make up for what's wasted by jobs running fine but with too short time.

pontus · 2025-03-18T09:09:54Z

+    withName: MAG_DEPTHS {
+        cpus   = { 16 * task.attempt }
+        memory = { 28.GB * task.attempt }
+        time   = { 24.h * (2 ** (task.attempt - 1)) }


I naturally don't know anything about your system, but the two main reasons to play with time is either fit in what's allowed for jobs to a specific partition or possibly to get squeezed in by the backfill (and in some very specific cases not being stopped because the job as submitted would be more than the total allocation).

Are you sure these give any benefit and just not make things needlessly complicated (with a not-insignificant risk of well functioning jobs timing out when they wouldn't need to)?

pontus · 2025-03-18T09:11:02Z

Using https://www.nextflow.io/docs/latest/reference/process.html#resourcelimits is highly recommended.

jfy133 · 2025-03-18T09:29:13Z

@nf-core-bot fix linting

jfy133 · 2025-03-19T03:58:33Z

pre-commit linnitng error:

docs/pipeline/mag/snellius.md:
	69: Wrong amount of left-padding spaces(want multiple of 2)
	70: Wrong amount of left-padding spaces(want multiple of 2)
	71: Wrong amount of left-padding spaces(want multiple of 2)
	72: Wrong amount of left-padding spaces(want multiple of 2)
	73: Wrong amount of left-padding spaces(want multiple of 2)
	74: Wrong amount of left-padding spaces(want multiple of 2)
	75: Wrong amount of left-padding spaces(want multiple of 2)
	76: Wrong amount of left-padding spaces(want multiple of 2)
	```

jfy133 · 2025-03-19T04:01:24Z

And tests aren't working because you haven't got all files for both the global and pipeline specific configs:

The specific error is:

Tests don't seem to test these profiles properly. Please check whether you added the profile to the Github Actions testing YAML.

{'snellius'}

maxulysse · 2026-04-07T11:06:30Z

@vdejager any update?

vdejager · 2026-04-07T11:17:32Z

In fact yes, I started working on it again after a year of other priorities. I still need to figure out how to add the tests

maxulysse · 2026-04-07T11:20:57Z

In fact yes, I started working on it again after a year of other priorities. I still need to figure out how to add the tests

Due to Nextflow strict syntax coming up, we are refactoring how we test.
So for now, just add your profile name to the list of profiles in .github/workflows/main.yml (alphabetical order please).

Victor de Jager added 3 commits March 18, 2025 09:34

added nf-core config for 'Snellius'

00e18b7

added ng-core config specific for pipeline 'mag'

added snellius profile to nfcore_custom.config

082b27f

added todo list for future enhancements

2aef58d

pontus reviewed Mar 18, 2025

View reviewed changes

[automated] Fix code linting

b351d60

Merge branch 'nf-core:master' into snellius_mag

6544ff1

Victor de Jager and others added 2 commits April 7, 2026 14:31

adapted snellius minimal CPU count for genoa and rome partitions

1bd34a7

Merge branch 'nf-core:master' into snellius_mag

550726b

Conversation

vdejager commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

name: New Config for Snellius HPC (Dutch National Supercomputer faclility about: A new cluster config for both Snellius in general and one config for the 'mag' pipeline

Uh oh!

pontus Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pontus commented Mar 18, 2025

Uh oh!

jfy133 commented Mar 18, 2025

Uh oh!

jfy133 commented Mar 19, 2025

Uh oh!

jfy133 commented Mar 19, 2025

Uh oh!

maxulysse commented Apr 7, 2026

Uh oh!

vdejager commented Apr 7, 2026

Uh oh!

maxulysse commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vdejager commented Mar 18, 2025 •

edited

Loading

name: New Config for Snellius HPC (Dutch National Supercomputer faclility
about: A new cluster config for both Snellius in general and one config for the 'mag' pipeline

pontus Mar 18, 2025 •

edited

Loading