'versions' directive in process

[adamrtalbot]

New feature

Many pipelines attempt to catch the software versions running within the container. The most common method is to catch a file with the software version written to it, typically using some fun string parsing:

https://github.com/nf-core/modules/blob/72f633b541815ae9565599e5b3cf66d863736ae1/modules/nf-core/fastqc/main.nf#L38-L41

https://github.com/epi2me-labs/wf-template/blob/3edea44cdf9449093a2adc2f08bb16ec143373bf/main.nf#L20-L30

This populates the output directive with an additional channel and prevents people using the pipe operator amongst other things.

Why not get this into a separate, native directive? It could automatically populate some property of the process which is available to use as a string (or map of strings).

Usage scenario

Anyone who wants to record their software at runtime.

Suggest implementation

process FASTQC {
    // process goes here
    versions:
    fastqc = 'fastqc --version | sed -e "s/FastQC v//g"'
    python = 'python --version'
}

Somewhere within the .command.run the .task.versions file is created:

fastqc: 0.12.1
python: 3.11.2

Back in the pipeline code:

FASTQC.versions == [ name: ${task.process}, versions: [ fastqc: "0.12.1", python: "3.11.2" ]  ]

// It's syntactic sugar around the FASTQC.out.versions channel, so can be used like so:
ch_versions = ch_versions.mix(FASTQC.versions)

[ewels]

I think that this is basically a duplicate of an issue that I wrote with @pditommaso back in 2018 😬 #879

But same motivation, similar syntax even (this is slightly better though, agree that the ability to have more than 1 version per process is necessary 👍🏻 )

[adamrtalbot]

Ah bother. I did search for versions but I see that feature is slightly more generic. You could probably use my this feature to implement generic info as well as versions.

The biggest flaw I can see right now is stealing a secret. From the example above:

versions:
sneaky-secret-steal = "echo $SECRET"

[pditommaso]

I like the general design, still find it horrible having to run and parse all these CLI option. would not be better to rely on the version declared by conda and spack packages?

[adamrtalbot]

In my personal experience, conda only contains about 25% of tools used in bioinformatics (although they are the big and important ones).

[ewels]

Yup, we've gone back and forth on this a lot over the years. At the end of the day, the only thing you can really trust is to ask the tool itself. Also many pipelines (eg. nf-core) specify both conda and docker. Both can be overwritten by the end user. So if running with some random Docker image there's really no way to know what the version is.

[pditommaso]

Fair enough. Other points address 1) cardinality: should this be collected once per process or once per task execution; 2) how do you expect this information to be consumed?

[adamrtalbot]

Yes, once per task. It's easy enough to reduce the data to per-process later. Likely is to consume it for an output file (collectFile), or wrap into an output tool like nf-prov.

[ewels]

My hope is to use the new MultiQC software versions YAML files for reporting. One per task is fine, MultiQC will handle collapsing the duplicates automatically.

And yeah, would also be nice to go into nf-prov / other tooling 👍🏻

[jordeu]

I've opened a PR #4425 with an alternative and more generic way of approaching this problem. Give it a try or add your comments about that alternative.

[jordeu]

Explaining a bit what I don't like about adding a "versions" directive and why I proposed this "custom traces" #4425 approach.

To add a directive for collecting the versions of the tools at runtime seems a bit like an odd feature. It suggests that something might be failing in the software management system; otherwise, all the versions should be fixed and known beforehand without any doubt.

However, I understand that reality can be different. Therefore, what we need is a directive to collect runtime metadata of the executed processes. This is why I decided to give the "customTraces" directive approach a try. To me, it also makes no sense to inject this metadata into a channel because it would mean mixing logics. This metadata is not necessary for the execution of the actual data flows. In other words, it should be possible to completely remove this metadata collection, and the pipeline should still function as intended (similar to how trace metrics are optionally collected).