topic channel output breaks storeDir

[CormacKinsella]

Bug report

Expected behavior and actual behavior

Expected

• including a storeDir directive in a process will catch only file outputs. I expected it to ignore topic channels since they are collected elsewhere.

Actual

• storeDir doesn't ignore topic channels and fails to store any files:

Steps to reproduce the problem

• Run script below - no files stored, warning and error thrown
• Comment out the topic channel line - works fine

nextflow.preview.topic = true

process FOO {

	storeDir 'output'

	output:
	path "output.txt"
	tuple val(task.process), val('grep'), eval('grep --version | head -n 1'), topic: versions

	script:
	"""
	touch output.txt
	"""
}

workflow {
	FOO()
}

Program output

WARN: [FOO] StoreDir can only be used when using 'file' outputs

ERROR ~ No such file or directory: /home/cormac/sandbox/output/.command.env

Environment

• Nextflow version: 24.10.4.5934
• Java version: openjdk version "21.0.6" 2025-01-21
• Operating system: Linux
• Bash version: GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

[bentsherman]

I think the problem is not the topic channel, but the eval output. storeDir can only be used with val and path outputs.

It's a bit uglier, but you could make it work by converting the eval output to a path output:

process FOO {
    storeDir 'output'

    output:
    path "output.txt"
    tuple val(task.process), val('grep'), path('version.grep.txt'), topic: versions

    script:
    """
    touch output.txt

    grep --version | head -n 1 > version.grep.txt
    """
}

[mahesh-panchal]

There's still a bug there it looks like. If you do that, the process still runs even with the path.

nextflow.preview.topic = true

process FOO {
    storeDir 'output'

    output:
    path "output.txt"
    tuple val(task.process), val('grep'), path('version.grep.txt'), topic: versions

    script:
    """
    touch output.txt

    grep --version | head -n 1 > version.grep.txt
    """
}
workflow {
	FOO()
}

terminal:

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main2.nf

 N E X T F L O W   ~  version 24.10.4

Launching `main2.nf` [gigantic_torricelli] DSL2 - revision: aef9650a3c

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[17/de4e61] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main2.nf

 N E X T F L O W   ~  version 24.10.4

Launching `main2.nf` [tender_snyder] DSL2 - revision: aef9650a3c

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[d1/23d825] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main2.nf -resume

 N E X T F L O W   ~  version 24.10.4

Launching `main2.nf` [chaotic_bohr] DSL2 - revision: aef9650a3c

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[fd/1beef3] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

It creates a new process each time. It's not using the stored version.

It's the same outputting a val.

nextflow.preview.topic = true

process FOO {

	storeDir 'output'

	output:
	path "output.txt"
	tuple val(task.process), val('grep'), val(version_text), topic: versions

	script:
        def version = 'grep --version'.execute() | 'head -n 1'.execute()
        version.waitFor()
        version_text = version.text
	"""
	touch output.txt
	"""
}

workflow {
	FOO()
}

terminal:

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf

 N E X T F L O W   ~  version 24.10.4

Launching `main.nf` [voluminous_pare] DSL2 - revision: ecf4603cee

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[14/a3c426] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf

 N E X T F L O W   ~  version 24.10.4

Launching `main.nf` [lethal_mclean] DSL2 - revision: ecf4603cee

WARN: CHANNEL TOPICS ARE A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
executor >  local (1)
[d1/1af4e7] process > FOO [100%] 1 of 1 ✔
WARN: [FOO] StoreDir can only be used when using 'file' outputs

[CormacKinsella]

Thanks for the reply Ben.

When running the workaround above, while it does correctly produce the output, it still breaks storeDir - the process is neither stored nor cached and repeats on a rerun. Something about the tuple?

Also, in the workaround provided the version report ends up in the storeDir directory, which in the use case of storing genome indexes alongside input FASTA is not ideal, if we're also trying to produce a unified report of the software versions used by a pipeline. This would typically need to be stored elsewhere, e.g. results/package_versions

Applying a map operator to the topic channel itself only reports the path to the version.grep.txt file, which while usable is not desirable.

If storeDir could match specific output patterns, like the old publishDir option.

Edit: @mahesh-panchal beat me to it :)

[CormacKinsella]

@bentsherman - just following up on this as I ran into the issue again on another pipeline.

As Mahesh pointed out, it does seem to be a bug, and on testing it looks like the tuple causes issues.

To get it to work, I needed to move the version reporting back to the script and output the resulting file in a topic channel with just a path (no process name or tool values), i.e. path "vg_version.txt", topic: versions

So while this works in principle in the process, it's incompatible with all the existing topic channels, which are received by a map operator in the workflow expecting three elements.

We'd need to gather all versions for storeDir processes in a second topic channel, with a new map, publish, etc, and end up with two files.

As I see it the only option to avoid this right now would be to have all topic channels receive a single file containing all the relevant content from the script blocks, but since we'd then be going back to version reporting in the script, it would remove some of the advantage of topic channels for version reports in the first place.

Thanks for any feedback!

[bentsherman]

After some more testing, it looks like the issue is not with the tuple or the topic channel, but with the val outputs. The storeDir was being overly strict for some reason. In principle, all value outputs should be compatible with storeDir because they don't relying on caching.

[mahesh-panchal]

Can this issue be reopened. I feel like it hasn't been adequately resolved. Nearly all nf-core modules are now being converted to using topic channels and so are using the eval type output. If we want to use storeDir with any of these nf-core modules for example those that make indices, or databases, we get the "error" that storeDir is only compatible with path and val output types. While we can use nf-core modules patch to "revert" the module back to the old versions way, we don't really want to write out a versions.yml also in our database directory. This leaves us to patch the explicit version number as the value instead, which is then fragile as any nf-core modules update is then going to update the module, apply the patch ( emitting potentially the wrong version if the module version was updated ), and keep emitting the hard-coded version.