When and if not identical in behaviour

[apeltzer]

Bug report

I have a process called bwa_indexhere:

https://github.com/nf-core/eager/blob/master/main.nf#L379-L401

that I'd like to be executed whenever the when directive is true.
This worked fine, but I'm also filling the same channel when a user specifies input using a simple Channel.fromPath()....

Expected behavior and actual behavior

If I use the when directive, I get an error message:

[warm up] executor > local
ERROR ~ Channel `ch_bwa_index` has been used twice as an output by process `makeBWAIndex` and another operator

 -- Check script 'main.nf' at line: 375 or see '.nextflow.log' file for more details

When I replace the when with an if clause around the respective process, I get no error and everything works fine.

I do like to use when where possible but had also issues when mixing channels with the multiple index files being consumed once and then not being able to process more than just one sample.

I'm not sure what this behaviour means / what I can do about it or whether this is intended.

Environment

• Nextflow version: latest stable (18.10.1)
• Java version: 1.8
• Operating system: OSX

Additional context

Working on this Pull Request atm: nf-core/eager#127

Problematic are calls that use the respective variable ch_bwa_index.

I create the references using this command for example (which simulates an entire run of the pipeline and is used by us to run the entire pipeline for testing purposes):

nextflow run . -profile test,docker --pairedEnd --saveReference

Now I can reproduce using the previously generated reference genome indices:

nextflow run . -profile test,docker --singleEnd --bwa_index results/reference_genome/bwa_index/

Ideally, I'd expect being able to use when and not having to take care that the ch_bwa_index isn't reached at all ...

[pditommaso]

Can you upload the .nextflow.log file including that error.

[apeltzer]

Ah sorry, just saw that your responded ... Will do now 👍

[Leon-Bichmann]

Hey, we just experienced the same problem, using the mhcquant workflow.

[apeltzer]

Jan-24 10:21:26.684 [main] DEBUG nextflow.cli.Launcher - $> nextflow run nf-core/mhcquant --mzmls '/private/path/untreated/*.mzML' -profile cfc --alleles /private/path/HROC113_62495347_S27_alleles.tsv --vcf /private/path/Strelka_Mutect2_hroc113_tumor_vs_hroc113_normal_somatic_snvs_passed_annotated.vcf --include_proteins_from_vcf --run_prediction
Jan-24 10:21:26.966 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 19.01.0
Jan-24 10:21:29.032 [main] DEBUG nextflow.scm.AssetManager - Git config: /home-link/iismo01/.nextflow/assets/nf-core/mhcquant/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/mhcquant.git
Jan-24 10:21:29.065 [main] DEBUG nextflow.scm.AssetManager - Git config: /home-link/iismo01/.nextflow/assets/nf-core/mhcquant/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/mhcquant.git
Jan-24 10:21:29.479 [main] DEBUG nextflow.scm.AssetManager - Git config: /home-link/iismo01/.nextflow/assets/nf-core/mhcquant/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/mhcquant.git
Jan-24 10:21:29.480 [main] INFO  nextflow.cli.CmdRun - Launching `nf-core/mhcquant` [goofy_edison] - revision: 8f2b0ae8e0 [master]
Jan-24 10:21:31.964 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home-link/iismo01/.nextflow/assets/nf-core/mhcquant/nextflow.config
Jan-24 10:21:31.969 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home-link/iismo01/.nextflow/assets/nf-core/mhcquant/nextflow.config
Jan-24 10:21:32.003 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `cfc`
Jan-24 10:21:32.752 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, cfc, uzh, debug, test, singularity, awsbatch, none, docker, binac]
Jan-24 10:21:32.892 [main] DEBUG nextflow.Session - Session uuid: 7c366a04-d6df-499a-b72e-e15827d9e818
Jan-24 10:21:32.894 [main] DEBUG nextflow.Session - Run name: goofy_edison
Jan-24 10:21:32.902 [main] DEBUG nextflow.Session - Executor pool size: 4
Jan-24 10:21:33.006 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 19.01.0 build 5050
  Modified: 22-01-2019 11:19 UTC (12:19 CEST)
  System: Linux 3.10.0-514.26.2.el7.x86_64
  Runtime: Groovy 2.5.5 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13
  Encoding: UTF-8 (UTF-8)
  Process: 13110@ilogin02 [172.20.4.2]
  CPUs: 4 - Mem: 7.6 GB (5.5 GB) - Swap: 8 GB (7.9 GB)
Jan-24 10:21:33.063 [main] DEBUG nextflow.Session - Work-dir: /private/path/work [fhgfs]
Jan-24 10:21:33.867 [main] DEBUG nextflow.Session - Session start invoked
Jan-24 10:21:33.876 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
Jan-24 10:21:33.878 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /private/path/results/pipeline_info/nf-core/mhcquant_trace.txt
Jan-24 10:21:33.896 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing
Jan-24 10:21:34.422 [main] DEBUG nextflow.Session - Session aborted -- Cause: startup failed:
_nf_script_69ff676d: 681: Process `filter_by_q_value` is already define @ line 681, column 9.
   process filter_by_q_value {
           ^

1 error

Jan-24 10:21:34.535 [main] ERROR nextflow.cli.Launcher - startup failed:
_nf_script_69ff676d: 681: Process `filter_by_q_value` is already define @ line 681, column 9.
   process filter_by_q_value {
           ^

1 error

However, these processes should be explicitly just run exclusively, so not interfere with each other:

https://github.com/nf-core/mhcquant/blob/8f2b0ae8e055a0b2a84ea16cb442196c47566d4c/main.nf#L653-L703

I'll probably have to dig out the old eager issue, which basically said the same thing:

if( true) 
  process bla {}
else { 
  process blub {}
}

When I try running these exclusively just using when I always end up having to mix the output channels for example (which is fine). However, I'd expect to be able to use the same channels from both processes, e.g. by using a simple

bwa_index = Channel.empty()

(and then fill either with bla or blub the bwa_index channel, since I know that these will never be interfering with other as they exclude each other using a when statement.

With an if around, it works fine - if I use the same condition using when they're not able to do that... "same channel used multiple times"

[pditommaso]

You cannot have two processes with the same name. In the if it works because only one definition takes place.

[apeltzer]

Thanks a lot Paolo!

Ok, and that also applies if I use the same channel? It's not applied when using an if statement, but whenever I use a when in a process, they conflict with each other?

Something like this:

process bla { 

when: params.fancy

output: 
file bla into ch_index
} 

process blub { 
when: !params.fancy

output:
file bla into ch_index
}

I guess its the same thing here ...?

The only problem with this is, that mixing requires both channels that need to be mixed to be defined already in both cases, e.g. I have to set up at least a channel_x = Channel.empty() respectively to mix these channels. Is there another way I missed?

[pditommaso]

that also applies if I use the same channel?

The channels do have effect it. Each declared process is expected to have its own name.

I would also add that using two mutually exclusive processes with the when condition as you are doing makes more sense the input structure is different. When having the same inputs and outputs as you are doing it could be easily refactor as a unique process with a conditional command script.

Does it make sense?

[apeltzer]

I think it makes sense to refactor it yes, didn't think about it earlier these days before but will try to achieve that and report back 👍 THanks again for your input :-)

[pditommaso]

Nice! therefore I close this.