Multiple samtools '.stats' files lead to MultiQC crash

[jfy133]

Describe the bug
This is related to current -r dev

Some of the prefix creation procedures removes anything after the first '.', however sample names can sometimes have '.' (all MPI-SHH samples have this). Thus, in this case samples that have the same IDs before the first full-stop, the resulting files are printed with the same name and causes conflicts.

For example:

ABM006.A0101_S0_L001_R1_000.fq.gz
ABM006.A0102_S0_L001_R1_000.fq.gz

would be converted to

ABM006.stats
ABM006.stats

and the conflict occurs.

The error begins on lines 695/704 of main.nf in in the bwa module, but is also used in the FastQC module (662 - but $prefix isn't used); bwa_mem module 764; circularmapper 764/774.

The prefix issue may also be related to #149 (although the error line in that issue has been removed from the bwa module, it still remains in bwa_mem).

Mar-16 16:00:56.579 [Actor Thread 80] DEBUG nextflow.processor.TaskProcessor - Process multiqc > collision check staing file names: [fastp/ABM006.A0101_S0_L001_R1_001.fastq_fastp.json:1, qualimap/qualimap4/ABM006.sorted.bam.filtered_stats:1, qualimap/qualimap3/ABM006.sorted.bam.filtered_stats:1, fastqc_raw/DJA002.A0101_S0_L001_R1_001_fastqc.html:1, fastqc_raw/DJA002.A0101_S0_L001_R1_001_fastqc.zip:1, fastqc_raw/ABM006.A0101_S0_L001_R1_001_fastqc.html:1, adapter_removal/ABM006.A0101_S0_L004_R1_001.fastq.pG.fq.settings:1, adapter_removal/DJA002.A0101_S0_L001_R1_001.fastq.pG.fq.settings:1, fastqc_raw/ABM006.A0101_S0_L003_R2_001_fastqc.zip:1, fastqc/DJA002.A0101_S0_L004_R1_001.fastq.pG.fq.combined_fastqc.html:1, damageprofiler/dmgprof8/dmgprof.json:1, fastqc/DJA002.A0101_S0_L003_R1_001.fastq.pG.fq.combined_fastqc.zip:1, fastp/DJA002.A0101_S0_L004_R1_001.fastq_fastp.json:1, fastqc_raw/DJA002.A0101_S0_L002_R2_001_fastqc.html:1, dedup6/dedup.log:1, fastqc_raw/ABM006.A0101_S0_L004_R2_001_fastqc.zip:1, fastqc_raw/DJA002.A0101_S0_L002_R1_001_fastqc.zip:1, fastqc_raw/ABM006.A0101_S0_L002_R2_001_fastqc.html:1, qualimap/qualimap2/ABM006.sorted.bam.filtered_stats:1, qualimap/qualimap1/ABM006.sorted.bam.filtered_stats:1, fastqc_raw/ABM006.A0101_S0_L004_R1_001_fastqc.html:1, fastqc_raw/ABM006.A0101_S0_L002_R2_001_fastqc.zip:1, dedup5/dedup.log:1, fastqc/ABM006.A0101_S0_L003_R1_001.fastq.pG.fq.combined_fastqc.html:1, fastqc_raw/ABM006.A0101_S0_L004_R1_001_fastqc.zip:1, dedup1/dedup.log:1, fastqc_raw/DJA002.A0101_S0_L003_R2_001_fastqc.html:1, multiqc_config.yaml:1, fastqc/DJA002.A0101_S0_L004_R1_001.fastq.pG.fq.combined_fastqc.zip:1, fastqc/DJA002.A0101_S0_L002_R1_001.fastq.pG.fq.combined_fastqc.zip:1, fastqc_raw/DJA002.A0101_S0_L004_R1_001_fastqc.html:1, fastqc_raw/DJA002.A0101_S0_L002_R1_001_fastqc.html:1, dedup4/dedup.log:1, fastp/DJA002.A0101_S0_L001_R1_001.fastq_fastp.json:1, adapter_removal/DJA002.A0101_S0_L004_R1_001.fastq.pG.fq.settings:1, fastqc/ABM006.A0101_S0_L002_R1_001.fastq.pG.fq.combined_fastqc.zip:1, idxstats/ABM006.sorted.stats:4, workflow_summary_mqc.yaml:1, fastqc/ABM006.A0101_S0_L003_R1_001.fastq.pG.fq.combined_fastqc.zip:1, fastqc_raw/DJA002.A0101_S0_L002_R2_001_fastqc.zip:1, fastqc/ABM006.A0101_S0_L004_R1_001.fastq.pG.fq.combined_fastqc.zip:1, fastqc/DJA002.A0101_S0_L001_R1_001.fastq.pG.fq.combined_fastqc.html:1, fastqc_raw/DJA002.A0101_S0_L003_R1_001_fastqc.zip:1, dedup7/dedup.log:1, damageprofiler/dmgprof4/dmgprof.json:1, fastqc/ABM006.A0101_S0_L001_R1_001.fastq.pG.fq.combined_fastqc.zip:1, damageprofiler/dmgprof2/dmgprof.json:1, qualimap/qualimap5/DJA002.sorted.bam.filtered_stats:1, fastqc_raw/ABM006.A0101_S0_L003_R1_001_fastqc.html:1, damageprofiler/dmgprof6/dmgprof.json:1, dedup3/dedup.log:1, fastqc_raw/DJA002.A0101_S0_L004_R2_001_fastqc.html:1, fastqc/ABM006.A0101_S0_L001_R1_001.fastq.pG.fq.combined_fastqc.html:1, fastqc_raw/ABM006.A0101_S0_L004_R2_001_fastqc.html:1, fastp/ABM006.A0101_S0_L004_R1_001.fastq_fastp.json:1, fastqc_raw/ABM006.A0101_S0_L002_R1_001_fastqc.zip:1, fastqc_raw/DJA002.A0101_S0_L004_R2_001_fastqc.zip:1, adapter_removal/ABM006.A0101_S0_L001_R1_001.fastq.pG.fq.settings:1, fastqc/DJA002.A0101_S0_L001_R1_001.fastq.pG.fq.combined_fastqc.zip:1, fastqc_raw/ABM006.A0101_S0_L001_R1_001_fastqc.zip:1, fastqc_raw/DJA002.A0101_S0_L003_R1_001_fastqc.html:1, software_versions/software_versions_mqc5:1, software_versions/software_versions_mqc6:1, software_versions/software_versions_mqc3:1, software_versions/software_versions_mqc4:1, software_versions/software_versions_mqc1:1, software_versions/software_versions_mqc2:1, dedup2/dedup.log:1, fastp/ABM006.A0101_S0_L003_R1_001.fastq_fastp.json:1, adapter_removal/ABM006.A0101_S0_L002_R1_001.fastq.pG.fq.settings:1, fastqc/ABM006.A0101_S0_L004_R1_001.fastq.pG.fq.combined_fastqc.html:1, fastqc/DJA002.A0101_S0_L003_R1_001.fastq.pG.fq.combined_fastqc.html:1, adapter_removal/DJA002.A0101_S0_L003_R1_001.fastq.pG.fq.settings:1, fastqc_raw/DJA002.A0101_S0_L001_R2_001_fastqc.html:1, fastp/DJA002.A0101_S0_L002_R1_001.fastq_fastp.json:1, qualimap/qualimap8/DJA002.sorted.bam.filtered_stats:1, idxstats/DJA002.sorted.stats:4, fastqc_raw/ABM006.A0101_S0_L003_R1_001_fastqc.zip:1, qualimap/qualimap6/DJA002.sorted.bam.filtered_stats:1, fastqc_raw/ABM006.A0101_S0_L002_R1_001_fastqc.html:1, fastqc_raw/ABM006.A0101_S0_L001_R2_001_fastqc.zip:1, damageprofiler/dmgprof5/dmgprof.json:1, fastqc_raw/DJA002.A0101_S0_L004_R1_001_fastqc.zip:1, fastp/DJA002.A0101_S0_L003_R1_001.fastq_fastp.json:1, fastqc/DJA002.A0101_S0_L002_R1_001.fastq.pG.fq.combined_fastqc.html:1, dedup8/dedup.log:1, damageprofiler/dmgprof7/dmgprof.json:1, fastqc_raw/DJA002.A0101_S0_L001_R2_001_fastqc.zip:1, fastqc_raw/ABM006.A0101_S0_L003_R2_001_fastqc.html:1, fastqc_raw/ABM006.A0101_S0_L001_R2_001_fastqc.html:1, adapter_removal/DJA002.A0101_S0_L002_R1_001.fastq.pG.fq.settings:1, fastqc/ABM006.A0101_S0_L002_R1_001.fastq.pG.fq.combined_fastqc.html:1, software_versions/software_versions_mqc7:1, software_versions/software_versions_mqc8:1, fastp/ABM006.A0101_S0_L002_R1_001.fastq_fastp.json:1, damageprofiler/dmgprof1/dmgprof.json:1, adapter_removal/ABM006.A0101_S0_L003_R1_001.fastq.pG.fq.settings:1, fastqc_raw/DJA002.A0101_S0_L003_R2_001_fastqc.zip:1, qualimap/qualimap7/DJA002.sorted.bam.filtered_stats:1, damageprofiler/dmgprof3/dmgprof.json:1]

Mar-16 16:00:56.590 [Actor Thread 80] ERROR nextflow.processor.TaskProcessor - Error executing process > 'multiqc' 
 
Caused by: 
  Process `multiqc` input file name collision -- There are multiple input files for each of the following file names: idxstats/ABM006.sorted.stats, idxstats/DJA002.sorted.stats

To Reproduce
Command:

nextflow run nf-core/eager \
-profile shh \
--reads '/projects1/users/fellows/nextflow/eager2/dev_testing/data/{ABM006.A0101,DJA002.A0101}/*_R{1,2}*fastq.gz' \
--pairedEnd \
--fasta '/projects1/users/fellows/nextflow/eager2/dev_testing/references/tannerella_forsythia_actual/Tannerella_forsythia_9212.fa' \
--outdir '/projects1/users/fellows/nextflow/eager2/dev_testing/eager2/01-actual_reference/tannerella_forsythia/Multiple' \
--name 'eager_nextflow_dev' \
--email fellows@shh.mpg.de \
--complexity_filter_poly_g \
--skip_preseq \
--min_adap_overlap 1 \
--clip_readlength 30 \
--clip_min_read_quality 20 \
--bwaalnn 0.01 \
--bwalnl 32 \
--bam_discard_unmapped \
--bam_unmapped_type discard \
--dedupper dedup \
--dedup_all_merged \
-r dev \
-resume

[jfy133]

After discussion with @apeltzer , the problem is indeed the use of prefix variables. Trying to shorten file names consistently is very difficult because of split delimiters are not the same between different sequencing centres. E.g. some places will use _ to delimit sections of file names, others will use ..

As an alternative to the 'prefix' system, we propose a tagging system, where each module has a 'descriptive' and unique tag appended to the output BAM/FASTQ file (but with the input file's suffix stripped).

For example, each output file would be generated following something like this (using bash notation):

${input_file%%.bam}.xxX.bam

where xxX is the module name.

(Note that the first processing step of the 'raw' input file will need consider that both fastq/fq is accepted, everything downstream would be specified by the pipeline so not an issue)

I make the following proposal based on the current version of EAGER (-r 2.0.6 )

---

Tool	Variant	Proposed tag	Example
Input bam 2 fastq	NA	sfq	.sfq.fq.gz
FastP Poly-G	PolyG	plG	.plG.fq.gz
AdapterRemoval	trimmed & collapsed	arB	.arB.fq.gz
AdapterRemoval	collapsed only	arC	.arC.fq.gz
AdapterRemoval	trimmed only	arT	.arT.fq.gz
(AdapterRemoval)*	(merged only)	(arM)	.arM.fq.gz
bwa	aln	bwA	.bwA.bam
bwa	mem	bwM	.bwM.bam
circularMapper	bwa	bwC	.bwC.bam
samtools	sort	srt	.srt.bam
samtools	filter (mapped+unmapped)	flB	.flB.bam
samtools	filter (mapped only)	flM	.flM.bam
samtools	filter (unmapped only)	flU	.flU.bam
dedup	NA	Ddp	.Ddp.bam
markduplicates	NA	Mdp	.Mdp.bam
pmdtools	NA	pmd	.pmd.bam
bamutils	trimBam	buT	.buT.bam

(* not implemented AFAIK but maybe should be?)

Rules:

• three character code (alphanumeric)
• tools abbreviations are lower case
• if there are multiple variants of a particular tool, the variants are indicated with capitals
• steps applied multiple times in the pipeline (e.g. samtools sort) do not need a 'tool' code if a verb is used.
• each character code is unique to that particular step - if multi-step commands used, they must be combined with the previous processing step (e.g. bwA.srt.bam, and later bwA.srt.Ddp.srt.bam).

For example, an input file with the name: ABM006.A0101_S0_L001_R1_000.bam would have the following code:

ABM006.A0101_S0_L001_R1_000.sfq.plG.arB.bwA.srt.flM.Ddp.srt.pmd.buT.bam [76 characters]

Indicating that it has been converted to FASTQ, polyG trimmed, adapterRemoval trimmed and collapsed, mapped with bwA, sorted by samtools, filtered for mapped reads only, deduplicated with DeDup, sorted again with satmools, somethign with PMDtools, and ends trimmed with bamUtils.

[maxibor]

I agree on the tool specific filename extensions, however I'm a bit more sceptical with regarding accepting . in filenames.
As a general rule, it should always be avoided when possible to include special characters (" ! # $ % & ' ( ) * + , - . / : ; & < = > ? @ [ \ ] ^ _ { | } ~) in filenenames/directory names.
Even if Eager might be able to handle them, some tools will most likely not, and this is kind of a bad practice anyway.

[apeltzer]

We might actually also solve this by having the TSV/CSV input 👍

[jfy133]

I agree on the tool specific filename extensions, however I'm a bit more sceptical with regarding accepting . in filenames.
As a general rule, it should always be avoided when possible to include special characters (" ! # $ % & ' ( ) * + , - . / : ; & < = > ? @ [ \ ] ^ _ { | } ~) in filenenames/directory names.
Even if Eager might be able to handle them, some tools will most likely not, and this is kind of a bad practice anyway.

Fair point. I agree . can be replaced with _ in my description above. I admit that at that point, it because of the MPI-SHH naming system which I thought it doesn't really matter 'anymore'. But indeed we should be as strict as possible regardless.

Lets see how the TSV/CSV system works.

[jfy133]

This appears to have been solved a while ago, well done @apeltzer ;)

e.g.

eager/main.nf

Line 904 in d4a2004

prefix = "$bam" - ~/(\.bam)?$/