Skip to content

modules config update - preprocessing refactor #485

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maxulysse merged 33 commits into from
Feb 20, 2022
Merged

modules config update - preprocessing refactor #485

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
f291ad0
nf-core modules update -a
maxulysse Feb 15, 2022
587a48c
add back params.publish_dir_mode + use ext.when
maxulysse Feb 15, 2022
74ad51f
feat: simplify prepare_genome with ext.when
maxulysse Feb 16, 2022
7fa9d52
fix: publish_dir_more does not inherit
maxulysse Feb 16, 2022
0acbc6a
fix: typos
maxulysse Feb 16, 2022
1898618
feat: code polishing
maxulysse Feb 16, 2022
577e131
feat: code polishing
maxulysse Feb 16, 2022
d11f6a7
feat: code polishing
maxulysse Feb 16, 2022
a51f985
feat: code polishing + refactor fastqc_trimgalore + refactor (skip_qc…
maxulysse Feb 16, 2022
e8808ea
feat: refactor gatk4_mapping/main.nf
maxulysse Feb 16, 2022
dd3a301
feat: move split_fastq in its own subworkflow
maxulysse Feb 16, 2022
2644527
fix: add when directive to local modules
maxulysse Feb 17, 2022
3957f87
feat: refactor markduplicates.nf
maxulysse Feb 17, 2022
649fb51
feat: add comments
maxulysse Feb 18, 2022
2041071
feat: add comments
maxulysse Feb 18, 2022
cdb8254
feat: add comments
maxulysse Feb 18, 2022
af59565
Update nextflow.config
maxulysse Feb 18, 2022
1e97f42
feat: better comments
maxulysse Feb 18, 2022
323fa54
fix: comments polishing
maxulysse Feb 18, 2022
50f6fa3
feat: uncouple intervals from prepare_genome + comments
maxulysse Feb 18, 2022
21a0f74
feat: refactor prepare_recalibration
maxulysse Feb 18, 2022
dc385a5
feat: refactor recalibrate.nf
maxulysse Feb 18, 2022
44f17d5
Update conf/modules.config
maxulysse Feb 18, 2022
367d2be
fix: split_fastq tests
maxulysse Feb 18, 2022
57f766e
Merge branch 'dev_skip' of github.com:maxulysse/nf-core_sarek into de…
maxulysse Feb 18, 2022
89720c5
fix: more tests
maxulysse Feb 18, 2022
774f35a
Update subworkflows/local/split_fastq.nf
maxulysse Feb 18, 2022
30b9cc8
feat: code polishing
maxulysse Feb 19, 2022
c0d36e7
fix: missing channel
maxulysse Feb 19, 2022
31a4aa3
fix: --skip_tools markduplicates
maxulysse Feb 19, 2022
f360d4e
fix: bwa issues with multiple samples
maxulysse Feb 20, 2022
2f80245
fix: targeted tests
maxulysse Feb 20, 2022
096f97d
code polishing
maxulysse Feb 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
452 changes: 259 additions & 193 deletions conf/modules.config
View file
Open in desktop

Large diffs are not rendered by default.

18 changes: 9 additions & 9 deletions conf/test.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,9 @@ profiles {
params.input = 'https://raw.githubusercontent.com/nf-core/test-datasets/sarek/testdata/csv/tiny-vcf-https.csv'
params.step = 'annotate'
}
no_intervals {
params.no_intervals = true
}
pair {
params.input = "${baseDir}/tests/csv/3.0/fastq_pair.csv"
}
Expand All @@ -61,15 +64,12 @@ profiles {
params.save_bam_mapped = true
}
skip_markduplicates {
params.skip_markduplicates = true
params.skip_tools = "markduplicates"
}
split_fastq {
params.split_fastq = 150000
params.save_split_fastqs = true
}
no_intervals {
params.no_intervals = true
}
targeted {
params.intervals = "${params.genomes_base}/data/genomics/homo_sapiens/genome/multi_intervals.bed"
params.wes = true
Expand All @@ -96,21 +96,21 @@ profiles {
params.three_prime_clip_r2 = 1
params.trim_fastq = true
}
use_gatk_spark {
params.use_gatk_spark = 'bqsr,markduplicates'
}
umi {
params.input = "${baseDir}/tests/csv/3.0/fastq_umi.csv"
params.umi_read_structure = '7M1S+T'
}
use_gatk_spark {
params.use_gatk_spark = 'baserecalibrator,markduplicates'
}
}

//This is apparently useless as it won't overwrite things in the modules.config
process {
Copy link
Copy Markdown
Contributor

@FriederikeHanssen FriederikeHanssen Feb 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think we can delete this now, no?

Copy link
Copy Markdown
Member Author

@maxulysse maxulysse Feb 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, we still want to test this no?

Copy link
Copy Markdown
Contributor

@FriederikeHanssen FriederikeHanssen Feb 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't get overwritten but will need to be moved to the modules.config and activated somehow on test

withName:SNPEFF {
withName:ENSEMBLVEP {
maxForks = 1
}
withName:ENSEMBLVEP {
withName:SNPEFF {
maxForks = 1
}
}
104 changes: 52 additions & 52 deletions modules.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,106 +4,106 @@
"repos": {
"nf-core/modules": {
"bcftools/stats": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"bwa/index": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"bwa/mem": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"bwamem2/index": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"bwamem2/mem": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"cat/fastq": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"cnvkit/batch": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"custom/dumpsoftwareversions": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"deepvariant": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"ensemblvep": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"fastqc": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"fgbio/callmolecularconsensusreads": {
"git_sha": "9d0cad583b9a71a6509b754fdf589cbfbed08961"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"fgbio/fastqtobam": {
"git_sha": "e3285528aca2733ff2d544cb5e5fcc34599226f3"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"fgbio/groupreadsbyumi": {
"git_sha": "9d0cad583b9a71a6509b754fdf589cbfbed08961"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"freebayes": {
"git_sha": "9d0cad583b9a71a6509b754fdf589cbfbed08961"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/applybqsr": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/applyvqsr": {
"git_sha": "598d7abdb2a8df1aa3471c48d9186a9e3465983f"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/baserecalibrator": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/calculatecontamination": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/createsequencedictionary": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/estimatelibrarycomplexity": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/filtermutectcalls": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/gatherbqsrreports": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/genomicsdbimport": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "967fb22dedc2c8855f00e64c3d7b5814c85242a6"
},
"gatk4/genotypegvcfs": {
"git_sha": "598d7abdb2a8df1aa3471c48d9186a9e3465983f"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/getpileupsummaries": {
"git_sha": "f5d5926516d2319c1af83fb4b33834cc4461ce62"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/haplotypecaller": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/learnreadorientationmodel": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/markduplicates": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/mutect2": {
"git_sha": "f5d5926516d2319c1af83fb4b33834cc4461ce62"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"gatk4/variantrecalibrator": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"manta/germline": {
"git_sha": "9d0cad583b9a71a6509b754fdf589cbfbed08961"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"manta/somatic": {
"git_sha": "9d0cad583b9a71a6509b754fdf589cbfbed08961"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"manta/tumoronly": {
"git_sha": "9d0cad583b9a71a6509b754fdf589cbfbed08961"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"msisensorpro/msi_somatic": {
"git_sha": "c8ebd0de36c649a14fc92f2f73cbd9f691a8ce0a"
Expand All @@ -112,58 +112,58 @@
"git_sha": "c8ebd0de36c649a14fc92f2f73cbd9f691a8ce0a"
},
"multiqc": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"samblaster": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"samtools/bam2fq": {
"git_sha": "e751e5040af57e1b4e06ed4e0f3efe6de25c1683"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"samtools/faidx": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"samtools/index": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"samtools/merge": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"samtools/mpileup": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"samtools/stats": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"samtools/view": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"seqkit/split2": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"snpeff": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"strelka/germline": {
"git_sha": "f5d5926516d2319c1af83fb4b33834cc4461ce62"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"strelka/somatic": {
"git_sha": "f5d5926516d2319c1af83fb4b33834cc4461ce62"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"tabix/bgziptabix": {
"git_sha": "e22966ce74340cb671576143e5fdbbd71670cffa"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"tabix/tabix": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"tiddit/sv": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"trimgalore": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
},
"vcftools": {
"git_sha": "e20e57f90b6787ac9a010a980cf6ea98bd990046"
"git_sha": "e745e167c1020928ef20ea1397b6b4d230681b4d"
}
}
}
Expand Down
3 changes: 3 additions & 0 deletions modules/local/bgzip.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ process BGZIP {
tuple val(meta), path("*.vcf"), emit: vcf
path "versions.yml" , emit: versions

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"
Expand Down
3 changes: 3 additions & 0 deletions modules/local/build_intervals/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ process BUILD_INTERVALS {
output:
path "*.bed", emit: bed

when:
task.ext.when == null || task.ext.when

script:
"""
awk -v FS='\t' -v OFS='\t' '{ print \$1, \"0\", \$2 }' ${fasta_fai} > ${fasta_fai.baseName}.bed
Expand Down
5 changes: 4 additions & 1 deletion modules/local/concat_vcf/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,16 @@ process CONCAT_VCF {
tuple val(meta), path("${prefix}.vcf.gz.tbi"), emit: tbi
path "versions.yml" , emit: versions

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
prefix = task.ext.prefix ?: "${meta.id}"
def target_options = target_bed ? "-t ${target_bed}" : ""

"""
concatenateVCFs.sh -i ${fasta_fai} -c ${task.cpus} -o ${prefix}.vcf ${target_options} $args

cat <<-END_VERSIONS > versions.yml
"${task.process}":
bcftools: \$(bcftools --version 2>&1 | head -n1 | sed 's/^.*bcftools //; s/ .*\$//')
Expand Down
3 changes: 3 additions & 0 deletions modules/local/create_intervals_bed/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ process CREATE_INTERVALS_BED {
path ("*.bed"), emit: bed
//TODO version number missing

when:
task.ext.when == null || task.ext.when

script:
// If intervals file is in BED format,
// Fifth column is interpreted to contain runtime estimates
Expand Down
4 changes: 3 additions & 1 deletion modules/local/deeptools/bamcoverage.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,12 @@ process DEEPTOOLS_BAMCOVERAGE {
tuple val(meta), path("*.bigWig"), emit: bigwig
path "versions.yml" , emit: versions

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"

"""
bamCoverage \
--bam $bam \
Expand Down
3 changes: 3 additions & 0 deletions modules/local/gatk4/applybqsrspark/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ process GATK4_APPLYBQSR_SPARK {
tuple val(meta), path("*.cram"), emit: cram
path "versions.yml" , emit: versions

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
def avail_mem = 3
Expand Down
3 changes: 3 additions & 0 deletions modules/local/gatk4/baserecalibratorspark/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ process GATK4_BASERECALIBRATOR_SPARK {
tuple val(meta), path("*.table"), emit: table
path "versions.yml" , emit: versions

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
def avail_mem = 3
Expand Down
3 changes: 3 additions & 0 deletions modules/local/gatk4/gatherpileupsummaries.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ process GATK4_GATHERPILEUPSUMMARIES {
tuple val(meta), path("*.table"), emit: table
path "versions.yml" , emit: versions

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"
Expand Down
3 changes: 3 additions & 0 deletions modules/local/gatk4/intervallisttobed.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ process GATK4_INTERVALLISTTOBED {
tuple val(meta), path("*.bed"), emit: bed
path "versions.yml" , emit: versions

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"
Expand Down
Loading
⚡