Add dsl2 module for deepvariant by abhi18av · Pull Request #394 · nf-core/sarek

abhi18av · 2021-07-06T16:54:20Z

Hi team,

I'm initiating the draft PR for the integration of deepvariant tool, we had a discussion a while back in the nf-core's #deepvariant channel regarding this but for I couldn't find the time to take it forward then. Now, I find myself with some time to finally see this through.

I'll update the checklist once I have finalized the code and it's ready for a wider/final review. In the meantime, I've marked this PR as a draft to have a placeholder for early discussions

PR checklist

github-actions · 2021-07-06T16:55:20Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit f038a67

+| ✅ 135 tests passed       |+
#| ❔  11 tests were ignored |#
!| ❗  84 tests had warnings |!

Details

❗ Test warnings:

files_exist - File not found: environment.yml
files_exist - File not found: Dockerfile
nextflow_config - Config variable not found: process.container
params_used - Config variable not found in main.nf: params.input
params_used - Config variable not found in main.nf: params.step
params_used - Config variable not found in main.nf: params.genome
params_used - Config variable not found in main.nf: params.genomes_base
params_used - Config variable not found in main.nf: params.save_reference
params_used - Config variable not found in main.nf: params.help
params_used - Config variable not found in main.nf: params.no_intervals
params_used - Config variable not found in main.nf: params.nucleotides_per_second
params_used - Config variable not found in main.nf: params.sentieon
params_used - Config variable not found in main.nf: params.skip_qc
params_used - Config variable not found in main.nf: params.target_bed
params_used - Config variable not found in main.nf: params.tools
params_used - Config variable not found in main.nf: params.trim_fastq
params_used - Config variable not found in main.nf: params.clip_r1
params_used - Config variable not found in main.nf: params.clip_r2
params_used - Config variable not found in main.nf: params.three_prime_clip_r1
params_used - Config variable not found in main.nf: params.three_prime_clip_r2
params_used - Config variable not found in main.nf: params.trim_nextseq
params_used - Config variable not found in main.nf: params.save_trimmed
params_used - Config variable not found in main.nf: params.split_fastq
params_used - Config variable not found in main.nf: params.aligner
params_used - Config variable not found in main.nf: params.markdup_java_options
params_used - Config variable not found in main.nf: params.use_gatk_spark
params_used - Config variable not found in main.nf: params.save_bam_mapped
params_used - Config variable not found in main.nf: params.skip_markduplicates
params_used - Config variable not found in main.nf: params.ascat_ploidy
params_used - Config variable not found in main.nf: params.ascat_purity
params_used - Config variable not found in main.nf: params.cf_coeff
params_used - Config variable not found in main.nf: params.cf_contamination
params_used - Config variable not found in main.nf: params.cf_contamination_adjustment
params_used - Config variable not found in main.nf: params.cf_ploidy
params_used - Config variable not found in main.nf: params.cf_window
params_used - Config variable not found in main.nf: params.generate_gvcf
params_used - Config variable not found in main.nf: params.no_strelka_bp
params_used - Config variable not found in main.nf: params.pon
params_used - Config variable not found in main.nf: params.pon_index
params_used - Config variable not found in main.nf: params.ignore_soft_clipped_bases
params_used - Config variable not found in main.nf: params.umi
params_used - Config variable not found in main.nf: params.read_structure1
params_used - Config variable not found in main.nf: params.read_structure2
params_used - Config variable not found in main.nf: params.annotate_tools
params_used - Config variable not found in main.nf: params.annotation_cache
params_used - Config variable not found in main.nf: params.cadd_cache
params_used - Config variable not found in main.nf: params.cadd_indels
params_used - Config variable not found in main.nf: params.cadd_indels_tbi
params_used - Config variable not found in main.nf: params.cadd_wg_snvs
params_used - Config variable not found in main.nf: params.cadd_wg_snvs_tbi
params_used - Config variable not found in main.nf: params.genesplicer
params_used - Config variable not found in main.nf: params.snpeff_cache
params_used - Config variable not found in main.nf: params.config_profile_contact
params_used - Config variable not found in main.nf: params.config_profile_description
params_used - Config variable not found in main.nf: params.config_profile_url
params_used - Config variable not found in main.nf: params.outdir
params_used - Config variable not found in main.nf: params.publish_dir_mode
params_used - Config variable not found in main.nf: params.sequencing_center
params_used - Config variable not found in main.nf: params.multiqc_config
params_used - Config variable not found in main.nf: params.monochrome_logs
params_used - Config variable not found in main.nf: params.email
params_used - Config variable not found in main.nf: params.email_on_fail
params_used - Config variable not found in main.nf: params.plaintext_email
params_used - Config variable not found in main.nf: params.max_multiqc_email_size
params_used - Config variable not found in main.nf: params.hostnames
params_used - Config variable not found in main.nf: params.validate_params
params_used - Config variable not found in main.nf: params.tracedir
params_used - Config variable not found in main.nf: params.enable_conda
params_used - Config variable not found in main.nf: params.pull_docker_container
params_used - Config variable not found in main.nf: params.cpus
params_used - Config variable not found in main.nf: params.max_cpus
params_used - Config variable not found in main.nf: params.max_memory
params_used - Config variable not found in main.nf: params.max_time
actions_awsfulltest - .github/workflows/awsfulltest.yml should test full datasets, not -profile test
readme - README did not have a Nextflow minimum version badge.
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
pipeline_todos - TODO string in awstest.yml: You can customise CI pipeline run tests as required
pipeline_todos - TODO string in main.nf: It MUST be possible to pass additional parameters to the tool as a command-line string via the "$ioptions.args" variable
pipeline_todos - TODO string in main.nf: If the tool supports multi-threading then you MUST provide the appropriate parameter
pipeline_todos - TODO string in main.nf: A module file SHOULD only define input and output files as command-line parameters.
pipeline_todos - TODO string in main.nf: Where applicable all sample-specific information e.g. "id", "single_end", "read_group"
pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
schema_description - No description provided in schema for parameter: cpus

❔ Tests ignored:

files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/feature_request.md
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo.png
files_unchanged - File ignored due to lint config: lib/NfcoreSchema.groovy
files_unchanged - File ignored due to lint config: .gitignore or foo
files_unchanged - File does not exist: .github/workflows/push_dockerhub_dev.yml
files_unchanged - File does not exist: .github/workflows/push_dockerhub_release.yml
conda_env_yaml - No environment.yml file found - skipping conda_env_yaml test
conda_dockerfile - No environment.yml / Dockerfile file found - skipping conda_dockerfile test

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: CHANGELOG.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File found: .github/markdownlint.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo.png
files_exist - File found: bin/markdown_to_html.py
files_exist - File found: docs/images/nf-core-sarek_logo.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.version
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .svg
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '3.0dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
params_used - Config variable found in main.nf: params.vep_cache
params_used - Config variable found in main.nf: params.ac_loci
params_used - Config variable found in main.nf: params.ac_loci_gc
params_used - Config variable found in main.nf: params.bwa
params_used - Config variable found in main.nf: params.chr_dir
params_used - Config variable found in main.nf: params.chr_length
params_used - Config variable found in main.nf: params.dbsnp
params_used - Config variable found in main.nf: params.dbsnp_index
params_used - Config variable found in main.nf: params.dict
params_used - Config variable found in main.nf: params.fasta
params_used - Config variable found in main.nf: params.fasta_fai
params_used - Config variable found in main.nf: params.germline_resource
params_used - Config variable found in main.nf: params.germline_resource_index
params_used - Config variable found in main.nf: params.intervals
params_used - Config variable found in main.nf: params.known_indels
params_used - Config variable found in main.nf: params.known_indels_index
params_used - Config variable found in main.nf: params.mappability
params_used - Config variable found in main.nf: params.snpeff_db
params_used - Config variable found in main.nf: params.vep_cache_version
params_used - Config variable found in main.nf: params.vep_genome
params_used - Config variable found in main.nf: params.vep_species
files_unchanged - .gitattributes matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/markdownlint.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - bin/markdown_to_html.py matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - assets/multiqc_config.yaml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (236 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: local_modules.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files

Run details

nf-core/tools version 1.14
Run at 2021-07-07 08:22:38

abhi18av

I've added some comments to confirm certain design decisions and request feedback.

I'm a bit confused about how to use the options.args to pass arbitrary args to the workflow, could you please point me to the right direction?

Also, is there a way to skip triggering the CI for these interim changes?

maxulysse · 2021-07-07T08:40:48Z

I've added some comments to confirm certain design decisions and request feedback.

I'm a bit confused about how to use the options.args to pass arbitrary args to the workflow, could you please point me to the right direction?

You need to add the args (args2 or even args3) to https://github.com/nf-core/sarek/blob/dsl2/conf/modules.config

Also, is there a way to skip triggering the CI for these interim changes?

Add skip ci to your commit:

https://github.blog/changelog/2021-02-08-github-actions-skip-pull-request-and-push-workflows-with-skip-ci/

abhi18av · 2021-07-12T11:30:51Z

Hi @maxulysse , I've now opened up the PR for review

implemented the basic module
intergrated the settings into modules.config
integrated the module into germline_variant_caller.nf subworkflow

Questions:

Regarding the testing, how should I test the variant calling steps? nextflow run . -profile test,docker doesn't seem to invoke the variant calling steps.
Would you like to see anyone of these outputs to be included in the output channels?

output.g.vcf.gz.tbi
output.vcf.gz.tbi
output.visual_report.html

Please let me know what further changes are necessary before this PR could be mergeable - apart from the things mentioned in the PR template checklist.

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

abhi18av · 2021-07-15T08:24:13Z

Thanks @maxulysse ! 😍

Add the basic stubs for deepvariant

0fa42ea

abhi18av self-assigned this Jul 6, 2021

This was referenced Jul 6, 2021

[FEATURE] Fix bioconda recipe for deepvariant nf-core/modules#1754

Open

new module: deepvariant nf-core/modules#234

Closed

Integrate stubs into the germline caller subworkflow

f038a67

abhi18av commented Jul 7, 2021

View reviewed changes

Comment thread subworkflows/local/germline_variant_calling.nf

Comment thread subworkflows/local/germline_variant_calling.nf

Comment thread modules/local/deepvariant/functions.nf

abhi18av added 2 commits July 12, 2021 13:22

Update the modules conf for deepvariant [ci skip]

0312af1

Tweak the output channel [ci skip]

f360c13

abhi18av marked this pull request as ready for review July 12, 2021 11:26

Link the output of deepvariant to germline variant caller [ci skip]

b0bd8e9

abhi18av added 2 commits July 12, 2021 13:31

Update version scraper [ci skip]

1e94be3

Update dv for cpus and minor fixes [ci skip]

cfdb91c

maxulysse approved these changes Jul 15, 2021

View reviewed changes

maxulysse reviewed Jul 15, 2021

View reviewed changes

Comment thread subworkflows/local/germline_variant_calling.nf Outdated

Accomodate the code review

b446514

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

maxulysse merged commit 18d5b61 into nf-core:dsl2 Jul 15, 2021

abhi18av deleted the abhinav/deepvariant branch July 15, 2021 08:24

abhi18av mentioned this pull request Jul 22, 2021

module: deeptrio nf-core/modules#573

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dsl2 module for deepvariant#394

Add dsl2 module for deepvariant#394
maxulysse merged 8 commits intonf-core:dsl2from
abhi18av:abhinav/deepvariant

abhi18av commented Jul 6, 2021 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 6, 2021 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

abhi18av left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxulysse commented Jul 7, 2021

Uh oh!

abhi18av commented Jul 12, 2021 •

edited

Loading

Uh oh!

Uh oh!

abhi18av commented Jul 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

abhi18av commented Jul 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

github-actions Bot commented Jul 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

abhi18av left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxulysse commented Jul 7, 2021

Uh oh!

abhi18av commented Jul 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

abhi18av commented Jul 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abhi18av commented Jul 6, 2021 •

edited

Loading

github-actions Bot commented Jul 6, 2021 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

abhi18av left a comment •

edited

Loading

abhi18av commented Jul 12, 2021 •

edited

Loading