Dsl2 annotation by maxulysse · Pull Request #386 · nf-core/sarek

maxulysse · 2021-06-08T20:50:56Z

I'm starting to add annotation.

Based on the branch from #379, so git history should be fine once it's merged.

I think it make sense to have that as (a) separate subworkflow(s).

Already created the dockerhub images for snpeff/vep:
https://hub.docker.com/r/nfcore/snpeff
https://hub.docker.com/r/nfcore/vep

PR checklist

github-actions · 2021-06-08T20:52:04Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit b7781ea

+| ✅ 135 tests passed       |+
#| ❔  11 tests were ignored |#
!| ❗  82 tests had warnings |!

Details

❗ Test warnings:

files_exist - File not found: environment.yml
files_exist - File not found: Dockerfile
nextflow_config - Config variable not found: process.container
params_used - Config variable not found in main.nf: params.input
params_used - Config variable not found in main.nf: params.step
params_used - Config variable not found in main.nf: params.genome
params_used - Config variable not found in main.nf: params.genomes_base
params_used - Config variable not found in main.nf: params.save_reference
params_used - Config variable not found in main.nf: params.help
params_used - Config variable not found in main.nf: params.no_intervals
params_used - Config variable not found in main.nf: params.nucleotides_per_second
params_used - Config variable not found in main.nf: params.sentieon
params_used - Config variable not found in main.nf: params.skip_qc
params_used - Config variable not found in main.nf: params.target_bed
params_used - Config variable not found in main.nf: params.tools
params_used - Config variable not found in main.nf: params.trim_fastq
params_used - Config variable not found in main.nf: params.clip_r1
params_used - Config variable not found in main.nf: params.clip_r2
params_used - Config variable not found in main.nf: params.three_prime_clip_r1
params_used - Config variable not found in main.nf: params.three_prime_clip_r2
params_used - Config variable not found in main.nf: params.trim_nextseq
params_used - Config variable not found in main.nf: params.save_trimmed
params_used - Config variable not found in main.nf: params.split_fastq
params_used - Config variable not found in main.nf: params.aligner
params_used - Config variable not found in main.nf: params.markdup_java_options
params_used - Config variable not found in main.nf: params.use_gatk_spark
params_used - Config variable not found in main.nf: params.save_bam_mapped
params_used - Config variable not found in main.nf: params.skip_markduplicates
params_used - Config variable not found in main.nf: params.ascat_ploidy
params_used - Config variable not found in main.nf: params.ascat_purity
params_used - Config variable not found in main.nf: params.cf_coeff
params_used - Config variable not found in main.nf: params.cf_contamination
params_used - Config variable not found in main.nf: params.cf_contamination_adjustment
params_used - Config variable not found in main.nf: params.cf_ploidy
params_used - Config variable not found in main.nf: params.cf_window
params_used - Config variable not found in main.nf: params.generate_gvcf
params_used - Config variable not found in main.nf: params.no_strelka_bp
params_used - Config variable not found in main.nf: params.pon
params_used - Config variable not found in main.nf: params.pon_index
params_used - Config variable not found in main.nf: params.ignore_soft_clipped_bases
params_used - Config variable not found in main.nf: params.umi
params_used - Config variable not found in main.nf: params.read_structure1
params_used - Config variable not found in main.nf: params.read_structure2
params_used - Config variable not found in main.nf: params.annotate_tools
params_used - Config variable not found in main.nf: params.annotation_cache
params_used - Config variable not found in main.nf: params.cadd_cache
params_used - Config variable not found in main.nf: params.cadd_indels
params_used - Config variable not found in main.nf: params.cadd_indels_tbi
params_used - Config variable not found in main.nf: params.cadd_wg_snvs
params_used - Config variable not found in main.nf: params.cadd_wg_snvs_tbi
params_used - Config variable not found in main.nf: params.genesplicer
params_used - Config variable not found in main.nf: params.snpeff_cache
params_used - Config variable not found in main.nf: params.config_profile_contact
params_used - Config variable not found in main.nf: params.config_profile_description
params_used - Config variable not found in main.nf: params.config_profile_url
params_used - Config variable not found in main.nf: params.outdir
params_used - Config variable not found in main.nf: params.publish_dir_mode
params_used - Config variable not found in main.nf: params.sequencing_center
params_used - Config variable not found in main.nf: params.multiqc_config
params_used - Config variable not found in main.nf: params.monochrome_logs
params_used - Config variable not found in main.nf: params.email
params_used - Config variable not found in main.nf: params.email_on_fail
params_used - Config variable not found in main.nf: params.plaintext_email
params_used - Config variable not found in main.nf: params.max_multiqc_email_size
params_used - Config variable not found in main.nf: params.hostnames
params_used - Config variable not found in main.nf: params.validate_params
params_used - Config variable not found in main.nf: params.tracedir
params_used - Config variable not found in main.nf: params.enable_conda
params_used - Config variable not found in main.nf: params.pull_docker_container
params_used - Config variable not found in main.nf: params.cpus
params_used - Config variable not found in main.nf: params.max_cpus
params_used - Config variable not found in main.nf: params.max_memory
params_used - Config variable not found in main.nf: params.max_time
actions_awsfulltest - .github/workflows/awsfulltest.yml should test full datasets, not -profile test
readme - README did not have a Nextflow minimum version badge.
pipeline_todos - TODO string in main.nf: It MUST be possible to pass additional parameters to the tool as a command-line string via the "$ioptions.args" variable
pipeline_todos - TODO string in main.nf: If the tool supports multi-threading then you MUST provide the appropriate parameter
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
pipeline_todos - TODO string in awstest.yml: You can customise CI pipeline run tests as required
pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
schema_description - No description provided in schema for parameter: cpus

❔ Tests ignored:

files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/feature_request.md
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo.png
files_unchanged - File ignored due to lint config: lib/NfcoreSchema.groovy
files_unchanged - File ignored due to lint config: .gitignore or foo
files_unchanged - File does not exist: .github/workflows/push_dockerhub_dev.yml
files_unchanged - File does not exist: .github/workflows/push_dockerhub_release.yml
conda_env_yaml - No environment.yml file found - skipping conda_env_yaml test
conda_dockerfile - No environment.yml / Dockerfile file found - skipping conda_dockerfile test

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: CHANGELOG.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File found: .github/markdownlint.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo.png
files_exist - File found: bin/markdown_to_html.py
files_exist - File found: docs/images/nf-core-sarek_logo.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.version
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .svg
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '3.0dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
params_used - Config variable found in main.nf: params.vep_cache
params_used - Config variable found in main.nf: params.ac_loci
params_used - Config variable found in main.nf: params.ac_loci_gc
params_used - Config variable found in main.nf: params.bwa
params_used - Config variable found in main.nf: params.chr_dir
params_used - Config variable found in main.nf: params.chr_length
params_used - Config variable found in main.nf: params.dbsnp
params_used - Config variable found in main.nf: params.dbsnp_index
params_used - Config variable found in main.nf: params.dict
params_used - Config variable found in main.nf: params.fasta
params_used - Config variable found in main.nf: params.fasta_fai
params_used - Config variable found in main.nf: params.germline_resource
params_used - Config variable found in main.nf: params.germline_resource_index
params_used - Config variable found in main.nf: params.intervals
params_used - Config variable found in main.nf: params.known_indels
params_used - Config variable found in main.nf: params.known_indels_index
params_used - Config variable found in main.nf: params.mappability
params_used - Config variable found in main.nf: params.snpeff_db
params_used - Config variable found in main.nf: params.vep_cache_version
params_used - Config variable found in main.nf: params.vep_genome
params_used - Config variable found in main.nf: params.vep_species
files_unchanged - .gitattributes matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/markdownlint.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - bin/markdown_to_html.py matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - assets/multiqc_config.yaml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (234 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: local_modules.yml
actions_schema_validation - Workflow validation passed: awstest.yml
merge_markers - No merge markers found in pipeline files

Run details

nf-core/tools version 1.14
Run at 2021-06-25 09:38:05

…into dsl2_annotation

maxulysse · 2021-06-18T14:31:10Z

ok, now need to do something smart with the tag for the different images.
and need to figure out how to reuse a submodules while changing its parameters.

maxulysse · 2021-06-23T11:17:42Z

and need to figure out how to reuse a submodules while changing its parameters.

Just need to spell the options properly actually...
🤦

FriederikeHanssen · 2021-06-23T12:28:25Z

ok, now need to do something smart with the tag for the different images.

What do you mean here?

maxulysse · 2021-06-23T12:31:20Z

ok, now need to do something smart with the tag for the different images.

What do you mean here?

Just did the commit B-)

maxulysse · 2021-06-23T12:45:03Z

@KevinMenden @drpatelh can you check out this PR?

drpatelh · 2021-06-24T09:33:03Z

Looks good to me guys. It would be much easier to see what is happening when these are submitted as separate nf-core/modules but otherwise 👌🏽 I have had to use customish modules for snpEff in viralrecon because I needed to create the db and run via the genome fasta and gff as opposed to an existing snpEff db. Be great to see these on nf-core modules though 💥

maxulysse · 2021-06-24T09:44:32Z

Looks good to me guys. It would be much easier to see what is happening when these are submitted as separate nf-core/modules but otherwise 👌🏽 I have had to use customish modules for snpEff in viralrecon because I needed to create the db and run via the genome fasta and gff as opposed to an existing snpEff db. Be great to see these on nf-core modules though boom

Ok, then I start migrating modules to the nf-core/modules repo

FriederikeHanssen

🚀

maxulysse added 2 commits June 8, 2021 22:15

feat: remove no longer used files

a7b829f

feat: move modules from local to nf-core

2b705c9

maxulysse added 3 commits June 11, 2021 15:36

Merge branch 'dsl2' into dsl2_annotation

81bcdcb

fix: restore samtools to 1.12

e8570cb

Merge branch 'dsl2_annotation' of github.com:maxulysse/nf-core_sarek …

531eecc

…into dsl2_annotation

maxulysse mentioned this pull request Jun 11, 2021

Update to DSL2 Best Practices #379

Merged

11 tasks

maxulysse added 5 commits June 18, 2021 10:00

Merge remote-tracking branch 'upstream/dsl2' into dsl2_annotation

8205da4

feat: add annotation

30bf2f4

fix: update json

d79d39b

feat: add tests for annotation

cca46d7

fix: output

df4fb5a

fix: merge + add tests

d6a6f03

feat: containers à la carte

7f886cb

maxulysse marked this pull request as ready for review June 23, 2021 12:31

maxulysse added 3 commits June 23, 2021 15:34

feat: move args to conf/modules.config

01a0293

fix: get rid of warning message

8b9812e

fix: code polishing

a3ab368

maxulysse requested review from FriederikeHanssen, KevinMenden and drpatelh June 23, 2021 14:03

FriederikeHanssen reviewed Jun 23, 2021

View reviewed changes

Comment thread modules/nf-core/software/snpeff/main.nf

maxulysse added 2 commits June 24, 2021 11:19

feat: add tests for subworkflow annotate

cc749c5

fix: annotate tests

44594c5

maxulysse added 6 commits June 24, 2021 17:14

code polishing

3da89f7

feat: add test for snpeff_annotate subworkflow

11f39a6

feat: add test for vep_annotate subworkflow

b0982ac

fix: update meta.yml file for snpEff

bf2a182

fix: update meta.yml file for VEP

1a9d53c

chores: update files

b7781ea

maxulysse mentioned this pull request Jun 28, 2021

feat: add VEP module nf-core/modules#547

Merged

14 tasks

FriederikeHanssen approved these changes Jun 29, 2021

View reviewed changes

maxulysse merged commit 88ac395 into nf-core:dsl2 Jun 29, 2021

maxulysse deleted the dsl2_annotation branch June 29, 2021 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dsl2 annotation#386

Dsl2 annotation#386
maxulysse merged 23 commits intonf-core:dsl2from
maxulysse:dsl2_annotation

maxulysse commented Jun 8, 2021

Uh oh!

github-actions Bot commented Jun 8, 2021 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

maxulysse commented Jun 18, 2021

Uh oh!

maxulysse commented Jun 23, 2021

Uh oh!

FriederikeHanssen commented Jun 23, 2021 •

edited

Loading

Uh oh!

maxulysse commented Jun 23, 2021

Uh oh!

maxulysse commented Jun 23, 2021

Uh oh!

Uh oh!

drpatelh commented Jun 24, 2021

Uh oh!

maxulysse commented Jun 24, 2021

Uh oh!

FriederikeHanssen left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

maxulysse commented Jun 8, 2021

PR checklist

Uh oh!

github-actions Bot commented Jun 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

maxulysse commented Jun 18, 2021

Uh oh!

maxulysse commented Jun 23, 2021

Uh oh!

FriederikeHanssen commented Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxulysse commented Jun 23, 2021

Uh oh!

maxulysse commented Jun 23, 2021

Uh oh!

Uh oh!

drpatelh commented Jun 24, 2021

Uh oh!

maxulysse commented Jun 24, 2021

Uh oh!

FriederikeHanssen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jun 8, 2021 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

FriederikeHanssen commented Jun 23, 2021 •

edited

Loading