Skip to content

Update modules once again#459

Merged
maxulysse merged 22 commits intonf-core:devfrom
maxulysse:dev_get_versions
Dec 2, 2021
Merged

Update modules once again#459
maxulysse merged 22 commits intonf-core:devfrom
maxulysse:dev_get_versions

Conversation

@maxulysse
Copy link
Copy Markdown
Member

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
    • If you've added a new tool - add to the software_versions process and a regex to scrape_software_versions.py
    • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
    • If necessary, also make a PR on the nf-core/sarek branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint .).
  • Ensure the test suite passes (nextflow run . -profile test,docker).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Nov 29, 2021

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 18209c0

+| ✅ 133 tests passed       |+
#| ❔   5 tests were ignored |#
!| ❗   5 tests had warnings |!
Details

❗ Test warnings:

  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo.png
  • files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
  • files_unchanged - File ignored due to lint config: assets/multiqc_config.yaml

✅ Tests passed:

Run details

  • nf-core/tools version 2.2.dev0
  • Run at 2021-12-01 21:49:56

Comment thread conf/modules.config Outdated
@maxulysse maxulysse marked this pull request as ready for review December 1, 2021 21:33
@maxulysse
Copy link
Copy Markdown
Member Author

@drpatelh at first I was a big annoyed because I just had finally finished my refactoring of everything with the newest modules syntax a week ago, and I had to do it again with the newer newest syntax.
But the simplification is so good, and I had a lot of fun playing with it.
It definitively will allow us to port a lot of subworkflows into the nf-core/modules repo.
It's even a bit scary how much you can control the processes within the modules.config.

I'm particularly proud of the rework of our local bwamem and bwamem2 modules that are now stripped of every sarek logic, and the rework it allowed for the mapping subworkflow.
<3
I'm really looking forward a tools release now.

Comment thread conf/modules.config
// MAPPING
process {

withName: "BWA.*MEM" {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍

Comment thread conf/modules.config
// Only name sort if Spark for Markduplicates + duplicate marking is not skipped
if (('markduplicates' in params.use_gatk_spark) && (!params.skip_markduplicates)) {
process {
withName: "BWA.*_MEM" { ext.args2 = 'sort -n' }
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This then overwrites the above config ext.args2 for BWAMEM?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hoping that we can also remove the part thingy from the bwa*mem processes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will think about how to do this. I am sure there is a way

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are you using it? I don't see it in the process anymore

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, that's what I wanted to figure it out.
I'll let someone else try that, I'm going to the NGI retreat today, so won't be able to work much

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep that is what I thought of, to also pull that in the modules config. I am glad I wasn't completely off here. I'll give it a try probably tomorrow afternoon

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plus by then maybe the yet new modules syntax is there and I can catch two birds with one stone 😆

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FriederikeHanssen will you be adding the split_fastq logic in the bwamem modules in #462 or will you be doing that in a separate PR?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll see that I can add it in :)

Copy link
Copy Markdown
Contributor

@FriederikeHanssen FriederikeHanssen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my side this looks all great. Taking the 'parameter logic' out of the subworkflow is pretty nice, makes the workflow look a lot simpler and less crowded. I can also now better see how we can make it work with UMI based subworkflow + mapping.

@maxulysse maxulysse merged commit 6f05f41 into nf-core:dev Dec 2, 2021
@maxulysse maxulysse deleted the dev_get_versions branch December 2, 2021 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants