refactor-workflow-tests by sateeshperi · Pull Request #539 · nf-core/methylseq

sateeshperi · 2025-06-26T06:17:58Z

This PR contains significant improvements to the FASTA index subworkflow, updates to multiple nf-core modules, test consolidation, and enhancements to the targeted sequencing functionality.

Major Changes

🔧 FASTA Index Subworkflow Refactoring - remove implicit params
Improved channel handling for FASTA files with better branching logic for gzipped vs unzipped files
Added aligner and collecthsmetrics parameters to the subworkflow inputs

📦 Module Updates

Updated multiple nf-core modules to latest versions

🧪 Test Suite Consolidation

Removed individual test files
Created comprehensive variant-based test suites:
bismark_variants.nf.test (
bwameth_variants.nf.test
targeted_sequencing_variants.nf.test
index_downloads.nf.test for testing index download scenarios

🎯 Targeted Sequencing Improvements

Enhanced targeted sequencing subworkflow
Better parameter handling with collecthsmetrics boolean parameter

⚙️ Main Workflow Updates

Improved parameter handling in main.nf:
Converted file parameters to proper channel initialization
Added channel mapping for ch_fasta, ch_fasta_index, ch_bismark_index, and ch_bwameth_index
Passed aligner and collecthsmetrics parameters to the subworkflow

…seq into update-fasta-index-sub-wf

…kflows

…lows, removing outdated tests and introducing new structured tests for variants and index downloads.

…PU support and alignment parameters. Remove deprecated special input tests.

github-actions · 2025-06-26T06:19:58Z

`nf-core pipelines lint` overall result: Passed ✅

Posted for pipeline commit c2f1074

+| ✅ 236 tests passed       |+
#| ❔   4 tests were ignored |#

Details

❔ Tests ignored:

files_exist - File is ignored: lib/nfcore_external_java_deps.jar
files_exist - File is ignored: conf/modules.config
files_exist - File is ignored: .github/workflows/ci.yml
modules_config - modules_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/nf-test.yml
files_exist - File found: .github/actions/get-shards/action.yml
files_exist - File found: .github/actions/nf-test/action.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-methylseq_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-methylseq_logo_light.png
files_exist - File found: docs/images/nf-core-methylseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: nf-test.config
files_exist - File found: tests/default.nf.test
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File found: ro-crate-metadata.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-methylseq_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowMethylseq.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 3.1.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.aligner= bismark
nextflow_config - Config default value correct: params.clip_r1= 0
nextflow_config - Config default value correct: params.clip_r2= 0
nextflow_config - Config default value correct: params.three_prime_clip_r1= 0
nextflow_config - Config default value correct: params.three_prime_clip_r2= 0
nextflow_config - Config default value correct: params.nextseq_trim= 0
nextflow_config - Config default value correct: params.skip_trimming_presets= false
nextflow_config - Config default value correct: params.num_mismatches= 0.6
nextflow_config - Config default value correct: params.no_overlap= true
nextflow_config - Config default value correct: params.ignore_r1= 0
nextflow_config - Config default value correct: params.ignore_r2= 2
nextflow_config - Config default value correct: params.ignore_3prime_r1= 0
nextflow_config - Config default value correct: params.ignore_3prime_r2= 2
nextflow_config - Config default value correct: params.min_depth= 0
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/methylseq/
nf_test_content - 'tests/targeted_sequencing_variants.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/targeted_sequencing_variants.nf.test' contains outdir parameter
nf_test_content - 'tests/bismark_hisat_variants.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bismark_hisat_variants.nf.test' contains outdir parameter
nf_test_content - 'tests/bwameth_variants.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwameth_variants.nf.test' contains outdir parameter
nf_test_content - 'tests/bismark_variants.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bismark_variants.nf.test' contains outdir parameter
nf_test_content - 'tests/index_downloads.nf.test' contains outdir parameter
nf_test_content - 'tests/index_downloads.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/bwameth_use_gpu.nf.test' contains outdir parameter
nf_test_content - 'tests/bwameth_use_gpu.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/default.nf.test' contains outdir parameter
nf_test_content - 'tests/default.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/default.nf.test' snapshots a 'versions.yml' file
nf_test_content - 'tests/default.nf.test' contains outdir parameter
nf_test_content - 'tests/nextflow.config' contains modules_testdata_base_path
nf_test_content - 'tests/nextflow.config' contains pipelines_testdata_base_path
nf_test_content - 'tests/nextflow.config' contains pipelines_testdata_base_path
nf_test_content - 'tests/nextflow.config' contains pipelines_testdata_base_path
nf_test_content - 'nf-test.config' sets a testsDir
nf_test_content - 'nf-test.config' sets a workDir
nf_test_content - 'nf-test.config' sets a configFile
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-methylseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-methylseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-methylseq_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_nf_test - '.github/workflows/nf-test.yml' is triggered on expected events
actions_nf_test - '.github/workflows/nf-test.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 24.10.2, Config: 24.10.2
readme - README nf-core template version badge found.
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_if_empty_null - No ifEmpty(null) strings found
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: fix_linting.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: nf-test.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: nf-test-gpu.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: template-version-comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
local_component_structure - local subworkflows directory structure is correct 'subworkflows/local/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - PRESEQ_LCEXTRAP found in conf/base.config and Nextflow scripts.
base_config - BISMARK_ALIGN found in conf/base.config and Nextflow scripts.
base_config - BISMARK_DEDUPLICATE found in conf/base.config and Nextflow scripts.
base_config - BISMARK_METHYLATIONEXTRACTOR found in conf/base.config and Nextflow scripts.
base_config - BWAMETH_ALIGN found in conf/base.config and Nextflow scripts.
base_config - PARABRICKS_FQ2BAMMETH found in conf/base.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.3.1
rocrate_readme_sync - RO-Crate descriptions are in sync with README.md.

Run details

nf-core/tools version 3.3.1
Run at 2025-06-30 07:38:27

GallVp · 2025-06-26T10:25:44Z

Hi @sateeshperi

This is a big one. Can we break it into a couple of smaller ones?

sateeshperi · 2025-06-26T11:28:20Z

I am afraid its too late for that sorry.

If you collapse the module and subworkflow updates, you should see the changes to the channel wrings.

Please take your time and lmk if you would like me to provide a walk-through via zoom. Thanks a lot!

…r each repeater instead use combine and multimap

…run FastQC and MultiQC. Enhance test cases for Bismark and Bwameth workflows to include skip_fastqc functionality. Fix workflow versions collection.

…ks, and enhance usage parameters section. Initialize channels for bedgraph and aligner multiqc in main workflow.

…ed sequencing subworkflow

sateeshperi · 2025-06-30T04:25:28Z

one last fix to support igenomes incoming

sateeshperi · 2025-06-30T04:49:02Z

fix to properly handle decompressed files

…ove unnecessary use of the first operator, improving clarity and consistency in the workflow setup.

GallVp

Thank you! This is really good work.

I am happy with the PR now.

sateeshperi · 2025-06-30T07:29:13Z

Thank you. could you review an easy PR to properly handle genome param
nf-core/modules#8732

sateeshperi added 18 commits June 17, 2025 17:03

update nf-core modules + sub-wf

696a9de

targeted meta.yml

95560af

fix typo and restore samplesheet

f2ce847

update bismark snaps after version updates

e27d93a

rename collecthsmetric param

3ba41a7

fix targeted sequencing workflow and test

ffbf7ed

udpate snapshots after version updates

d05b31b

add target_regions_file param for tests

a99834d

Merge branch 'update-fasta-index-sub-wf' of github.com:nf-core/methyl…

728dcc3

…seq into update-fasta-index-sub-wf

Update FASTA index parameter in targeted sequencing and methylseq wor…

150ae40

…kflows

Refactor and consolidate test scenarios for Bismark and Bwameth workf…

ea56d1e

…lows, removing outdated tests and introducing new structured tests for variants and index downloads.

fix tests workflow trace scope

6a12df6

mv multiqc data parquet to nftignore

0f405f6

consolidate bwameth tests into one file+snap

2906d7e

remove old test snapshots

fc7b010

Add new test scenarios for Bismark and Bwameth workflows, including G…

f2e3269

…PU support and alignment parameters. Remove deprecated special input tests.

add to bismark and bwameth tests

52dd376

remove alignment variants test for now

0f7e847

sateeshperi requested a review from a team as a code owner June 26, 2025 06:17

sateeshperi requested a review from FelixKrueger June 26, 2025 06:17

sateeshperi added 3 commits June 26, 2025 06:39

add version in snapshot for index tests

762dbbb

update ro-crate

8185c00

update changelog

8bec8bc

update GPU test snapshot

ee29919

GallVp reviewed Jun 27, 2025

View reviewed changes

Comment thread nextflow.config

GallVp reviewed Jun 27, 2025

View reviewed changes

Comment thread subworkflows/nf-core/fasta_index_bismark_bwameth/main.nf

GallVp reviewed Jun 27, 2025

View reviewed changes

Comment thread main.nf Outdated

GallVp reviewed Jun 27, 2025

View reviewed changes

Comment thread tests/bismark_hisat_variants.nf.test

GallVp reviewed Jun 27, 2025

View reviewed changes

Comment thread tests/bismark_hisat_variants.nf.test Outdated

sateeshperi added 5 commits June 29, 2025 11:44

apply suggestions from code review - replace use of first operator fo…

bdad2e6

…r each repeater instead use combine and multimap

Add skip_fastqc parameter and update workflow logic to conditionally …

ca370fd

…run FastQC and MultiQC. Enhance test cases for Bismark and Bwameth workflows to include skip_fastqc functionality. Fix workflow versions collection.

update changelog

76af38a

update snapshots to capture all module versions

235efdc

update gpu snapshot

4aeaacb

sateeshperi requested a review from GallVp June 29, 2025 12:06

sateeshperi added 4 commits June 29, 2025 20:11

Add flowchart for pipeline stages in README, update documentation lin…

579ac33

…ks, and enhance usage parameters section. Initialize channels for bedgraph and aligner multiqc in main workflow.

remove unnecessary channel setups for fasta and fasta_index in target…

cee4864

…ed sequencing subworkflow

mv mermaid chart to usage docs

9998efe

restore docs/README

5c96b1e

sateeshperi self-assigned this Jun 30, 2025

sateeshperi added this to the 3.1.0 milestone Jun 30, 2025