Skip to content

Commit a0ac594

Browse files
pditommasoclaudejorgeeewelsbentsherman
authored
Module system (#6650)
* Add Nextflow Development Constitution Introduce comprehensive development constitution documenting core principles and practices for Nextflow development including modular architecture, test-driven quality assurance, dataflow programming model, licensing compliance, DCO requirements, semantic versioning, and Groovy code standards. The constitution codifies existing best practices from CLAUDE.md and CONTRIBUTING.md to provide clear governance and quality standards for the project. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Module adr v1 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Module adr v2 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * ADR: Nextflow remote modules system v2.1 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Add detailed CLI command descriptions to module system ADR [ci skip] - Add comprehensive documentation for all module CLI commands - Add `nextflow module run` command for standalone module execution - Remove `module update` command to simplify the design - Use single-dash prefix for Nextflow options, double-dash for module inputs - Remove @ prefix from scope in CLI commands (keep only in DSL syntax) Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Simplify module spec: unify dependencies under requires - Remove separate `dependencies` and `components` fields - Expand `requires` to include: - `nextflow`: version constraint (unchanged) - `plugins`: array with name@constraint syntax - `modules`: array of module dependencies - `workflows`: array of workflow dependencies - Unified version constraint syntax: `[scope/]name[@constraint]` - Mark `components` as deprecated (use requires.modules) - Update all examples in ADR and schema Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Add structured tool arguments to module spec schema - Add `args` property to tools section for type-safe argument configuration - Define toolArgSpec with flag, type, description, default, enum, required - Support implicit variable `tools.<toolname>.args.<argname>` returning formatted flag+value (e.g., "-K 100000000") - Support `tools.<toolname>.args` to return all args concatenated - Document deprecation of ext.args/ext.args2/ext.args3 pattern - Update ADR with Tool Arguments Configuration section and appendix [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update adr/20251114-module-system.md [ci skip] Expand deprecation notice to cover all ext.* custom directives Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Fix inconsistencies in module system ADR [ci skip] - Use consistent module path format with version: modules/@scope/name@version/ - Fix directory structure example: samtools-view -> samtools/view - Standardize on 'license' spelling (American English) - Fix author -> authors (plural array format) Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Fix registry API path in comparison table [ci skip] Remove @ prefix from scope in API path to match API definition Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Fix checksum format: sha256- -> sha256: [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update module API to match registry implementation [ci skip] - Update API endpoints to match seqeralabs/nextflow-registry#266 - Use /api/modules base path (no v1 prefix) - Use single {name} parameter with namespace (e.g., "nf-core/fastqc") - Add separate /releases endpoint - Simplify publish to single POST endpoint Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update adr [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Simplify module storage and verification model [ci skip] - Remove checksums from nextflow.config (registry is source of truth) - Use single version per module locally (no version in directory path) - Add .checksum file in module directory for integrity verification - Simplify freeze command to only pin transitive dependency versions - Checksum mismatch reports warning instead of automatic re-download - Remove resolved open question about checksum verification Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Clarify module resolution rules for version changes vs local modifications [ci skip] - Added Resolution Rules table clearly specifying behavior for each scenario: - Version change (local unmodified): automatically replace with declared version - Local modification (checksum mismatch): warn and protect local changes - Use -force flag to override locally modified modules - Updated install command behavior to reflect checksum verification - Updated Technical Details with expanded resolution flow Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * ADR v2.3: Fix resolution rules consistency and add version entry [ci skip] - Fixed Resolution Rules table: locally modified modules will NOT be replaced unless -force is used (was incorrectly saying "will be replaced") - Added Version 2.3 changelog entry documenting: - Resolution Rules table with clear behavior matrix - Local modification protection with -force flag - Simplified storage model (single version per module) - .checksum file for fast integrity verification Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Fix Version 2.1 date typo: 2025 -> 2024 [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Add module system client specification [ci skip] Specification for Nextflow module system client implementation based on ADR 20251114-module-system.md. Covers: P1 (Core): - Install and use registry modules via @scope/name syntax - Run modules directly from CLI without wrapper workflow - Structured tool arguments replacing ext.args pattern P2 (Important): - Module version management and freeze command - Module integrity protection with checksum validation P3 (Nice to have): - Remove module command - Search and discover modules - Publish module to registry Registry backend is out of scope (assumed implemented). Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Remove transitive dependency resolution from module system [ci skip] - Remove `freeze` command from CLI - Remove transitive dependency install behavior - Remove orphaned transitive dependency removal from `remove` command - Update rationale and consequences sections - Simplify dependency resolution flow - Update ADR to version 2.4 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update adr/20251114-module-system.md [ci skip] Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update adr/20251114-module-system.md [ci skip] Co-authored-by: Phil Ewels <phil.ewels@seqera.io> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update modules plan [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Remove tools config [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Simplify module system requires definition [ci skip] - Remove plugins, modules, subworkflows from requires block - Spec now focused on process modules only - Dependencies managed via nextflow.config Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Replace tool arguments with module parameters in ADR [ci skip] - Module parameters defined in meta.yaml params section with name, type, description, and example attributes - Removed tools args property and specification - Updated schema and examples throughout Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Update specs + adr [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Resolve some conflicts with nextflow-io/schemas#10 Signed-off-by: Ben Sherman <bentshermann@gmail.com> * Update adr/20251114-module-system.md [ci skip] Co-authored-by: Ben Sherman <bentshermann@gmail.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Extract module parameters to separate spec [ci skip] Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * docs: clarify path name for staged task inputs (#6869) Co-authored-by: Ben Sherman <bentshermann@gmail.com> * Add devcontainer (#6792) Co-authored-by: Phil Ewels <phil.ewels@seqera.io> Co-authored-by: Ben Sherman <bentshermann@gmail.com> * Fix vertical alignment in timeline report (#6794) * Ensure main script is first in the WorkflowRun lineage record (#6845) * Add whitespace rules to .editorconfig (#5606) * docs: Update your first script to use outputs (#6500) * Update AWS CLI install docs to bypass conda install TOS prompt (#6685) * Clarify onError and onComplete handler descriptions (#6709) [ci fast] * Use explicit memory units for LSF executor (#5217) Signed-off-by: Ben Sherman <bentshermann@gmail.com> * Treat LSF job status UNKWN as HOLD (#5756) * Allow boolean params to implicitly default to false (#6764) * Sched core implementation alpha1 (#6242) Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Lorenzo Fontana <fontanalorenz@gmail.com> * Revert version 26.01.1-edge Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Bump copyright 2026 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Handle Azure Batch ActiveJobAndScheduleQuotaReached with retry (#6874) * Handle Azure Batch ActiveJobAndScheduleQuotaReached with retry (#5575) Add retry logic when Azure Batch returns HTTP 409 with ActiveJobAndScheduleQuotaReached error code during job creation, instead of failing immediately. Configurable via azure.batch.maxJobQuotaRetries (default 3) and azure.batch.jobQuotaRetryDelay (default 2 min). Generated by Claude Code Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> * Refactor job quota retry to use Failsafe RetryPolicy Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> --------- Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * [ci fast] Add agent output mode (NXF_AGENT_MODE) Add AI agent-friendly output mode activated via NXF_AGENT_MODE=1. When enabled, Nextflow replaces the interactive ANSI progress display with minimal, structured tagged lines optimized for AI agent context windows: [PIPELINE] name version | profile=X [WORKDIR] /path/to/work [PROCESS hash] name (tag) [WARN] deduplicated warning [ERROR] name with exit/cmd/stderr/workdir [SUCCESS|FAILED] completed=N failed=N cached=N Key changes: - LogObserver interface as common abstraction for AnsiLogObserver and AgentLogObserver - Session holds a single logObserver field instead of two concrete fields - DefaultObserverFactory creates both observer types using the same pattern - AgentLogObserver filters hash-prefixed log lines to avoid duplicating [PROCESS] output - Workflow output (view operator) passes through to stdout in agent mode - Renamed env var from NXF_AGENT to NXF_AGENT_MODE - Added NXF_AGENT_MODE to environment variables documentation * Bump nf-amazon@3.7.1 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Bump nf-azure@1.22.0 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Bump nf-google@1.26.1 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Bump nf-tower@1.21.0 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Bump nf-seqera@0.12.0 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Bump nf-tower@1.21.0 nextflowVersion Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Bump nf-seqera@0.12.0 nextflowVersion and README Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Fix AgentLogObserver println clash with Groovy GDK method Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Add Summary section to plugin README files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * [release] [ci fast] Nextflow version 26.02.0-edge Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Fix docs [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> * Add logStreamId to Tower task record (#6877) Propagate the log stream identifier from TraceRecord to the Tower task record so that Seqera Platform can link tasks to their cloud log streams. Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Don't include container in task trace when containers are not enabled (#6824) * Propagate errors from workflow outputs (#6876) * Fix type detection of CLI params in v2 config parser (#6765) * Add NVMe disk allocation and diskMountPath support (#6879) [ci fast] * Add NVMe disk allocation and diskMountPath support for Seqera executor For task/node allocation the default is /tmp; for nvme allocation the default is the server-configured NVMe mount path (typically /scratch). Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Version 2.7: rename `.checksum` to `.module-info`, remove `@` prefix from module scopes, remove version pinning from `nextflow.config` Signed-off-by: Ben Sherman <bentshermann@gmail.com> * Module System Implementation (#6768) Signed-off-by: jorgee <jorge.ejarque@seqera.io> * minor edits Signed-off-by: Ben Sherman <bentshermann@gmail.com> * Cleanup module run output Signed-off-by: Ben Sherman <bentshermann@gmail.com> * Fix meta.yaml -> meta.yml, module manifest -> module spec Signed-off-by: Ben Sherman <bentshermann@gmail.com> * Fix failing test Signed-off-by: Ben Sherman <bentshermann@gmail.com> * fix tests Signed-off-by: jorgee <jorge.ejarque@seqera.io> * modify module list to skip subdirectories of directories that already contains the .module-info. This folders can't be a module Signed-off-by: jorgee <jorge.ejarque@seqera.io> * Update docs Signed-off-by: Ben Sherman <bentshermann@gmail.com> * Apply suggestion from @pditommaso [ci skip] Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> --------- Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Signed-off-by: Ben Sherman <bentshermann@gmail.com> Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: jorgee <jorge.ejarque@seqera.io> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com> Co-authored-by: Phil Ewels <phil.ewels@seqera.io> Co-authored-by: Ben Sherman <bentshermann@gmail.com> Co-authored-by: Abi <llawlietbagsum@gmail.com> Co-authored-by: Peter Kneale <PeterKneale@gmail.com> Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io> Co-authored-by: Rintze Zelle <78232505+rzelle-lallemand@users.noreply.github.com> Co-authored-by: Colin J. Brislawn <cbrisl@gmail.com> Co-authored-by: Lorenzo Fontana <fontanalorenz@gmail.com> Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Co-authored-by: Edmund Miller <20095261+edmundmiller@users.noreply.github.com> Co-authored-by: jorgee <jorge.ejarque@seqera.io>
1 parent 3d2e4c4 commit a0ac594

64 files changed

Lines changed: 11962 additions & 33 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

adr/20251114-module-system.md

Lines changed: 897 additions & 0 deletions
Large diffs are not rendered by default.

adr/module-spec-schema.json

Lines changed: 467 additions & 0 deletions
Large diffs are not rendered by default.

docs/cli.md

Lines changed: 132 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -262,7 +262,137 @@ $ nextflow secrets set AWS_ACCESS_KEY_ID
262262
$ nextflow secrets delete AWS_ACCESS_KEY_ID
263263
```
264264

265-
See {ref}`cli-secrets` for more information.
265+
See {ref}`cli-secrets` for more information.
266+
267+
## Module management
268+
269+
:::{versionadded} 26.04.0
270+
:::
271+
272+
The `module` command enables working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and sharing improvements with the community.
273+
274+
Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use.
275+
276+
### Searching for modules
277+
278+
The `module search` command queries the module registry to discover available modules by keyword or name.
279+
280+
Use this to find modules for specific tasks, explore available tools, or discover community modules.
281+
282+
```console
283+
$ nextflow module search alignment
284+
$ nextflow module search "quality control" -limit 10
285+
$ nextflow module search bwa -output json
286+
```
287+
288+
Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for JSON-formatted output.
289+
290+
See {ref}`cli-module-search` for more information.
291+
292+
### Installing modules
293+
294+
The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL.
295+
296+
Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions.
297+
298+
```console
299+
$ nextflow module install nf-core/fastqc
300+
$ nextflow module install nf-core/fastqc -version 1.0.0
301+
```
302+
303+
The installed module will be available in `modules/nf-core/fastqc`.
304+
305+
Use the `-force` flag to reinstall a module even if local modifications exist.
306+
307+
See {ref}`cli-module-install` for more information.
308+
309+
### Listing modules
310+
311+
The `module list` command displays all modules currently installed in your project, showing their versions and integrity status.
312+
313+
Use this to review installed modules, check module versions, or detect local modifications.
314+
315+
```console
316+
$ nextflow module list
317+
$ nextflow module list -output json
318+
```
319+
320+
The output shows each module's name, installed version, and whether it has been modified locally. Use `-o json` for JSON-formatted output.
321+
322+
See {ref}`cli-module-list` for more information.
323+
324+
### Viewing module information
325+
326+
The `module info` command displays detailed metadata and usage information for a specific module from the registry.
327+
328+
Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module.
329+
330+
```console
331+
$ nextflow module info nf-core/fastqc
332+
$ nextflow module info nf-core/fastqc -version 1.0.0
333+
$ nextflow module info nf-core/fastqc -output json
334+
```
335+
336+
The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access.
337+
338+
See {ref}`cli-module-info` for more information.
339+
340+
### Running modules directly
341+
342+
The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing.
343+
344+
Use this to quickly run a module, test module functionality, or execute one-off data processing tasks.
345+
346+
```console
347+
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
348+
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0
349+
```
350+
351+
The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.):
352+
353+
```console
354+
$ nextflow module run nf-core/salmon \
355+
--reads reads.fq \
356+
--index salmon_index \
357+
-profile docker \
358+
-resume
359+
```
360+
361+
Process inputs can be specified like params on the command line. For example, `--reads reads.fq` corresponds to the `reads` input in the `nf-core/salmon` module. Run `nextflow module info nf-core/salmon` to see the available params for the module.
362+
363+
See {ref}`cli-module-run` for more information.
364+
365+
### Removing modules
366+
367+
The `module remove` command deletes modules from your project, removing local files and configuration entries.
368+
369+
Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline.
370+
371+
```console
372+
$ nextflow module remove nf-core/fastqc
373+
$ nextflow module remove nf-core/fastqc -keep-files
374+
```
375+
376+
By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files.
377+
378+
See {ref}`cli-module-remove` for more information.
379+
380+
### Publishing modules
381+
382+
The `module publish` command uploads modules to a registry, making them available for others to install and use.
383+
384+
Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization.
385+
386+
```console
387+
$ nextflow module publish myorg/my-module
388+
$ nextflow module publish myorg/my-module -dry-run
389+
```
390+
391+
Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. The module must include `main.nf`, `meta.yml`, and `README.md` files.
392+
393+
Use `-dry-run` to validate your module structure without uploading.
394+
395+
See {ref}`cli-module-publish` for more information.
266396

267397
## Configuration and validation
268398

@@ -384,7 +514,7 @@ Use this to understand input/output relationships between tasks, trace data flow
384514
$ nextflow lineage
385515
```
386516

387-
See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information.
517+
See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information.
388518

389519
## Seqera Platform
390520

docs/module.md

Lines changed: 157 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -279,8 +279,164 @@ This feature requires the use of a local or shared file system for the pipeline
279279

280280
## Sharing modules
281281

282-
Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. While Nextflow does not provide an explicit mechanism for sharing modules, there are several ways to do it:
282+
Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules:
283283

284+
- Use the Nextflow module registry (recommended, see below)
284285
- Simply copy the module files into your pipeline repository
285286
- Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy
286287
- Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community
288+
289+
(module-registry)=
290+
291+
## Registry-based modules
292+
293+
:::{versionadded} 26.04.0
294+
:::
295+
296+
Nextflow provides a module registry that enables you to install, publish, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow language.
297+
298+
### Installing modules from a registry
299+
300+
Use the `module install` command to download modules from a registry:
301+
302+
```console
303+
$ nextflow module install nf-core/fastqc
304+
$ nextflow module install nf-core/fastqc -version 1.0.0
305+
```
306+
307+
Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path:
308+
309+
```nextflow
310+
include { FASTQC } from 'nf-core/fastqc'
311+
312+
workflow {
313+
reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz')
314+
FASTQC(reads)
315+
}
316+
```
317+
318+
### Running modules directly
319+
320+
For ad-hoc tasks or testing, you can run a module directly without creating a workflow:
321+
322+
```console
323+
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
324+
```
325+
326+
This command accepts all standard `nextflow run` options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed.
327+
328+
### Discovering modules
329+
330+
Search for available modules using the `module search` command:
331+
332+
```console
333+
$ nextflow module search alignment
334+
$ nextflow module search "quality control" -limit 10
335+
```
336+
337+
List installed modules in your project:
338+
339+
```console
340+
$ nextflow module list
341+
```
342+
343+
### Module checksum verification
344+
345+
Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites:
346+
347+
```console
348+
$ nextflow module install nf-core/fastqc -version 1.1.0
349+
Warning: Module nf-core/fastqc has local modifications. Use -force to override.
350+
```
351+
352+
Use the `-force` flag to override local modifications when needed.
353+
354+
### Removing modules
355+
356+
Use the `module remove` command to uninstall a module:
357+
358+
```console
359+
$ nextflow module remove nf-core/fastqc
360+
```
361+
362+
By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour:
363+
364+
- `-keep-files`: Remove the `.module-info` file created at install but keep the rest of files
365+
- `-force`: Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications
366+
367+
### Viewing module information
368+
369+
Use the `module info` command to display metadata and a usage template for a module:
370+
371+
```console
372+
$ nextflow module info nf-core/fastqc
373+
$ nextflow module info nf-core/fastqc -version 1.0.0
374+
```
375+
376+
The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-o json` to get machine-readable output.
377+
378+
### Publishing modules
379+
380+
To share your own modules, use the `module publish` command:
381+
382+
```console
383+
$ nextflow module publish myorg/my-module
384+
```
385+
386+
The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files.
387+
388+
Your module directory must include:
389+
390+
- `main.nf`: The module entry point
391+
- `meta.yml`: Module spec (name, description, version, etc.)
392+
- `README.md`: Module documentation
393+
394+
Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration:
395+
396+
```groovy
397+
registry {
398+
apiKey = 'YOUR_REGISTRY_TOKEN'
399+
}
400+
```
401+
402+
Use `-dry-run` to validate your module structure without uploading:
403+
404+
```console
405+
$ nextflow module publish myorg/my-module -dry-run
406+
```
407+
408+
### Registry configuration
409+
410+
By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries:
411+
412+
```groovy
413+
registry {
414+
url = [
415+
'https://private.registry.myorg.com',
416+
'https://registry.nextflow.io'
417+
]
418+
apiKey = '${MYORG_TOKEN}'
419+
}
420+
```
421+
422+
Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry.
423+
424+
### Module directory structure
425+
426+
Registry modules follow a standard directory structure:
427+
428+
```
429+
modules/
430+
└── scope/
431+
└── module-name/
432+
├── .module-info # Integrity checksum (generated automatically)
433+
├── README.md # Documentation (required for publishing)
434+
├── main.nf # Module script (required)
435+
├── meta.yml # Module spec (required for publishing)
436+
├── resources/ # Optional: module binaries and resources
437+
└── templates/ # Optional: process templates
438+
```
439+
440+
The `modules/` directory should be committed to your Git repository to ensure reproducibility.
441+
442+
See the {ref}`cli-page` documentation for more information about module commands.

0 commit comments

Comments
 (0)