- Authors: Paolo Di Tommaso
- Status: draft
- Date: 2025-01-06
- Tags: modules, dsl, registry, versioning, architecture
- Version: 2.7
- Renamed
.checksumto.module-info: Leaves room for additional properties in the future - Removed
@prefix from module scopes: Local modules are distinguished from remote modules by presence/absence of./prefix - Removed version pinning from config: Installed module versions are now inferred from the
meta.yamlof each module in themodules/directory instead of being declared innextflow.config
- Removed module parameters: Module parameters specification moved to separate spec document.
- Module parameters: Replaced structured tool arguments with general module parameters defined in
meta.yaml - Simplified tools section: Removed
argsproperty from tools; tool arguments now configured via module parameters - Simplified
requiresblock: Removedplugins,modules, andsubworkflowssub-properties;requiresnow only containsnextflowversion constraint - Process modules focus: Removed sub-workflow references; spec is now focused on process modules only
- Removed transitive dependency resolution: Module dependencies are explicit only; no automatic transitive resolution
- Removed
freezecommand: No longer needed without transitive dependency management - Simplified model: Each module explicitly declares its dependencies in
nextflow.config
- Resolution Rules table: Added clear table specifying behavior for each combination of local state and declared version
- Local modification protection: Locally modified modules (checksum mismatch) are NOT overridden unless
-forceflag is used - Simplified storage model: Single version per module locally (
modules/@scope/name/without version in path) .checksumfile: Registry checksum cached locally for fast integrity verification without network calls
- Structured tool arguments: Added
argsproperty totoolssection for type-safe argument configuration - New implicit variables:
tools.<toolname>.args.<argname>returns formatted flag+value;tools.<toolname>.argsreturns all args concatenated - Deprecation: All
ext.*custom directives (e.g.,ext.args,ext.args2,ext.args3,ext.prefix,ext.suffix) deprecated in favor of structured tool arguments - Note: Tool arguments replaced by module parameters in v2.5
- Unified dependencies: Consolidated
components,dependencies, andrequiresinto singlerequiresfield - Unified version syntax:
[scope/]name[@constraint]format across plugins and modules - Deprecation:
componentsfield deprecated (use top-levelmodulesinstead)
Nextflow supports local script inclusion via include directive but lacks standardized mechanisms for package management, versioning, and distribution of reusable process definitions. This limits code reuse and reproducibility across the ecosystem.
Discussion/request goes back to at least 2019, see GitHub issues #1376, #1463 and #4122.
Implement a module system with four core capabilities:
- Remote module inclusion via registry
- Semantic versioning with dependency resolution
- Unified Nextflow Registry (rebrand existing Nextflow registry)
- First-class CLI support (install, publish, search, list, remove, run)
DSL Syntax:
// Include from registry (scoped module name without `./` prefix)
include { BWA_ALIGN } from 'nf-core/bwa-align'
// Existing file-based includes remain supported
include { MY_PROCESS } from './modules/my-process.nf'Module Naming: Scoped modules scope/name (e.g., nf-core/salmon, myorg/custom). Local paths supported for backwards compatibility. No nested paths with the module are allowed - each module must have a main.nf as the entry point.
Version Resolution: Installed module versions are inferred from the meta.yaml of each module in the modules/ directory. If a module is not present locally, the latest available version is downloaded from the registry.
Resolution Order:
- Check local
modules/scope/name/exists - Verify integrity against
.module-infofile - Apply resolution rules (see below)
Resolution Rules:
| Local State | Action |
|---|---|
| Missing | Download latest from registry |
| Exists, checksum valid | Use local module (version from meta.yaml) |
| Exists, checksum mismatch | Warn: locally modified, will NOT replace unless -force is used |
Key Behaviors:
- Local modification: When the local module content was manually changed (checksum mismatch with
.module-info), Nextflow warns and does NOT override to prevent accidental loss of local changes - Force flag: Use
-forcewithnextflow module installto override locally modified modules
Resolution Timing: Modules resolved at workflow parse time (after plugin resolution at startup).
Local Storage: Downloaded modules stored in modules/scope/name/ directory in project root (not global cache). Each module must contain a main.nf file as the required entry point. It is intended that module source code will be committed to the pipeline git repository.
Version Format: MAJOR.MINOR.PATCH
- MAJOR: Breaking changes to process signatures, inputs, or outputs
- MINOR: New processes, backward-compatible enhancements
- PATCH: Bug fixes, documentation updates
Registry Configuration (nextflow.config):
registry {
url = 'https://registry.nextflow.io' // Default registry
// allow the use of multiple registry url for resolving module
// across custom registries, e.g.
// url = [ 'https://custom.registry.com', 'https://registry.nextflow.io' ]
auth {
'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}'
'npm.myorg.com' = '${MYORG_TOKEN}'
}
}Module Spec (meta.yaml):
name: nf-core/bwa-align
version: 1.2.4 # This module's version
requires:
nextflow: ">=24.04.0"Version Constraints (unified name@constraint syntax):
name: Any version (latest)name@1.2.3: Exact versionname@>=1.2.3: Greater or equalname@>=1.2.3,<2.0.0: Range (comma-separated)
Version Notation Consistency:
Modules use the same version constraint syntax already supported by both nextflowVersion and plugins:
| Notation | Meaning | nextflowVersion | Plugins | Modules |
|---|---|---|---|---|
| 1.2.3 | Exact version | ✓ | ✓ | ✓ |
| >=1.2.3 | Greater or equal | ✓ | ✓ | ✓ |
| <=1.2.3 | Less or equal | ✓ | ✓ | ✓ |
| >1.2.3 | Greater than | ✓ | ✓ | ✓ |
| <1.2.3 | Less than | ✓ | ✓ | ✓ |
| >=1.2, <2.0 | Range (comma) | ✓ | ✓ | ✓ |
| !=1.2.3 | Not equal | ✓ | - | - |
| 1.2+ | >=1.2.x <2.0 | ✓ | - | - |
| 1.2.+ | >=1.2.0 <1.3.0 | ✓ | - | - |
| ~1.2.3 | >=1.2.3 <1.3.0 | - | ✓ | - |
Using comparison operators (>=, <) with comma-separated ranges provides the same expressive power as
npm-style ^ and ~ notation while maintaining consistency with existing Nextflow version constraint syntax.
This avoids introducing new notation that would require additional parser support.
Module Resolution:
Installed module versions are inferred from the meta.yaml file for each module in the modules/ directory.
Architecture Decision: Extend existing Nextflow registry at registry.nextflow.io to host both plugins and modules.
Current Plugin API (reference: https://registry.nextflow.io/openapi/):
GET /api/v1/plugins # List/search plugins
GET /api/v1/plugins/{pluginId} # Get plugin + all releases
GET /api/v1/plugins/{pluginId}/{version} # Get specific release
GET /api/v1/plugins/{pluginId}/{version}/download/{fileName} # Download artifact
POST /api/v1/plugins/release # Create draft release
POST /api/v1/plugins/release/{releaseId}/upload # Upload artifact
Module API (reference: https://github.com/seqeralabs/plugin-registry/pull/266):
GET /api/modules?query=<text> # Search modules (semantic search)
GET /api/modules/{name} # Get module + latest release
GET /api/modules/{name}/releases # List all releases
GET /api/modules/{name}/{version} # Get specific release
GET /api/modules/{name}/{version}/download # Download module bundle
POST /api/modules/{name} # Publish module version (authenticated)
Note: The {name} parameter includes the namespace prefix (e.g., "nf-core/fastqc").
Registry URL: registry.nextflow.io
Artifact Types:
- Plugins: JAR files with JSON metadata, resolved at startup
- Modules: Source archives (.nf + meta.yaml), resolved at parse time
Benefits:
- Reuses existing infrastructure (HTTP service, S3 storage, authentication)
- Consistent API patterns for both artifact types
- Operational simplicity (one service vs. two)
- Internal module API already partially implemented
Commands:
nextflow module run scope/name # Run a module directly without a wrapper script
nextflow module search <query> # Search registry
nextflow module install scope/name # Install a module
nextflow module list # Show installed vs configured
nextflow module remove scope/name # Remove from config + local cache
nextflow module publish scope/name # Publish to registry (requires api key)General Notes:
- All commands respect the
registry.urlconfiguration for custom registries
Run a module directly without requiring a wrapper workflow script. This command enables standalone execution of any module by automatically mapping command-line arguments to the module's process inputs. If the module is not available locally, it is automatically installed before execution.
Arguments:
scope/name: Module identifier to run (required)
Options:
-version <ver>: Run a specific version (default: latest or configured version)--<input_name> <value>: Map value to the corresponding module process input channel- All standard
nextflow runoptions (e.g.,-profile,-work-dir,-resume, etc.)
Behavior:
- Checks if module is installed locally; if not, downloads from registry
- Parses the module's
main.nfto identify the main process and its input declarations - Validates command-line arguments against the process input declarations
- Generates an implicit workflow that wires CLI arguments to process inputs
- Executes the workflow using standard Nextflow runtime
Input Mapping:
- Named arguments (
--reads,--reference) are mapped to corresponding process inputs - File paths are automatically converted to files for process file inputs
- Multiple values can be provided for inputs expecting collections
- Required inputs without defaults must be provided; optional inputs use declared defaults
Example:
# Run BWA alignment module with input files
nextflow module run nf-core/bwa-align \
--reads 'samples/*_{1,2}.fastq.gz' \
--reference genome.fa
# Run a specific version with Nextflow options
nextflow module run nf-core/fastqc -version 1.0.0 \
--input 'data/*.fastq.gz' \
-profile docker \
-resume
# Run with work directory and output specification
nextflow module run nf-core/salmon \
--reads reads.fq \
--index salmon_index \
-work-dir /tmp/work \
-output-dir results/Search the Nextflow registry for available modules matching the specified query. The search operates against module names, descriptions, tags, and author information. Results are displayed with module name, latest version, description, and download statistics.
Arguments:
<query>: Search term (required) - matches against module metadata
Options:
-limit <n>: Maximum number of results to return (default: 10)-json: Output results in JSON format for programmatic use
Example:
nextflow module search bwa
nextflow module search "alignment" -limit 50Download and install a module to the local modules/ directory.
Arguments:
<scope/name>: Module identifier.
Options:
-version <ver>: Install a specific version (default: latest)-force: Overwrite any local changes
Behavior:
- If
-versionnot specified, queries registry for the latest available version - Checks if local module exists and verifies integrity against
.module-infofile - If local module is unmodified and version differs: replaces with requested version
- If local module was modified (checksum mismatch): warns and aborts unless
-forceis used - Downloads the module archive from the registry
- Extracts to
modules/scope/name/directory - Stores
.module-infofile from registry's X-Checksum response header
Example:
nextflow module install nf-core/bwa-align # Install specific module (latest)
nextflow module install nf-core/salmon -version 1.2.0Display the status of all modules, comparing what is configured in nextflow.config against what is actually installed in the modules/ directory.
Options:
-json: Output in JSON format-outdated: Only show modules with available updates
Output columns:
- Module name (
scope/name) - Installed version (from
modules/scope/name/meta.yaml) - Latest available version (from registry)
- Status indicator (up-to-date, outdated, missing)
Example:
nextflow module list
nextflow module list -outdatedRemove a module from the local modules/ directory.
Arguments:
scope/name: Module identifier to remove (required)
Options:
-keep-files: Remove.module-infofile but keep local files
Behavior:
- Removes the module directory from
modules/scope/name/
Example:
nextflow module remove nf-core/bwa-align
nextflow module remove myorg/custom -keep-filesPublish a module to the Nextflow registry, making it available for others to install. Requires authentication via API key and appropriate permissions for the target scope.
Arguments:
scope/name: Module identifier to publish (required)
Options:
-registry <url>: Target registry URL (default:registry.nextflow.io)-tag <tag>: Additional tags for discoverability-dry-run: Validate without publishing
Behavior:
- Validates
meta.yamlschema and required fields (name, version, description) - Verifies that
main.nfexists and is valid Nextflow syntax - Verifies that
README.mddocumentation is present - Authenticates with registry using configured credentials
- Creates a release draft and uploads the module archive
- Publishes the release, making it available for installation
Requirements:
- Valid
meta.yamlwith name, version, and description main.nfentry point fileREADME.mddocumentation- Authentication token configured in
registry.authorNXF_REGISTRY_TOKEN - Write permission for the target scope
Example:
nextflow module publish myorg/my-process
nextflow module publish myorg/my-process -dry-runDirectory Layout: Everything within the module directory should be uploaded. Module bundle should not exceed 1MB (uncompressed). Typically this is expected to look something like this:
my-module/
├── main.nf # Required: entry point for module
├── meta.yaml # Required: Module spec (version, metadata, I/O specs)
├── README.md # Required: Module description
└── tests/ # Optional tests
Module Spec extension (meta.yaml):
name: nf-core/bwa-align
version: 1.2.4 # This module's version
description: Align reads using BWA-MEM
authors:
- nf-core community
license: MIT
requires:
nextflow: ">=24.04.0"Local Storage Structure:
project-root/
├── nextflow.config
├── main.nf
└── modules/ # Local module cache
├── nf-core/
│ ├── bwa-align/
│ │ ├── .module-info # Cached registry checksum
│ │ ├── meta.yaml
│ │ └── main.nf # Required entry point
│ └── samtools/view/
│ ├── .module-info
│ ├── meta.yaml
│ └── main.nf # Required entry point
└── myorg/
└── custom-process/
├── .module-info
├── meta.yaml
└── main.nf # Required entry point
Module Integrity Verification:
- On install:
.module-infofile created from registry's X-Checksum response header - On run: Local module checksum compared against
.module-infofile - If match: Proceed without network call
- If mismatch: Report warning (module may have been locally modified)
Phase 1: Module schema, local module loading, validation tools
Phase 2: Extend Nextflow registry for modules, implement caching, add install and search commands
Phase 3: Extend DSL parser for from module syntax
Phase 4: Implement publish command with authentication and run command
Phase 5: Advanced features (search UI, language server integration, ontology validation)
Module Resolution Flow:
- Parse
includestatements → extract module names (e.g.,nf-core/bwa-align) - For each module:
a. Check local
modules/scope/name/exists- If exists → read installed version from
modules/scope/name/meta.yaml - If missing → download latest version from registry
b. Verify local module integrity against
.module-infofile - Checksum mismatch → warn and do NOT override (local changes detected)
- If exists → read installed version from
- On download: store module to
modules/scope/name/with.module-infofile - Read
meta.yamlfile: Validates Nextflow requirement → Fail if not fulfilled - Parse module's
main.nffile → make processes available
Security:
- SHA-256 checksum verification on download (stored in
.module-infofile) - Integrity verification on run (local checksum vs
.module-infofile) - Authentication required for publishing
- Support for private registries
Integration with Plugin System:
- Both plugins and modules query same registry
- Single authentication system
- Separate cache locations:
$NXF_HOME/plugins/(global) vsmodules/(per-project)
| Aspect | Plugins | Modules |
|---|---|---|
| Purpose | Extend runtime | Reusable processes |
| Format | JAR files | Source code (.nf) |
| Resolution | Startup | Parse time |
| Metadata | JSON spec | YAML spec |
| Naming | nf-amazon |
nf-core/salmon |
| Cache Location | $NXF_HOME/plugins/ |
modules/scope/name/ |
| Version Config | plugins {} in config |
meta.yaml in modules/ directory |
| Registry Path | /api/v1/plugins/ |
/api/modules/{name} |
Why unified registry?
- Reuses battle-tested infrastructure (HTTP API, S3, auth)
- Single discovery experience for ecosystem
- Lower operational overhead
- Type-specific handling maintains separation of concerns
Why infer versions from meta.yaml instead of pinning in a separate file?
- Simple: install a version once and it is captured in the module files
- Reproducibility via committing the
modules/directory (includingmeta.yaml) to the project git repository - Reduces configuration burden: no need to keep config in sync with installed state
Why parse-time resolution?
- Modules are source code, not compiled artifacts
- Allows inspection/modification for reproducibility
- Enables dependency analysis before execution
Why scoped modules?
- Organization namespacing prevents name collisions (
nf-core/salmonvsmyorg/salmon) - Clear ownership and provenance of modules
- Supports private registries per scope
- Industry-standard pattern (NPM, Terraform, others)
- Enables ecosystem organization by maintainer/organization
Why semantic versioning?
- Clear compatibility guarantees
- Industry standard (npm, cargo, Go modules)
Positive:
- Enables ecosystem-wide code reuse
- Reproducible workflows via committing the
modules/directory (includingmeta.yaml) to the project git repository - Centralized discovery and distribution via unified registry
- Minimal operational overhead (single registry for both plugins and modules)
- Module scoping enables organization namespaces and private registries
- Local
modules/directory provides project isolation - No version duplication: installed
meta.yamlis the single source of truth - Simple module structure: each module has single
main.nfentry point
Negative:
- Registry becomes critical infrastructure (requires HA setup)
- Type-specific handling adds registry complexity
- Parse-time resolution adds latency to workflow startup
- Local
modules/directory duplicates storage across projects (unlike global cache)
Neutral:
- Modules and plugins conceptually distinct but share infrastructure
- Different resolution timing supported by same API
- Related: Plugin Spec ADR
- Inspired by: Go Modules, npm, Cargo
- Related: nf-core modules
This appendix defines the JSON schema for module meta.yaml files. The schema maintains backward compatibility with existing nf-core module metadata patterns while supporting the new Nextflow module system features.
Schema File: module-spec-schema.json
Published URL: https://registry.nextflow.io/schemas/module-spec/v1.0.0
These fields are already widely adopted in the nf-core community and remain fully supported:
| Field | Type | Required | Description |
|---|---|---|---|
name |
string | Yes | Module identifier |
description |
string | Yes | Brief description of module functionality |
keywords |
array[string] | Recommended | Discovery and categorization keywords |
authors |
array[string] | Recommended | Original authors (GitHub handles) |
maintainers |
array[string] | Recommended | Current maintainers |
tools |
array[object] | Conditional | Software tools wrapped by the module |
input |
array/object | Recommended | Input channel specifications |
output |
object/array | Recommended | Output channel specifications |
These fields extend the schema to support the new Nextflow module system:
| Field | Type | Required | Description |
|---|---|---|---|
version |
string | Registry | Semantic version (MAJOR.MINOR.PATCH) |
license |
string | Registry | SPDX license identifier for module code |
requires |
object | Optional | Runtime requirements |
requires.nextflow |
string | Optional | Nextflow version constraint |
The module name must be a fully qualified scoped identifier in scope/name format:
name: nf-core/fastqc
name: nf-core/bwa-mem
name: myorg/custom-alignerNaming Rules:
- Format:
scope/name(e.g.,nf-core/salmon,myorg/custom) - Scope: lowercase alphanumeric with hyphens (organization/owner identifier)
- Name: lowercase alphanumeric with underscores/hyphens (module identifier)
- Pattern:
^[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$
Semantic version following SemVer 2.0.0:
version: "1.0.0"
version: "2.3.1"
version: "1.0.0-beta.1"Version Semantics:
- MAJOR: Breaking changes to process signatures, inputs, or outputs
- MINOR: New processes, backward-compatible enhancements
- PATCH: Bug fixes, documentation updates
Requirement: Mandatory for registry-published modules (scoped names in scope/name format).
Specifies runtime requirements for the module.
requires:
nextflow: ">=24.04.0"requires.nextflow - Nextflow version constraint:
requires:
nextflow: ">=24.04.0" # minimum version
nextflow: ">=24.04.0,<25.0.0" # version rangeDocuments the software tools wrapped by the module:
tools:
- bwa:
description: BWA aligner
homepage: http://bio-bwa.sourceforge.net/
license: ["GPL-3.0-or-later"]
identifier: biotools:bwaTool Properties:
| Property | Required | Description |
|---|---|---|
description |
Yes | Tool description |
homepage |
One of these | Tool homepage URL |
documentation |
One of these | Documentation URL |
tool_dev_url |
One of these | Development/source URL |
doi |
One of these | Publication DOI |
arxiv |
No | arXiv identifier |
license |
Recommended | SPDX license(s) |
identifier |
Recommended | bio.tools identifier |
manual |
No | User manual URL |
The schema supports both nf-core patterns to ensure backward compatibility:
Module Pattern (Tuple-based):
input:
- - meta:
type: map
description: Sample metadata
- reads:
type: file
description: Input FastQ files
ontologies:
- edam: "http://edamontology.org/format_1930"
- - index:
type: directory
description: Reference index
output:
bam:
- - meta:
type: map
description: Sample metadata
- "*.bam":
type: file
description: Aligned BAM file
pattern: "*.bam"
versions:
- versions.yml:
type: file
description: Software versionsChannel Element Properties:
| Property | Type | Description |
|---|---|---|
type |
string | Data type: map, file, directory, string, integer, float, boolean, list, val |
description |
string | Human-readable description |
pattern |
string | File glob pattern or value pattern |
optional |
boolean | Whether input is optional (default: false) |
default |
any | Default value if not provided |
enum |
array | List of allowed values |
ontologies |
array | EDAM or other ontology annotations |
Before (nf-core local):
name: bwa_mem
description: Align reads using BWA-MEM
keywords:
- alignment
- bwa
tools:
- bwa:
description: BWA software
homepage: http://bio-bwa.sourceforge.net/
license: ["GPL-3.0-or-later"]
identifier: biotools:bwa
authors:
- "@drpatelh"
maintainers:
- "@drpatelh"
input:
# ... existing input spec
output:
# ... existing output specAfter (Registry-ready):
name: nf-core/bwa-mem # Added scope prefix
version: "1.0.0" # Added version
description: Align reads using BWA-MEM
keywords:
- alignment
- bwa
license: MIT # Added module license
requires: # Added requirements
nextflow: ">=24.04.0"
tools:
- bwa:
description: BWA software
homepage: http://bio-bwa.sourceforge.net/
license: ["GPL-3.0-or-later"]
identifier: biotools:bwa
authors:
- "@drpatelh"
maintainers:
- "@drpatelh"
input:
# ... unchanged
output:
# ... unchangedUse the schema reference in your meta.yaml:
# yaml-language-server: $schema=https://registry.nextflow.io/schemas/module-spec/v1.0.0
name: nf-core/my-module
version: "1.0.0"
# ...| Feature | nf-core Current | Nextflow Module System |
|---|---|---|
| Simple names | Yes | Yes (local only) |
| Scoped names | No | Yes (registry) |
| Version field | No | Yes (required for registry) |
tools section |
Yes | Yes |
components |
Yes | Deprecated |
requires |
No | Yes (Nextflow version constraint) |
| I/O specifications | Yes | Yes |
| Ontologies | Yes | Yes |
The following attributes from the nf-core meta schema are not supported in the Nextflow module system:
| Attribute | Reason | Alternative |
|---|---|---|
extra_args |
Not adopted in practice by nf-core modules | To be defined |
components |
No longer supported | Module dependencies are managed via nextflow.config |
name: fastqc
description: Run FastQC on sequenced reads
keywords:
- quality control
- qc
- fastq
tools:
- fastqc:
description: FastQC quality metrics
homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/
license: ["GPL-2.0-only"]
identifier: biotools:fastqc
authors:
- "@drpatelh"
maintainers:
- "@drpatelh"
output:
html:
- "*.html":
type: file
description: FastQC HTML report
versions:
- versions.yml:
type: file
description: Software versionsname: nf-core/bwa-align
version: "1.2.4"
description: Align reads to reference genome using BWA-MEM algorithm
keywords:
- alignment
- mapping
- bwa
- bam
- fastq
license: MIT
requires:
nextflow: ">=24.04.0"
tools:
- bwa:
description: |
BWA is a software package for mapping DNA sequences
against a large reference genome.
homepage: http://bio-bwa.sourceforge.net/
documentation: https://bio-bwa.sourceforge.net/bwa.shtml
doi: 10.1093/bioinformatics/btp324
license: ["GPL-3.0-or-later"]
identifier: biotools:bwa
authors:
- "@nf-core"
maintainers:
- "@drpatelh"
- "@maxulysse"
input:
- - meta:
type: map
description: Sample metadata map (e.g., [ id:'sample1', single_end:false ])
- reads:
type: file
description: Input FastQ files
ontologies:
- edam: "http://edamontology.org/format_1930"
- - meta2:
type: map
description: Reference metadata
- index:
type: directory
description: BWA index directory
ontologies:
- edam: "http://edamontology.org/data_3210"
output:
bam:
- - meta:
type: map
description: Sample metadata
- "*.bam":
type: file
description: Aligned BAM file
pattern: "*.bam"
ontologies:
- edam: "http://edamontology.org/format_2572"
versions:
- versions.yml:
type: file
description: Software versions
pattern: "versions.yml"