Skip to content
Merged
Show file tree
Hide file tree
Changes from 74 commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
59983c1
Add Nextflow Development Constitution
pditommaso Nov 17, 2025
fccfc74
Module adr v1
pditommaso Nov 24, 2025
20020a7
Module adr v2
pditommaso Nov 25, 2025
226d4ee
ADR: Nextflow remote modules system v2.1
pditommaso Dec 11, 2025
513fa6a
Merge master into 251117-module-system [ci skip]
pditommaso Dec 11, 2025
beaaa08
Merge branch 'master' into 251117-module-system
pditommaso Dec 15, 2025
8e681e9
Add detailed CLI command descriptions to module system ADR [ci skip]
pditommaso Dec 15, 2025
c6ca725
Merge branch 'master' into 251117-module-system
pditommaso Jan 6, 2026
5df12cd
Simplify module spec: unify dependencies under requires
pditommaso Jan 6, 2026
01c777a
Add structured tool arguments to module spec schema
pditommaso Jan 6, 2026
7e12f3a
Update adr/20251114-module-system.md [ci skip]
pditommaso Jan 7, 2026
5a01d00
Update adr/20251114-module-system.md [ci skip]
pditommaso Jan 7, 2026
4779d50
Update adr/20251114-module-system.md [ci skip]
pditommaso Jan 7, 2026
5b1d4e5
Update adr/20251114-module-system.md [ci skip]
pditommaso Jan 14, 2026
a06ab7a
Fix inconsistencies in module system ADR [ci skip]
pditommaso Jan 14, 2026
f7dab8d
Fix registry API path in comparison table [ci skip]
pditommaso Jan 14, 2026
0242a4a
Fix checksum format: sha256- -> sha256: [ci skip]
pditommaso Jan 14, 2026
5461f5b
Update module API to match registry implementation [ci skip]
pditommaso Jan 14, 2026
926052c
Update adr [ci skip]
pditommaso Jan 15, 2026
ec187fd
Simplify module storage and verification model [ci skip]
pditommaso Jan 15, 2026
46c6060
Update adr/20251114-module-system.md [ci skip]
pditommaso Jan 15, 2026
1b0ab1a
Clarify module resolution rules for version changes vs local modifica…
pditommaso Jan 15, 2026
7a1e62e
ADR v2.3: Fix resolution rules consistency and add version entry [ci …
pditommaso Jan 15, 2026
c6eaabb
Fix Version 2.1 date typo: 2025 -> 2024 [ci skip]
pditommaso Jan 15, 2026
daf01be
Merge branch 'master' into 251117-module-system [ci skip]
pditommaso Jan 15, 2026
c8bc50d
Add module system client specification [ci skip]
pditommaso Jan 15, 2026
68b202f
Remove transitive dependency resolution from module system [ci skip]
pditommaso Jan 15, 2026
d118d5a
Update adr/20251114-module-system.md [ci skip]
pditommaso Jan 21, 2026
ffe1116
Update adr/20251114-module-system.md [ci skip]
pditommaso Jan 21, 2026
806a41f
Update modules plan [ci skip]
pditommaso Jan 19, 2026
f7e9def
Remove tools config [ci skip]
pditommaso Jan 21, 2026
474eb75
Simplify module system requires definition [ci skip]
pditommaso Jan 22, 2026
5581257
Replace tool arguments with module parameters in ADR [ci skip]
pditommaso Jan 23, 2026
903c740
Update specs + adr [ci skip]
pditommaso Jan 23, 2026
82d90d6
Resolve some conflicts with nextflow-io/schemas#10
bentsherman Jan 25, 2026
d86c211
Update adr/20251114-module-system.md [ci skip]
pditommaso Jan 26, 2026
8fa6478
Extract module parameters to separate spec [ci skip]
pditommaso Jan 28, 2026
3bce3eb
Merge branch 'master' into 251117-module-system
jorgee Feb 10, 2026
6cfa59f
Merge branch 'master' into 251117-module-system
jorgee Feb 25, 2026
1d6005b
docs: clarify path name for staged task inputs (#6869)
LarytheLord Feb 26, 2026
d9952fb
Add devcontainer (#6792)
PeterKneale Feb 26, 2026
481b90f
Fix vertical alignment in timeline report (#6794)
PeterKneale Feb 26, 2026
4ad9e34
Ensure main script is first in the WorkflowRun lineage record (#6845)
jorgee Feb 26, 2026
4d8ef30
Add whitespace rules to .editorconfig (#5606)
ewels Feb 26, 2026
0228a4e
docs: Update your first script to use outputs (#6500)
christopher-hakkaart Feb 26, 2026
a1a046a
Update AWS CLI install docs to bypass conda install TOS prompt (#6685)
rzelle-lallemand Feb 27, 2026
3830af3
Clarify onError and onComplete handler descriptions (#6709) [ci fast]
colinbrislawn Feb 27, 2026
28458b5
Use explicit memory units for LSF executor (#5217)
bentsherman Feb 27, 2026
911d399
Treat LSF job status UNKWN as HOLD (#5756)
bentsherman Feb 27, 2026
d17e6c1
Allow boolean params to implicitly default to false (#6764)
bentsherman Feb 27, 2026
d0a9fbd
Sched core implementation alpha1 (#6242)
pditommaso Feb 27, 2026
153d937
Revert version 26.01.1-edge
pditommaso Feb 27, 2026
5c67090
Bump copyright 2026
pditommaso Feb 27, 2026
b0a1877
Handle Azure Batch ActiveJobAndScheduleQuotaReached with retry (#6874)
adamrtalbot Feb 27, 2026
d8a7fc2
[ci fast] Add agent output mode (NXF_AGENT_MODE)
edmundmiller Feb 28, 2026
f13573d
Bump nf-amazon@3.7.1
pditommaso Feb 28, 2026
837af5f
Bump nf-azure@1.22.0
pditommaso Feb 28, 2026
6b58251
Bump nf-google@1.26.1
pditommaso Feb 28, 2026
daf9f36
Bump nf-tower@1.21.0
pditommaso Feb 28, 2026
3c8752a
Bump nf-seqera@0.12.0
pditommaso Feb 28, 2026
53df43c
Bump nf-tower@1.21.0 nextflowVersion
pditommaso Feb 28, 2026
7a17237
Bump nf-seqera@0.12.0 nextflowVersion and README
pditommaso Feb 28, 2026
d2cd6f0
Fix AgentLogObserver println clash with Groovy GDK method
pditommaso Feb 28, 2026
e79eb98
Add Summary section to plugin README files
pditommaso Feb 28, 2026
15e1eef
[release] [ci fast] Nextflow version 26.02.0-edge
pditommaso Feb 28, 2026
51098d3
Fix docs [ci skip]
pditommaso Mar 2, 2026
9a3c847
Add logStreamId to Tower task record (#6877)
pditommaso Mar 2, 2026
4bf4b75
Don't include container in task trace when containers are not enabled…
bentsherman Mar 2, 2026
3137069
Propagate errors from workflow outputs (#6876)
bentsherman Mar 2, 2026
8644993
Fix type detection of CLI params in v2 config parser (#6765)
bentsherman Mar 2, 2026
7d772ea
Add NVMe disk allocation and diskMountPath support (#6879) [ci fast]
pditommaso Mar 2, 2026
b0feed3
Merge remote-tracking branch 'origin/master' into 251117-module-syste…
jorgee Mar 4, 2026
f145979
Version 2.7: rename `.checksum` to `.module-info`, remove `@` prefix …
bentsherman Mar 10, 2026
583285c
Module System Implementation (#6768)
jorgee Mar 10, 2026
b2fa0f8
minor edits
bentsherman Mar 13, 2026
9b647ef
Cleanup module run output
bentsherman Mar 13, 2026
00824f3
Fix meta.yaml -> meta.yml, module manifest -> module spec
bentsherman Mar 13, 2026
24e14ee
Merge branch 'master' into 251117-module-system
bentsherman Mar 13, 2026
2e183b5
Fix failing test
bentsherman Mar 13, 2026
0a7ef22
fix tests
jorgee Mar 16, 2026
6c15bc8
modify module list to skip subdirectories of directories that already…
jorgee Mar 16, 2026
3be2f02
Update docs
bentsherman Mar 16, 2026
512b27d
Merge branch 'master' into 251117-module-system
bentsherman Mar 16, 2026
34426ea
Apply suggestion from @pditommaso [ci skip]
pditommaso Mar 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
897 changes: 897 additions & 0 deletions adr/20251114-module-system.md

Large diffs are not rendered by default.

467 changes: 467 additions & 0 deletions adr/module-spec-schema.json

Large diffs are not rendered by default.

132 changes: 130 additions & 2 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,135 @@ $ nextflow secrets set AWS_ACCESS_KEY_ID
$ nextflow secrets delete AWS_ACCESS_KEY_ID
```

See {ref}`cli-secrets` for more information.
See {ref}`cli-secrets` for more information.

## Module management

:::{versionadded} 26.04.0
:::

Module management commands enable working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and spreading improvements throughout the community.

Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use.

### Installing modules

The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL.

Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions.

```console
$ nextflow module install nf-core/fastqc
$ nextflow module install nf-core/fastqc -version 1.0.0
```

After installation, module will be available in `modules/nf-core/fastqc`.

Use the `-force` flag to reinstall a module even if local modifications exist.

See {ref}`cli-module-install` for more information.

### Running modules directly

The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing.

Use this to quickly run a module, test module functionality, or execute one-off data processing tasks.

```console
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0
```

The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.):

```console
$ nextflow module run nf-core/salmon \
--reads reads.fq \
--index salmon_index \
Comment thread
bentsherman marked this conversation as resolved.
Outdated
-profile docker \
-resume
```

See {ref}`cli-module-run` for more information.

### Listing modules

The `module list` command displays all modules currently installed in your project, showing their versions and integrity status.

Use this to review installed modules, check module versions, or detect local modifications.

```console
$ nextflow module list
$ nextflow module list -output json
```

The output shows each module's name, installed version, and whether it has been modified locally. Use `-json` for machine-readable output suitable for scripting.

See {ref}`cli-module-list` for more information.

### Searching for modules

The `module search` command queries the module registry to discover available modules by keyword or name.

Use this to find modules for specific tasks, explore available tools, or discover community contributions.

```console
$ nextflow module search alignment
$ nextflow module search "quality control" -limit 10
$ nextflow module search bwa -output json
```

Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for programmatic access.

See {ref}`cli-module-search` for more information.

### Viewing module information

The `module info` command displays detailed metadata and usage information for a specific module from the registry.

Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module.

```console
$ nextflow module info nf-core/fastqc
$ nextflow module info nf-core/fastqc -version 1.0.0
$ nextflow module info nf-core/fastqc -output json
```

The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access.

See {ref}`cli-module-info` for more information.

### Removing modules

The `module remove` command deletes modules from your project, removing local files and configuration entries.

Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline.

```console
$ nextflow module remove nf-core/fastqc
$ nextflow module remove nf-core/fastqc -keep-files
```

By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files.

See {ref}`cli-module-remove` for more information.

### Publishing modules

The `module publish` command uploads modules to a registry, making them available for others to install and use.

Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization.

```console
$ nextflow module publish myorg/my-module
$ nextflow module publish myorg/my-module -dry-run
```

Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` in the Nextflow configuration. The module must include `main.nf`, `meta.yaml`, and `README.md` files.

Use `-dry-run` to validate your module structure without uploading.

See {ref}`cli-module-publish` for more information.

## Configuration and validation

Expand Down Expand Up @@ -384,7 +512,7 @@ Use this to understand input/output relationships between tasks, trace data flow
$ nextflow lineage
```

See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information.
See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information.

## Seqera Platform

Expand Down
159 changes: 158 additions & 1 deletion docs/module.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,8 +279,165 @@ This feature requires the use of a local or shared file system for the pipeline

## Sharing modules

Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. While Nextflow does not provide an explicit mechanism for sharing modules, there are several ways to do it:
Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules:

- Use the Nextflow module registry (recommended, see below)
- Simply copy the module files into your pipeline repository
- Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy
- Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community

(module-registry)=

## Registry-based modules

:::{versionadded} 26.04.0
:::

Nextflow provides a module registry that enables you to install, share, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow DSL.

### Installing modules from a registry

Use the `module install` command to download modules from a registry:

```console
$ nextflow module install nf-core/fastqc
$ nextflow module install nf-core/fastqc -version 1.0.0
```

Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path:

```nextflow
include { FASTQC } from 'nf-core/fastqc'

workflow {
reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz')
FASTQC(reads)
}
```

### Running modules directly

For ad-hoc tasks or testing, you can run a module directly without creating a workflow:

```console
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
```

This command accepts all standard Nextflow options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed.


### Discovering modules

Search for available modules using the `module search` command:

```console
$ nextflow module search alignment
$ nextflow module search "quality control" -limit 10
```

List installed modules in your project:

```console
$ nextflow module list
```

### Module checksum verification

Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites:

```console
$ nextflow module install nf-core/fastqc -version 1.1.0
Warning: Module nf-core/fastqc has local modifications. Use -force to override.
```

Use the `-force` flag to override local modifications when needed.

### Removing modules

Use the `module remove` command to uninstall a module:

```console
$ nextflow module remove nf-core/fastqc
```

By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour:

- `-keep-files` — Remove the `.module-info` file created at install but keep the rest of files
- `-force` — Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications

### Viewing module information

Use the `module info` command to display metadata and a usage template for a module:

```console
$ nextflow module info nf-core/fastqc
$ nextflow module info nf-core/fastqc -version 1.0.0
```

The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-json` to get machine-readable output.

### Publishing modules

To share your own modules, use the `module publish` command:

```console
$ nextflow module publish myorg/my-module
```

The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files.

Your module directory must include:

- `main.nf`: The module entry point
- `meta.yaml`: Module metadata (name, description, version, etc.)
- `README.md`: Module documentation

Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration:

```groovy
registry {
apiKey = 'YOUR_REGISTRY_TOKEN'
}
```

Use `-dry-run` to validate your module structure without uploading:

```console
$ nextflow module publish myorg/my-module -dry-run
```

### Registry configuration

By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries:

```groovy
registry {
url = [
'https://private.registry.myorg.com',
'https://registry.nextflow.io'
]
apiKey = '${MYORG_TOKEN}'
}
```

Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry.

### Module directory structure

Registry modules follow a standard directory structure:

```
modules/
└── scope/
└── module-name/
├── .checksum # Integrity checksum (generated automatically)
├── README.md # Documentation (required for publishing)
├── main.nf # Module entry point (required)
├── meta.yaml # Module metadata (required for publishing)
├── resources/ # Optional: module binaries and resources
└── templates/ # Optional: process templates
```

The `modules/` directory should be committed to your Git repository to ensure reproducibility.

See the {ref}`cli-page` documentation for more information about module commands.
Loading
Loading