Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,17 @@ sharing
vscode
```

```{toctree}
:hidden:
:caption: Modules
:maxdepth: 1

modules/modules
modules/using-modules
modules/developing-modules
modules/module-registry
```
Comment thread
bentsherman marked this conversation as resolved.

```{toctree}
:hidden:
:caption: Software dependencies
Expand Down
162 changes: 5 additions & 157 deletions docs/module.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,164 +283,12 @@ When {ref}`wave-page` is enabled, the contents of the module `resources/` direct

## Sharing modules

Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules:
Nextflow makes modules easy to share and re-use across different pipelines, helping eliminate duplicate work and spread improvements throughout the community.
There are several ways to share modules:

- Use the Nextflow module registry (recommended, see below)
- Simply copy the module files into your pipeline repository
- Use the {ref}`Nextflow module registry <modules-page>` (recommended)
- Copy the module files into your pipeline repository
- Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy
- Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community

(module-registry)=

## Registry-based modules

:::{versionadded} 26.04.0
:::

Nextflow provides a module registry that enables you to install, publish, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow language.

### Installing modules from a registry

Use the `module install` command to download modules from a registry:

```console
$ nextflow module install nf-core/fastqc
$ nextflow module install nf-core/fastqc -version 1.0.0
```

Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path:

```nextflow
include { FASTQC } from 'nf-core/fastqc'

workflow {
reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz')
FASTQC(reads)
}
```

### Running modules directly

For ad-hoc tasks or testing, you can run a module directly without creating a workflow:

```console
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
```

This command accepts all standard `nextflow run` options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed.

### Discovering modules

Search for available modules using the `module search` command:

```console
$ nextflow module search alignment
$ nextflow module search "quality control" -limit 10
```

List installed modules in your project:

```console
$ nextflow module list
```

### Module checksum verification

Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites:

```console
$ nextflow module install nf-core/fastqc -version 1.1.0
Warning: Module nf-core/fastqc has local modifications. Use -force to override.
```

Use the `-force` flag to override local modifications when needed.

### Removing modules

Use the `module remove` command to uninstall a module:

```console
$ nextflow module remove nf-core/fastqc
```

By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour:

- `-keep-files`: Remove the `.module-info` file created at install but keep the rest of files
- `-force`: Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications

### Viewing module information

Use the `module info` command to display metadata and a usage template for a module:

```console
$ nextflow module info nf-core/fastqc
$ nextflow module info nf-core/fastqc -version 1.0.0
```

The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-o json` to get machine-readable output.

### Publishing modules

To share your own modules, use the `module publish` command:

```console
$ nextflow module publish myorg/my-module
```

The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files.

Your module directory must include:

- `main.nf`: The module entry point
- `meta.yml`: Module spec (name, description, version, etc.)
- `README.md`: Module documentation

Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration:

```groovy
registry {
apiKey = 'YOUR_REGISTRY_TOKEN'
}
```

Use `-dry-run` to validate your module structure without uploading:

```console
$ nextflow module publish myorg/my-module -dry-run
```

### Registry configuration

By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries:

```groovy
registry {
url = [
'https://private.registry.myorg.com',
'https://registry.nextflow.io'
]
apiKey = '${MYORG_TOKEN}'
}
```

Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry.

### Module directory structure

Registry modules follow a standard directory structure:

```
modules/
└── scope/
└── module-name/
├── .module-info # Integrity checksum (generated automatically)
├── README.md # Documentation (required for publishing)
├── main.nf # Module script (required)
├── meta.yml # Module spec (required for publishing)
├── resources/ # Optional: module binaries and resources
└── templates/ # Optional: process templates
```

The `modules/` directory should be committed to your Git repository to ensure reproducibility.

See the {ref}`cli-page` documentation for more information about module commands.
See {ref}`modules-page` for an overview of the module registry, {ref}`using-modules-page` to learn how to install and manage registry modules, and {ref}`dev-modules-page` to learn how to create and publish your own modules.
166 changes: 166 additions & 0 deletions docs/modules/developing-modules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
(dev-modules-page)=

# Developing modules

Learn how to create modules for sharing through the Nextflow module registry.
For information about modules syntax, see {ref}`module-page`.

(dev-modules-creating)=

## Creating a module

Use the `module create` command to scaffold a new module with the required files:

```console
$ nextflow module create myorg/my-module
```

If you omit the name, the command prompts you for details:

```console
$ nextflow module create
```

The command creates a module directory with the following files:

- `main.nf`: The module script containing your process definition.
- `meta.yml`: The module spec describing metadata, inputs, and outputs.
- `README.md`: Documentation for the module.

See {ref}`cli-module-create` for the full command reference.

(dev-modules-structure)=

## Module structure

Registry modules follow a standard directory structure:

```
modules/
└── myorg/
└── my-module/
├── .module-info # Integrity checksum (generated at install/creation)
├── README.md # Documentation (required for publishing)
├── main.nf # Module script (required)
├── meta.yml # Module spec (required for publishing)
├── resources/ # Optional: module binaries and resources
└── templates/ # Optional: process templates
```

### main.nf

The `main.nf` file contains the process (or workflow/function) definition. For example, a simple module wrapping FastQC:

```nextflow
process FASTQC {
tag "$meta.id"
label 'process_medium'

conda 'bioconda::fastqc=0.12.1'
container "${ workflow.containerEngine == 'singularity'
? 'https://depot.galaxyproject.org/singularity/fastqc:0.12.1--hdfd78af_0'
: 'biocontainers/fastqc:0.12.1--hdfd78af_0' }"

input:
tuple val(meta), path(reads)

output:
tuple val(meta), path("*.html"), emit: html
tuple val(meta), path("*.zip") , emit: zip

script:
"""
fastqc $reads --threads $task.cpus
"""
}
```

### meta.yml

The `meta.yml` file describes the module's metadata, including its name, version, description, authors, and input/output specifications. Publishing requires this file, and the registry uses it to display module information and generate usage templates.

### README.md

The `README.md` file provides documentation for the module. It should describe what the module does, the tools it wraps, and any configuration requirements.

### templates

Include process script {ref}`templates <process-template>` alongside a module in the `templates/` directory.
See {ref}`module-templates` for details.

### resources

Modules can include binary scripts in the `resources/usr/bin/` directory that are locally scoped to the module's processes. See {ref}`module-binaries` for details.

(dev-modules-spec)=

## Generating a module spec

Use the `module spec` command to generate or update the `meta.yml` file from the module's `main.nf`:

```console
$ nextflow module spec myorg/my-module
```

Provide metadata fields directly to avoid `TODO` placeholders in the generated file:

```console
$ nextflow module spec \
-namespace myorg \
-version 1.0.0 \
-description "Quality control of raw sequencing reads" \
-license MIT \
-author "@myname" \
./modules/myorg/my-module
```

Use `-dry-run` to preview the generated spec without writing to disk:

```console
$ nextflow module spec -dry-run myorg/my-module
```

If a `meta.yml` already exists, the command incorporates its content into the new file.

See {ref}`cli-module-spec` for the full command reference.

(dev-modules-validate)=

## Validating a module

Use the `module validate` command to check that a module is ready for publishing:

```console
$ nextflow module validate myorg/my-module
```

The command verifies that:

- All required files are present (`main.nf`, `meta.yml`, `README.md`).
- The module spec contains all required fields (name, version, description, and license).

See {ref}`cli-module-validate` for the full command reference.

(dev-modules-testing)=

## Testing a module

Before publishing, test your module by running it directly:

```console
$ nextflow module run myorg/my-module --input 'test-data/*.fastq.gz'
```

The command executes the module as a standalone workflow, allowing you to verify that inputs are correctly declared, the process runs successfully, and outputs appear as expected.

For more thorough testing, create a small wrapper workflow that exercises the module:

```nextflow
include { MY_MODULE } from './modules/myorg/my-module'

workflow {
input_ch = Channel.fromFilePairs('test-data/*_{1,2}.fastq.gz')
MY_MODULE(input_ch)
MY_MODULE.out.results.view()
}
```
Loading
Loading