Merge branch 'master' into 6779-nextflow-prioritizes-50001-over-exitcode-even-when-the-batch-job-is-retried-and-succeeds

Author: bentsherman

adr/20251114-module-system.md

@@ -262,7 +262,137 @@ $ nextflow secrets set AWS_ACCESS_KEY_ID
 $ nextflow secrets delete AWS_ACCESS_KEY_ID
 ```
 
-See {ref}`cli-secrets` for more information. 
+See {ref}`cli-secrets` for more information.
+
+## Module management
+
+:::{versionadded} 26.04.0
+:::
+
+The `module` command enables working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and sharing improvements with the community.
+
+Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use.
+
+### Searching for modules
+
+The `module search` command queries the module registry to discover available modules by keyword or name.
+
+Use this to find modules for specific tasks, explore available tools, or discover community modules.
+
+```console
+$ nextflow module search alignment
+$ nextflow module search "quality control" -limit 10
+$ nextflow module search bwa -output json
+```
+
+Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for JSON-formatted output.
+
+See {ref}`cli-module-search` for more information.
+
+### Installing modules
+
+The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL.
+
+Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions.
+
+```console
+$ nextflow module install nf-core/fastqc
+$ nextflow module install nf-core/fastqc -version 1.0.0
+```
+
+The installed module will be available in `modules/nf-core/fastqc`.
+
+Use the `-force` flag to reinstall a module even if local modifications exist.
+
+See {ref}`cli-module-install` for more information.
+
+### Listing modules
+
+The `module list` command displays all modules currently installed in your project, showing their versions and integrity status.
+
+Use this to review installed modules, check module versions, or detect local modifications.
+
+```console
+$ nextflow module list
+$ nextflow module list -output json
+```
+
+The output shows each module's name, installed version, and whether it has been modified locally. Use `-o json` for JSON-formatted output.
+
+See {ref}`cli-module-list` for more information.
+
+### Viewing module information
+
+The `module info` command displays detailed metadata and usage information for a specific module from the registry.
+
+Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module.
+
+```console
+$ nextflow module info nf-core/fastqc
+$ nextflow module info nf-core/fastqc -version 1.0.0
+$ nextflow module info nf-core/fastqc -output json
+```
+
+The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access.
+
+See {ref}`cli-module-info` for more information.
+
+### Running modules directly
+
+The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing.
+
+Use this to quickly run a module, test module functionality, or execute one-off data processing tasks.
+
+```console
+$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
+$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0
+```
+
+The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.):
+
+```console
+$ nextflow module run nf-core/salmon \
+    --reads reads.fq \
+    --index salmon_index \
+    -profile docker \
+    -resume
+```
+
+Process inputs can be specified like params on the command line. For example, `--reads reads.fq` corresponds to the `reads` input in the `nf-core/salmon` module. Run `nextflow module info nf-core/salmon` to see the available params for the module.
+
+See {ref}`cli-module-run` for more information.
+
+### Removing modules
+
+The `module remove` command deletes modules from your project, removing local files and configuration entries.
+
+Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline.
+
+```console
+$ nextflow module remove nf-core/fastqc
+$ nextflow module remove nf-core/fastqc -keep-files
+```
+
+By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files.
+
+See {ref}`cli-module-remove` for more information.
+
+### Publishing modules
+
+The `module publish` command uploads modules to a registry, making them available for others to install and use.
+
+Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization.
+
+```console
+$ nextflow module publish myorg/my-module
+$ nextflow module publish myorg/my-module -dry-run
+```
+
+Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. The module must include `main.nf`, `meta.yml`, and `README.md` files.
+
+Use `-dry-run` to validate your module structure without uploading.
+
+See {ref}`cli-module-publish` for more information.
 
 ## Configuration and validation
 
@@ -384,7 +514,7 @@ Use this to understand input/output relationships between tasks, trace data flow
 $ nextflow lineage
 ```
 
-See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information. 
+See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information.
 
 ## Seqera Platform
 

adr/module-spec-schema.json

@@ -279,8 +279,164 @@ This feature requires the use of a local or shared file system for the pipeline
 
 ## Sharing modules
 
-Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. While Nextflow does not provide an explicit mechanism for sharing modules, there are several ways to do it:
+Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules:
 
+- Use the Nextflow module registry (recommended, see below)
 - Simply copy the module files into your pipeline repository
 - Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy
 - Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community
+
+(module-registry)=
+
+## Registry-based modules
+
+:::{versionadded} 26.04.0
+:::
+
+Nextflow provides a module registry that enables you to install, publish, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow language.
+
+### Installing modules from a registry
+
+Use the `module install` command to download modules from a registry:
+
+```console
+$ nextflow module install nf-core/fastqc
+$ nextflow module install nf-core/fastqc -version 1.0.0
+```
+
+Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path:
+
+```nextflow
+include { FASTQC } from 'nf-core/fastqc'
+
+workflow {
+    reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz')
+    FASTQC(reads)
+}
+```
+
+### Running modules directly
+
+For ad-hoc tasks or testing, you can run a module directly without creating a workflow:
+
+```console
+$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
+```
+
+This command accepts all standard `nextflow run` options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed.
+
+### Discovering modules
+
+Search for available modules using the `module search` command:
+
+```console
+$ nextflow module search alignment
+$ nextflow module search "quality control" -limit 10
+```
+
+List installed modules in your project:
+
+```console
+$ nextflow module list
+```
+
+### Module checksum verification
+
+Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites:
+
+```console
+$ nextflow module install nf-core/fastqc -version 1.1.0
+Warning: Module nf-core/fastqc has local modifications. Use -force to override.
+```
+
+Use the `-force` flag to override local modifications when needed.
+
+### Removing modules
+
+Use the `module remove` command to uninstall a module:
+
+```console
+$ nextflow module remove nf-core/fastqc
+```
+
+By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour:
+
+- `-keep-files`: Remove the `.module-info` file created at install but keep the rest of files
+- `-force`: Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications
+
+### Viewing module information
+
+Use the `module info` command to display metadata and a usage template for a module:
+
+```console
+$ nextflow module info nf-core/fastqc
+$ nextflow module info nf-core/fastqc -version 1.0.0
+```
+
+The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-o json` to get machine-readable output.
+
+### Publishing modules
+
+To share your own modules, use the `module publish` command:
+
+```console
+$ nextflow module publish myorg/my-module
+```
+
+The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files.
+
+Your module directory must include:
+
+- `main.nf`: The module entry point
+- `meta.yml`: Module spec (name, description, version, etc.)
+- `README.md`: Module documentation
+
+Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration:
+
+```groovy
+registry {
+    apiKey = 'YOUR_REGISTRY_TOKEN'
+}
+```
+
+Use `-dry-run` to validate your module structure without uploading:
+
+```console
+$ nextflow module publish myorg/my-module -dry-run
+```
+
+### Registry configuration
+
+By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries:
+
+```groovy
+registry {
+    url = [
+        'https://private.registry.myorg.com',
+        'https://registry.nextflow.io'
+    ]
+    apiKey = '${MYORG_TOKEN}'
+}
+```
+
+Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry.
+
+### Module directory structure
+
+Registry modules follow a standard directory structure:
+
+```
+modules/
+└── scope/
+    └── module-name/
+        ├── .module-info     # Integrity checksum (generated automatically)
+        ├── README.md        # Documentation (required for publishing)
+        ├── main.nf          # Module script (required)
+        ├── meta.yml         # Module spec (required for publishing)
+        ├── resources/       # Optional: module binaries and resources
+        └── templates/       # Optional: process templates
+```
+
+The `modules/` directory should be committed to your Git repository to ensure reproducibility.
+
+See the {ref}`cli-page` documentation for more information about module commands.