diff --git a/docs/cli.md b/docs/cli.md index 5c6362d3bf..29c67ecf82 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -250,149 +250,26 @@ $ nextflow drop nextflow-io/hello See {ref}`cli-drop` for more information. -### Secret management - -The `secrets` command manages secure pipeline secrets. - -Use this to store credentials securely, reference them in pipelines without exposing values, and manage sensitive data centrally across your organization. - -```console -$ nextflow secrets list -$ nextflow secrets set AWS_ACCESS_KEY_ID -$ nextflow secrets delete AWS_ACCESS_KEY_ID -``` - -See {ref}`cli-secrets` for more information. - ## Module management :::{versionadded} 26.04.0 ::: -The `module` command enables working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and sharing improvements with the community. - -Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use. - -### Searching for modules - -The `module search` command queries the module registry to discover available modules by keyword or name. - -Use this to find modules for specific tasks, explore available tools, or discover community modules. - -```console -$ nextflow module search alignment -$ nextflow module search "quality control" -limit 10 -$ nextflow module search bwa -output json -``` - -Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for JSON-formatted output. - -See {ref}`cli-module-search` for more information. - -### Installing modules - -The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL. - -Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions. - -```console -$ nextflow module install nf-core/fastqc -$ nextflow module install nf-core/fastqc -version 1.0.0 -``` - -The installed module will be available in `modules/nf-core/fastqc`. - -Use the `-force` flag to reinstall a module even if local modifications exist. - -See {ref}`cli-module-install` for more information. - -### Listing modules - -The `module list` command displays all modules currently installed in your project, showing their versions and integrity status. - -Use this to review installed modules, check module versions, or detect local modifications. - -```console -$ nextflow module list -$ nextflow module list -output json -``` - -The output shows each module's name, installed version, and whether it has been modified locally. Use `-o json` for JSON-formatted output. +The `module` command allows you to discover, install, and run modules through a centralized module registry. See {ref}`using-modules-page` and the {ref}`cli-module` command reference for more information. -See {ref}`cli-module-list` for more information. +## Secret management -### Viewing module information - -The `module view` command displays detailed metadata and usage information for a specific module from the registry. - -Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module. - -```console -$ nextflow module view nf-core/fastqc -$ nextflow module view nf-core/fastqc -version 1.0.0 -$ nextflow module view nf-core/fastqc -output json -``` - -The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access. - -See {ref}`cli-module-view` for more information. - -### Running modules directly - -The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing. - -Use this to quickly run a module, test module functionality, or execute one-off data processing tasks. - -```console -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 -``` - -The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.): - -```console -$ nextflow module run nf-core/salmon \ - --reads reads.fq \ - --index salmon_index \ - -profile docker \ - -resume -``` - -Process inputs can be specified like params on the command line. For example, `--reads reads.fq` corresponds to the `reads` input in the `nf-core/salmon` module. Run `nextflow module view nf-core/salmon` to see the available params for the module. - -See {ref}`cli-module-run` for more information. - -### Removing modules - -The `module remove` command deletes modules from your project, removing local files and configuration entries. - -Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline. - -```console -$ nextflow module remove nf-core/fastqc -$ nextflow module remove nf-core/fastqc -keep-files -``` - -By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files. - -See {ref}`cli-module-remove` for more information. - -### Publishing modules - -The `module publish` command uploads modules to a registry, making them available for others to install and use. +The `secrets` command manages secure pipeline secrets. -Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization. +Use this to store credentials securely, reference them in pipelines without exposing values, and manage sensitive data centrally across your organization. ```console -$ nextflow module publish myorg/my-module -$ nextflow module publish myorg/my-module -dry-run +$ nextflow secrets list +$ nextflow secrets set AWS_ACCESS_KEY_ID +$ nextflow secrets delete AWS_ACCESS_KEY_ID ``` -Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. The module must include `main.nf`, `meta.yml`, and `README.md` files. - -Use the {ref}`cli-module-spec` and {ref}`cli-module-validate` commands to prepare and validate a module before publishing. - -See {ref}`cli-module-publish` for more information. +See {ref}`cli-secrets` for more information. ## Configuration and validation diff --git a/docs/conf.py b/docs/conf.py index 47d5739387..9420b4d7ed 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -58,7 +58,8 @@ 'flux.md': 'tutorials/flux.md', 'developer/plugins.md': 'plugins/developing-plugins.md', 'plugins.md': 'plugins/plugins.md', - 'channel.md': 'workflow.md' + 'channel.md': 'workflow.md', + 'module.md': 'modules/modules.md' } # Add any paths that contain templates here, relative to this directory. diff --git a/docs/guides/gradle-plugin.md b/docs/guides/gradle-plugin.md index 23e9052cf6..75de793466 100644 --- a/docs/guides/gradle-plugin.md +++ b/docs/guides/gradle-plugin.md @@ -6,10 +6,6 @@ Nextflow provides a new `plugin create` command that simplifies the creation of The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) configures default dependencies needed for Nextflow integration and defines Gradle tasks for building, testing, and publishing Nextflow plugins. The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of improvements. This page introduces [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) and how to use it. -:::{note} -The Nextflow Gradle plugin and plugin registry are currently available as a public preview. See the {ref}`Migrating to the Nextflow plugin registry ` for more information. -::: - (gradle-plugin-create)= ## Creating a plugin diff --git a/docs/index.md b/docs/index.md index 877f0d8837..be9e7bc359 100644 --- a/docs/index.md +++ b/docs/index.md @@ -80,13 +80,23 @@ process process-typed workflow workflow-typed -module notifications secrets sharing vscode ``` +```{toctree} +:hidden: +:caption: Modules +:maxdepth: 1 + +modules/modules +modules/using-modules +modules/developing-modules +modules/module-registry +``` + ```{toctree} :hidden: :caption: Software dependencies diff --git a/docs/migrations/24-10.md b/docs/migrations/24-10.md index 9d6fe18c5f..2152c0e7db 100644 --- a/docs/migrations/24-10.md +++ b/docs/migrations/24-10.md @@ -51,7 +51,7 @@ The `task` variable in the process definition has two new properties, `task.prev ## Deprecations -- The use of `addParams` and `params` clauses in include declarations is deprecated. See {ref}`module-params` for details. +- The use of `addParams` and `params` clauses in include declarations is deprecated. See {ref}`strict-syntax-page` for details. ## Miscellaneous diff --git a/docs/migrations/26-04.md b/docs/migrations/26-04.md index e184efa224..42afdddb28 100644 --- a/docs/migrations/26-04.md +++ b/docs/migrations/26-04.md @@ -44,7 +44,7 @@ include { BWA_MEM } from 'nf-core/bwa/mem' All [nf-core modules](https://github.com/nf-core/modules) are automatically synced to the registry from GitHub. They are available as `nf-core/` as shown above. ::: -See {ref}`module-registry` and the {ref}`cli-module` command reference for details. +See the {ref}`Modules overview ` and {ref}`cli-module` command reference for details.

Record types

diff --git a/docs/migrations/dsl1.md b/docs/migrations/dsl1.md index a3b532d8f6..0a1094ab69 100644 --- a/docs/migrations/dsl1.md +++ b/docs/migrations/dsl1.md @@ -100,7 +100,7 @@ Similarly, in DSL2, process outputs can be consumed by multiple consumers automa In DSL1, the entire Nextflow pipeline must be defined in a single file. For example, `main.nf`. This restriction becomes cumbersome as a pipeline grows and hinders the sharing and reuse of pipeline components. -DSL2 introduces the concept of "module scripts" (or "modules" for short), which are Nextflow scripts that can be "included" by other scripts. While modules are not essential to migrating to DSL2, nor are they mandatory in DSL2, modules can help you organize a large pipeline into multiple smaller files and take advantage of modules created by others. See {ref}`module-page` to learn more about modules. +DSL2 introduces the concept of "module scripts" (or "modules" for short), which are Nextflow scripts that can be "included" by other scripts. While modules are not essential to migrating to DSL2, nor are they mandatory in DSL2, modules can help you organize a large pipeline into multiple smaller files and take advantage of modules created by others. See {ref}`modules-page` to learn more about modules. :::{note} DSL2 scripts cannot exceed 64 KB in size. Split large DSL1 scripts into modules to avoid this limit. diff --git a/docs/module.md b/docs/module.md deleted file mode 100644 index 97c82a3b50..0000000000 --- a/docs/module.md +++ /dev/null @@ -1,446 +0,0 @@ -(module-page)= - -# Modules - -Nextflow scripts can include **definitions** (workflows, processes, and functions) from other scripts. When a script is included in this way, it is referred to as a **module**. Modules can be included by other modules or pipeline scripts and can even be shared across workflows. - -:::{note} -Modules were introduced in DSL2. If you are still using DSL1, see the {ref}`dsl1-page` page to learn how to migrate your Nextflow pipelines to DSL2. -::: - -## Module inclusion - -You can include any definition from a module into a Nextflow script using the `include` keyword. - -For example: - -```nextflow -include { cat } from './some/module' - -workflow { - data = channel.fromPath('/some/data/*.txt') - cat(data) -} -``` - -The above snippet imports a process named `cat`, defined in the module, into the main execution context. This way, `cat` can be invoked in the `workflow` scope. - -Nextflow implicitly looks for the script file `./some/module.nf`, resolving the path against the *including* script location. - -Module includes are subject to the following rules: - -- Relative paths must begin with the `./` prefix. -- Include statements are not allowed from within a workflow. They must occur at the script level. - -(module-directory)= - -## Module directory - -:::{versionadded} 22.10.0 -::: - -A module can be defined as a directory with the same name as the module and with a script named `main.nf`. For example: - -``` -some -└── module - └── main.nf -``` - -When defined as a directory, the module must be included by specifying the module directory path: - -```nextflow -include { hello } from './some/module' -``` - -Module directories allow the use of module scoped binary scripts. See [Module binaries] for details. - -## Multiple inclusions - -A Nextflow script can include any number of modules, and an `include` statement can import any number of definitions from a module. Multiple definitions can be included from the same module by using the syntax shown below: - -```nextflow -include { cat; wc } from './some/module' - -workflow { - data = channel.fromPath('/some/data/*.txt') - cat(data) - wc(data) -} -``` - -(module-aliases)= - -## Module aliases - -When including definition from a module, it's possible to specify an *alias* with the `as` keyword. Aliasing allows you to avoid module name clashes, by assigning them different names in the including context. For example: - -```nextflow -include { cat as cat_alpha } from './some/module' -include { cat as cat_beta } from './other/module' - -workflow { - cat_alpha(some_data) - cat_beta(other_data) -} -``` - -You can also include the same definition multiple times under different names: - -```nextflow -include { cat as cat_alpha; cat as cat_beta } from './some/module' - -workflow { - cat_alpha(some_data) - cat_beta(other_data) -} -``` - -(module-params)= - -## Module parameters - -:::{deprecated} 24.07.0-edge -As a best practice, parameters should be used in the entry workflow and passed to workflows, processes, and functions as explicit inputs. -::: - -A module can define parameters using the same syntax as a Nextflow workflow script: - -```nextflow -params.message = 'Hello' -params.target = 'world!' - -def sayHello() { - println "$params.message $params.target" -} -``` - -When including a module, the module will first use parameters from the including context. For example: - -```nextflow -params.message = 'Hola' -params.target = 'Mundo' - -include { sayHello } from './some/module' - -workflow { - sayHello() -} -``` - -The above snippet prints: - -``` -Hola Mundo -``` - -:::{note} -The module inherits the parameters defined *before* the `include` statement, therefore any parameters set afterwards will not be used by the module. -::: - -:::{tip} -It is best to define all pipeline parameters *before* any `include` statements. -::: - -The `addParams` option can be used to pass parameters to the module without adding them to the including scope. - -```nextflow -params.message = 'Hola' -params.target = 'Mundo' - -include { sayHello } from './some/module' addParams(message: 'Ciao') - -workflow { - sayHello() -} -``` - -The above snippet prints: - -``` -Ciao Mundo -``` - -Alternatively, the `params` option can be used to pass parameters to module without adding them to the including scope, *and* without inheriting any parameters from the including scope. - -```nextflow -params.message = 'Hola' -params.target = 'Mundo' - -include { sayHello } from './some/module' params(message: 'Ciao') - -workflow { - sayHello() -} -``` - -The above snippet prints: - -``` -Ciao world! -``` - -(module-templates)= - -## Module templates - -Process script {ref}`templates ` can be included alongside a module in the `templates` directory. - -For example, suppose we have a project L with a module that defines two processes, P1 and P2, both of which use templates. The template files can be made available in the local `templates` directory: - -``` -Project L -|── myModules.nf -└── templates - |── P1-template.sh - └── P2-template.sh -``` - -Then, we have a second project A with a workflow that includes P1 and P2: - -``` -Pipeline A -└── main.nf -``` - -Finally, we have a third project B with a workflow that also includes P1 and P2: - -``` -Pipeline B -└── main.nf -``` - -With the possibility to keep the template files inside the project L, A and B can use the modules defined in L without any changes. A future project C would do the same, just cloning L (if not available on the system) and including its module. - -Beside promoting the sharing of modules across pipelines, there are several advantages to keeping the module template under the script path: - -1. Modules are self-contained -2. Modules can be tested independently from the pipeline(s) that import them -3. Modules can be made into libraries - -Having multiple template locations enables a structured project organization. If a project has several modules, and they all use templates, the project could group module scripts and their templates as needed. For example: - -``` -baseDir -|── main.nf -|── Phase0-Modules - |── mymodules1.nf - |── mymodules2.nf - └── templates - |── P1-template.sh - |── P2-template.sh -|── Phase1-Modules - |── mymodules3.nf - |── mymodules4.nf - └── templates - |── P3-template.sh - └── P4-template.sh -└── Phase2-Modules - |── mymodules5.nf - |── mymodules6.nf - └── templates - |── P5-template.sh - |── P6-template.sh - └── P7-template.sh -``` - -(module-binaries)= - -## Module binaries - -:::{versionadded} 22.10.0 -::: - -Modules can define binary scripts that are locally scoped to the processes defined by the tasks. - -To enable this feature, set the following flag in your pipeline script or configuration file: - -```nextflow -nextflow.enable.moduleBinaries = true -``` - -The binary scripts must be placed in the module directory named `/resources/usr/bin`: - -``` - -|── main.nf -└── resources - └── usr - └── bin - |── your-module-script1.sh - └── another-module-script2.py -``` - -Those scripts will be made accessible like any other command in the task environment, provided they have been granted the Linux execute permissions. - -:::{note} -This feature requires the use of a local or shared file system for the pipeline work directory, or {ref}`wave-page` when using cloud-based executors. -::: - -:::{note} -When {ref}`wave-page` is enabled, the contents of the module `resources/` directory are automatically included in the provisioned container. Wave treats the `resources/` directory as the container root. For example, `resources/some/data/file.txt` becomes `/some/data/file.txt` in the container. -::: - -## Sharing modules - -Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules: - -- Use the Nextflow module registry (recommended, see below) -- Simply copy the module files into your pipeline repository -- Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy -- Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community - -(module-registry)= - -## Registry-based modules - -:::{versionadded} 26.04.0 -::: - -Nextflow provides a module registry that enables you to install, publish, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow language. - -### Installing modules from a registry - -Use the `module install` command to download modules from a registry: - -```console -$ nextflow module install nf-core/fastqc -$ nextflow module install nf-core/fastqc -version 1.0.0 -``` - -Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path: - -```nextflow -include { FASTQC } from 'nf-core/fastqc' - -workflow { - reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') - FASTQC(reads) -} -``` - -### Running modules directly - -For ad-hoc tasks or testing, you can run a module directly without creating a workflow: - -```console -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -``` - -This command accepts all standard `nextflow run` options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed. - -### Discovering modules - -Search for available modules using the `module search` command: - -```console -$ nextflow module search alignment -$ nextflow module search "quality control" -limit 10 -``` - -List installed modules in your project: - -```console -$ nextflow module list -``` - -### Module checksum verification - -Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites: - -```console -$ nextflow module install nf-core/fastqc -version 1.1.0 -Warning: Module nf-core/fastqc has local modifications. Use -force to override. -``` - -Use the `-force` flag to override local modifications when needed. - -### Removing modules - -Use the `module remove` command to uninstall a module: - -```console -$ nextflow module remove nf-core/fastqc -``` - -By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour: - -- `-keep-files`: Remove the `.module-info` file created at install but keep the rest of files -- `-force`: Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications - -### Viewing module information - -Use the `module view` command to display metadata and a usage template for a module: - -```console -$ nextflow module view nf-core/fastqc -$ nextflow module view nf-core/fastqc -version 1.0.0 -``` - -The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-o json` to get machine-readable output. - -### Publishing modules - -To share your own modules, use the `module publish` command: - -```console -$ nextflow module publish myorg/my-module -``` - -The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. - -Your module directory must include: - -- `main.nf`: The module entry point -- `meta.yml`: Module spec (name, description, version, etc.) -- `README.md`: Module documentation - -Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration: - -```groovy -registry { - apiKey = 'YOUR_REGISTRY_TOKEN' -} -``` - -Use `-dry-run` to validate your module structure without uploading: - -```console -$ nextflow module publish myorg/my-module -dry-run -``` - -### Registry configuration - -By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries: - -```groovy -registry { - url = [ - 'https://private.registry.myorg.com', - 'https://registry.nextflow.io' - ] - apiKey = '${MYORG_TOKEN}' -} -``` - -Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry. - -### Module directory structure - -Registry modules follow a standard directory structure: - -``` -modules/ -└── scope/ - └── module-name/ - ├── .module-info # Integrity checksum (generated automatically) - ├── README.md # Documentation (required for publishing) - ├── main.nf # Module script (required) - ├── meta.yml # Module spec (required for publishing) - ├── resources/ # Optional: module binaries and resources - └── templates/ # Optional: process templates -``` - -The `modules/` directory should be committed to your Git repository to ensure reproducibility. - -See the {ref}`cli-page` documentation for more information about module commands. diff --git a/docs/modules/developing-modules.md b/docs/modules/developing-modules.md new file mode 100644 index 0000000000..910df03573 --- /dev/null +++ b/docs/modules/developing-modules.md @@ -0,0 +1,234 @@ +(dev-modules-page)= + +# Developing modules + +Learn how to create modules and share them through the Nextflow module registry. + +(dev-modules-creating)= + +## Creating a module + +Use the `module create` command to scaffold a new module with the required files: + +```console +$ nextflow module create myorg/my-module +``` + +If you omit the name, the command prompts you for details: + +```console +$ nextflow module create +``` + +The command creates a module directory with the following files: + +- `main.nf`: The module script containing your process definition. +- `meta.yml`: The module spec describing metadata, inputs, and outputs. +- `README.md`: Documentation for the module. + +See {ref}`module create ` for the full command reference. + +(dev-modules-structure)= + +## Module structure + +Registry modules follow a standard directory structure: + +``` +modules/ +└── myorg/ + └── my-module/ + ├── .module-info # Integrity checksum + ├── README.md # Documentation + ├── main.nf # Module script + ├── meta.yml # Module spec + ├── resources/ # Optional: Module resources + └── templates/ # Optional: Process script templates +``` + +Local modules that are not intended for publishing do not need to follow this structure, although it is recommended as a best practice. + +### main.nf + +The `main.nf` file contains the process definition. For example, a simple module wrapping FastQC: + +```nextflow +process FASTQC { + tag "$meta.id" + label 'process_medium' + + conda 'bioconda::fastqc=0.12.1' + container 'biocontainers/fastqc:0.12.1--hdfd78af_0' + + input: + tuple val(meta), path(reads) + + output: + tuple val(meta), path("*.html"), emit: html + tuple val(meta), path("*.zip") , emit: zip + + script: + """ + fastqc $reads --threads $task.cpus + """ +} +``` + +Registry modules are subject to the following constraints: + +- The module must contain a single script called `main.nf` +- The module must define a single process +- The module must not define any named workflows +- The module may define an entry workflow to override the default behavior of `module run`. +- The module may define any number of functions + +Local modules can define any number of processes, workflows, and functions. As a best practice, each process and named workflow should be defined in its own script. + +### meta.yml + +The `meta.yml` file contains the module's metadata, including its name, version, description, authors, and input/output specifications. The registry uses this file to display module information and generate usage templates. + +### README.md + +The `README.md` file provides documentation for the module. It should describe what the module does, the tools it wraps, and any configuration requirements. + +(module-resources)= + +### resources + +Modules can include resource files in the `resources/` directory. + +When running the module with {ref}`wave-page`, the contents of `resources/` are mounted into the root directory of the task container. + +For example, given a module with the following structure: + +``` +my-module/ +├── main.nf +└── resources/ + ├── data/ + | └── file.txt + └── usr/ + └── bin/ + └── hello.sh +``` + +The process script can use these files as follows: + +```nextflow +process hello { + container 'quay.io/nextflow/bash' + + script: + """ + cat /data/file.txt + hello.sh + """ +} +``` + +Module resources can be used without Wave or containerization, with the following limitations: + +- The `nextflow.enable.moduleBinaries` feature flag must be enabled in the pipeline script. + +- The pipeline work directory must be in a local or shared file system. Remote object storage is not supported without Wave. + +- Only executable scripts in `resources/usr/bin/` are made accessible to the process script. + +### templates + +Modules can include process script {ref}`templates ` in the `templates/` directory. + +For example, given a module with the following structure: + +``` +my-module/ +|── main.nf +└── templates/ + └── hello.sh +``` + +The module's process can use the script template as follows: + +```nextflow +process hello { + input: + val STR + + script: + template 'hello.sh' +} +``` + +(dev-modules-spec)= + +## Generating a module spec + +Use the `module spec` command to generate or update the `meta.yml` file from the module's `main.nf`: + +```console +$ nextflow module spec myorg/my-module +``` + +Use `-dry-run` to preview the generated spec without writing to disk: + +```console +$ nextflow module spec -dry-run myorg/my-module +``` + +When generating the module spec for the first time, provide required fields directly to avoid `TODO` placeholders in the generated file: + +```console +$ nextflow module spec \ + -namespace myorg \ + -version 1.0.0 \ + -description "Quality control of raw sequencing reads" \ + -license MIT \ + -author "@myname" \ + ./modules/myorg/my-module +``` + +When updating an existing module spec, it is incorporated into the new file. + +See {ref}`module spec ` for the full command reference. + +(dev-modules-validate)= + +## Validating a module + +Use the `module validate` command to check that a module is ready for publishing: + +```console +$ nextflow module validate myorg/my-module +``` + +The command verifies that: + +- All required files are present (`main.nf`, `meta.yml`, `README.md`). +- The module spec contains all required fields (name, version, description, and license). + +See {ref}`module validate ` for the full command reference. + +(dev-modules-testing)= + +## Testing a module + +Before publishing, test your module by running it directly: + +```console +$ nextflow module run myorg/my-module --input 'test-data/*.fastq.gz' +``` + +The command executes the module as a standalone run, allowing you to verify that inputs are correctly declared, the process runs successfully, and the correct outputs are produced. + +For more thorough testing, create a small wrapper workflow that exercises the module: + +```nextflow +include { MY_MODULE } from './modules/myorg/my-module' + +workflow { + input_ch = channel.fromPath('test-data/*.fastq.gz') + results_ch = MY_MODULE(input_ch) + results_ch.view() +} +``` diff --git a/docs/modules/module-registry.md b/docs/modules/module-registry.md new file mode 100644 index 0000000000..5cdefdd6c8 --- /dev/null +++ b/docs/modules/module-registry.md @@ -0,0 +1,175 @@ +(module-registry-page)= + +# Nextflow module registry + +:::{versionadded} 26.04.0 +::: + +The [Nextflow module registry](https://registry.nextflow.io) is a central repository for sharing Nextflow modules. +It provides module discovery, version tracking, and integrity verification. + +The module registry shares its login, access tokens, and web interface with the {ref}`Nextflow plugin registry `. + +(module-registry-claim)= + +## Claiming a namespace + +A namespace is an organizational prefix (e.g., the namespace `nf-core` owns modules like `nf-core/fastqc` and `nf-core/samtools`). +As a namespace member, you can publish new modules and release new versions under that prefix. +Each namespace can have multiple members who share publishing rights. + +To claim a namespace: + +1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser. + +2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address. + + :::{note} + If you are logging in for the first time, Seqera sends an authentication link to your email address to complete the login process. + ::: + +3. Go to **My Resources** and select the **My Namespaces** tab. + +4. Select **Claim Namespace** + +4. Enter your **Namespace name**, **Organization**, **Project URL**, and a **Description**. + +5. Select **Submit Request**. + +The namespace shows as **PENDING REVIEW** until an administrator approves the request. +Admin approval is required only once. +Once approved, you can publish modules using the `nextflow module publish` command. + +(module-registry-access-token)= + +## Creating an access token + +An API access token is required to publish modules to the registry. + +To create an API access token: + +1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser. + +2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address. + + :::{note} + If you are logging in for the first time, Seqera sends an authentication link to your email address to complete the login process. + ::: + +3. Go to **My Resources** and select the **My access tokens** tab. + +4. Under **Create New Access Token**, enter a descriptive name for the **Token name** and select a token duration from the **Expiry** drop down. + +5. Select **Generate Token**. + +6. Copy the token and store it somewhere safe. + You cannot view it again after this step. + +Once you have your token, see [Publishing modules](#publishing-modules) for instructions on how to use it. + +## Viewing modules + +View the modules you have published in the registry: + +1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser. + +2. Log in and go to **My Resources**. + +3. Select the **My Modules** tab. + +The page lists all modules published under your namespaces, along with their versions and status. + +(publish-modules)= + +## Publishing modules + +Publishing requires a {ref}`claimed namespace ` and an {ref}`access token `. + +Provide your registry token in one of two ways: + +**Environment variable:** + +```console +$ export NXF_REGISTRY_TOKEN= +``` + +**Nextflow configuration:** + +```groovy +registry { + apiKey = '${MYORG_TOKEN}' +} +``` + +Use the `module publish` command to upload a module to the registry: + +```console +$ nextflow module publish myorg/my-module +``` + +The module directory must include the following files: + +- `main.nf`: The module script. +- `meta.yml`: The module spec with name, version, description, and license. +- `README.md`: Module documentation. + +Provide a `namespace/name` reference or a relative path to the module directory. + +:::{tip} +Before publishing, verify your module is ready: + +- Run {ref}`module spec ` to generate or update `meta.yml` from your `main.nf`. +- Run {ref}`module validate ` to check that all required files and spec fields are present. + +See {ref}`dev-modules-page` for a complete guide on creating modules. +::: + +See {ref}`module publish ` for the full command reference. + +## Registry configuration + +By default, Nextflow uses the public registry at `https://registry.nextflow.io`. +Configure alternative or additional registries using the `registry` scope in your Nextflow configuration. + +### Use a private registry + +Replace the default registry with your organization's private registry: + +```groovy +registry { + url = 'https://registry.myorg.com' + apiKey = '${MYORG_TOKEN}' +} +``` + +Nextflow uses this registry for all module operations (search, install, and publish). + +### Use multiple registries + +Specify multiple registries as a list. +Nextflow queries them in order when searching for or installing a module: + +```groovy +registry { + url = [ + 'https://registry.myorg.com', + 'https://private.registry.nextflow.io' + ] + apiKey = '${MYORG_TOKEN}' +} +``` + +In this example, Nextflow searches the private registry first and falls back to the public registry. + +:::{note} +The `apiKey` authenticates with the primary (first) registry. +::: + +:::{tip} +Override the target registry with the `-registry` flag: + +```console +$ export NXF_REGISTRY_TOKEN= +$ nextflow module publish myorg/my-module -registry 'https://private.registry.myorg.com' +``` +::: diff --git a/docs/modules/modules.md b/docs/modules/modules.md new file mode 100644 index 0000000000..f756c6d653 --- /dev/null +++ b/docs/modules/modules.md @@ -0,0 +1,63 @@ +(modules-page)= + +# Overview + +Nextflow scripts can include **definitions** (workflows, processes, and functions) from other scripts. +When a script is included in this way, it is referred to as a **module**. +Modules can be re-used within a pipeline and can be shared across projects. +By packaging definitions as modules, you avoid duplicating code and benefit from community improvements. + +There are two ways to use modules in Nextflow: + +- [Local modules](#local-modules) +- [Registry modules](#registry-modules) + +See {ref}`using-modules-page` to learn how to install and use modules in a project. +See {ref}`dev-modules-page` to learn how to create and publish your own modules. +See {ref}`module-registry-page` to learn how to use the Nextflow module registry. + +## Local modules + +Local modules are stored and maintained directly in your project. You include definitions from local modules using the `include` keyword with a relative path: + +```nextflow +include { CAT } from './modules/cat' + +workflow { + data = channel.fromPath('data/*.txt') + CAT(data) +} +``` + +The above snippet imports a process named `CAT` from the *included module* into the *including script*. The include source `./modules/cat` refers to the script `./modules/cat.nf` relative to the including script path. + +Local modules are well suited for project-specific components that are not intended for sharing. + +## Registry modules + +:::{versionadded} 26.04.0 +::: + +Registry modules are sourced from the [Nextflow registry](https://registry.nextflow.io) or a compatible module registry. +Like local modules, registry modules are stored in your project, but they are installed and managed using the `nextflow module` command. + +Key features of registry modules: + +- **Discoverability**: Search for modules by keyword or name and browse available versions. +- **Version management**: Pin specific versions or use the latest release, with automatic integrity checking via checksums. +- **Direct execution**: Run modules as standalone workflows for ad-hoc tasks or testing without writing a wrapper script. +- **Standard structure**: Each module includes a script (`main.nf`), spec (`meta.yml`), and documentation (`README.md`), providing a consistent structure for tooling and automation. + +:::{note} +Modules from the [nf-core](https://nf-co.re/) community are automatically mirrored to the Nextflow module registry under the `nf-core` namespace. +::: + +Install registry modules into your project and include them by name: + +```console +$ nextflow module install nf-core/fastqc +``` + +```nextflow +include { FASTQC } from 'nf-core/fastqc' +``` diff --git a/docs/modules/using-modules.md b/docs/modules/using-modules.md new file mode 100644 index 0000000000..4a9dec2c23 --- /dev/null +++ b/docs/modules/using-modules.md @@ -0,0 +1,163 @@ +(using-modules-page)= + +# Using modules + +:::{versionadded} 26.04.0 +::: + +The Nextflow module system allows you to discover, install, and manage reusable modules from centralized registries. +This page describes how to use modules in your pipelines. + +## Discovering modules + +Search for available modules using the `module search` command: + +```console +$ nextflow module search alignment +$ nextflow module search "quality control" -limit 10 +``` + +Results include module names and descriptions. +Use `-output json` for machine-readable output. + +See {ref}`module search ` for the full command reference. + +## Installing modules + +Use the `module install` command to download modules from a registry into your project: + +```console +$ nextflow module install nf-core/fastqc +$ nextflow module install nf-core/fastqc -version 0.0.0-0c7146d +``` + +:::{note} +Modules mirrored from nf-core do not follow standard semantic versioning. +Instead, they use the format `0.0.0-`, where the suffix is a short portion of the nf-core module's commit hash. +::: + +Nextflow stores installed modules in the `modules/` directory and creates a `.module-info` file alongside the module to record installation metadata such as the module checksum and registry URL. + +:::{tip} +Commit the `modules/` directory to your Git repository to ensure reproducibility. +::: + +See {ref}`module install ` for the full command reference. + +## Listing installed modules + +View all modules installed in your project with the `module list` command: + +```console +$ nextflow module list +``` + +The output shows each module's name, installed version, and whether it has been modified locally. +Use `-output json` for machine-readable output. + +See {ref}`module list ` for the full command reference. + +## Viewing module information + +Use the `module view` command to view metadata and a usage template for a module: + +```console +$ nextflow module view nf-core/fastqc +$ nextflow module view nf-core/fastqc -version 0.0.0-0c7146d +``` + +The output includes the module's version, URL, description, authors, maintainers, keywords, tools, input/output channels, and a generated usage template. +Use `-output json` for machine-readable output. + +See {ref}`module view ` for the full command reference. + +## Including modules + +Modules installed from a registry can be included by name: + +```nextflow +include { FASTQC } from 'nf-core/fastqc' + +workflow { + reads = channel.fromPath('data/*.fastq') + reads = reads.map { fastq -> tuple([id: fastq.baseName], fastq) } + FASTQC(reads) +} +``` + +Local modules must be included by relative path: + +```nextflow +include { FASTQC } from './modules/local/fastqc' +``` + +See {ref}`syntax-include` for a full description of the include syntax. + +## Running modules directly + +For ad-hoc tasks or testing, run a module directly without creating a wrapper workflow: + +```console +$ nextflow module run nf-core/fastqc --meta.id test_sample --reads sample1_R1.fastq.gz +``` + +:::{tip} +Run `nextflow module view` to see the available inputs for a module. +::: + +The command automatically downloads the module if it is not already installed. +It accepts all standard Nextflow run options (`-profile`, `-resume`, etc.): + +```console +$ nextflow module run nf-core/fastqc \ + --meta.id test_sample \ + --reads sample1_R1.fastq.gz \ + -with-docker +``` + +Run a local module by specifying a path starting with `./` or `../`: + +```console +$ nextflow module run ./modules/local/fastqc/main.nf \ + --meta.id test_sample \ + --reads sample1_R1.fastq.gz \ + -with-docker +``` + +:::{note} +The local module should define a single process, or else the command will fail. +::: + +See {ref}`module run ` for the full command reference. + +## Updating modules + +To update a module to a newer version, reinstall it with the desired version: + +```console +$ nextflow module install nf-core/fastqc -version 0.0.0-c9h0bv4 +``` + +Nextflow automatically verifies module integrity using a checksum stored in the `.module-info` file. +If the module has local modifications, it will not be updated. +Use the `-force` flag to overwrite local changes: + +```console +$ nextflow module install nf-core/fastqc -version 0.0.0-c9h0bv4 -force +``` + +## Removing modules + +Use the `module remove` command to uninstall a module from your project: + +```console +$ nextflow module remove nf-core/fastqc +``` + +By default, Nextflow removes both the module files and the `.module-info` file. +Use flags to control this behavior: + +- `-keep-files`: Remove the `.module-info` file but keep the module files in the `modules/` directory. +- `-force`: Remove the module directory even if it has no `.module-info` file or has local modifications. + +See {ref}`module remove ` for the full command reference. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 1c01085e4a..7a636d15ac 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -12,10 +12,6 @@ The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template You can use the `nextflow plugin create` sub-command to create plugins using the plugin template. See {ref}`gradle-plugin-create` for more information. -:::{note} -The Nextflow Gradle plugin is currently available as a public preview. See {ref}`Migrating to the Nextflow plugin registry ` for more information. -::: - (dev-plugins-structure)= ### Structure diff --git a/docs/plugins/plugin-registry.md b/docs/plugins/plugin-registry.md index b0f5b96d55..ef1c691e0d 100644 --- a/docs/plugins/plugin-registry.md +++ b/docs/plugins/plugin-registry.md @@ -6,10 +6,6 @@ The [Nextflow plugin registry](https://registry.nextflow.io/) is a central repos Nextflow 25.10 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. See {ref}`migrate-plugin-page` for more information about migrating to the Nextflow plugin registry. -:::{note} -The Nextflow plugin registry is currently available as a public preview. Plugin developers can access the registry by contacting [info@nextflow.io](mailto:info@nextflow.io) to obtain access to the registry. -::: - See {ref}`gradle-plugin-publish` for instructions on how to publish plugins to the registry, including the {ref}`README.md requirement `. (plugin-registry-claim)= diff --git a/docs/process.md b/docs/process.md index ffb623c3d5..cedbc33143 100644 --- a/docs/process.md +++ b/docs/process.md @@ -166,12 +166,15 @@ process hello { input: val STR + output: + stdout + script: template 'hello.sh' } workflow { - channel.of('this', 'that') | hello + hello('Hello!').view() } ``` @@ -189,7 +192,7 @@ echo "process completed" Variables prefixed with the dollar character (`$`) are interpreted as Nextflow variables when the template script is executed by Nextflow and Bash variables when executed directly. For example, the above script can be executed from the command line by providing each input as an environment variable: ```bash -STR='Hello!' bash templates/my_script.sh +STR='Hello!' bash templates/hello.sh ``` The following caveats should be considered: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 1c8b70f6b4..70480dbffa 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1157,20 +1157,12 @@ The `module` command provides a comprehensive system for managing registry-based **Subcommands** +(cli-module-create)= + `create [namespace/name]` : Create a new module with a basic `main.nf`, `meta.yml`, and `README.md`. -: **Examples:** - - ```console - # Create a module (it will prompt for details) - $ nextflow module create - - # Create a module with the given name - $ nextflow module create myorg/my-module - ``` - (cli-module-install)= `install [options] [namespace/name]` @@ -1186,19 +1178,6 @@ The `module` command provides a comprehensive system for managing registry-based `-force` : Force reinstall even if the module exists locally with modifications. Without this flag, Nextflow prevents overwriting locally modified modules. -: **Examples:** - - ```console - # Install latest version - $ nextflow module install nf-core/fastqc - - # Install specific version - $ nextflow module install nf-core/fastqc -version 1.0.0 - - # Force reinstall over local modifications - $ nextflow module install nf-core/fastqc -force - ``` - (cli-module-list)= `list [options]` @@ -1210,16 +1189,6 @@ The `module` command provides a comprehensive system for managing registry-based `-o, -output` (`table`) : Output mode for list results. Options: `table` (default), `json`. -: **Examples:** - - ```console - # Display installed modules in formatted table - $ nextflow module list - - # Output as JSON - $ nextflow module list -output 'json' - ``` - (cli-module-publish)= `publish [options] [namespace/name | path]` @@ -1236,21 +1205,6 @@ The `module` command provides a comprehensive system for managing registry-based `-registry` : Specify the registry to publish the module (default: `https://registry.nextflow.io`) -: **Examples:** - - ```console - # Validate module structure without publishing - $ nextflow module publish myorg/my-module -dry-run - - # Publish to nextflow registry - $ export NXF_REGISTRY_TOKEN=your-token - $ nextflow module publish myorg/my-module - - # Publish to a custom registry - $ export NXF_REGISTRY_TOKEN=your-token - $ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' - ``` - (cli-module-remove)= `remove [options] [namespace/name]` @@ -1265,16 +1219,6 @@ The `module` command provides a comprehensive system for managing registry-based `-keep-files` : Remove the `.module-info` but keep local files in the `modules/` directory. -: **Examples:** - - ```console - # Remove module completely - $ nextflow module remove nf-core/fastqc - - # Remove from config but keep local files - $ nextflow module remove nf-core/fastqc -keep-files - ``` - (cli-module-run)= `run [options] [namespace/name | path] [-- ]` @@ -1286,25 +1230,6 @@ The `module` command provides a comprehensive system for managing registry-based `-version` : Specify the module version to run (e.g., `1.0.0`). If not specified, uses the latest version. -: **Examples:** - - ```console - # Run remote module - $ nextflow module run nf-core/fastqc \ - --input 'data/*.fastq.gz' - - # Run remote module with specific version and run options - $ nextflow module run nf-core/fastqc \ - -version 1.0.0 \ - --input 'data/*.fastq.gz' \ - -with-conda \ - -resume - - # Run local module - $ nextflow module run ./modules/nf-core/fastqc/main.nf \ - --input 'data/*.fastq.gz' - ``` - (cli-module-search)= `search [options] [query]` @@ -1319,19 +1244,6 @@ The `module` command provides a comprehensive system for managing registry-based `-o, -output` (`simple`) : Output mode for search results. Options: `simple` (default), `json`. -: **Examples:** - - ```console - # Search for alignment-related modules - $ nextflow module search alignment - - # Search with limited results - $ nextflow module search "quality control" -limit 10 - - # Get results as JSON - $ nextflow module search bwa -output json - ``` - (cli-module-spec)= `spec [options] ` @@ -1359,29 +1271,6 @@ The `module` command provides a comprehensive system for managing registry-based `-dry-run` : Print the generated spec to stdout without writing any file. -: **Examples:** - - ```console - # Generate spec for local module by name - $ nextflow module spec nf-core/fastqc - - # Generate spec for local module by path (namespace required) - $ nextflow module spec -namespace nf-core ./modules/my-module - - # Provide additional fields to avoid TODO placeholders - $ nextflow module spec \ - -namespace nf-core \ - -version 1.0.0 \ - -description "Quality control of raw sequencing reads" \ - -license MIT \ - -author "@drpatelh" \ - -author "@joseespinosa" \ - ./modules/nf-core/fastqc - - # Print module spec without saving it - $ nextflow module spec -dry-run -namespace nf-core ./modules/my-module - ``` - (cli-module-validate)= `validate [options] ` @@ -1389,16 +1278,6 @@ The `module` command provides a comprehensive system for managing registry-based : Validate a module before publishing to the registry. : Verifies that all required files are present (`main.nf`, `meta.yml`, `README.md`) and that the module spec contains all required fields (name, version, description, license). -: **Examples:** - - ```console - # Validate module by name - $ nextflow module validate myorg/my-module - - # Validate module by path - $ nextflow module validate ./modules/myorg/my-module - ``` - (cli-module-view)= `view [options] [namespace/name]` @@ -1413,18 +1292,77 @@ The `module` command provides a comprehensive system for managing registry-based `-o, -output` (`text`) : Output format. Options: `text` (default), `json`. -: **Examples:** +**Examples** - ```console - # Display information for latest version - $ nextflow module view nf-core/fastqc +Search for modules related to "alignment": - # Display information for specific version - $ nextflow module view nf-core/fastqc -version 1.0.0 +```console +$ nextflow module search alignment +``` - # Get results as JSON - $ nextflow module view nf-core/fastqc -output json - ``` +View information for a module: + +```console +$ nextflow module view nf-core/fastqc +``` + +Install a module: + +```console +$ nextflow module install nf-core/fastqc +``` + +List installed modules: + +```console +$ nextflow module list +``` + +Run a module: + +```console +$ nextflow module run nf-core/fastqc \ + --input 'data/*.fastq.gz' \ + -with-docker + +Run a local module: + +```console +$ nextflow module run ./modules/local/fastqc/main.nf \ + --input 'data/*.fastq.gz' \ + -with-docker +``` + +Remove a module: + +```console +$ nextflow module remove nf-core/fastqc +``` + +Create a module with a given name: + +```console +$ nextflow module create myorg/my-module +``` + +Generate a spec for a local module: + +```console +$ nextflow module spec -namespace myorg ./modules/myorg/my-module +``` + +Validate a local module: + +```console +$ nextflow module validate ./modules/myorg/my-module +``` + +Publish a module to the Nextflow registry: + +```console +$ export NXF_REGISTRY_TOKEN= +$ nextflow module publish myorg/my-module +``` (cli-plugin)= @@ -1813,29 +1751,29 @@ $ nextflow secrets [OPTIONS] **Examples** -- Set a secret: +Set a secret: - ```console - $ nextflow secrets set FOO "Hello world" - ``` +```console +$ nextflow secrets set FOO "Hello world" +``` -- List secrets: +List secrets: - ```console - $ nextflow secrets list - ``` +```console +$ nextflow secrets list +``` -- Get a secret: +Get a secret: - ```console - $ nextflow secrets get FOO - ``` +```console +$ nextflow secrets get FOO +``` -- Delete a secret: +Delete a secret: - ```console - $ nextflow secrets delete FOO - ``` +```console +$ nextflow secrets delete FOO +``` ### `self-update` @@ -1853,7 +1791,7 @@ The `self-update` command directs the `nextflow` CLI to update itself to the lat **Examples** -Update Nextflow. +Update Nextflow: ```console $ nextflow self-update @@ -1903,7 +1841,7 @@ The `view` command is used to inspect the pipelines that are already stored in t **Examples** -Viewing the contents of a downloaded pipeline. +View the contents of a downloaded pipeline: ```console $ nextflow view nextflow-io/hello diff --git a/docs/reference/feature-flags.md b/docs/reference/feature-flags.md index 594066aa50..2eea415904 100644 --- a/docs/reference/feature-flags.md +++ b/docs/reference/feature-flags.md @@ -20,7 +20,7 @@ Feature flags with the `nextflow.preview` prefix can cause pipelines run with ne : Defines the DSL version to use (`1` or `2`). `nextflow.enable.moduleBinaries` -: When `true`, enables the use of modules with binary scripts. See {ref}`module-binaries` for more information. +: When `true`, enables the use of module-scoped executable scripts via {ref}`module resources `. `nextflow.enable.strict` : :::{deprecated} 26.04.0 diff --git a/docs/reference/syntax.md b/docs/reference/syntax.md index 64985f0a9a..99682ac7df 100644 --- a/docs/reference/syntax.md +++ b/docs/reference/syntax.md @@ -73,6 +73,8 @@ A feature flag declaration is an assignment. The target should be a valid {ref}` nextflow.preview.recursion = true ``` +(syntax-include)= + ### Include An include declaration consists of an *include source* and one or more *include clauses*: @@ -1053,6 +1055,8 @@ Compound expressions are evaluated in the following order: The following legacy features were excluded from this page because they are deprecated: -- The `addParams` and `params` clauses of include declarations. See {ref}`module-params` for more information. -- The `when:` section of a process definition. See {ref}`process-when` for more information. -- The `shell:` section of a process definition. See {ref}`process-shell` for more information. +- The `addParams` and `params` clauses of include declarations. +- The `when:` section of a process definition. +- The `shell:` section of a process definition. + +See {ref}`strict-syntax-page` for more information. diff --git a/docs/script.md b/docs/script.md index d4c80ee07b..3f606b2e9d 100644 --- a/docs/script.md +++ b/docs/script.md @@ -561,4 +561,4 @@ workflow { } ``` -See {ref}`workflow-page`, {ref}`process-page`, and {ref}`module-page` for more information about how to use these features in your Nextflow scripts. +See {ref}`Workflows `, {ref}`Processes `, and {ref}`Modules ` for more information about how to use these features in your Nextflow scripts. diff --git a/docs/strict-syntax.md b/docs/strict-syntax.md index 00037698e8..8e6d949847 100644 --- a/docs/strict-syntax.md +++ b/docs/strict-syntax.md @@ -260,7 +260,7 @@ The following patterns are still supported but have been restricted. That is, so ### Include declarations -In Nextflow DSL2, include declarations can have an `addParams` or `params` clause as described in {ref}`module-params`: +In Nextflow DSL2, include declarations can have an `addParams` or `params` clause: ```nextflow params.message = 'Hola' diff --git a/docs/wave.md b/docs/wave.md index 44e882d307..ea733ed61c 100644 --- a/docs/wave.md +++ b/docs/wave.md @@ -48,22 +48,16 @@ tower { Wave can build and provision container images on-demand for your Nextflow pipelines. -To enable this feature, add the Dockerfile of the container to be built in the {ref}`module directory ` where the pipeline process is defined. When Wave is enabled, it automatically uses the Dockerfile to build the required container, upload to the registry, and it uses the container to carry out the tasks defined in the module. +To enable this feature, add the Dockerfile of the container image to the module directory where the process is defined. When Wave is enabled, it automatically uses the Dockerfile to build the required container, upload to the registry, and it uses the container to carry out the tasks defined in the module. -:::{tip} -Make sure the process does not declare a `container` directive, otherwise it will take precedence over the Dockerfile definition. -::: - -If a process uses a `container` directive and you still want to build the container using the Dockerfile provided in the module directory, add the following setting to the pipeline config file: +If a process has a `container` directive, it will take precedence over the Dockerfile. Use the `wave.strategy` config option to prioritize the Dockerfile over the `container` directive: ```groovy wave.strategy = ['dockerfile','container'] ``` -This setting instructs Wave to prioritize the module Dockerfile over process `container` directives. - :::{note} -When building containers, Wave does not support `ADD`, `COPY`, or any other Dockerfile commands that access files in the host file system. To include custom scripts or files, use the module `resources/` directory instead. Wave automatically includes its contents in the container build context. See {ref}`module-binaries` for more details. +When building container images, Wave does not support `ADD`, `COPY`, or any other Dockerfile commands that access files in the host file system. To include custom scripts or files, use the module `resources/` directory instead. Wave automatically includes its contents in the container build context. See {ref}`module-resources` for more details. ::: ### Build Conda based containers diff --git a/docs/workflow.md b/docs/workflow.md index 6e1b2489c1..46146d50a9 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -346,8 +346,8 @@ A *named workflow* is a workflow that can be called by other workflows: ```nextflow workflow my_workflow { - hello() - bye( hello.out.collect() ) + ch_hello = hello() + bye( ch_hello.collect() ) } workflow { @@ -368,8 +368,8 @@ workflow my_workflow { data2 main: - hello(data1, data2) - bye(hello.out) + ch_hello = hello(data1, data2) + bye(hello) } ``` @@ -385,31 +385,34 @@ The `emit:` section declares the outputs of a named workflow: ```nextflow workflow my_workflow { + take: + data + main: - hello(data) - bye(hello.out) + ch_bye = bye(hello(data)) emit: - bye.out + ch_bye } ``` -When calling the workflow, the output can be accessed using the `out` property, i.e. `my_workflow.out`. - If an output is assigned to a name, the name can be used to reference the output from the calling workflow. For example: ```nextflow workflow my_workflow { main: - hello(data) - bye(hello.out) + ch_hello = hello(data) + ch_bye = bye(ch_hello) emit: - my_data = bye.out + my_data = ch_bye } -``` -The result of the above workflow can be accessed using `my_workflow.out.my_data`. +workflow { + result = my_workflow() + result.my_data.view() +} +``` :::{note} Every output must be assigned to a name when multiple outputs are declared. @@ -421,9 +424,7 @@ Every output must be assigned to a name when multiple outputs are declared. Workflows consist of *dataflow* logic, in which processes are connected to each other through *dataflow channels* and *dataflow values*. -(dataflow-type-channel)= - -### Channels +### Channels and values A *dataflow channel* (or simply *channel*) is an asynchronous sequence of values. @@ -439,7 +440,21 @@ channel emits 2 channel emits 3 ``` -**Factories** +A *dataflow value* is an asynchronous value. + +Dataflow values can be created using the {ref}`channel.value ` factory, and they are created by processes (under {ref}`certain conditions `). + +A dataflow value cannot be accessed directly, but only through an operator or process. For example: + +```nextflow +channel.value(1).view { v -> "dataflow value is ${v}" } +``` + +```console +dataflow value is 1 +``` + +### Factories A channel can be created by factories in the `channel` namespace. For example, the `channel.fromPath()` factory creates a channel from a file name or glob pattern, similar to the `files()` function: @@ -449,7 +464,7 @@ channel.fromPath('input/*.txt').view() See {ref}`channel-factory` for the full list of channel factories. -**Operators** +### Operators Channel operators, or *operators* for short, are functions that consume and produce channels. Because channels are asynchronous, operators are necessary to manipulate the values in a channel. Operators are particularly useful for implementing glue logic between processes. @@ -473,27 +488,7 @@ Commonly used operators include: - {ref}`operator-view`: print each channel value to standard output -See {ref}`operator-page` for the full set of operators. - -(dataflow-type-value)= - -### Values - -A *dataflow value* is an asynchronous value. - -Dataflow values can be created using the {ref}`channel.value ` factory, and they are created by processes (under {ref}`certain conditions `). - -A dataflow value cannot be accessed directly, but only through an operator or process. For example: - -```nextflow -channel.value(1).view { v -> "dataflow value is ${v}" } -``` - -```console -dataflow value is 1 -``` - -See {ref}`stdlib-types-value` for the set of available methods for dataflow values. +See {ref}`operator-page` for the full set of operators. See {ref}`stdlib-types-value` for the set of available methods for dataflow values. (workflow-process-invocation)= @@ -540,11 +535,19 @@ workflow { } ``` -Processes and workflows have a few extra rules for how they can be called: +Processes and workflows can only be called by workflows. A given process or workflow can only be called once in a given workflow. To use a process or workflow multiple times in the same workflow, {ref}`include ` it from another script with multiple aliases: -- Processes and workflows can only be called by workflows +```nextflow +include { hello_bye as hello_bye1 } from './modules/hello_bye' +include { hello_bye as hello_bye2 } from './modules/hello_bye' -- A given process or workflow can only be called once in a given workflow. To use a process or workflow multiple times in the same workflow, use {ref}`module-aliases`. +workflow { + data1 = channel.fromPath('data1/*.txt') + data2 = channel.fromPath('data2/*.txt') + hello_bye1(data1) + hello_bye2(data2) +} +``` The "return value" of a process or workflow call is the process outputs or workflow emits, respectively. The return value can be assigned to a variable or passed into another call: @@ -614,10 +617,6 @@ workflow { Process named outputs are defined using the `emit` option on a process output. See {ref}`naming process outputs ` for more information. ::: -:::{note} -Process and workflow outputs can also be accessed by index (e.g., `hello.out[0]`, `hello.out[1]`, etc.). As a best practice, multiple outputs should be accessed by name. -::: - Workflows can be composed in the same way: ```nextflow @@ -625,36 +624,25 @@ workflow flow1 { take: data - main: - tick(data) - tack(tick.out) - emit: - tack.out + tack(tick(data)) } workflow flow2 { take: data - main: - tick(data) - tock(tick.out) - emit: - tock.out + tock(tick(data)) } workflow { data = channel.fromPath('/some/path/*.txt') - flow1(data) - flow2(flow1.out) + flow2(flow1(data)) } ``` -:::{note} The same process can be called in different workflows without using an alias, like `tick` in the above example, which is used in both `flow1` and `flow2`. The workflow call stack determines the *fully qualified process name*, which is used to distinguish the different process calls, i.e. `flow1:tick` and `flow2:tick` in the above example. -::: :::{tip} The fully qualified process name can be used as a {ref}`process selector ` in a Nextflow configuration file, and it takes priority over the simple process name. @@ -662,55 +650,18 @@ The fully qualified process name can be used as a {ref}`process selector ` is enabled. Using these operators will prevent the type checker from validating your code. +:::{deprecated} 26.04.0 +These operators are not supported when {ref}`static typing ` is enabled. Use standard method calls and assignments instead. ::: -**Pipe `|`** - -The `|` *pipe* operator can be used to chain processes, operators, and workflows: - -```nextflow -process greet { - input: - val data - - output: - val result - - exec: - result = "$data world" -} - -workflow { - channel.of('Hello', 'Hola', 'Ciao') - | greet - | map { v -> v.toUpperCase() } - | view -} -``` - -The above snippet defines a process named `greet` and invokes it with the input channel. The result is then piped to the {ref}`operator-map` operator, which converts each string to uppercase, and finally to the {ref}`operator-view` operator which prints it. +The following operators have a special meaning when used with process and workflow calls in a workflow: -The same code can also be written as: +- The `|` *pipe* operator can be used to chain processes, operators, and workflows. +- The `&` *and* operator can be used to call multiple processes in parallel with the same channel(s). -```nextflow -workflow { - ch_input = channel.of('Hello', 'Hola', 'Ciao') - ch_greet = greet(ch_input) - ch_greet - .map { v -> v.toUpperCase() } - .view() -} -``` - -**And `&`** - -The `&` *and* operator can be used to call multiple processes in parallel with the same channel(s): +For example: ```nextflow process greet { diff --git a/modules/nf-lang/src/main/java/nextflow/script/dsl/FeatureFlagDsl.java b/modules/nf-lang/src/main/java/nextflow/script/dsl/FeatureFlagDsl.java index ce9f6f27e3..1e9ee17738 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/dsl/FeatureFlagDsl.java +++ b/modules/nf-lang/src/main/java/nextflow/script/dsl/FeatureFlagDsl.java @@ -33,7 +33,7 @@ Defines the DSL version (`1` or `2`). @FeatureFlag("nextflow.enable.moduleBinaries") @Description(""" - When `true`, enables the use of modules with executable scripts i.e. [module binaries](https://nextflow.io/docs/latest/module.html#module-binaries). + When `true`, enables the use of module-scoped executable scripts via [module resources](https://nextflow.io/docs/latest/module.html#module-resources). """) public boolean moduleBinaries;