Merge branch 'master' into rename-preview-types-flag

Author: bentsherman

VERSION

@@ -1 +1 @@
-26.03.3-edge
+26.03.4-edge

adr/20260323-hints-process-directive.md

@@ -0,0 +1,180 @@
+# `hints` process directive for executor-specific scheduling hints
+
+- Authors: Rob Syme
+- Status: accepted
+- Deciders: Paolo Di Tommaso, Ben Sherman, Rob Syme
+- Date: 2026-03-23
+- Tags: directive, executor, scheduling
+
+## Summary
+
+Introduce a `hints` process directive for executor-specific scheduling hints that don't map to existing directives.
+
+## Problem Statement
+
+Many executors can be configured in various ways on a per-task basis. For example:
+
+- AWS Batch jobs can use *consumable resources* to limit concurrent job execution based on non-standard resources such as software license seats.
+
+- Google Batch jobs can specify a *provisioning model* to control the use of spot vs on-demand VMs on a per-task basis.
+
+- Seqera Scheduler supports a variety of resource and scheduling settings, including spot/on-demand provisioning.
+
+These settings can be exposed by Nextflow as executor-specific config options, such as `google.batch.spot`, but config options are applied globally. In order to apply a setting to specific processes or tasks, it must be exposed as a process directive.
+
+Process directives in Nextflow aim to provide a common vocabulary for executing tasks in many different environments. Directives such as `cpus`, `memory`, and `time` have broadly the same meaning across most executors, making it easier for users to write portable pipelines.
+
+At the same time, many executors have custom settings not shared by other executors, and it is not practical to create a new process directive for every new setting. There are over 40 [process directives](https://docs.seqera.io/nextflow/reference/process#directives) at the time of writing, and every new directive adds cognitive load when a user is trying to find the right directive for a given situation.
+
+There exist a few generic process directives already:
+
+- The `clusterOptions` directive can be used to specify command-line arguments, primarily for HPC schedulers
+- The `ext` directive supports arbitrary key-values, but is designed primarily to customize the task script (e.g. tool arguments), not executor behavior
+- The `resourceLabels` directive also supports arbitrary key-values, but is intended for tagging and tracking resources, not controlling them
+
+A new directive is needed to support executor-specific settings at a per-task level in a structured manner, without bloating the process directives for every new custom setting.
+
+## Goals
+
+- Provide a way to apply executor-specific settings to individual processes or tasks
+
+- Avoid the proliferation of narrow, executor-specific directives (e.g. `consumableResources`, `schedulingPolicy`, etc.)
+
+- Provide a single extension point that executors can consume selectively
+
+- Allow settings to be specified as key-values, providing validation where possible
+
+## Non-goals
+
+- Replacing existing directives (`cpus`, `memory`, `accelerator`, `queue`) — those remain the right place for standard resources
+
+## Decision
+
+Introduce a `hints` process directive with namespaced keys. Executors consume the hints they understand and silently ignore the rest.
+
+## Core Capabilities
+
+### Syntax
+
+The `hints` directive accepts a map of key-value pairs:
+
+```groovy
+// process definition
+process runDragen {
+    cpus 4
+    memory '16 GB'
+    hints consumableResources: ['my-dragen-license': 1, 'other-license': 2]
+
+    script:
+    """
+    dragen --ref-dir /ref ...
+    """
+}
+```
+
+```groovy
+// process config
+process {
+    withName: 'runDragen' {
+        hints = [
+            consumableResources: ['my-dragen-license': 1, 'other-license': 2]
+        ]
+    }
+}
+```
+
+Keys are strings. Values may be any raw data type: strings, numbers, booleans, lists, or maps. Executors are responsible for defining which hints they recognize and what value type each hint expects.
+
+In the above example, the `consumableResources` hint is given as a map of resource name to quantity. The AWS Batch executor supplies it to each job request using `ConsumableResourceProperties`.
+
+### Namespacing
+
+Keys can use dot-separated scopes to namespace settings as needed:
+
+```groovy
+hints consumableResources: ['my-dragen-license': 1]
+hints 'scheduling.priority': 10
+hints 'scheduling.provisioningModel': 'spot'
+```
+
+Keys can be routed to specific executors by prefixing with the executor name and a slash (`/`):
+
+```groovy
+hints 'awsbatch/consumableResources': ['my-dragen-license': 1]
+hints 'seqera/scheduling.provisioningModel': 'spot'
+hints 'k8s/nodeSelector': 'gpu=true'
+```
+
+The executor prefix gives pipeline developers the ability to target specific executors and have assurance that it won't accidentally apply to other executors (e.g. if another executor adds support for the same hint in the future).
+
+### Validation
+
+Nextflow should validate hints to the best of its ability, to catch errors such as typos:
+
+- **Prefixed hints** can be validated against the set of hints declared by the corresponding executor. Unrecognized hints should be reported as errors.
+
+- **Unprefixed hints** can be validated against the union of hints declared by all executors. Since unprefixed hints might be supported by executors that aren't currently loaded, unrecognized hints should be reported as warnings.
+
+### Multiple hint resolution
+
+The `hints` directive uses *replacement semantics* when specified multiple times, meaning that each `hints` setting completely replaces any previous settings:
+
+```groovy
+process {
+    // generic hint
+    hints = [provisioningModel: 'spot']
+
+    // specific hint replaces generic hint
+    withLabel: 'dragen' {
+        hints = [consumableResources: ['my-dragen-license': 1]]
+    }
+}
+```
+
+Within a process definition, the `hints` directive uses *accumulation semantics*, meaning that subsequent `hints` directives are accumulated:
+
+```groovy
+process runDragen {
+    // multiple separate hints
+    hints provisioningModel: 'spot'
+    hints consumableResources: ['my-dragen-license': 1, 'other-license': 2]
+
+    // equivalent to...
+    hints (
+        provisioningModel: 'spot',
+        consumableResources: ['my-dragen-license': 1, 'other-license': 2]
+    )
+
+    // ...
+}
+```
+
+This behavior is consistent with other directives such as `pod` and `resourceLabels`. In practice, this means that a given `hints` setting should specify all relevant hints for the given context.
+
+For example, the `withLabel` selector above should also specify the `provisioningModel` hint if the intention is to preserve that hint for the selected processes:
+
+```groovy
+process {
+    hints = [provisioningModel: 'spot']
+
+    withLabel: 'dragen' {
+        hints = [provisioningModel: 'spot', consumableResources: ['my-dragen-license': 1]]
+    }
+}
+```
+
+While this approach may lead to duplication, it gives users and developers more control over which hints are applied in a given context.
+
+### Initial hint catalog
+
+The following hints should be supported initially:
+
+| Hint name | Value type | Executors | Use case |
+|--|--|--|--|
+| `consumableResources` | `Map<String, Integer>` | AWS Batch | License-aware scheduling ([#5917](https://github.com/nextflow-io/nextflow/issues/5917)) |
+| `scheduling.priority` | `Integer` | AWS Batch | Job scheduling priority ([#6998](https://github.com/nextflow-io/nextflow/issues/6998)) |
+| `scheduling.provisioningModel` | `String` | Google Batch | Spot VM scheduling ([#3530](https://github.com/nextflow-io/nextflow/issues/3530)) |
+
+## Links
+
+- [Community issue](https://github.com/nextflow-io/nextflow/issues/5917)

changelog.txt

@@ -1,5 +1,29 @@
 NEXTFLOW CHANGE-LOG
 ===================
+26.03.4-edge - 25 Apr 2026
+- Abort execution when platform telemetry error (#6827) [b1ad3f720]
+- Add $schema ref to generated module spec (#7056) [c40d742f3]
+- Add Apple container engine support (#7073) [2f7a3c455]
+- Add hints process directive for executor-specific scheduling hints (#7034) [406358e03]
+- Add Seqera NIO filesystem for datasets and refactor TowerClient/TowerObserver (#6946) [433b10a1f]
+- Add workspaceId/computeEnvId to nf-seqera auto labels (#7059) [5e8276c00]
+- Allow `-with-docker` to be used without a default container image (#7054) [41759d36e]
+- Allow module run to run modules with local path (#7057) [e2c77c6b7]
+- Default NXF_FUSION_TRACE to false (#7071) [5b4c8f0c1]
+- Fix IllegalArgumentException when process.resourceLabels is a closure (#7068) [944977e3f]
+- Fix resolution of params in resolved config text (#7072) [cb7133def]
+- Propagate task.containerPlatform through Fusion container command (#7074) [b58a590bd]
+- Remove arch config option from Seqera MachineRequirement (#7063) [da06e9a9d]
+- Replace current cloud info URL call with cloudInfo client (#7065) [629184251]
+- Restructure modules docs as a section and add registry steps (#7030) [29370f4bc]
+- Update workflow outputs tutorial (#7060) [68d144b9c]
+- Use toUriString for paths in work-dir and FilesEx error messages (#7075) [b535377cc]
+- Bump nf-amazon@3.9.0
+- Bump nf-google@1.27.2
+- Bump nf-seqera@0.19.0
+- Bump nf-tower@1.26.0
+- Bump nf-wave@1.20.0
+
 26.03.3-edge - 20 Apr 2026
 - Add -files-from option to lint command to avoid ARG_MAX limit (#6858) [5a3cd830c]
 - Add 26.04 migration docs (#7000) [89ec31bbf]

docs/cli.md

@@ -250,149 +250,26 @@ $ nextflow drop nextflow-io/hello
 
 See {ref}`cli-drop` for more information.
 
-### Secret management
-
-The `secrets` command manages secure pipeline secrets.
-
-Use this to store credentials securely, reference them in pipelines without exposing values, and manage sensitive data centrally across your organization.
-
-```console
-$ nextflow secrets list
-$ nextflow secrets set AWS_ACCESS_KEY_ID
-$ nextflow secrets delete AWS_ACCESS_KEY_ID
-```
-
-See {ref}`cli-secrets` for more information.
-
 ## Module management
 
 :::{versionadded} 26.04.0
 :::
 
-The `module` command enables working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and sharing improvements with the community.
-
-Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use.
-
-### Searching for modules
-
-The `module search` command queries the module registry to discover available modules by keyword or name.
-
-Use this to find modules for specific tasks, explore available tools, or discover community modules.
-
-```console
-$ nextflow module search alignment
-$ nextflow module search "quality control" -limit 10
-$ nextflow module search bwa -output json
-```
-
-Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for JSON-formatted output.
-
-See {ref}`cli-module-search` for more information.
-
-### Installing modules
-
-The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL.
-
-Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions.
-
-```console
-$ nextflow module install nf-core/fastqc
-$ nextflow module install nf-core/fastqc -version 1.0.0
-```
-
-The installed module will be available in `modules/nf-core/fastqc`.
-
-Use the `-force` flag to reinstall a module even if local modifications exist.
-
-See {ref}`cli-module-install` for more information.
-
-### Listing modules
-
-The `module list` command displays all modules currently installed in your project, showing their versions and integrity status.
-
-Use this to review installed modules, check module versions, or detect local modifications.
-
-```console
-$ nextflow module list
-$ nextflow module list -output json
-```
-
-The output shows each module's name, installed version, and whether it has been modified locally. Use `-o json` for JSON-formatted output.
+The `module` command allows you to discover, install, and run modules through a centralized module registry. See {ref}`using-modules-page` and the {ref}`cli-module` command reference for more information.
 
-See {ref}`cli-module-list` for more information.
+## Secret management
 
-### Viewing module information
-
-The `module view` command displays detailed metadata and usage information for a specific module from the registry.
-
-Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module.
-
-```console
-$ nextflow module view nf-core/fastqc
-$ nextflow module view nf-core/fastqc -version 1.0.0
-$ nextflow module view nf-core/fastqc -output json
-```
-
-The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access.
-
-See {ref}`cli-module-view` for more information.
-
-### Running modules directly
-
-The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing.
-
-Use this to quickly run a module, test module functionality, or execute one-off data processing tasks.
-
-```console
-$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
-$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0
-```
-
-The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.):
-
-```console
-$ nextflow module run nf-core/salmon \
-    --reads reads.fq \
-    --index salmon_index \
-    -profile docker \
-    -resume
-```
-
-Process inputs can be specified like params on the command line. For example, `--reads reads.fq` corresponds to the `reads` input in the `nf-core/salmon` module. Run `nextflow module view nf-core/salmon` to see the available params for the module.
-
-See {ref}`cli-module-run` for more information.
-
-### Removing modules
-
-The `module remove` command deletes modules from your project, removing local files and configuration entries.
-
-Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline.
-
-```console
-$ nextflow module remove nf-core/fastqc
-$ nextflow module remove nf-core/fastqc -keep-files
-```
-
-By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files.
-
-See {ref}`cli-module-remove` for more information.
-
-### Publishing modules
-
-The `module publish` command uploads modules to a registry, making them available for others to install and use.
+The `secrets` command manages secure pipeline secrets.
 
-Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization.
+Use this to store credentials securely, reference them in pipelines without exposing values, and manage sensitive data centrally across your organization.
 
 ```console
-$ nextflow module publish myorg/my-module
-$ nextflow module publish myorg/my-module -dry-run
+$ nextflow secrets list
+$ nextflow secrets set AWS_ACCESS_KEY_ID
+$ nextflow secrets delete AWS_ACCESS_KEY_ID
 ```
 
-Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. The module must include `main.nf`, `meta.yml`, and `README.md` files.
-
-Use the {ref}`cli-module-spec` and {ref}`cli-module-validate` commands to prepare and validate a module before publishing.
-
-See {ref}`cli-module-publish` for more information.
+See {ref}`cli-secrets` for more information.
 
 ## Configuration and validation
 

docs/conf.py

@@ -58,7 +58,8 @@
     'flux.md': 'tutorials/flux.md',
     'developer/plugins.md': 'plugins/developing-plugins.md',
     'plugins.md': 'plugins/plugins.md',
-    'channel.md': 'workflow.md'
+    'channel.md': 'workflow.md',
+    'module.md': 'modules/modules.md'
 }
 
 # Add any paths that contain templates here, relative to this directory.