Skip to content

Commit bbbb2b3

Browse files
docs: Update config page (#6642) [ci fast]
Co-authored-by: Ben Sherman <bentshermann@gmail.com>
1 parent 34bdffd commit bbbb2b3

3 files changed

Lines changed: 54 additions & 82 deletions

File tree

docs/config.md

Lines changed: 51 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -2,50 +2,48 @@
22

33
# Configuration
44

5-
## Configuration file
5+
## Configuration files
66

7-
When a pipeline script is launched, Nextflow looks for configuration files in multiple locations. Since each configuration file may contain conflicting settings, they are applied in the following order (from lowest to highest priority):
7+
When you launch a pipeline script, Nextflow detects configuration files from multiple sources and applies them in the following order (from lowest to highest priority):
88

9-
1. The config file `$HOME/.nextflow/config` (or `$NXF_HOME/config` when {ref}`NXF_HOME <nxf-env-vars>` is set).
10-
2. The config file `nextflow.config` in the project directory
11-
3. The config file `nextflow.config` in the launch directory
12-
4. Config files specified using the `-c <config-files>` option
9+
1. `$NXF_HOME/config` (defaults to `$HOME/.nextflow/config`)
10+
2. `nextflow.config` in the project directory
11+
3. `nextflow.config` in the launch directory
12+
4. Config files specified with `-c <config-files>`
1313

1414
:::{tip}
15-
You can alternatively use the `-C <config-file>` option to specify a fixed set of configuration files and ignore all other files.
15+
You can use the `-C <config-file>` option to specify a fixed set of configuration files and ignore all other files.
1616
:::
1717

1818
(config-syntax)=
1919

2020
## Syntax
2121

22-
The Nextflow configuration syntax is based on the Nextflow script syntax. It is designed for setting configuration options in a declarative manner while also allowing for dynamic expressions where appropriate.
22+
Nextflow configuration uses a similar syntax as Nextflow scripts. Configuration options must be set declaratively, but the value of a config option can be an arbitrary expression.
2323

24-
A Nextflow config file may consist of any number of *assignments*, *blocks*, and *includes*. Config files may also contain comments in the same manner as scripts.
25-
26-
See {ref}`syntax-page` for more information about the Nextflow script syntax.
24+
A config file may contain any number of [assignments](#assignments), [blocks](#blocks), and [includes](#includes). You can add comments just like in scripts.
2725

2826
### Assignments
2927

30-
A config assignment consists of a config option and an expression separated by an equals sign:
28+
A config assignment sets a config option to a value:
3129

3230
```groovy
3331
workDir = 'work'
3432
docker.enabled = true
3533
process.maxErrors = 10
3634
```
3735

38-
A config option consists of an *option name* prefixed by any number of *scopes* separated by dots. Config scopes are used to group related config options. See {ref}`config-options` for the full set of config options.
36+
The config option consists of an *option name* prefixed by any number of *scopes*. Scopes group related config options. See {ref}`config-options` for the full set of config options.
3937

40-
The expression is typically a literal value such as a number, boolean, or string. However, any expression can be used:
38+
The value is typically a literal value, such as a number, boolean, or string. However, you can use any {ref}`expression <syntax-expressions>`:
4139

4240
```groovy
4341
params.helper_file = "${projectDir}/assets/helper.txt"
4442
```
4543

4644
### Blocks
4745

48-
A config scope can also be specified as a block, which may contain multiple configuration options. For example:
46+
You can specify config options in the same scope as a block:
4947

5048
```groovy
5149
// dot syntax
@@ -59,7 +57,7 @@ docker {
5957
}
6058
```
6159

62-
As a result, deeply nested config options can be assigned in various ways. For example, the following three assignments are equivalent:
60+
As a result, deeply nested config options can be assigned in multiple ways. The following three assignments are equivalent:
6361

6462
```groovy
6563
executor.retry.maxAttempt = 5
@@ -77,7 +75,7 @@ executor {
7775

7876
### Includes
7977

80-
A configuration file can include any number of other configuration files using the `includeConfig` keyword:
78+
You can include configuration files in other configuration files with the `includeConfig` keyword:
8179

8280
```groovy
8381
process.executor = 'sge'
@@ -90,12 +88,10 @@ includeConfig 'path/extra.config'
9088
Relative paths are resolved against the location of the including file.
9189

9290
:::{note}
93-
Config includes can also be specified within config blocks. However, config files should only be included at the top level or in a [profile](#config-profiles) so that the included config file is valid on its own and in the context in which it is included.
91+
Config files should only be included at the top level or in a [profile](#config-profiles). This ensures the included config file is valid on its own and in the context in which it is included.
9492
:::
9593

96-
(config-constants)=
97-
98-
## Constants
94+
## Constants and functions
9995

10096
The following constants are globally available in a Nextflow configuration file:
10197

@@ -113,20 +109,18 @@ The following constants are globally available in a Nextflow configuration file:
113109
`secrets: Map<String,String>`
114110
: Map of pipeline secrets. See {ref}`secrets-page` for more information.
115111

116-
## Functions
117-
118112
The following functions are globally available in a Nextflow configuration file:
119113

120114
`env( name: String ) -> String`
121-
: :::{versionadded} 24.11.0-edge
115+
: :::{versionadded} 25.04.0
122116
:::
123117
: Get the value of the environment variable with the specified name in the Nextflow launch environment.
124118

125119
(config-params)=
126120

127121
## Parameters
128122

129-
Pipeline parameters can be defined in the config file using the `params` scope:
123+
You can define pipeline parameters in config files using the `params` scope:
130124

131125
```groovy
132126
// dot syntax
@@ -141,10 +135,10 @@ params {
141135
```
142136

143137
:::{note}
144-
When including a config file, the included config is evaluated with the parameters that are defined before the include. Parameters defined after the include are not visible to the included config.
138+
When including a config file, the included config is evaluated with the parameters defined before the include. The included config cannot see parameters defined after the include.
145139
:::
146140

147-
As a best practice, declare parameters in the config file only if they are used by other config options. If a parameter is used within the script, declare it there and override it in config profiles as needed. For example:
141+
You should declare parameters in the config file only when other config options use them. When you use a parameter in the script, you should declare it there and override it in config profiles as needed:
148142

149143
```nextflow
150144
// main.nf
@@ -166,13 +160,13 @@ profiles {
166160
}
167161
```
168162

169-
See {ref}`cli-params` for information about how pipeline parameters are resolved at runtime.
163+
See {ref}`cli-params` for information about how Nextflow resolves pipeline parameters at runtime.
170164

171165
(config-process)=
172166

173167
## Process configuration
174168

175-
The `process` scope allows you to specify {ref}`process directives <process-reference>` separately from the pipeline code.
169+
You can use the `process` scope to specify {ref}`process directives <process-reference>` separately from the pipeline code.
176170

177171
For example:
178172

@@ -184,13 +178,13 @@ process {
184178
}
185179
```
186180

187-
By using this configuration, all processes in your pipeline will be executed through the SGE cluster, with the specified settings.
181+
This configuration executes all processes using the SGE executor with the given settings.
188182

189183
(config-process-selectors)=
190184

191185
### Process selectors
192186

193-
The `withLabel` selectors allow the configuration of all processes annotated with a {ref}`process-label` directive as shown below:
187+
You can use the `withLabel` selector to configure all processes annotated with a {ref}`process-label` directive:
194188

195189
```groovy
196190
process {
@@ -202,9 +196,9 @@ process {
202196
}
203197
```
204198

205-
The above configuration example assigns 16 cpus, 64 Gb of memory and the `long` queue to all processes annotated with the `big_mem` label.
199+
This configuration assigns 16 CPUs, 64 GB of memory, and the `long` queue to all processes with the `big_mem` label.
206200

207-
In the same manner, the `withName` selector allows the configuration of a specific process in your pipeline by its name. For example:
201+
You can use the `withName` selector to configure a specific process by its name:
208202

209203
```groovy
210204
process {
@@ -216,19 +210,22 @@ process {
216210
}
217211
```
218212

219-
The `withName` selector applies both to processes defined with the same name and processes included under the same alias. For example, `withName: hello` will apply to any process originally defined as `hello`, as well as any process included under the alias `hello`.
213+
The `withName` selector matches both:
220214

221-
Furthermore, selectors for the alias of an included process take priority over selectors for the original name of the process. For example, given a process defined as `hello` and included as `sayHello`, the selectors `withName: hello` and `withName: sayHello` will both be applied to the process, with the second selector taking priority over the first.
215+
- Processes defined with that name
216+
- Processes included under that alias
217+
218+
When you include a process with an alias, selectors for the alias take priority over selectors for the original name. For example, if you define a process as `hello` and include it as `sayHello`, both `withName: hello` and `withName: sayHello` apply, with `withName: sayHello` taking priority.
222219

223220
:::{tip}
224-
Label and process names do not need to be enclosed with quotes, provided the name does not include special characters (`-`, `!`, etc) and is not a keyword or a built-in type identifier. When in doubt, you can enclose the label name or process name with single or double quotes.
221+
You don't need to enclose label and process names in quotes unless they contain special characters (`-`, `!`, etc.) or are keywords or built-in type identifiers.
225222
:::
226223

227224
(config-selector-expressions)=
228225

229226
### Selector expressions
230227

231-
Both label and process name selectors allow the use of a regular expression in order to apply the same configuration to all processes matching the specified pattern condition. For example:
228+
You can use regular expressions in process selectors to apply the same configuration to all processes matching the pattern:
232229

233230
```groovy
234231
process {
@@ -239,9 +236,9 @@ process {
239236
}
240237
```
241238

242-
The above configuration snippet requests 2 cpus and 4 GB of memory for processes labeled as `hello` or `bye`.
239+
This configuration requests 2 CPUs and 4 GB of memory for processes labeled as `hello` or `bye`.
243240

244-
A process selector can be negated prefixing it with the special character `!`. For example:
241+
You can negate a selector expression by prefixing it with the special character `!`:
245242

246243
```groovy
247244
process {
@@ -251,13 +248,13 @@ process {
251248
}
252249
```
253250

254-
The above configuration snippet sets 2 cpus for every process labeled as `hello` and 4 cpus to every process *not* labeled as `hello`. It also specifies the `long` queue for every process whose name does *not* start with `align`.
251+
This configuration sets 2 CPUs for processes labeled as `hello` and 4 CPUs for all processes *not* labeled as `hello`. It also specifies the `long` queue for processes whose name does *not* start with `align`.
255252

256253
(config-selector-priority)=
257254

258255
### Selector priority
259256

260-
Process configuration settings are applied to a process in the following order (from lowest to highest priority):
257+
Nextflow applies process configuration settings in the following order (from lowest to highest priority):
261258

262259
1. Process configuration settings (without a selector)
263260
2. Process directives in the process definition
@@ -277,17 +274,18 @@ process {
277274
}
278275
```
279276

280-
With the above configuration:
281-
- All processes will use 4 cpus (unless otherwise specified in their process definition).
282-
- Processes annotated with the `hello` label will use 8 cpus.
283-
- Any process named `bye` (or imported as `bye`) will use 16 cpus.
284-
- Any process named `bye` (or imported as `bye`) invoked by a workflow named `aloha` will use 32 cpus.
277+
This configuration:
278+
279+
- Sets 4 CPUs for all processes (unless otherwise specified in their process definition)
280+
- Sets 8 CPUs for processes annotated with the `hello` label
281+
- Sets 16 CPUs for any process named `bye` (or imported as `bye`)
282+
- Sets 32 CPUs for any process named `bye` (or imported as `bye`) invoked by a workflow named `aloha`
285283

286284
(config-profiles)=
287285

288286
## Config profiles
289287

290-
Configuration files can define one or more *profiles*. A profile is a set of configuration settings that can be selected during pipeline execution using the `-profile` command line option.
288+
Configuration files can define one or more *profiles*. A profile is a set of configuration settings that can be selected at runtime using the `-profile` command line option.
291289

292290
Configuration profiles are defined in the `profiles` scope. For example:
293291

@@ -311,44 +309,18 @@ profiles {
311309
}
312310
```
313311

314-
The above configuration defines three profiles: `standard`, `cluster`, and `cloud`. Each profile provides a different configuration for a given execution environment. The `standard` profile is used by default when no profile is specified.
312+
This configuration defines three profiles: `standard`, `cluster`, and `cloud`. Each profile provides a different configuration for a given execution environment. When you do not specify a profile, Nextflow uses the `standard` profile by default.
315313

316-
Configuration profiles can be specified at runtime as a comma-separated list:
314+
You can enable configuration profiles at runtime as a comma-separated list:
317315

318316
```bash
319-
nextflow run <your script> -profile standard,cloud
320-
```
321-
322-
Config profiles are applied in the order in which they were defined in the config file, regardless of the order they are specified on the command line.
323-
324-
:::{versionadded} 25.02.0-edge
325-
When using the {ref}`strict config syntax <updating-config-syntax>`, profiles are applied in the order in which they are specified on the command line.
326-
:::
327-
328-
:::{danger}
329-
When defining a profile in the config file, avoid using both the dot and block syntax for the same scope. For example:
330-
331-
```groovy
332-
profiles {
333-
cluster {
334-
process.memory = '2 GB'
335-
process {
336-
cpus = 2
337-
}
338-
}
339-
}
317+
nextflow run main.nf -profile standard,cloud
340318
```
341319

342-
Due to a limitation of the legacy config parser, the first setting will be overwritten by the second:
343-
344-
```console
345-
$ nextflow config -profile cluster
346-
process {
347-
cpus = 2
348-
}
349-
```
320+
Nextflow applies config profiles in the order in which they were defined in the config, regardless of the order you specify them on the command line.
350321

351-
This limitation can be avoided by using the {ref}`strict config syntax <updating-config-syntax>`.
322+
:::{versionadded} 25.04.0
323+
When using the {ref}`strict syntax <strict-syntax-page>`, Nextflow applies profiles in the order you specify them on the command line.
352324
:::
353325

354326
(config-workflow-handlers)=
@@ -363,7 +335,6 @@ You can define workflow event handlers in the config file:
363335

364336
```groovy
365337
workflow.onComplete = {
366-
// any workflow property can be used here
367338
println "Pipeline complete"
368339
println "Command line: $workflow.commandLine"
369340
}

docs/reference/stdlib-types.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -731,7 +731,7 @@ The following methods are available for listing and traversing directories:
731731
: Returns the first-level elements (files and directories) in a directory.
732732

733733
`listFiles() -> Iterable<Path>`
734-
: :::{deprecated}
734+
: :::{deprecated} 26.04.0
735735
Use `listDirectory()` instead.
736736
:::
737737
: Returns the first-level elements (files and directories) in a directory.

docs/reference/syntax.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -602,6 +602,8 @@ catch( IOException e ) {
602602

603603
The try block will be executed, and if an error is raised and matches the expected error type of a catch clause, the code in that catch clause will be executed. If no catch clause is matched, the error will be raised to the next enclosing try/catch statement, or to the Nextflow runtime.
604604

605+
(syntax-expressions)=
606+
605607
## Expressions
606608

607609
An expression represents a value. A *literal* value is an expression whose value is known at compile-time, such as a number, string, or boolean. All other expressions must be evaluated at run-time.
@@ -1010,4 +1012,3 @@ The following legacy features were excluded from this page because they are depr
10101012
- The `addParams` and `params` clauses of include declarations. See {ref}`module-params` for more information.
10111013
- The `when:` section of a process definition. See {ref}`process-when` for more information.
10121014
- The `shell:` section of a process definition. See {ref}`process-shell` for more information.
1013-
- The implicit `it` closure parameter. See {ref}`script-closure` for more information.

0 commit comments

Comments
 (0)