Improve secret loading documentation and rename ConfigNullProvider to EmptySecretProvider

This commit enhances the documentation and naming of Nextflow's 2-phase
configuration loading strategy to better explain how plugin-provided secrets
are handled during startup.

## Changes

### Renamed ConfigNullProvider → EmptySecretProvider
- Better reflects the class purpose as a mock provider
- Updated all references and imports
- Maintains backward compatibility in functionality

### Enhanced Documentation
- Added comprehensive JavaDoc with ASCII flow chart
- Improved inline comments in CmdRun explaining the rationale
- Clarified that this is a mock provider, not a secret detector

## 2-Phase Configuration Loading Strategy

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   PHASE 1       │    │   BETWEEN       │    │   PHASE 2       │
│   (Mock Mode)   │    │   PHASES        │    │   (Real Mode)   │
├─────────────────┤    ├─────────────────┤    ├─────────────────┤
│ EmptySecretProv │    │ Load Plugins    │    │ Real Secrets    │
│ secrets → ""    │    │ Load Secret     │    │ secrets → value │
│ Config: fallback│ -> │ Providers       │ -> │ Config: actual  │
│ accessed=true   │    │                 │    │ values          │
└─────────────────┘    └─────────────────┘    └─────────────────┘

## Problem Solved
Configuration files may reference secrets provided by plugins (e.g., AWS secrets),
but plugins are loaded AFTER configuration parsing. The 2-phase approach:

1. **Phase 1**: Parse config with EmptySecretProvider (returns empty strings)
2. **Load plugins**: Initialize plugin-provided secret providers
3. **Phase 2**: Conditionally reload config with real secret values

## Key Benefits
- No parsing failures (config always parses successfully)
- Plugin compatibility (supports plugin-provided secret providers)
- Performance optimization (only reloads when secrets are actually used)
- Forces defensive configuration patterns

Example defensive config pattern:
```groovy
outputDir = secrets.MY_SECRET ? "results-${secrets.MY_SECRET}" : "results"

This improvement makes the codebase more maintainable by clearly documenting
the sophisticated secret loading mechanism that enables plugin-provided secrets
to work seamlessly with Nextflow's configuration system.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

Author: bentsherman

docs/cli.md

@@ -286,6 +286,10 @@ Or in JSON format:
 }
 ```
 
+:::{note}
+Parameter values in the params file can reference the following {ref}`built-in variables <config-constants>`: `baseDir`, `projectDir`, `launchDir`.
+:::
+
 Parameters are applied in the following order (from lowest to highest priority):
 
 1. Parameters defined in pipeline scripts (e.g. `main.nf`)

docs/config.md

@@ -93,6 +93,8 @@ Relative paths are resolved against the location of the including file.
 Config includes can also be specified within config blocks. However, config files should only be included at the top level or in a [profile](#config-profiles) so that the included config file is valid on its own and in the context in which it is included.
 :::
 
+(config-constants)=
+
 ## Constants
 
 The following constants are globally available in a Nextflow configuration file:

docs/developer/index.md

@@ -137,6 +137,13 @@ cd tests/checks
 ./qrun.sh
 ```
 
+To run a specific integration test:
+
+```bash
+cd tests/checks
+./qrun.sh <FOLDER>
+```
+
 To test the documentation snippets:
 
 ```bash

docs/secrets.md

@@ -13,6 +13,8 @@ This feature allows you to decouple the use of secrets in your pipelines from th
 
 When a pipeline is launched, Nextflow injects the secrets into the run without leaking them into temporary execution files. Secrets are provided to tasks as environment variables.
 
+Secrets can be used with the local executor and grid executors (e.g., Slurm or Grid Engine). Secrets can be used with the AWS Batch executor when launched from [Seqera Platform](https://seqera.io/blog/pipeline-secrets-secure-handling-of-sensitive-information-in-tower/).
+
 ## Command line
 
 The Nextflow {ref}`cli-secrets` sub-command can be used to manage secrets:
@@ -45,9 +47,32 @@ aws {
 The above snippet accesses the secrets `MY_ACCESS_KEY` and `MY_SECRET_KEY` and assigns them to the corresponding AWS config settings.
 
 :::{warning}
-Secrets cannot be assigned to pipeline parameters.
+Secrets should not be assigned to pipeline parameters, as they can be leaked by the pipeline.
+:::
+
+:::{versionadded} 25.10.0
 :::
 
+Nextflow supports the use of secrets provided by plugins (e.g., AWS secrets) in configuration. However, due to the way that plugins are loaded, there are specific considerations when using config secrets:
+
+- **Initial config load**: Nextflow first loads the configuration _without_ secrets enabled. Any reference to a secret will return the empty string `''`.
+
+- **Plugin resolution**: Plugins are resolved after the initial configuration load. This is because the configuration can specify additional plugins.
+
+- **Config reloading**: If secrets are accessed during configuration and the initial load succeeds, Nextflow will reload the configuration with secrets enabled.
+
+As a result, config secrets must be used in a way that does not cause the config resolution to fail when secrets are not present.
+
+For example:
+
+```groovy
+includeConfig secrets.MY_SECRET
+    ? "https://example.com/extra.config?secret=${secrets.MY_SECRET}"
+    : '/dev/null'
+```
+
+The above snippet includes a secured config only if the secret is present. Otherwise, it includes `/dev/null`, which is equivalent to including an empty file. The reference to `secrets.MY_SECRET` in the condition causes the config to be reloaded with secrets enabled, including secrets from plugins such as AWS secrets.
+
 (secrets-pipeline-script)=
 
 ## Pipeline script
@@ -67,10 +92,6 @@ workflow {
 The above example is only meant to demonstrate how to access a secret, not how to use it. In practice, sensitive information should not be printed to the console or output files.
 :::
 
-:::{note}
-Secrets can only be used with the local or grid executors (e.g., Slurm or Grid Engine). Secrets can be used with the AWS Batch executor when launched from [Seqera Platform](https://seqera.io/blog/pipeline-secrets-secure-handling-of-sensitive-information-in-tower/).
-:::
-
 ## Process directive
 
 Secrets can be accesses by processes using the {ref}`process-secret` directive. For example:
@@ -92,7 +113,3 @@ In the above example, the secrets `MY_ACCESS_KEY` and `MY_SECRET_KEY` are inject
 :::{warning}
 Secrets are made available as environment variables in the process script. To prevent evaluation in the Nextflow script context, escape variable names with a backslash (e.g., `\$MY_ACCESS_KEY`) as shown above.
 :::
-
-:::{note}
-Secrets can only be used with the local or grid executors (e.g., Slurm or Grid Engine). Secrets can be used with the AWS Batch executor when launched from [Seqera Platform](https://seqera.io/blog/pipeline-secrets-secure-handling-of-sensitive-information-in-tower/).
-:::

modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy

@@ -44,6 +44,8 @@ import nextflow.plugin.Plugins
 import nextflow.scm.AssetManager
 import nextflow.script.ScriptFile
 import nextflow.script.ScriptRunner
+import nextflow.secret.EmptySecretProvider
+import nextflow.secret.SecretsLoader
 import nextflow.util.CustomPoolFactory
 import nextflow.util.Duration
 import nextflow.util.HistoryFile
@@ -320,31 +322,72 @@ class CmdRun extends CmdBase implements HubOptions {
         checkRunName()
 
         printBanner()
-        Plugins.init()
 
-        // -- specify the arguments
+        // -- resolve main script
         final scriptFile = getScriptFile(pipeline)
 
         // -- load command line params
         final baseDir = scriptFile.parent
-        final cliParams = parsedParams(ConfigBuilder.getConfigVars(baseDir))
+        final cliParams = parsedParams(ConfigBuilder.getConfigVars(baseDir, null))
+
+        /*
+         * 2-PHASE CONFIGURATION LOADING STRATEGY
+         * 
+         * Problem: Configuration files may reference secrets provided by plugins (e.g., AWS secrets),
+         * but plugins are loaded AFTER configuration parsing. This creates a chicken-and-egg problem:
+         * - Config parsing needs secret values to complete
+         * - Plugin loading needs config to determine which plugins to load  
+         * - Secret providers are registered by plugins
+         * 
+         * Solution: Parse configuration twice when secrets are referenced
+         * 
+         * PHASE 1: Parse config with EmptySecretProvider (returns "" for all secrets)
+         * - Configuration must use defensive patterns: secrets.FOO ? "value-${secrets.FOO}" : "fallback"
+         * - Config parses successfully with fallback values
+         * - EmptySecretProvider tracks if ANY secrets were accessed
+         */
+
+        // -- PHASE 1: Load config with mock secrets provider
+        final secretsProvider = new EmptySecretProvider()
+        ConfigBuilder builder = new ConfigBuilder()
+            .setOptions(launcher.options)
+            .setCmdRun(this)
+            .setBaseDir(scriptFile.parent)
+            .setCliParams(cliParams)
+            .setSecretsProvider(secretsProvider)  // Mock provider returns empty strings
+        ConfigMap config = builder.build()
+        Map configParams = builder.getConfigParams()
+
+        // -- Load plugins (may register secret providers)
+        Plugins.init()
+        Plugins.load(config)
 
-        // create the config object
-        final builder = new ConfigBuilder()
+        // -- Initialize real secrets system
+        SecretsLoader.getInstance().load()
+
+        /*
+         * PHASE 2: Conditionally reload config with real secrets
+         * - Only reload if Phase 1 actually accessed any secrets
+         * - This time, real secret providers are available (including plugin-provided ones)
+         * - Same config expressions now resolve with actual secret values
+         */
+
+        // -- PHASE 2: Reload config if secrets were used in Phase 1
+        if( secretsProvider.usedSecrets() ) {
+            log.debug "Config file used secrets -- reloading config with secrets provider"
+            builder = new ConfigBuilder()
                 .setOptions(launcher.options)
                 .setCmdRun(this)
-                .setBaseDir(baseDir)
+                .setBaseDir(scriptFile.parent)
                 .setCliParams(cliParams)
-        final config = builder.build()
-        final configParams = builder.getConfigParams()
+                // No .setSecretsProvider() - uses real secrets system now
+            config = builder.build()
+            configParams = builder.getConfigParams()
+        }
 
         // check DSL syntax in the config
         launchInfo(config, scriptFile)
 
-        // -- load plugins
-        final cfg = plugins ? [plugins: plugins.tokenize(',')] : config
-        Plugins.load(cfg)
-
         // -- validate config options
         if( NF.isSyntaxParserV2() )
             new ConfigValidator().validate(config)

modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy

@@ -34,6 +34,7 @@ import nextflow.cli.CmdRun
 import nextflow.exception.AbortOperationException
 import nextflow.exception.ConfigParseException
 import nextflow.secret.SecretsLoader
+import nextflow.secret.SecretsProvider
 import nextflow.util.HistoryFile
 import nextflow.util.SecretHelper
 /**
@@ -76,6 +77,8 @@ class ConfigBuilder {
 
     boolean showMissingVariables
 
+    SecretsProvider secretsProvider
+
     Map<ConfigObject, String> emptyVariables = new LinkedHashMap<>(10)
 
     Map<String,String> env = new HashMap<>(SysEnv.get())
@@ -104,6 +107,11 @@ class ConfigBuilder {
         return this
     }
 
+    ConfigBuilder setSecretsProvider(SecretsProvider value) {
+        this.secretsProvider = value
+        return this
+    }
+
     ConfigBuilder setOptions( CliOptions options ) {
         this.options = options
         return this
@@ -337,17 +345,20 @@ class ConfigBuilder {
         // this is needed to make sure to reuse the same
         // instance of the config vars across different instances of the ConfigBuilder
         // and prevent multiple parsing of the same params file (which can even be remote resource)
-        return getConfigVars(baseDir)
+        final secretContext = secretsProvider
+            ? SecretsLoader.secretContext(secretsProvider)
+            : SecretsLoader.secretContext()
+        return getConfigVars(baseDir, secretContext)
     }
 
     @Memoized
-    static Map getConfigVars(Path base) {
+    static Map getConfigVars(Path base, Object secretContext) {
         final binding = new HashMap(10)
         binding.put('baseDir', base)
         binding.put('projectDir', base)
         binding.put('launchDir', Paths.get('.').toRealPath())
         binding.put('outputDir', Paths.get('results').complete())
-        binding.put('secrets', SecretsLoader.secretContext())
+        binding.put('secrets', secretContext)
         return binding
     }
 
@@ -560,6 +571,9 @@ class ConfigBuilder {
         if( cmdRun.preview )
             config.preview = cmdRun.preview
 
+        if( cmdRun.plugins )
+            config.plugins = cmdRun.plugins.tokenize(',')
+
         // -- sets the working directory
         if( cmdRun.workDir )
             config.workDir = cmdRun.workDir

modules/nextflow/src/main/groovy/nextflow/secret/EmptySecretProvider.groovy

@@ -0,0 +1,120 @@
+/*
+ * Copyright 2013-2024, Seqera Labs
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package nextflow.secret
+
+import groovy.transform.CompileStatic
+
+/**
+ * A mock secrets provider that returns empty strings for all secret requests.
+ * 
+ * <p>This provider is a critical component of Nextflow's 2-phase configuration loading
+ * strategy, which solves the chicken-and-egg problem between configuration parsing 
+ * and plugin loading.
+ * 
+ * <h2>The Problem</h2>
+ * Configuration files may reference secrets provided by plugins (e.g., AWS secrets), 
+ * but plugins are loaded AFTER configuration parsing. This creates a dependency cycle:
+ * <ul>
+ *   <li>Config parsing needs secret values to complete</li>
+ *   <li>Plugin loading needs config to determine which plugins to load</li>
+ *   <li>Secret providers are registered by plugins</li>
+ * </ul>
+ * 
+ * <h2>The Solution: 2-Phase Configuration Loading</h2>
+ * <pre>
+ * ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+ * │   PHASE 1       │    │   BETWEEN       │    │   PHASE 2       │
+ * │   (Mock Mode)   │    │   PHASES        │    │   (Real Mode)   │
+ * ├─────────────────┤    ├─────────────────┤    ├─────────────────┤
+ * │ EmptySecretProv │    │ Load Plugins    │    │ Real Secrets    │
+ * │ secrets → ""    │    │ Load Secret     │    │ secrets → value │
+ * │ Config: fallback│ -> │ Providers       │ -> │ Config: actual  │
+ * │ accessed=true   │    │                 │    │ values          │
+ * └─────────────────┘    └─────────────────┘    └─────────────────┘
+ * </pre>
+ * 
+ * <h3>Phase 1: Mock Configuration Loading</h3>
+ * <ol>
+ *   <li>EmptySecretProvider is used as the secrets provider</li>
+ *   <li>Configuration is parsed normally</li>
+ *   <li>When {@code secrets.SOME_NAME} is referenced, this provider returns empty string</li>
+ *   <li>The {@code accessed} flag is set to {@code true}</li>
+ *   <li>Config must use defensive patterns: {@code secrets.FOO ? "value-${secrets.FOO}" : "fallback"}</li>
+ *   <li>Result: Configuration parses successfully with fallback values</li>
+ * </ol>
+ * 
+ * <h3>Between Phases: Plugin and Secrets Loading</h3>
+ * <ol>
+ *   <li>Plugins are loaded using the successfully parsed configuration</li>
+ *   <li>Plugin-provided secrets providers are registered</li>
+ *   <li>The real secrets loading system is initialized</li>
+ * </ol>
+ * 
+ * <h3>Phase 2: Real Configuration Loading (Conditional)</h3>
+ * <ol>
+ *   <li>Check if {@code usedSecrets()} returns {@code true}</li>
+ *   <li>If secrets were accessed in Phase 1, reload the entire configuration</li>
+ *   <li>This time, real secret providers are available</li>
+ *   <li>Same config expressions now resolve with actual secret values</li>
+ *   <li>Result: Configuration contains real secret values instead of fallbacks</li>
+ * </ol>
+ * 
+ * <h2>Example Usage</h2>
+ * Given a config file:
+ * <pre>
+ * outputDir = secrets.MY_SECRET ? "results-${secrets.MY_SECRET}" : "results"
+ * </pre>
+ * 
+ * <b>Phase 1:</b> {@code secrets.MY_SECRET} returns {@code ""} → {@code outputDir = "results"}
+ * <b>Phase 2:</b> {@code secrets.MY_SECRET} returns {@code "hello-world"} → {@code outputDir = "results-hello-world"}
+ * 
+ * <h2>Key Benefits</h2>
+ * <ul>
+ *   <li><b>No parsing failures:</b> Configuration always parses successfully</li>
+ *   <li><b>Plugin compatibility:</b> Supports plugin-provided secret providers</li>
+ *   <li><b>Performance:</b> Only reloads config when secrets are actually used</li>
+ *   <li><b>Defensive configs:</b> Forces robust configuration patterns</li>
+ * </ul>
+ * 
+ * <h2>Important Notes</h2>
+ * <ul>
+ *   <li>This provider does NOT remember which specific secrets were accessed</li>
+ *   <li>It only tracks WHETHER any secrets were accessed at all</li>
+ *   <li>Configuration files MUST handle empty secret values gracefully</li>
+ *   <li>The 2-phase loading is transparent to the user</li>
+ * </ul>
+ *
+ * @author Ben Sherman <bentshermann@gmail.com>
+ * @see nextflow.cli.CmdRun#run()
+ * @see nextflow.secret.SecretsLoader
+ */
+@CompileStatic
+class EmptySecretProvider extends NullProvider {
+
+    private boolean accessed
+
+    @Override
+    Secret getSecret(String name) {
+        accessed = true
+        return new SecretImpl(name, '')
+    }
+
+    boolean usedSecrets() {
+        return accessed
+    }
+}

modules/nextflow/src/main/groovy/nextflow/secret/SecretsLoader.groovy

@@ -78,5 +78,9 @@ class SecretsLoader {
         final provider = isEnabled() ? getInstance().load() : new NullProvider()
         return makeSecretsContext(provider)
     }
-    
+
+    static Object secretContext(SecretsProvider provider) {
+        return makeSecretsContext(provider)
+    }
+
 }

modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy

@@ -21,7 +21,7 @@ import org.pf4j.ExtensionPoint
 /**
  * Factory class for creating {@link TraceObserverV2} instances
  *
- * @author Ben Shermann <bentshermann@gmail.com>
+ * @author Ben Sherman <bentshermann@gmail.com>
  */
 interface TraceObserverFactoryV2 extends ExtensionPoint {