diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 3613344ae8..7502adc387 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -133,7 +133,7 @@ jobs: fail-fast: false matrix: java_version: [17, 23] - test_mode: ["test_integration", "test_docs", "test_aws", "test_azure", "test_google", "test_wave"] + test_mode: ["test_integration", "test_parser_v2", "test_docs", "test_aws", "test_azure", "test_google", "test_wave"] steps: - name: Checkout uses: actions/checkout@v4 diff --git a/build.gradle b/build.gradle index 03de4ffe27..af43ab13e5 100644 --- a/build.gradle +++ b/build.gradle @@ -85,6 +85,10 @@ allprojects { maven { url 'https://oss.sonatype.org/content/repositories/snapshots' } maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases" } maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots" } + maven { + url 'https://jitpack.io' + content { includeGroup 'com.github.nextflow-io.language-server' } + } } configurations { diff --git a/docs/config.md b/docs/config.md index 0818f2e3c1..75cbb5775e 100644 --- a/docs/config.md +++ b/docs/config.md @@ -254,13 +254,12 @@ With the above configuration: ## Config profiles -Configuration files can contain the definition of one or more *profiles*. A profile is a set of configuration attributes that can be selected during pipeline execution by using the `-profile` command line option. +Configuration files can define one or more *profiles*. A profile is a set of configuration settings that can be selected during pipeline execution using the `-profile` command line option. -Configuration profiles are defined by using the special scope `profiles`, which group the attributes that belong to the same profile using a common prefix. For example: +Configuration profiles are defined in the `profiles` scope. For example: ```groovy profiles { - standard { process.executor = 'local' } @@ -276,41 +275,47 @@ profiles { process.container = 'cbcrg/imagex' docker.enabled = true } - } ``` -This configuration defines three different profiles: `standard`, `cluster`, and `cloud`, that each set different process -configuration strategies depending on the target runtime platform. The `standard` profile is used by default when no profile is specified. +The above configuration defines three profiles: `standard`, `cluster`, and `cloud`. Each profile provides a different configuration for a given execution environment. The `standard` profile is used by default when no profile is specified. -:::{tip} -Multiple configuration profiles can be specified by separating the profile names with a comma, for example: +Configuration profiles can be specified at runtime as a comma-separated list: ```bash nextflow run -profile standard,cloud ``` Config profiles are applied in the order in which they were defined in the config file, regardless of the order they are specified on the command line. + +:::{versionadded} 25.02.0-edge +When using the {ref}`strict config syntax `, profiles are applied in the order in which they are specified on the command line. ::: :::{danger} -When using the `profiles` feature in your config file, do NOT set attributes in the same scope both inside and outside a `profiles` context. For example: +When defining a profile in the config file, avoid using both the dot and block syntax for the same scope. For example: ```groovy -process.cpus = 1 - profiles { - foo { - process.memory = '2 GB' - } + foo { + process.memory = '2 GB' + process { + cpus = 2 + } + } +} +``` - bar { - process.memory = '4 GB' - } +Due to a limitation of the legacy config parser, the first setting will be overwritten by the second: + +```console +$ nextflow config -profile foo +process { + cpus = 2 } ``` -In the above example, the `process.cpus` attribute is not correctly applied because the `process` scope is also used in the `foo` and `bar` profiles. +This limitation can be avoided by using the {ref}`strict config syntax `. ::: ## Workflow handlers diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index 8bb17797bd..480a725470 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -182,6 +182,11 @@ The following environment variables control the configuration of the Nextflow ru ::: : Enable the use of Spack recipes defined by using the {ref}`process-spack` directive. (default: `false`). +`NXF_SYNTAX_PARSER` +: :::{versionadded} 25.02.0-edge + ::: +: Set to `'v2'` to use the {ref}`strict syntax ` for Nextflow config files (default: `'v1'`). + `NXF_TEMP` : Directory where temporary files are stored diff --git a/docs/updating-syntax.md b/docs/updating-syntax.md index ce0d3224e4..10b48e8033 100644 --- a/docs/updating-syntax.md +++ b/docs/updating-syntax.md @@ -487,8 +487,14 @@ The process `when` section is deprecated. Use conditional logic, such as an `if` The process `shell` section is deprecated. Use the `script` block instead. The VS Code extension provides syntax highlighting and error checking to help distinguish between Nextflow variables and Bash variables. +(updating-config-syntax)= + ### Configuration syntax +:::{versionadded} 25.02.0-edge +The strict config syntax can be enabled in Nextflow by setting `NXF_SYNTAX_PARSER=v2`. +::: + See {ref}`Configuration ` for a comprehensive description of the configuration language. Currently, Nextflow parses config files as Groovy scripts, allowing the use of scripting constructs like variables, helper functions, try-catch blocks, and conditional logic for dynamic configuration: diff --git a/modules/nextflow/build.gradle b/modules/nextflow/build.gradle index c005504dc3..75a398b364 100644 --- a/modules/nextflow/build.gradle +++ b/modules/nextflow/build.gradle @@ -20,6 +20,7 @@ compileGroovy { dependencies { api(project(':nf-commons')) api(project(':nf-httpfs')) + api 'com.github.nextflow-io.language-server:compiler:main-SNAPSHOT' api "org.apache.groovy:groovy:4.0.25" api "org.apache.groovy:groovy-nio:4.0.25" api "org.apache.groovy:groovy-xml:4.0.25" diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy b/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy index 76cada7d75..f8500d63f2 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy @@ -71,8 +71,6 @@ class ConfigBuilder { List parsedConfigFiles = [] - List parsedProfileNames - boolean showClosures boolean stripSecrets @@ -347,7 +345,7 @@ class ConfigBuilder { assert env != null final ignoreIncludes = options ? options.ignoreConfigIncludes : false - final slurper = new ConfigParser() + final slurper = ConfigParserFactory.create() .setRenderClosureAsString(showClosures) .setStripSecrets(stripSecrets) .setIgnoreIncludes(ignoreIncludes) @@ -384,9 +382,8 @@ class ConfigBuilder { } } - this.parsedProfileNames = new ArrayList<>(slurper.getProfileNames()) if( validateProfile ) { - checkValidProfile(slurper.getConditionalBlockNames()) + checkValidProfile(slurper.getProfiles()) } } @@ -421,7 +418,7 @@ class ConfigBuilder { log.debug "Applying config profile: `${profile}`" def allNames = profile.tokenize(',') - slurper.registerConditionalBlock('profiles', allNames) + slurper.setProfiles(allNames) def config = parse0(slurper,entry) validate(config,entry) diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigParser.groovy b/modules/nextflow/src/main/groovy/nextflow/config/ConfigParser.groovy index 787f1da57d..9a51a02f81 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigParser.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/ConfigParser.groovy @@ -18,503 +18,70 @@ package nextflow.config import java.nio.file.Path -import ch.artecat.grengine.Grengine -import com.google.common.hash.Hashing -import groovy.transform.PackageScope -import nextflow.ast.NextflowXform -import nextflow.exception.ConfigParseException -import nextflow.extension.Bolts -import nextflow.file.FileHelper -import nextflow.util.Duration -import nextflow.util.MemoryUnit -import org.codehaus.groovy.control.CompilerConfiguration -import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer -import org.codehaus.groovy.control.customizers.ImportCustomizer -import org.codehaus.groovy.runtime.InvokerHelper - -/* - * Copyright 2003-2013 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - - /** - * A ConfigSlurper that allows to include a config file into another. For example: - * - * 
- *     process {
- *         foo = 1
- *         that = 2
- *
- *         includeConfig( 'path/to/another/config/file' )
+ * Interface for Nextflow config parsers.
  *
- *     }
- *
- *
- * - * See http://naleid.com/blog/2009/07/30/modularizing-groovy-config-files-with-a-dash-of-meta-programming - * - * @author Paolo Di Tommaso - * - */ - -/** - *

- * ConfigSlurper is a utility class for reading configuration files defined in the form of Groovy - * scripts. Configuration settings can be defined using dot notation or scoped using closures - * - * 


- *   grails.webflow.stateless = true
- *    smtp {
- *        mail.host = 'smtp.myisp.com'
- *        mail.auth.user = 'server'
- *    }
- *    resources.URL = "http://localhost:80/resources"
- *
- * - *

Settings can either be bound into nested maps or onto a specified JavaBean instance. In the case - * of the latter an error will be thrown if a property cannot be bound. - * - * @author Graeme Rocher - * @author Andres Almiray - * @since 1.5 + * @author Ben Sherman */ -class ConfigParser { - private static final ENVIRONMENTS_METHOD = 'environments' - - private Map bindingVars = [:] - private Map paramVars = [:] - - private final Map> conditionValues = [:] - private final Stack> conditionalBlocks = new Stack>() - private final Set conditionalNames = new HashSet<>() - private final Set profileNames = new HashSet<>() - - private boolean ignoreIncludes - - private boolean renderClosureAsString - - private boolean stripSecrets - - private Grengine grengine - - ConfigParser() { - this('') - } - - /** - * Constructs a new IncludeConfigSlurper instance using the given environment - * @param env The Environment to use - */ - ConfigParser(String env) { - conditionValues[ENVIRONMENTS_METHOD] = [env] - } - - ConfigParser registerConditionalBlock(String blockName, String blockValue) { - if (blockName) { - if (!blockValue) { - conditionValues.remove(blockName) - } - else { - conditionValues[blockName] = [blockValue] - } - } - return this - } - - ConfigParser registerConditionalBlock(String blockName, List blockValues) { - if (blockName) { - if (!blockValues) { - conditionValues.remove(blockName) - } - else { - conditionValues[blockName] = blockValues - } - } - return this - } - - /** - * @return - * When a conditional block is registered this method returns the collection - * of block names visited during the parsing - */ - Set getConditionalBlockNames() { - Collections.unmodifiableSet(conditionalNames) - } +interface ConfigParser { /** - * Returns the profile names defined in the config file + * Toggle whether config include statements should be ignored. * - * @return The set of profile names. + * @param value */ - Set getProfileNames() { profileNames } - - private Grengine getGrengine() { - if( grengine ) { - return grengine - } - - // set the required base script - def config = new CompilerConfiguration() - config.scriptBaseClass = ConfigBase.class.name - if( stripSecrets ) - config.addCompilationCustomizers(new ASTTransformationCustomizer(StripSecretsXform)) - def params = [:] - if( renderClosureAsString ) - params.put('renderClosureAsString', true) - config.addCompilationCustomizers(new ASTTransformationCustomizer(params, ConfigTransform)) - config.addCompilationCustomizers(new ASTTransformationCustomizer(NextflowXform)) - // add implicit types - def importCustomizer = new ImportCustomizer() - importCustomizer.addImports( Duration.name ) - importCustomizer.addImports( MemoryUnit.name ) - config.addCompilationCustomizers(importCustomizer) - grengine = new Grengine(config) - } - - ConfigParser setRenderClosureAsString(boolean value) { - this.renderClosureAsString = value - return this - } - - ConfigParser setStripSecrets(boolean value) { - this.stripSecrets = value - return this - } + ConfigParser setIgnoreIncludes(boolean value) /** - * Sets any additional variables that should be placed into the binding when evaluating Config scripts - */ - ConfigParser setBinding(Map vars) { - this.bindingVars = vars - return this - } - - ConfigParser setParams(Map vars) { - // deep clone the map to prevent side-effect - // see https://github.com/nextflow-io/nextflow/issues/1923 - this.paramVars = Bolts.deepClone(vars) - return this - } - - - /** - * Creates a unique name for the config class in order to avoid collision - * with top level configuration scopes + * Toggle whether to strip secrets when rendering the config. * - * @param text - * @return + * @param value */ - private String createUniqueName(String text) { - def hash = Hashing - .murmur3_32() - .newHasher() - .putUnencodedChars(text) - .hash() - return "_nf_config_$hash" - } - - private Script loadScript(String text) { - (Script)getGrengine().load(text, createUniqueName(text)).newInstance() - } + ConfigParser setStripSecrets(boolean value) /** - * Parses a ConfigObject instances from an instance of java.util.Properties - * @param The java.util.Properties instance + * Toggle whether to render the source code of closures. + * + * @param value */ - ConfigObject parse(Properties properties) { - ConfigObject config = new ConfigObject() - for (key in properties.keySet()) { - def tokens = key.split(/\./) - - def current = config - def last - def lastToken - def foundBase = false - for (token in tokens) { - if (foundBase) { - // handle not properly nested tokens by ignoring - // hierarchy below this point - lastToken += "." + token - current = last - } else { - last = current - lastToken = token - current = current."${token}" - if (!(current instanceof ConfigObject)) foundBase = true - } - } + ConfigParser setRenderClosureAsString(boolean value) - if (current instanceof ConfigObject) { - if (last[lastToken]) { - def flattened = last.flatten() - last.clear() - flattened.each { k2, v2 -> last[k2] = v2 } - last[lastToken] = properties.get(key) - } - else { - last[lastToken] = properties.get(key) - } - } - current = config - } - return config - } /** - * Parse the given script as a string and return the configuration object + * Toggle whether to raise an error if a missing property is accessed. * - * @see ConfigParser#parse(groovy.lang.Script) + * @param value */ - ConfigObject parse(String text) { - return parse(loadScript(text)) - } + ConfigParser setStrict(boolean value) /** - * Parse the given script into a configuration object (a Map) - * (This method creates a new class to parse the script each time it is called.) - * @param script The script to parse - * @return A Map of maps that can be navigating with dot de-referencing syntax to obtain configuration entries + * Define variables which will be made available to the config script. + * + * @param vars */ - @Deprecated - ConfigObject parse(Script script) { - return parse(script, null) - } + ConfigParser setBinding(Map vars) /** - * Parses a Script represented by the given URL into a ConfigObject + * Define pipeline parameters which will be made available to the config script. * - * @param location The location of the script to parse - * @return The ConfigObject instance + * @param vars */ - @Deprecated - ConfigObject parse(URL location) { - return parse(loadScript(location.text), FileHelper.asPath(location.toURI())) - } - - ConfigObject parse(File file) { - return parse(file.toPath()) - } - - ConfigObject parse(Path path) { - return parse(loadScript(path.text), path) - } + ConfigParser setParams(Map vars) /** - * Parses the passed groovy.lang.Script instance using the second argument to allow the ConfigObject - * to retain an reference to the original location other Groovy script - * - * @param script The groovy.lang.Script instance - * @param location The original location of the Script as a URL - * @return The ConfigObject instance + * Parse a config object from the given source. */ - ConfigObject parse(Script _script, Path location) { - final script = (ConfigBase)_script - Stack currentConditionalBlock = new Stack() - def config = location ? new ConfigObject(location.toUri().toURL()) : new ConfigObject() - GroovySystem.metaClassRegistry.removeMetaClass(script.class) - def mc = script.class.metaClass - def prefix = "" - LinkedList stack = new LinkedList() - LinkedList profileStack = new LinkedList() - stack << [config: config, scope: [:]] - boolean withinProfile = false - - def pushStack = { co -> - stack << [config: co, scope: stack.last.scope.clone()] - } - def assignName = { name, co -> - def current = stack.last - current.config[name] = co - current.scope[name] = co - } - mc.getProperty = { String name -> - def current = stack.last - def result - if (current.config.get(name)) { - result = current.config.get(name) - } else if (current.scope.get(name)) { - result = current.scope[name] - } else { - try { - result = InvokerHelper.getProperty(this, name) - } catch (GroovyRuntimeException e) { - result = new ConfigObject() - assignName.call(name, result) - } - } - if( name=='params' && result instanceof Map && paramVars ) { - result.putAll(Bolts.deepMerge(result, paramVars)) - } - return result - } - - ConfigObject overrides = new ConfigObject() - mc.invokeMethod = { String name, args -> - def result - if (args.length == 1 && args[0] instanceof Closure) { - if( profileStack && profileStack.last == 'profiles' ) - profileNames.add(name) - - if (name in conditionValues.keySet()) { - try { - if( name == 'profiles' ){ - withinProfile=true - } - currentConditionalBlock.push(name) - conditionalBlocks.push([:]) - args[0].call() - } finally { - currentConditionalBlock.pop() - for (entry in conditionalBlocks.pop().entrySet()) { - def c = stack.last.config - (c != config? c : overrides).merge(entry.value) - } - if( name == 'profiles' ){ - withinProfile=false - } - } - } else if (currentConditionalBlock.size() > 0) { - String conditionalBlockKey = currentConditionalBlock.peek() - conditionalNames.add(name) - if (name in conditionValues[conditionalBlockKey]) { - def co = conditionalBlocks.peek()[conditionalBlockKey] - if( co == null ) { - co = new ConfigObject() - conditionalBlocks.peek()[conditionalBlockKey] = co - } - - pushStack.call(co) - try { - currentConditionalBlock.pop() - args[0].call() - } finally { - currentConditionalBlock.push(conditionalBlockKey) - } - stack.removeLast() - } - } - else if( name == 'plugins' ) { - if( stack.size()>1 ) - throw new ConfigParseException("Plugins definition is only allowed in config top-most scope") - // Implements `plugins` mini-dsl for plugins definition - def dsl = new PluginsDsl() - def clo = args[0] as Closure - clo.delegate = dsl - clo.resolveStrategy = Closure.DELEGATE_ONLY - clo.call() - assignName.call(name, dsl.plugins) - } - else { - def current = name=='profiles' || withinProfile ? stack.first : stack.last - def co - if (current.config.containsKey(name) && current.config.get(name) instanceof ConfigObject) { - co = current.config.get(name) - } - else if (current.scope.containsKey(name) && current.scope.get(name) instanceof ConfigObject) { - co = current.scope.get(name).clone() - } - else { - co = new ConfigObject() - } - - profileStack.add(name) - assignName.call(name, co) - pushStack.call(co) - args[0].call() - stack.removeLast() - profileStack.removeLast() - - if (current.scope.containsKey(name) && current.scope.get(name) instanceof ConfigObject) { - if( current.scope.get(name) != co) { - current.scope.get(name).merge(co) - } - } else { - current.scope.put(name,co) - } - } - } else if (args.length == 2 && args[1] instanceof Closure) { - try { - prefix = name + '.' - assignName.call(name, args[0]) - args[1].call() - } finally { prefix = "" } - } else { - MetaMethod mm = mc.getMetaMethod(name, args) - if (mm) { - result = mm.invoke(delegate, args) - } else { - throw new MissingMethodException(name, getClass(), args) - } - } - result - } - script.metaClass = mc - - def setProperty = { String name, value -> - assignName.call(prefix + name, value) - } - def binding = new ConfigBinding(setProperty) - if (this.bindingVars) { - binding.getVariables().putAll(this.bindingVars) - } - - // add the script file location into the binding - if( location ) { - script.setConfigPath(location) - } - - // disable include parsing when required - script.setIgnoreIncludes(ignoreIncludes) - script.setRenderClosureAsString(renderClosureAsString) - script.setStripSecrets(stripSecrets) - - // -- set the binding and run - script.binding = binding - script.run() - config.merge(overrides) - - return config - } + ConfigObject parse(String text) + ConfigObject parse(File file) + ConfigObject parse(Path path) /** - * Disable parsing of {@code includeConfig} directive - * - * @param value A boolean value, when {@code true} includes are disabled - * @return The {@link ConfigParser} object itself + * Set the profiles that should be applied. */ - ConfigParser setIgnoreIncludes(boolean value) { - this.ignoreIncludes = value - return this - } + ConfigParser setProfiles(List profiles) /** - * Since Groovy Script doesn't support overriding setProperty, we have to using a trick with the Binding to provide this - * functionality + * Get the set of available profiles. */ - @PackageScope - static class ConfigBinding extends Binding { - Closure callable + Set getProfiles() - ConfigBinding(Closure c) { - this.callable = c - } - - void setVariable(String name, Object value) { - callable(name, value) - } - } } - diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigParserFactory.groovy b/modules/nextflow/src/main/groovy/nextflow/config/ConfigParserFactory.groovy new file mode 100644 index 0000000000..d61200e7d3 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/ConfigParserFactory.groovy @@ -0,0 +1,47 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package nextflow.config + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.SysEnv +import nextflow.config.parser.v1.ConfigParserV1 +import nextflow.config.parser.v2.ConfigParserV2 + +/** + * Factory for creating an instance of {@link ConfigParser}. + * + * @author Ben Sherman + */ +@Slf4j +@CompileStatic +class ConfigParserFactory { + + static ConfigParser create() { + final parser = SysEnv.get('NXF_SYNTAX_PARSER', 'v1') + if( parser == 'v1' ) { + return new ConfigParserV1() + } + if( parser == 'v2' ) { + log.debug "Using config parser v2" + return new ConfigParserV2() + } + throw new IllegalStateException("Invalid NXF_SYNTAX_PARSER setting -- should be either 'v1' or 'v2'") + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/PluginsDsl.groovy b/modules/nextflow/src/main/groovy/nextflow/config/PluginsDsl.groovy deleted file mode 100644 index e840df8028..0000000000 --- a/modules/nextflow/src/main/groovy/nextflow/config/PluginsDsl.groovy +++ /dev/null @@ -1,23 +0,0 @@ -package nextflow.config - -import groovy.transform.CompileStatic - -/** - * Model a mini-dsl for plugins configuration - * - * @author Paolo Di Tommaso - */ -@CompileStatic -class PluginsDsl { - - private Set plugins = [] - - Set getPlugins() { plugins } - - void id( String plg ) { - if( !plg ) - throw new IllegalArgumentException("Plugin id cannot be empty or null") - plugins << plg - } - -} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigBase.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigBase.groovy similarity index 98% rename from modules/nextflow/src/main/groovy/nextflow/config/ConfigBase.groovy rename to modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigBase.groovy index a97e23f19a..e970d1e80a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigBase.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigBase.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -package nextflow.config +package nextflow.config.parser.v1 import java.nio.file.NoSuchFileException import java.nio.file.Path @@ -22,6 +22,7 @@ import java.nio.file.Path import ch.artecat.grengine.Grengine import groovy.transform.Memoized import nextflow.SysEnv +import nextflow.config.StripSecretsXform import nextflow.exception.IllegalConfigException import nextflow.file.FileHelper import org.codehaus.groovy.control.CompilerConfiguration diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigParserV1.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigParserV1.groovy new file mode 100644 index 0000000000..a259185a96 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigParserV1.groovy @@ -0,0 +1,500 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v1 + +import java.nio.file.Path + +import ch.artecat.grengine.Grengine +import com.google.common.hash.Hashing +import groovy.transform.PackageScope +import nextflow.ast.NextflowXform +import nextflow.config.ConfigParser +import nextflow.config.StripSecretsXform +import nextflow.exception.ConfigParseException +import nextflow.extension.Bolts +import nextflow.file.FileHelper +import nextflow.util.Duration +import nextflow.util.MemoryUnit +import org.codehaus.groovy.control.CompilerConfiguration +import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer +import org.codehaus.groovy.control.customizers.ImportCustomizer +import org.codehaus.groovy.runtime.InvokerHelper + +/* + * Copyright 2003-2013 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + + +/** + * A ConfigSlurper that allows to include a config file into another. For example: + * + * 

+ *     process {
+ *         foo = 1
+ *         that = 2
+ *
+ *         includeConfig( 'path/to/another/config/file' )
+ *
+ *     }
+ *
+ *
+ * + * See http://naleid.com/blog/2009/07/30/modularizing-groovy-config-files-with-a-dash-of-meta-programming + * + * @author Paolo Di Tommaso + * + */ + +/** + *

+ * ConfigSlurper is a utility class for reading configuration files defined in the form of Groovy + * scripts. Configuration settings can be defined using dot notation or scoped using closures + * + * 


+ *   grails.webflow.stateless = true
+ *    smtp {
+ *        mail.host = 'smtp.myisp.com'
+ *        mail.auth.user = 'server'
+ *    }
+ *    resources.URL = "http://localhost:80/resources"
+ *
+ * + *

Settings can either be bound into nested maps or onto a specified JavaBean instance. In the case + * of the latter an error will be thrown if a property cannot be bound. + * + * @author Graeme Rocher + * @author Andres Almiray + * @since 1.5 + */ +class ConfigParserV1 implements ConfigParser { + private Map bindingVars = [:] + private Map paramVars = [:] + + private final Map> conditionValues = [:] + private final Stack> conditionalBlocks = new Stack>() + private final Set conditionalNames = new HashSet<>() + private final Set profileNames = new HashSet<>() + + private boolean ignoreIncludes + + private boolean renderClosureAsString + + private boolean stripSecrets + + private Grengine grengine + + @Override + ConfigParser setProfiles(List profiles) { + final blockName = 'profiles' + if (!profiles) { + conditionValues.remove(blockName) + } + else { + conditionValues[blockName] = profiles + } + return this + } + + @Override + Set getProfiles() { + Collections.unmodifiableSet(conditionalNames) + } + + private Grengine getGrengine() { + if( grengine ) { + return grengine + } + + // set the required base script + def config = new CompilerConfiguration() + config.scriptBaseClass = ConfigBase.class.name + if( stripSecrets ) + config.addCompilationCustomizers(new ASTTransformationCustomizer(StripSecretsXform)) + def params = [:] + if( renderClosureAsString ) + params.put('renderClosureAsString', true) + config.addCompilationCustomizers(new ASTTransformationCustomizer(params, ConfigTransform)) + config.addCompilationCustomizers(new ASTTransformationCustomizer(NextflowXform)) + // add implicit types + def importCustomizer = new ImportCustomizer() + importCustomizer.addImports( Duration.name ) + importCustomizer.addImports( MemoryUnit.name ) + config.addCompilationCustomizers(importCustomizer) + grengine = new Grengine(config) + } + + @Override + ConfigParser setRenderClosureAsString(boolean value) { + this.renderClosureAsString = value + return this + } + + @Override + ConfigParser setStrict(boolean value) { + // not supported + return this + } + + @Override + ConfigParser setStripSecrets(boolean value) { + this.stripSecrets = value + return this + } + + /** + * Sets any additional variables that should be placed into the binding when evaluating Config scripts + */ + @Override + ConfigParser setBinding(Map vars) { + this.bindingVars = vars + return this + } + + @Override + ConfigParser setParams(Map vars) { + // deep clone the map to prevent side-effect + // see https://github.com/nextflow-io/nextflow/issues/1923 + this.paramVars = Bolts.deepClone(vars) + return this + } + + + /** + * Creates a unique name for the config class in order to avoid collision + * with top level configuration scopes + * + * @param text + * @return + */ + private String createUniqueName(String text) { + def hash = Hashing + .murmur3_32() + .newHasher() + .putUnencodedChars(text) + .hash() + return "_nf_config_$hash" + } + + private Script loadScript(String text) { + (Script)getGrengine().load(text, createUniqueName(text)).newInstance() + } + + /** + * Parses a ConfigObject instances from an instance of java.util.Properties + * @param The java.util.Properties instance + */ + ConfigObject parse(Properties properties) { + ConfigObject config = new ConfigObject() + for (key in properties.keySet()) { + def tokens = key.split(/\./) + + def current = config + def last + def lastToken + def foundBase = false + for (token in tokens) { + if (foundBase) { + // handle not properly nested tokens by ignoring + // hierarchy below this point + lastToken += "." + token + current = last + } else { + last = current + lastToken = token + current = current."${token}" + if (!(current instanceof ConfigObject)) foundBase = true + } + } + + if (current instanceof ConfigObject) { + if (last[lastToken]) { + def flattened = last.flatten() + last.clear() + flattened.each { k2, v2 -> last[k2] = v2 } + last[lastToken] = properties.get(key) + } + else { + last[lastToken] = properties.get(key) + } + } + current = config + } + return config + } + + /** + * Parse the given script as a string and return the configuration object + * + * @see ConfigParser#parse(groovy.lang.Script) + */ + @Override + ConfigObject parse(String text) { + return parse(loadScript(text)) + } + + /** + * Parse the given script into a configuration object (a Map) + * (This method creates a new class to parse the script each time it is called.) + * @param script The script to parse + * @return A Map of maps that can be navigating with dot de-referencing syntax to obtain configuration entries + */ + @Deprecated + ConfigObject parse(Script script) { + return parse(script, null) + } + + /** + * Parses a Script represented by the given URL into a ConfigObject + * + * @param location The location of the script to parse + * @return The ConfigObject instance + */ + @Deprecated + ConfigObject parse(URL location) { + return parse(loadScript(location.text), FileHelper.asPath(location.toURI())) + } + + @Override + ConfigObject parse(File file) { + return parse(file.toPath()) + } + + @Override + ConfigObject parse(Path path) { + return parse(loadScript(path.text), path) + } + + /** + * Parses the passed groovy.lang.Script instance using the second argument to allow the ConfigObject + * to retain an reference to the original location other Groovy script + * + * @param script The groovy.lang.Script instance + * @param location The original location of the Script as a URL + * @return The ConfigObject instance + */ + ConfigObject parse(Script _script, Path location) { + final script = (ConfigBase)_script + Stack currentConditionalBlock = new Stack() + def config = location ? new ConfigObject(location.toUri().toURL()) : new ConfigObject() + GroovySystem.metaClassRegistry.removeMetaClass(script.class) + def mc = script.class.metaClass + def prefix = "" + LinkedList stack = new LinkedList() + LinkedList profileStack = new LinkedList() + stack << [config: config, scope: [:]] + boolean withinProfile = false + + def pushStack = { co -> + stack << [config: co, scope: stack.last.scope.clone()] + } + def assignName = { name, co -> + def current = stack.last + current.config[name] = co + current.scope[name] = co + } + mc.getProperty = { String name -> + def current = stack.last + def result + if (current.config.get(name)) { + result = current.config.get(name) + } else if (current.scope.get(name)) { + result = current.scope[name] + } else { + try { + result = InvokerHelper.getProperty(this, name) + } catch (GroovyRuntimeException e) { + result = new ConfigObject() + assignName.call(name, result) + } + } + if( name=='params' && result instanceof Map && paramVars ) { + result.putAll(Bolts.deepMerge(result, paramVars)) + } + return result + } + + ConfigObject overrides = new ConfigObject() + mc.invokeMethod = { String name, args -> + def result + if (args.length == 1 && args[0] instanceof Closure) { + if( profileStack && profileStack.last == 'profiles' ) + profileNames.add(name) + + if (name in conditionValues.keySet()) { + try { + if( name == 'profiles' ){ + withinProfile=true + } + currentConditionalBlock.push(name) + conditionalBlocks.push([:]) + args[0].call() + } finally { + currentConditionalBlock.pop() + for (entry in conditionalBlocks.pop().entrySet()) { + def c = stack.last.config + (c != config? c : overrides).merge(entry.value) + } + if( name == 'profiles' ){ + withinProfile=false + } + } + } else if (currentConditionalBlock.size() > 0) { + String conditionalBlockKey = currentConditionalBlock.peek() + conditionalNames.add(name) + if (name in conditionValues[conditionalBlockKey]) { + def co = conditionalBlocks.peek()[conditionalBlockKey] + if( co == null ) { + co = new ConfigObject() + conditionalBlocks.peek()[conditionalBlockKey] = co + } + + pushStack.call(co) + try { + currentConditionalBlock.pop() + args[0].call() + } finally { + currentConditionalBlock.push(conditionalBlockKey) + } + stack.removeLast() + } + } + else if( name == 'plugins' ) { + if( stack.size()>1 ) + throw new ConfigParseException("Plugins definition is only allowed in config top-most scope") + // Implements `plugins` mini-dsl for plugins definition + def dsl = new PluginsDsl() + def clo = args[0] as Closure + clo.delegate = dsl + clo.resolveStrategy = Closure.DELEGATE_ONLY + clo.call() + assignName.call(name, dsl.plugins) + } + else { + def current = name=='profiles' || withinProfile ? stack.first : stack.last + def co + if (current.config.containsKey(name) && current.config.get(name) instanceof ConfigObject) { + co = current.config.get(name) + } + else if (current.scope.containsKey(name) && current.scope.get(name) instanceof ConfigObject) { + co = current.scope.get(name).clone() + } + else { + co = new ConfigObject() + } + + profileStack.add(name) + assignName.call(name, co) + pushStack.call(co) + args[0].call() + stack.removeLast() + profileStack.removeLast() + + if (current.scope.containsKey(name) && current.scope.get(name) instanceof ConfigObject) { + if( current.scope.get(name) != co) { + current.scope.get(name).merge(co) + } + } else { + current.scope.put(name,co) + } + } + } else if (args.length == 2 && args[1] instanceof Closure) { + try { + prefix = name + '.' + assignName.call(name, args[0]) + args[1].call() + } finally { prefix = "" } + } else { + MetaMethod mm = mc.getMetaMethod(name, args) + if (mm) { + result = mm.invoke(delegate, args) + } else { + throw new MissingMethodException(name, getClass(), args) + } + } + result + } + script.metaClass = mc + + def setProperty = { String name, value -> + assignName.call(prefix + name, value) + } + def binding = new ConfigBinding(setProperty) + if (this.bindingVars) { + binding.getVariables().putAll(this.bindingVars) + } + + // add the script file location into the binding + if( location ) { + script.setConfigPath(location) + } + + // disable include parsing when required + script.setIgnoreIncludes(ignoreIncludes) + script.setRenderClosureAsString(renderClosureAsString) + script.setStripSecrets(stripSecrets) + + // -- set the binding and run + script.binding = binding + script.run() + config.merge(overrides) + + return config + } + + /** + * Disable parsing of {@code includeConfig} directive + * + * @param value A boolean value, when {@code true} includes are disabled + * @return The {@link ConfigParser} object itself + */ + @Override + ConfigParser setIgnoreIncludes(boolean value) { + this.ignoreIncludes = value + return this + } + + /** + * Since Groovy Script doesn't support overriding setProperty, we have to using a trick with the Binding to provide this + * functionality + */ + @PackageScope + static class ConfigBinding extends Binding { + Closure callable + + ConfigBinding(Closure c) { + this.callable = c + } + + void setVariable(String name, Object value) { + callable(name, value) + } + } +} + diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigTransform.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransform.groovy similarity index 97% rename from modules/nextflow/src/main/groovy/nextflow/config/ConfigTransform.groovy rename to modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransform.groovy index 4dc8164aa4..bb6430c9b5 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigTransform.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransform.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -package nextflow.config +package nextflow.config.parser.v1 import java.lang.annotation.ElementType import java.lang.annotation.Retention diff --git a/modules/nextflow/src/main/groovy/nextflow/config/ConfigTransformImpl.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransformImpl.groovy similarity index 98% rename from modules/nextflow/src/main/groovy/nextflow/config/ConfigTransformImpl.groovy rename to modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransformImpl.groovy index e6d8723388..65c5b9bf4a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/config/ConfigTransformImpl.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/ConfigTransformImpl.groovy @@ -14,11 +14,12 @@ * limitations under the License. */ -package nextflow.config +package nextflow.config.parser.v1 import groovy.transform.CompileStatic import groovy.util.logging.Slf4j +import nextflow.config.ConfigClosurePlaceholder import org.codehaus.groovy.ast.ASTNode import org.codehaus.groovy.ast.AnnotationNode import org.codehaus.groovy.ast.ClassCodeVisitorSupport diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy new file mode 100644 index 0000000000..a46ebb935d --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v1/PluginsDsl.groovy @@ -0,0 +1,38 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.config.parser.v1 + +import groovy.transform.CompileStatic + +/** + * Model a mini-dsl for plugins configuration + * + * @author Paolo Di Tommaso + */ +@CompileStatic +class PluginsDsl { + + private Set plugins = [] + + Set getPlugins() { plugins } + + void id( String plg ) { + if( !plg ) + throw new IllegalArgumentException("Plugin id cannot be empty or null") + plugins << plg + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringVisitor.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringVisitor.groovy new file mode 100644 index 0000000000..e3b9518cc7 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringVisitor.groovy @@ -0,0 +1,103 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import groovy.transform.CompileStatic +import nextflow.config.ConfigClosurePlaceholder +import org.codehaus.groovy.ast.ClassCodeVisitorSupport +import org.codehaus.groovy.ast.ClassNode +import org.codehaus.groovy.ast.expr.ArgumentListExpression +import org.codehaus.groovy.ast.expr.ClosureExpression +import org.codehaus.groovy.ast.expr.ConstantExpression +import org.codehaus.groovy.ast.expr.ConstructorCallExpression +import org.codehaus.groovy.ast.expr.Expression +import org.codehaus.groovy.ast.expr.MapExpression +import org.codehaus.groovy.ast.expr.MethodCallExpression +import org.codehaus.groovy.control.SourceUnit +/** + * AST transformation to render closure source text + * + * @author Ben Sherman + */ +@CompileStatic +class ClosureToStringVisitor extends ClassCodeVisitorSupport { + + protected SourceUnit sourceUnit + + ClosureToStringVisitor(SourceUnit sourceUnit) { + this.sourceUnit = sourceUnit + } + + @Override + protected SourceUnit getSourceUnit() { sourceUnit } + + @Override + void visitMethodCallExpression(MethodCallExpression methodCall) { + final name = methodCall.methodAsString + if( name != 'assign' ) { + super.visitMethodCallExpression(methodCall) + return + } + + final arguments = (ArgumentListExpression)methodCall.arguments + if( arguments.size() != 2 ) + return + + final arg = arguments.last() + if( arg instanceof MapExpression ) { + for( final entry : arg.mapEntryExpressions ) { + if( entry.valueExpression instanceof ClosureExpression ) + entry.valueExpression = closureToString(entry.valueExpression) + } + } + if( arg instanceof ClosureExpression ) { + final placeholder = closureToString(arg) + methodCall.arguments = new ArgumentListExpression(arguments[0], placeholder) + } + } + + protected Expression closureToString(Expression closure) { + final buffer = new StringBuilder() + readSource(closure, buffer) + final str = new ConstantExpression(buffer.toString()) + + final type = new ClassNode(ConfigClosurePlaceholder) + final args = new ArgumentListExpression(str) + return new ConstructorCallExpression(type, args) + } + + protected void readSource(Expression expr, StringBuilder buffer) { + final colBegin = Math.max(expr.getColumnNumber()-1, 0) + final colEnd = Math.max(expr.getLastColumnNumber()-1, 0) + final lineFirst = expr.getLineNumber() + final lineLast = expr.getLastLineNumber() + + for( int i=lineFirst; i<=lineLast; i++ ) { + def line = sourceUnit.source.getLine(i, null) + if( i==lineFirst ) { + def str = i==lineLast ? line.substring(colBegin,colEnd) : line.substring(colBegin) + buffer.append(str) + } + else { + def str = i==lineLast ? line.substring(0, colEnd) : line + buffer.append('\n') + buffer.append(str) + } + } + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringXform.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringXform.groovy new file mode 100644 index 0000000000..dcd60f84e1 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ClosureToStringXform.groovy @@ -0,0 +1,47 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.lang.annotation.ElementType +import java.lang.annotation.Retention +import java.lang.annotation.RetentionPolicy +import java.lang.annotation.Target + +import groovy.transform.CompileStatic +import org.codehaus.groovy.ast.ASTNode +import org.codehaus.groovy.ast.ClassNode +import org.codehaus.groovy.control.CompilePhase +import org.codehaus.groovy.control.SourceUnit +import org.codehaus.groovy.transform.ASTTransformation +import org.codehaus.groovy.transform.GroovyASTTransformation +import org.codehaus.groovy.transform.GroovyASTTransformationClass + +@Retention(RetentionPolicy.SOURCE) +@Target(ElementType.METHOD) +@GroovyASTTransformationClass(classes = [ClosureToStringXformImpl]) +@interface ClosureToStringXform { + + @CompileStatic + @GroovyASTTransformation(phase = CompilePhase.CONVERSION) + class ClosureToStringXformImpl implements ASTTransformation { + @Override + void visit(ASTNode[] astNodes, SourceUnit sourceUnit) { + final clazz = (ClassNode)astNodes[1] + new ClosureToStringVisitor(sourceUnit).visitClass(clazz) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigDsl.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigDsl.groovy new file mode 100644 index 0000000000..e0a1bcfc96 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigDsl.groovy @@ -0,0 +1,226 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.nio.file.NoSuchFileException +import java.nio.file.Path +import java.nio.file.Paths + +import groovy.transform.CompileStatic +import groovy.transform.Memoized +import groovy.util.logging.Slf4j +import nextflow.SysEnv +import nextflow.exception.ConfigParseException +import nextflow.extension.Bolts +import nextflow.file.FileHelper +/** + * Builder DSL for Nextflow config files. + * + * @author Ben Sherman + * @author Paolo Di Tommaso + */ +@Slf4j +@CompileStatic +class ConfigDsl extends Script { + + private boolean ignoreIncludes + + private boolean renderClosureAsString + + private boolean strict + + private Path configPath + + private Map target = [:] + + void setIgnoreIncludes(boolean value) { + this.ignoreIncludes = value + } + + void setRenderClosureAsString(boolean value) { + this.renderClosureAsString = value + } + + void setStrict(boolean value) { + this.strict = value + } + + void setConfigPath(Path path) { + this.configPath = path + } + + void setParams(Map params) { + target.params = params + } + + Map getTarget() { + if( !target.params ) + target.remove('params') + return target + } + + Object run() {} + + @Override + def getProperty(String name) { + if( name == 'params' ) + return target.params + + try { + return super.getProperty(name) + } + catch( MissingPropertyException e ) { + if( strict ) + throw e + else + return null + } + } + + void append(List names, Object right) { + final values = (Set) navigate(names.init()).computeIfAbsent(names.last(), (k) -> new HashSet<>()) + values.add(right) + } + + void assign(List names, Object right) { + navigate(names.init()).put(names.last(), right) + } + + private Map navigate(List names) { + Map ctx = target + for( final name : names ) { + if( name !in ctx ) ctx[name] = [:] + ctx = ctx[name] as Map + } + return ctx + } + + void block(String name, Closure closure) { + block([name], closure) + } + + void block(List names, Closure closure) { + final delegate = new ConfigBlockDsl(this, names) + final cl = (Closure)closure.clone() + cl.setResolveStrategy(Closure.DELEGATE_FIRST) + cl.setDelegate(delegate) + cl.call() + } + + /** + * Get the value of an environment variable from the launch environment. + * + * @param name + */ + String env(String name) { + return SysEnv.get(name) + } + + void includeConfig(String includeFile) { + includeConfig([], includeFile) + } + + void includeConfig(List names, String includeFile) { + assert includeFile + + if( ignoreIncludes ) + return + + Path includePath = FileHelper.asPath(includeFile) + log.trace "Include config file: $includeFile [parent: $configPath]" + + if( !includePath.isAbsolute() && configPath ) + includePath = configPath.resolveSibling(includeFile) + + final configText = readConfigFile(includePath) + final config = new ConfigParserV2() + .setIgnoreIncludes(ignoreIncludes) + .setRenderClosureAsString(renderClosureAsString) + .setStrict(strict) + .setBinding(binding.getVariables()) + .parse(configText, includePath) + + final ctx = navigate(names) + ctx.putAll(Bolts.deepMerge(ctx, config)) + } + + /** + * Read the content of a config file. The result is cached to + * avoid multiple reads. + * + * @param includePath + */ + @Memoized + protected static String readConfigFile(Path includePath) { + try { + return includePath.getText() + } + catch (NoSuchFileException | FileNotFoundException ignored) { + throw new NoSuchFileException("Config file does not exist: ${includePath.toUriString()}") + } + catch (IOException e) { + throw new IOException("Cannot read config file include: ${includePath.toUriString()}", e) + } + } + + static class ConfigBlockDsl { + private ConfigDsl dsl + private List scope + + ConfigBlockDsl(ConfigDsl dsl, List scope) { + this.dsl = dsl + this.scope = scope + } + + void append(String name, Object right) { + dsl.append(scope, right) + } + + void assign(List names, Object right) { + dsl.assign(scope + names, right) + } + + void block(String name, Closure closure) { + dsl.block(scope + [name], closure) + } + + void withLabel(String label, Closure closure) { + if( !isWithinProcessScope() ) + throw new ConfigParseException("Process selectors are only allowed in the `process` scope (offending scope: `${scope.join('.')}`)") + dsl.block(scope + ["withLabel:${label}".toString()], closure) + } + + void withName(String selector, Closure closure) { + if( !isWithinProcessScope() ) + throw new ConfigParseException("Process selectors are only allowed in the `process` scope (offending scope: `${scope.join('.')}`)") + dsl.block(scope + ["withName:${selector}".toString()], closure) + } + + private boolean isWithinProcessScope() { + if( scope.size() == 1 ) + return scope.first() == 'process' + if( scope.size() == 3 ) + return scope.first() == 'profiles' && scope.last() == 'process' + return false + } + + void includeConfig(String includeFile) { + dsl.includeConfig(scope, includeFile) + } + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigParserV2.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigParserV2.groovy new file mode 100644 index 0000000000..1b79af0ce6 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigParserV2.groovy @@ -0,0 +1,193 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.nio.file.Path + +import com.google.common.hash.Hashing +import groovy.transform.CompileStatic +import nextflow.ast.NextflowXform +import nextflow.config.ConfigParser +import nextflow.config.StripSecretsXform +import nextflow.config.parser.ConfigParserPluginFactory +import nextflow.extension.Bolts +import nextflow.util.Duration +import nextflow.util.MemoryUnit +import org.codehaus.groovy.control.CompilerConfiguration +import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer +import org.codehaus.groovy.control.customizers.ImportCustomizer + +/** + * The parser for Nextflow config files. + * + * @author Ben Sherman + */ +@CompileStatic +class ConfigParserV2 implements ConfigParser { + + private Map bindingVars = [:] + + private Map paramVars = [:] + + private boolean ignoreIncludes = false + + private boolean renderClosureAsString = false + + private boolean strict = true + + private boolean stripSecrets + + private List appliedProfiles + + private Set parsedProfiles = [] + + private GroovyShell groovyShell + + @Override + ConfigParserV2 setProfiles(List profiles) { + this.appliedProfiles = profiles + return this + } + + @Override + Set getProfiles() { + return parsedProfiles + } + + @Override + ConfigParserV2 setIgnoreIncludes(boolean value) { + this.ignoreIncludes = value + return this + } + + @Override + ConfigParserV2 setRenderClosureAsString(boolean value) { + this.renderClosureAsString = value + return this + } + + @Override + ConfigParserV2 setStrict(boolean value) { + this.strict = value + return this + } + + @Override + ConfigParser setStripSecrets(boolean value) { + this.stripSecrets = value + return this + } + + @Override + ConfigParserV2 setBinding(Map vars) { + this.bindingVars = vars + return this + } + + @Override + ConfigParserV2 setParams(Map vars) { + // deep clone the map to prevent side-effect + // see https://github.com/nextflow-io/nextflow/issues/1923 + this.paramVars = Bolts.deepClone(vars) + return this + } + + /** + * Parse the given script as a string and return the configuration object + * + * @param text + * @param location + */ + @Override + ConfigObject parse(String text) { + parse(text, null) + } + + ConfigObject parse(String text, Path location) { + final groovyShell = getGroovyShell() + final script = (ConfigDsl) groovyShell.parse(text, uniqueClassName(text)) + if( location ) + script.setConfigPath(location) + script.setIgnoreIncludes(ignoreIncludes) + script.setRenderClosureAsString(renderClosureAsString) + if( location ) + script.setConfigPath(location) + + script.setBinding(new Binding(bindingVars)) + script.setParams(paramVars) + script.run() + + final result = Bolts.toConfigObject(script.getTarget()) + final profiles = (result.profiles ?: [:]) as ConfigObject + parsedProfiles.addAll(profiles.keySet()) + if( appliedProfiles ) { + for( final profile : appliedProfiles ) { + if( profile in profiles.keySet() ) + result.merge(profiles[profile] as ConfigObject) + } + result.remove('profiles') + } + + return result + } + + @Override + ConfigObject parse(File file) { + return parse(file.toPath()) + } + + @Override + ConfigObject parse(Path path) { + return parse(path.text, path) + } + + private GroovyShell getGroovyShell() { + if( groovyShell ) + return groovyShell + final classLoader = new GroovyClassLoader() + final config = new CompilerConfiguration() + config.setScriptBaseClass(ConfigDsl.class.getName()) + config.setPluginFactory(new ConfigParserPluginFactory()) + config.addCompilationCustomizers(new ASTTransformationCustomizer(ConfigToGroovyXform)) + if( stripSecrets ) + config.addCompilationCustomizers(new ASTTransformationCustomizer(StripSecretsXform)) + if( renderClosureAsString ) + config.addCompilationCustomizers(new ASTTransformationCustomizer(ClosureToStringXform)) + config.addCompilationCustomizers(new ASTTransformationCustomizer(NextflowXform)) + final importCustomizer = new ImportCustomizer() + importCustomizer.addImports( Duration.name ) + importCustomizer.addImports( MemoryUnit.name ) + config.addCompilationCustomizers(importCustomizer) + return groovyShell = new GroovyShell(classLoader, new Binding(), config) + } + + /** + * Creates a unique name for the config class in order to avoid collision + * with config DSL + * + * @param text + */ + private String uniqueClassName(String text) { + def hash = Hashing + .sipHash24() + .newHasher() + .putUnencodedChars(text) + .hash() + return "_nf_config_$hash" + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyVisitor.java b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyVisitor.java new file mode 100644 index 0000000000..dd7d447f26 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyVisitor.java @@ -0,0 +1,112 @@ +/* + * Copyright 2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.config.parser.v2; + +import java.util.ArrayList; +import java.util.stream.Collectors; + +import nextflow.config.ast.ConfigAppendNode; +import nextflow.config.ast.ConfigAssignNode; +import nextflow.config.ast.ConfigBlockNode; +import nextflow.config.ast.ConfigIncludeNode; +import nextflow.config.ast.ConfigNode; +import nextflow.config.ast.ConfigVisitorSupport; +import org.codehaus.groovy.ast.VariableScope; +import org.codehaus.groovy.ast.expr.Expression; +import org.codehaus.groovy.ast.stmt.ReturnStatement; +import org.codehaus.groovy.ast.stmt.Statement; +import org.codehaus.groovy.control.SourceUnit; + +import static org.codehaus.groovy.ast.tools.GeneralUtils.*; + +/** + * Visitor to convert a Nextflow config AST into a + * Groovy AST which is executed against {@link ConfigDsl}. + * + * @author Ben Sherman + */ +public class ConfigToGroovyVisitor extends ConfigVisitorSupport { + + private SourceUnit sourceUnit; + + private ConfigNode moduleNode; + + public ConfigToGroovyVisitor(SourceUnit sourceUnit) { + this.sourceUnit = sourceUnit; + this.moduleNode = (ConfigNode) sourceUnit.getAST(); + } + + @Override + protected SourceUnit getSourceUnit() { + return sourceUnit; + } + + public void visit() { + if( moduleNode == null ) + return; + super.visit(moduleNode); + if( moduleNode.isEmpty() ) + moduleNode.addStatement(ReturnStatement.RETURN_NULL_OR_VOID); + } + + @Override + public void visitConfigAssign(ConfigAssignNode node) { + moduleNode.addStatement(transformConfigAssign(node)); + } + + protected Statement transformConfigAssign(ConfigAssignNode node) { + if( node instanceof ConfigAppendNode ) { + var name = node.names.get(0); + return stmt(callThisX("append", args(constX(name), node.value))); + } + var names = listX( + node.names.stream() + .map(name -> (Expression) constX(name)) + .collect(Collectors.toList()) + ); + return stmt(callThisX("assign", args(names, node.value))); + } + + @Override + public void visitConfigBlock(ConfigBlockNode node) { + moduleNode.addStatement(transformConfigBlock(node)); + } + + protected Statement transformConfigBlock(ConfigBlockNode node) { + var statements = new ArrayList(); + for( var stmt : node.statements ) { + if( stmt instanceof ConfigAssignNode can ) + statements.add(transformConfigAssign(can)); + else if( stmt instanceof ConfigBlockNode cbn ) + statements.add(transformConfigBlock(cbn)); + else if( stmt instanceof ConfigIncludeNode cin ) + statements.add(transformConfigInclude(cin)); + } + var code = block(new VariableScope(), statements); + var kind = node.kind != null ? node.kind : "block"; + return stmt(callThisX(kind, args(constX(node.name), closureX(code)))); + } + + @Override + public void visitConfigInclude(ConfigIncludeNode node) { + moduleNode.addStatement(transformConfigInclude(node)); + } + + protected Statement transformConfigInclude(ConfigIncludeNode node) { + return stmt(callThisX("includeConfig", args(node.source))); + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyXform.groovy b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyXform.groovy new file mode 100644 index 0000000000..677ca16d11 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/config/parser/v2/ConfigToGroovyXform.groovy @@ -0,0 +1,45 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.lang.annotation.ElementType +import java.lang.annotation.Retention +import java.lang.annotation.RetentionPolicy +import java.lang.annotation.Target + +import groovy.transform.CompileStatic +import org.codehaus.groovy.ast.ASTNode +import org.codehaus.groovy.control.CompilePhase +import org.codehaus.groovy.control.SourceUnit +import org.codehaus.groovy.transform.ASTTransformation +import org.codehaus.groovy.transform.GroovyASTTransformation +import org.codehaus.groovy.transform.GroovyASTTransformationClass + +@Retention(RetentionPolicy.SOURCE) +@Target(ElementType.METHOD) +@GroovyASTTransformationClass(classes = [ConfigToGroovyXformImpl]) +@interface ConfigToGroovyXform { + + @CompileStatic + @GroovyASTTransformation(phase = CompilePhase.CONVERSION) + class ConfigToGroovyXformImpl implements ASTTransformation { + @Override + public void visit(ASTNode[] astNodes, SourceUnit sourceUnit) { + new ConfigToGroovyVisitor(sourceUnit).visit() + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy index a1b7d57683..5aa65364e6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy @@ -28,7 +28,7 @@ import groovy.transform.ToString import groovy.transform.TupleConstructor import groovy.util.logging.Slf4j import nextflow.cli.HubOptions -import nextflow.config.ConfigParser +import nextflow.config.ConfigParserFactory import nextflow.config.Manifest import nextflow.exception.AbortOperationException import nextflow.exception.AmbiguousPipelineNameException @@ -455,7 +455,7 @@ class AssetManager { } if( text ) try { - def config = new ConfigParser().setIgnoreIncludes(true).parse(text) + def config = ConfigParserFactory.create().setIgnoreIncludes(true).setStrict(false).parse(text) result = (ConfigObject)config.manifest } catch( Exception e ) { diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/ProviderConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/ProviderConfig.groovy index 63000e72f4..28064f9aaa 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/ProviderConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/ProviderConfig.groovy @@ -23,7 +23,7 @@ import groovy.transform.CompileStatic import groovy.transform.PackageScope import groovy.util.logging.Slf4j import nextflow.Const -import nextflow.config.ConfigParser +import nextflow.config.ConfigParserFactory import nextflow.exception.AbortOperationException import nextflow.exception.ConfigParseException import nextflow.file.FileHelper @@ -245,7 +245,7 @@ class ProviderConfig { @PackageScope static Map parse(String text) { - def slurper = new ConfigParser() + def slurper = ConfigParserFactory.create() slurper.setBinding(env) return slurper.parse(text) } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/ProviderPath.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/ProviderPath.groovy index 8d73d965fd..366f57e0db 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/ProviderPath.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/ProviderPath.groovy @@ -34,9 +34,7 @@ import groovy.util.logging.Slf4j * project hosted in a source code repository * * @see nextflow.config.ConfigParser - * @see nextflow.config.ConfigBase * - * * @author Paolo Di Tommaso */ @EqualsAndHashCode diff --git a/modules/nextflow/src/test/groovy/FunctionalTests.groovy b/modules/nextflow/src/test/groovy/FunctionalTests.groovy index 018690cfff..14318032cd 100644 --- a/modules/nextflow/src/test/groovy/FunctionalTests.groovy +++ b/modules/nextflow/src/test/groovy/FunctionalTests.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -import nextflow.config.ConfigParser +import nextflow.config.ConfigParserFactory import nextflow.exception.AbortRunException import nextflow.processor.TaskProcessor import nextflow.util.MemoryUnit @@ -80,7 +80,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -116,7 +116,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -155,7 +155,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -196,7 +196,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -255,7 +255,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -282,7 +282,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -308,7 +308,7 @@ class FunctionalTests extends Dsl2Spec { workflow { foo() } ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -337,7 +337,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -389,7 +389,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -415,7 +415,7 @@ class FunctionalTests extends Dsl2Spec { workflow { bar() } ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -448,7 +448,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -479,7 +479,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -509,7 +509,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -535,7 +535,7 @@ class FunctionalTests extends Dsl2Spec { workflow { foo() } ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -564,7 +564,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -594,7 +594,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -626,7 +626,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -656,7 +656,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() def processor = TaskProcessor.currentProcessor() @@ -685,7 +685,7 @@ class FunctionalTests extends Dsl2Spec { ''' and: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() processor = TaskProcessor.currentProcessor() @@ -716,7 +716,7 @@ class FunctionalTests extends Dsl2Spec { ''' when: - new MockScriptRunner(new ConfigParser().parse(config)) + new MockScriptRunner(ConfigParserFactory.create().parse(config)) .setScript(script) .execute() then: diff --git a/modules/nextflow/src/test/groovy/nextflow/config/ConfigBuilderTest.groovy b/modules/nextflow/src/test/groovy/nextflow/config/ConfigBuilderTest.groovy index b3781a92ee..ead6b64d59 100644 --- a/modules/nextflow/src/test/groovy/nextflow/config/ConfigBuilderTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/config/ConfigBuilderTest.groovy @@ -1784,7 +1784,7 @@ class ConfigBuilderTest extends Specification { def 'should collect config files' () { given: - def slurper = new ConfigParser() + def slurper = ConfigParserFactory.create() def file1 = Files.createTempFile('test1', null) def file2 = Files.createTempFile('test2', null) def result = new ConfigObject() diff --git a/modules/nextflow/src/test/groovy/nextflow/config/ConfigParserTest.groovy b/modules/nextflow/src/test/groovy/nextflow/config/parser/v1/ConfigParserV1Test.groovy similarity index 83% rename from modules/nextflow/src/test/groovy/nextflow/config/ConfigParserTest.groovy rename to modules/nextflow/src/test/groovy/nextflow/config/parser/v1/ConfigParserV1Test.groovy index a128f0fd36..43fbf761cd 100644 --- a/modules/nextflow/src/test/groovy/nextflow/config/ConfigParserTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/config/parser/v1/ConfigParserV1Test.groovy @@ -14,7 +14,7 @@ * limitations under the License. */ -package nextflow.config +package nextflow.config.parser.v1 import java.nio.file.Files import java.nio.file.NoSuchFileException @@ -25,6 +25,8 @@ import com.sun.net.httpserver.HttpExchange import com.sun.net.httpserver.HttpHandler import com.sun.net.httpserver.HttpServer import nextflow.SysEnv +import nextflow.config.ConfigBuilder +import nextflow.config.ConfigClosurePlaceholder import nextflow.exception.ConfigParseException import nextflow.util.Duration import nextflow.util.MemoryUnit @@ -35,7 +37,7 @@ import spock.lang.Specification * * @author Paolo Di Tommaso */ -class ConfigParserTest extends Specification { +class ConfigParserV1Test extends Specification { def 'should get an environment variable' () { given: @@ -45,7 +47,7 @@ class ConfigParserTest extends Specification { def CONFIG = ''' process.cpus = env('MAX_CPUS') ''' - def config = new ConfigParser().parse(CONFIG) + def config = new ConfigParserV1().parse(CONFIG) then: config.process.cpus == '1' @@ -61,16 +63,16 @@ class ConfigParserTest extends Specification { id 'foo' id 'bar' id 'bar' - } - + } + process { cpus = 1 - mem = 2 + mem = 2 } ''' when: - def config = new ConfigParser().parse(CONFIG) + def config = new ConfigParserV1().parse(CONFIG) then: config.plugins == ['foo','bar'] as Set @@ -84,12 +86,12 @@ class ConfigParserTest extends Specification { profiles { plugins { id 'foo' - } + } } ''' when: - def config = new ConfigParser().parse(CONFIG) + def config = new ConfigParserV1().parse(CONFIG) then: def e = thrown(ConfigParseException) @@ -129,7 +131,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().parse(text) + def config = new ConfigParserV1().parse(text) then: config.process.name == 'alpha' config.process.resources.cpus == 4 @@ -141,20 +143,20 @@ class ConfigParserTest extends Specification { when: def buffer = new StringWriter() config.writeTo(buffer) - def str = buffer.toString() + def str = buffer.toString().replaceAll('\t', ' ') then: str == ''' process { - name='alpha' - resources { - disk='1TB' - cpus=4 - memory='8GB' - nested { - foo=1 - bar=2 - } - } + name='alpha' + resources { + disk='1TB' + cpus=4 + memory='8GB' + nested { + foo=1 + bar=2 + } + } } '''.stripIndent().leftTrim() @@ -193,7 +195,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().setBinding().parse(text) + def config = new ConfigParserV1().setBinding().parse(text) then: config.params.xxx == 'x' config.params.yyy == 'y' @@ -251,7 +253,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().setBinding([MIN: 1, MAX: 32]).parse(main) + def config = new ConfigParserV1().setBinding([MIN: 1, MAX: 32]).parse(main) then: config.profiles.proc1.cpus == 4 config.profiles.proc1.memory == '8GB' @@ -308,7 +310,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().parse(main) + def config = new ConfigParserV1().parse(main) then: config.process.name == 'foo' config.process.resources.cpus == 4 @@ -372,8 +374,8 @@ class ConfigParserTest extends Specification { """ when: - def config1 = new ConfigParser() - .registerConditionalBlock('profiles','slow') + def config1 = new ConfigParserV1() + .setProfiles(['slow']) .parse(configText) then: config1.workDir == '/my/scratch' @@ -382,8 +384,8 @@ class ConfigParserTest extends Specification { config1.process.disk == '100GB' when: - def config2 = new ConfigParser() - .registerConditionalBlock('profiles','fast') + def config2 = new ConfigParserV1() + .setProfiles(['fast']) .parse(configText) then: config2.workDir == '/fast/scratch' @@ -409,77 +411,19 @@ class ConfigParserTest extends Specification { b = 2 } } - - servers { - local { - x = 1 - } - test { - y = 2 - } - prod { - z = 3 - } - } ''' when: - def slurper = new ConfigParser().registerConditionalBlock('profiles','alpha') - slurper.parse(text) - then: - slurper.getConditionalBlockNames() == ['alpha','beta'] as Set - - when: - slurper = new ConfigParser().registerConditionalBlock('profiles','omega') + def slurper = new ConfigParserV1().setProfiles(['alpha']) slurper.parse(text) then: - slurper.getConditionalBlockNames() == ['alpha','beta'] as Set - - when: - slurper = new ConfigParser().registerConditionalBlock('servers','xxx') - slurper.parse(text) - then: - slurper.getConditionalBlockNames() == ['local','test','prod'] as Set - - when: - slurper = new ConfigParser().registerConditionalBlock('foo','bar') - slurper.parse(text) - then: - slurper.getConditionalBlockNames() == [] as Set - } - - def 'should return the profile names' () { - given: - def text = ''' - profiles { - alpha { - a = 1 - } - beta { - b = 2 - } - } - - servers { - local { - x = 1 - } - test { - y = 2 - } - prod { - z = 3 - } - } - ''' + slurper.getProfiles() == ['alpha','beta'] as Set when: - def slurper = new ConfigParser() + slurper = new ConfigParserV1().setProfiles(['omega']) slurper.parse(text) then: - slurper.getProfileNames() == ['alpha','beta'] as Set - slurper.getConditionalBlockNames() == [] as Set - + slurper.getProfiles() == ['alpha','beta'] as Set } def 'should disable includeConfig parsing' () { @@ -494,12 +438,12 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().setIgnoreIncludes(true).parse(text) + def config = new ConfigParserV1().setIgnoreIncludes(true).parse(text) then: config.manifest.description == 'some text ..' when: - new ConfigParser().parse(text) + new ConfigParserV1().parse(text) then: thrown(NoSuchFileException) @@ -512,7 +456,7 @@ class ConfigParserTest extends Specification { configFile.text = 'XXX.enabled = true' when: - new ConfigParser().parse(configFile) + new ConfigParserV1().parse(configFile) then: noExceptionThrown() @@ -525,7 +469,7 @@ class ConfigParserTest extends Specification { given: ConfigObject result - def CONFIG = ''' + def CONFIG = ''' str1 = 'hello' str2 = "${str1} world" closure1 = { "$str" } @@ -533,7 +477,7 @@ class ConfigParserTest extends Specification { ''' when: - result = new ConfigParser() + result = new ConfigParserV1() .parse(CONFIG) then: result.str1 instanceof String @@ -542,7 +486,7 @@ class ConfigParserTest extends Specification { result.map1.bar instanceof Closure when: - result = new ConfigParser() + result = new ConfigParserV1() .setRenderClosureAsString(false) .parse(CONFIG) then: @@ -552,7 +496,7 @@ class ConfigParserTest extends Specification { result.map1.bar instanceof Closure when: - result = new ConfigParser() + result = new ConfigParserV1() .setRenderClosureAsString(true) .parse(CONFIG) then: @@ -567,19 +511,19 @@ class ConfigParserTest extends Specification { def 'should handle extend mem and duration units' () { ConfigObject result - def CONFIG = ''' + def CONFIG = ''' mem1 = 1.GB mem2 = 1_000_000.toMemory() mem3 = MemoryUnit.of(2_000) time1 = 2.hours time2 = 60_000.toDuration() time3 = Duration.of(120_000) - flag = 10000 < 1.GB + flag = 10000 < 1.GB ''' when: - result = new ConfigParser() - .parse(CONFIG) + result = new ConfigParserV1() + .parse(CONFIG) then: result.mem1 instanceof MemoryUnit result.mem1 == MemoryUnit.of('1 GB') @@ -615,11 +559,11 @@ class ConfigParserTest extends Specification { folder.resolve('conf/remote.config').text = ''' process { - cpus = 4 + cpus = 4 memory = '10GB' } ''' - + when: def url = 'http://localhost:9900/nextflow.config' as Path def cfg = new ConfigBuilder().buildGivenFiles(url) @@ -652,7 +596,7 @@ class ConfigParserTest extends Specification { ''' when: - def config = new ConfigParser().parse(CONFIG) + def config = new ConfigParserV1().parse(CONFIG) then: config.params.foo.bar == 'bar1' @@ -670,7 +614,7 @@ class ConfigParserTest extends Specification { } ''' and: - config = new ConfigParser().parse(CONFIG) + config = new ConfigParserV1().parse(CONFIG) then: config.params.foo.bar == 'bar1' @@ -725,7 +669,7 @@ class ConfigParserTest extends Specification { """ when: - def config1 = new ConfigParser() + def config1 = new ConfigParserV1() .parse(configText) then: config1.workDir == '/my/scratch' @@ -739,6 +683,26 @@ class ConfigParserTest extends Specification { } + def 'should apply profiles in the order they were defined' () { + given: + def CONFIG = ''' + profiles { + foo { + params.input = 'foo' + } + + bar { + params.input = 'bar' + } + } + ''' + + when: + def config = new ConfigParserV1().setProfiles(['bar', 'foo']).parse(CONFIG) + + then: + config.params.input == 'bar' + } static class ConfigFileHandler implements HttpHandler { diff --git a/modules/nextflow/src/test/groovy/nextflow/config/parser/v2/ConfigParserV2Test.groovy b/modules/nextflow/src/test/groovy/nextflow/config/parser/v2/ConfigParserV2Test.groovy new file mode 100644 index 0000000000..78f4228fcf --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/config/parser/v2/ConfigParserV2Test.groovy @@ -0,0 +1,664 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config.parser.v2 + +import java.nio.file.Files +import java.nio.file.NoSuchFileException +import java.nio.file.Path + +import com.sun.net.httpserver.Headers +import com.sun.net.httpserver.HttpExchange +import com.sun.net.httpserver.HttpHandler +import com.sun.net.httpserver.HttpServer +import nextflow.SysEnv +import nextflow.config.ConfigBuilder +import nextflow.config.ConfigClosurePlaceholder +import nextflow.util.Duration +import nextflow.util.MemoryUnit +import spock.lang.Specification + +/** + * + * @author Paolo Di Tommaso + */ +class ConfigParserV2Test extends Specification { + + def 'should get an environment variable' () { + given: + SysEnv.push(MAX_CPUS: '1') + + when: + def CONFIG = ''' + process.cpus = env('MAX_CPUS') + ''' + def config = new ConfigParserV2().parse(CONFIG) + + then: + config.process.cpus == '1' + + cleanup: + SysEnv.pop() + } + + def 'should parse plugin ids' () { + given: + def CONFIG = ''' + plugins { + id 'foo' + id 'bar' + id 'bar' + } + + process { + cpus = 1 + mem = 2 + } + ''' + + when: + def config = new ConfigParserV2().parse(CONFIG) + + then: + config.plugins == ['foo','bar'] as Set + and: + config.process.cpus == 1 + } + + def 'should parse composed config files' () { + + given: + def folder = File.createTempDir() + def snippet1 = new File(folder,'config1.txt').absoluteFile + def snippet2 = new File(folder,'config2.txt').absoluteFile + + def text = """ + process { + name = 'alpha' + resources { + disk = '1TB' + includeConfig "$snippet1" + } + } + """ + + snippet1.text = """ + cpus = 4 + memory = '8GB' + nested { + includeConfig("$snippet2") + } + """ + + snippet2.text = ''' + foo = 1 + bar = 2 + ''' + + when: + def config = new ConfigParserV2().parse(text) + then: + config.process.name == 'alpha' + config.process.resources.cpus == 4 + config.process.resources.memory == '8GB' + config.process.resources.disk == '1TB' + config.process.resources.nested.foo == 1 + config.process.resources.nested.bar == 2 + + when: + def buffer = new StringWriter() + config.writeTo(buffer) + def str = buffer.toString().replaceAll('\t', ' ') + then: + str == ''' + process { + name='alpha' + resources { + disk='1TB' + cpus=4 + memory='8GB' + nested { + foo=1 + bar=2 + } + } + } + '''.stripIndent().leftTrim() + + cleanup: + folder?.deleteDir() + + } + + def 'should parse include config using dot properties syntax' () { + + given: + def folder = File.createTempDir() + def snippet1 = new File(folder,'config1.txt').absoluteFile + def snippet2 = new File(folder,'config2.txt').absoluteFile + + def text = """ + process.name = 'alpha' + includeConfig "$snippet1" + """ + + snippet1.text = """ + params.xxx = 'x' + + process.cpus = 4 + process.memory = '8GB' + + includeConfig("$snippet2") + """ + + snippet2.text = ''' + params.yyy = 'y' + process { disk = '1TB' } + process.resources.foo = 1 + process.resources.bar = 2 + ''' + + when: + def config = new ConfigParserV2().setBinding().parse(text) + then: + config.params.xxx == 'x' + config.params.yyy == 'y' + config.process.name == 'alpha' + config.process.cpus == 4 + config.process.memory == '8GB' + config.process.disk == '1TB' + config.process.resources.foo == 1 + config.process.resources.bar == 2 + + cleanup: + folder?.deleteDir() + + } + + def 'should parse multiple relative files' () { + + given: + def folder = File.createTempDir() + def main = new File(folder, 'main.config') + def folder1 = new File(folder, 'dir1') + def folder2 = new File(folder, 'dir2') + def folder3 = new File(folder, 'dir3') + folder1.mkdirs() + folder2.mkdirs() + folder3.mkdirs() + + main. text = ''' + profiles { + includeConfig 'dir1/config' + includeConfig 'dir2/config' + includeConfig 'dir3/config' + } + ''' + + new File(folder,'dir1/config').text = ''' + proc1 { + cpus = 4 + memory = '8GB' + } + ''' + + new File(folder, 'dir2/config').text = ''' + proc2 { + cpus = MIN + memory = '6GB' + } + ''' + + new File(folder, 'dir3/config').text = ''' + proc3 { + cpus = MAX + disk = '500GB' + } + ''' + + when: + def config = new ConfigParserV2().setBinding([MIN: 1, MAX: 32]).parse(main) + then: + config.profiles.proc1.cpus == 4 + config.profiles.proc1.memory == '8GB' + config.profiles.proc2.cpus == 1 + config.profiles.proc2.memory == '6GB' + config.profiles.proc3.cpus == 32 + config.profiles.proc3.disk == '500GB' + + cleanup: + folder?.deleteDir() + + } + + def 'should parse nested relative files' () { + + given: + def folder = File.createTempDir() + def main = new File(folder, 'main.config') + def folder1 = new File(folder, 'dir1/dir3') + def folder2 = new File(folder, 'dir2') + folder1.mkdirs() + folder2.mkdirs() + + main. text = """ + process { + name = 'foo' + resources { + disk = '1TB' + includeConfig "dir1/nextflow.config" + } + + commands { + includeConfig "dir2/nextflow.config" + } + } + """ + + new File(folder,'dir1/nextflow.config').text = """ + cpus = 4 + memory = '8GB' + nested { + includeConfig 'dir3/nextflow.config' + } + """ + + new File(folder, 'dir1/dir3/nextflow.config').text = ''' + alpha = 1 + delta = 2 + ''' + + new File(folder, 'dir2/nextflow.config').text = ''' + cmd1 = 'echo true' + cmd2 = 'echo false' + ''' + + when: + def config = new ConfigParserV2().parse(main) + then: + config.process.name == 'foo' + config.process.resources.cpus == 4 + config.process.resources.memory == '8GB' + config.process.resources.disk == '1TB' + + config.process.resources.nested.alpha == 1 + config.process.resources.nested.delta == 2 + + config.process.commands.cmd1 == 'echo true' + config.process.commands.cmd2 == 'echo false' + + cleanup: + folder?.deleteDir() + + } + + def 'should load selected profile configuration' () { + + given: + def folder = File.createTempDir() + def snippet1 = new File(folder,'config1.txt').absoluteFile + def snippet2 = new File(folder,'config2.txt').absoluteFile + + snippet1.text = ''' + process { + cpus = 1 + memory = '2GB' + disk = '100GB' + } + ''' + + snippet2.text = ''' + process { + cpus = 8 + memory = '20GB' + disk = '2TB' + } + ''' + + def configText = """ + workDir = '/my/scratch' + + profiles { + + standard {} + + slow { + includeConfig "$snippet1" + } + + fast { + workDir = '/fast/scratch' + includeConfig "$snippet2" + } + + } + """ + + when: + def config1 = new ConfigParserV2() + .setProfiles(['slow']) + .parse(configText) + then: + config1.workDir == '/my/scratch' + config1.process.cpus == 1 + config1.process.memory == '2GB' + config1.process.disk == '100GB' + + when: + def config2 = new ConfigParserV2() + .setProfiles(['fast']) + .parse(configText) + then: + config2.workDir == '/fast/scratch' + config2.process.cpus == 8 + config2.process.memory == '20GB' + config2.process.disk == '2TB' + + + cleanup: + folder.deleteDir() + + } + + def 'should return the set of parsed profiles' () { + + given: + def text = ''' + profiles { + alpha { + a = 1 + } + beta { + b = 2 + } + } + ''' + + when: + def slurper = new ConfigParserV2().setProfiles(['alpha']) + slurper.parse(text) + then: + slurper.getProfiles() == ['alpha','beta'] as Set + + when: + slurper = new ConfigParserV2().setProfiles(['omega']) + slurper.parse(text) + then: + slurper.getProfiles() == ['alpha','beta'] as Set + } + + def 'should ignore config includes when specified' () { + given: + def text = ''' + manifest { + description = 'some text ..' + } + + includeConfig 'this' + includeConfig 'that' + ''' + + when: + def config = new ConfigParserV2().setIgnoreIncludes(true).parse(text) + then: + config.manifest.description == 'some text ..' + + when: + new ConfigParserV2().parse(text) + then: + thrown(NoSuchFileException) + + } + + def 'should parse file named as a top config scope' () { + given: + def folder = File.createTempDir() + def configFile = new File(folder, 'XXX.config') + configFile.text = 'XXX.enabled = true' + + when: + new ConfigParserV2().parse(configFile) + then: + noExceptionThrown() + + cleanup: + folder?.deleteDir() + } + + def 'should access node metadata' () { + + given: + Map params + def CONFIG = ''' + params.str1 = 'hello' + params.str2 = "${params.str1} world" + params.closure1 = { "$str" } + params.map1 = [foo: 'hello', bar: { world }] + ''' + + when: + params = new ConfigParserV2() + .parse(CONFIG) + .params + then: + params.str1 instanceof String + params.str2 instanceof GString + params.closure1 instanceof Closure + params.map1.bar instanceof Closure + + when: + params = new ConfigParserV2() + .setRenderClosureAsString(false) + .parse(CONFIG) + .params + then: + params.str1 instanceof String + params.str2 instanceof GString + params.closure1 instanceof Closure + params.map1.bar instanceof Closure + + when: + params = new ConfigParserV2() + .setRenderClosureAsString(true) + .parse(CONFIG) + .params + then: + params.str1 == 'hello' + params.str2 == 'hello world' + params.closure1 instanceof ConfigClosurePlaceholder + params.closure1 == new ConfigClosurePlaceholder('{ "$str" }') + params.map1.foo == 'hello' + params.map1.bar == new ConfigClosurePlaceholder('{ world }') + + } + + def 'should handle extend mem and duration units' () { + ConfigObject result + def CONFIG = ''' + mem1 = 1.GB + mem2 = 1_000_000.toMemory() + mem3 = MemoryUnit.of(2_000) + time1 = 2.hours + time2 = 60_000.toDuration() + time3 = Duration.of(120_000) + flag = 10000 < 1.GB + ''' + + when: + result = new ConfigParserV2() + .parse(CONFIG) + then: + result.mem1 instanceof MemoryUnit + result.mem1 == MemoryUnit.of('1 GB') + result.mem2 == MemoryUnit.of(1_000_000) + result.mem3 == MemoryUnit.of(2_000) + result.time1 instanceof Duration + result.time1 == Duration.of('2 hours') + result.time2 == Duration.of(60_000) + result.time3 == Duration.of(120_000) + result.flag == true + } + + def 'should parse a config from an http server' () { + given: + def folder = Files.createTempDirectory('test') + folder.resolve('conf').mkdir() + + HttpServer server = HttpServer.create(new InetSocketAddress(9900), 0); + server.createContext("/", new ConfigFileHandler(folder)); + server.start() + + folder.resolve('nextflow.config').text = ''' + includeConfig 'conf/base.config' + includeConfig 'http://localhost:9900/conf/remote.config' + ''' + + folder.resolve('conf/base.config').text = ''' + params.foo = 'Hello' + params.bar = 'world!' + ''' + + folder.resolve('conf/remote.config').text = ''' + process { + cpus = 4 + memory = '10GB' + } + ''' + + when: + def url = 'http://localhost:9900/nextflow.config' as Path + def cfg = new ConfigBuilder().buildGivenFiles(url) + then: + cfg.params.foo == 'Hello' + cfg.params.bar == 'world!' + cfg.process.cpus == 4 + cfg.process.memory == '10GB' + + cleanup: + server?.stop(0) + folder?.deleteDir() + } + + def 'should not overwrite param values with nested values of with the same name' () { + given: + def CONFIG = ''' + params { + foo { + bar = 'bar1' + } + baz = 'baz1' + nested { + baz = 'baz2' + foo { + bar = 'bar2' + } + } + } + ''' + + when: + def config = new ConfigParserV2().parse(CONFIG) + + then: + config.params.foo.bar == 'bar1' + config.params.baz == 'baz1' + config.params.nested.baz == 'baz2' + config.params.nested.foo.bar == 'bar2' + + when: + CONFIG = ''' + params { + foo.bar = 'bar1' + baz = 'baz1' + nested.baz = 'baz2' + nested.foo.bar = 'bar2' + } + ''' + and: + config = new ConfigParserV2().parse(CONFIG) + + then: + config.params.foo.bar == 'bar1' + config.params.baz == 'baz1' + config.params.nested.baz == 'baz2' + config.params.nested.foo.bar == 'bar2' + } + + def 'should apply profiles in the order they are specified at runtime' () { + given: + def CONFIG = ''' + profiles { + foo { + params.input = 'foo' + } + + bar { + params.input = 'bar' + } + } + ''' + + when: + def config = new ConfigParserV2().setProfiles(['bar', 'foo']).parse(CONFIG) + + then: + config.params.input == 'foo' + } + + def 'should allow mixed use of dot and block syntax in a profile' () { + given: + def CONFIG = ''' + profiles { + foo { + process.memory = '2 GB' + process { + cpus = 2 + } + } + } + ''' + + when: + def config = new ConfigParserV2().setProfiles(['foo']).parse(CONFIG) + + then: + config.process.memory == '2 GB' + config.process.cpus == 2 + } + + static class ConfigFileHandler implements HttpHandler { + + Path folder + + ConfigFileHandler(Path folder) { + this.folder = folder + } + + void handle(HttpExchange request) throws IOException { + def path = request.requestURI.toString().substring(1) + def file = folder.resolve(path) + + Headers header = request.getResponseHeaders() + header.add("Content-Type", "text/plain") + request.sendResponseHeaders(200, file.size()) + + OutputStream os = request.getResponseBody(); + os.write(file.getBytes()); + os.close(); + } + } + +} diff --git a/modules/nextflow/src/test/groovy/nextflow/script/ScriptRunnerTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/ScriptRunnerTest.groovy index 682337368d..bd2dc3e4bb 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/ScriptRunnerTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/ScriptRunnerTest.groovy @@ -17,7 +17,7 @@ package nextflow.script import groovyx.gpars.dataflow.DataflowVariable -import nextflow.config.ConfigParser +import nextflow.config.ConfigParserFactory import nextflow.exception.AbortRunException import nextflow.exception.ProcessUnrecoverableException import nextflow.processor.TaskProcessor @@ -330,7 +330,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -378,7 +378,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -413,7 +413,7 @@ class ScriptRunnerTest extends Dsl2Spec { workflow { hola() } ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -451,7 +451,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -482,7 +482,7 @@ class ScriptRunnerTest extends Dsl2Spec { workflow { hola() } ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: new MockScriptRunner(session).setScript(script).execute() @@ -530,7 +530,7 @@ class ScriptRunnerTest extends Dsl2Spec { } ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: def result = new MockScriptRunner(session) @@ -684,7 +684,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: def result = new MockScriptRunner(session).setScript(script).execute() @@ -721,7 +721,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: def result = new MockScriptRunner(session).setScript(script).execute() @@ -759,7 +759,7 @@ class ScriptRunnerTest extends Dsl2Spec { ''' and: - def session = new MockSession(new ConfigParser().parse(config)) + def session = new MockSession(ConfigParserFactory.create().parse(config)) when: def result = new MockScriptRunner(session).setScript(script).execute() diff --git a/tests/config-labels.included b/tests/config-labels-included.config similarity index 100% rename from tests/config-labels.included rename to tests/config-labels-included.config diff --git a/tests/config-labels.config b/tests/config-labels.config index 294fa3900a..51af5f4f80 100644 --- a/tests/config-labels.config +++ b/tests/config-labels.config @@ -32,6 +32,6 @@ profiles { } test3 { - includeConfig 'config-labels.included' + includeConfig 'config-labels-included.config' } } diff --git a/tests/config-vars.config b/tests/config-vars.config index a5bafc5237..f8af6330e8 100644 --- a/tests/config-vars.config +++ b/tests/config-vars.config @@ -18,19 +18,19 @@ * author Emilio Palumbo */ -l = [ - a: [1,2], - b: [3,4] -] +params { + a = [1,2] + b = [3,4] +} process { withName: foo { - ext.out = { l.a } + ext.out = { params.a } } withName: bar { - ext.out = { l.b } + ext.out = { params.b } } } diff --git a/tests/profiles.config b/tests/profiles.config index b74a3f87ef..95f0ac4ee9 100644 --- a/tests/profiles.config +++ b/tests/profiles.config @@ -16,8 +16,7 @@ echo = true -def x = 'delta' -includeConfig "${x}.config" +includeConfig "${'delta'}.config" profiles { diff --git a/validation/test.sh b/validation/test.sh index 7de964312a..b3c5d29fbe 100755 --- a/validation/test.sh +++ b/validation/test.sh @@ -11,11 +11,7 @@ export NXF_CMD=${NXF_CMD:-$(get_abs_filename ../launch.sh)} export NXF_ANSI_LOG=false export NXF_DISABLE_CHECK_LATEST=true -# -# Integration tests -# -if [[ $TEST_MODE == 'test_integration' ]]; then - +test_integration() { ( cd ../tests/ sudo bash cleanup.sh @@ -47,6 +43,21 @@ if [[ $TEST_MODE == 'test_integration' ]]; then $NXF_CMD run nextflow-io/rnaseq-nf -with-docker $OPTS -resume exit 0 +} + +# +# Integration tests +# +if [[ $TEST_MODE == 'test_integration' ]]; then + test_integration +fi + +# +# Integration tests (strict syntax) +# +if [[ $TEST_MODE == 'test_parser_v2' ]]; then + export NXF_SYNTAX_PARSER=v2 + test_integration fi #