diff --git a/docs/module.md b/docs/module.md index 73d92fe4e2..9f8afdd7b0 100644 --- a/docs/module.md +++ b/docs/module.md @@ -279,8 +279,179 @@ This feature requires the use of a local or shared file system for the pipeline ## Sharing modules -Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. While Nextflow does not provide an explicit mechanism for sharing modules, there are several ways to do it: +Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules: +- Use the Nextflow module registry (recommended, see below) - Simply copy the module files into your pipeline repository - Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy - Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community + +(module-registry)= + +## Registry-based modules + +:::{versionadded} 26.04.0 +::: + +Nextflow provides a module registry that enables you to install, share, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow DSL. + +### Installing modules from a registry + +Use the `module install` command to download modules from a registry: + +```console +$ nextflow module install nf-core/fastqc +$ nextflow module install nf-core/fastqc -version 1.0.0 +``` + +Installed modules are stored in the `modules/` directory and can be included using the registry syntax with the `@` prefix: + +```nextflow +include { FASTQC } from '@nf-core/fastqc' + +workflow { + reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') + FASTQC(reads) +} +``` + +### Running modules directly + +For ad-hoc tasks or testing, you can run a module directly without creating a workflow: + +```console +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +``` + +This command accepts all standard Nextflow options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed. + +### Managing module versions + +Module versions are tracked in `nextflow_spec.json` in your project directory: + +```json +{ + "modules": { + "@nf-core/fastqc": "1.0.0", + "@nf-core/bwa-align": "1.2.0" + } +} +``` + +When you run your workflow, Nextflow automatically installs or updates modules to match the specified versions. + +### Discovering modules + +Search for available modules using the `module search` command: + +```console +$ nextflow module search alignment +$ nextflow module search "quality control" -limit 10 +``` + +List installed modules in your project: + +```console +$ nextflow module list +``` + +### Module checksum verification + +Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites: + +```console +$ nextflow module install nf-core/fastqc -version 1.1.0 +Warning: Module @nf-core/fastqc has local modifications. Use -force to override. +``` + +Use the `-force` flag to override local modifications when needed. + +### Removing modules + +Use the `module remove` command to uninstall a module: + +```console +$ nextflow module remove nf-core/fastqc +``` + +By default, both the local module files and the entry in `nextflow_spec.json` are removed. Use the flags below to control this behaviour: + +- `-keep-files` — Remove the entry from `nextflow_spec.json` but keep the local module files +- `-keep-config` — Remove the local module files but keep the entry in `nextflow_spec.json` + +### Viewing module information + +Use the `module info` command to display metadata and a usage template for a module: + +```console +$ nextflow module info nf-core/fastqc +$ nextflow module info nf-core/fastqc -version 1.0.0 +``` + +The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-json` to get machine-readable output. + +### Publishing modules + +To share your own modules, use the `module publish` command: + +```console +$ nextflow module publish myorg/my-module +``` + +The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. + +Your module directory must include: + +- `main.nf` - The module entry point +- `meta.yaml` - Module metadata (name, description, version, etc.) +- `README.md` - Module documentation + +Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration: + +```groovy +registry { + apiKey = 'YOUR_REGISTRY_TOKEN' +} +``` + +Use `-dry-run` to validate your module structure without uploading: + +```console +$ nextflow module publish myorg/my-module -dry-run +``` + +### Registry configuration + +By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries: + +```groovy +registry { + url = [ + 'https://private.registry.myorg.com', + 'https://registry.nextflow.io' + ] + apiKey = '${MYORG_TOKEN}' +} +``` + +Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry. + +### Module directory structure + +Registry modules follow a standard directory structure: + +``` +modules/ +└── @scope/ + └── module-name/ + ├── .checksum # Integrity checksum (generated automatically) + ├── README.md # Documentation (required for publishing) + ├── main.nf # Module entry point (required) + ├── meta.yaml # Module metadata (required for publishing) + ├── resources/ # Optional: module binaries and resources + └── templates/ # Optional: process templates +``` + +The `modules/` directory should be committed to your Git repository to ensure reproducibility. + +See the {ref}`cli-page` documentation for complete details on all module commands. diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy index 56bc69708e..26a438a682 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy @@ -90,8 +90,7 @@ class CmdModuleInfo extends CmdBase { throw new AbortOperationException("Incorrect number of arguments") } - def moduleRef = '@' + args[0] - def reference = ModuleReference.parse(moduleRef) + def reference = ModuleReference.parse(args[0]) // Get config def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() @@ -117,10 +116,10 @@ class CmdModuleInfo extends CmdBase { log.warn "Failed to fetch metadata from registry: ${e.message}" } if( !release ) { - throw new AbortOperationException("No release information available for ${reference.nameWithoutPrefix}") + throw new AbortOperationException("No release information available for ${reference}") } if( !release.metadata ) { - log.info("No metadata found for $reference.nameWithoutPrefix ${release.version ? "($release.version)" : ''}") + log.info("No metadata found for $reference ${release.version ? "($release.version)" : ''}") } def moduleUrl = buildModuleUrl(registryConfig.url, reference, release.version) if( !output || output == 'text' ) { @@ -135,7 +134,7 @@ class CmdModuleInfo extends CmdBase { private void printFormattedInfo(ModuleReference reference, ModuleRelease release, String moduleUrl) { ModuleMetadata metadata = release.metadata println "" - println "Module: ${reference.nameWithoutPrefix}" + println "Module: ${reference}" println "Version: ${release.version}" println "URL: ${moduleUrl}" println "Description: ${metadata.description ?: release.description ?: 'N/A'}" @@ -218,7 +217,7 @@ class CmdModuleInfo extends CmdBase { private List generateUsageTemplate(ModuleReference reference, ModuleMetadata metadata) { def template = new ArrayList() - template.add("nextflow module run ${reference.nameWithoutPrefix}".toString()) + template.add("nextflow module run ${reference}".toString()) if( version ) template.add(" -version $version".toString()) @@ -281,7 +280,7 @@ class CmdModuleInfo extends CmdBase { private void printJsonInfo(ModuleReference reference, ModuleRelease release, String moduleUrl) { def metadata = release?.metadata def info = [ - name : reference.nameWithoutPrefix, + name : reference.toString(), fullName : reference.fullName, version : release.version, url : moduleUrl, diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy index f509aef73e..bcc52b6ed7 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy @@ -70,9 +70,7 @@ class CmdModuleInstall extends CmdBase { throw new AbortOperationException("Incorrect number of arguments") } - def moduleRef = '@' + args[0] - - def reference = ModuleReference.parse(moduleRef) + def reference = ModuleReference.parse(args[0]) // Get config def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() @@ -96,7 +94,7 @@ class CmdModuleInstall extends CmdBase { def installedVersion = version ?: resolver.resolveVersion(reference) specFile.addModuleEntry(reference.fullName, installedVersion) - println "Module ${reference.nameWithoutPrefix}@${installedVersion} installed and configured successfully" + println "Module ${reference}@${installedVersion} installed and configured successfully" } catch( AbortOperationException e ) { throw e diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy index b4919b0c68..5992196c2b 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy @@ -111,7 +111,7 @@ class CmdModuleList extends CmdBase { installed.each { module -> def status = getStatusString(module.integrity) - println "${module.reference.nameWithoutPrefix.padRight(40)}${(module.installedVersion ?: 'unknown').padRight(15)}${status}" + println "${module.reference.toString().padRight(40)}${(module.installedVersion ?: 'unknown').padRight(15)}${status}" } println "" } @@ -119,7 +119,7 @@ class CmdModuleList extends CmdBase { private void printJsonList(List installed) { def modules = installed.collect { module -> [ - name : module.reference.nameWithoutPrefix, + name : module.reference.toString(), version : module.installedVersion ?: 'unknown', integrity: module.integrity.toString(), directory: module.directory.toString() @@ -136,8 +136,8 @@ class CmdModuleList extends CmdBase { return 'OK' case ModuleIntegrity.MODIFIED: return 'MODIFIED' - case ModuleIntegrity.MISSING_CHECKSUM: - return 'NO CHECKSUM' + case ModuleIntegrity.NO_REMOTE_MODULE: + return 'LOCAL' case ModuleIntegrity.CORRUPTED: return 'CORRUPTED' default: diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy index a3b46056fc..23cc9f74cc 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy @@ -25,6 +25,7 @@ import nextflow.cli.CmdBase import nextflow.config.ConfigBuilder import nextflow.config.RegistryConfig import nextflow.exception.AbortOperationException +import nextflow.module.ModuleChecksum import nextflow.module.ModuleSpec import nextflow.module.ModuleReference import nextflow.module.ModuleRegistryClient @@ -60,6 +61,9 @@ class CmdModulePublish extends CmdBase { @TestOnly protected ModuleRegistryClient client + //Flag if publish is invoked from a scope/name. In this case we should create/update the .module-info with the correct checksum + private boolean useModuleReference = false + @Override String getName() { return 'publish' @@ -115,14 +119,10 @@ class CmdModulePublish extends CmdBase { private void publishModule(Path moduleDir, RegistryConfig registryConfig, ModuleSpec manifest){ log.info "Creating module bundle..." - def storage = new ModuleStorage(moduleDir.parent) def tempBundleFile = Files.createTempFile("nf-module-publish-", ".tar.gz") try { - storage.createBundle(moduleDir, tempBundleFile) - - // Compute bundle checksum - def checksum = storage.computeBundleChecksum(tempBundleFile) + def checksum = ModuleStorage.createBundle(moduleDir, tempBundleFile) log.info "Bundle checksum: ${checksum}" // Read bundle content as bytes @@ -139,6 +139,14 @@ class CmdModulePublish extends CmdBase { def registryClient = new ModuleRegistryClient(registryConfig) def response = registryClient.publishModule(manifest.name, request, registryUrl) + if (useModuleReference) { + // If publish is performed using the module reference we should create/update the .module-info with the correct checksum + try { + ModuleChecksum.save(moduleDir, ModuleChecksum.compute(moduleDir)) + }catch (Exception e){ + log.warn("Unable to save the checksum - ${e.message}") + } + } println "✓ Module published successfully!" println "" println "Module details:" @@ -246,13 +254,13 @@ class CmdModulePublish extends CmdBase { return Paths.get(module).toAbsolutePath().normalize() } - final ref = ModuleReference.parse('@' + module) + final ref = ModuleReference.parse(module) final localStorage = new ModuleStorage(root ?: Paths.get('.').toAbsolutePath().normalize()) if (!localStorage.isInstalled(ref)){ throw new AbortOperationException("No module diretory found for $module") } - + useModuleReference = true return localStorage.getModuleDir(ref) } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy index b793974f32..3c3ab9d026 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy @@ -68,9 +68,7 @@ class CmdModuleRemove extends CmdBase { throw new AbortOperationException("Cannot use both -keep-config and -keep-files flags together") } - def moduleRef = '@' + args[0] - - def reference = ModuleReference.parse(moduleRef) + def reference = ModuleReference.parse(args[0]) // Get config def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() @@ -87,15 +85,15 @@ class CmdModuleRemove extends CmdBase { // Remove local files unless -keep-files is set if( !keepFiles ) { - println "Removing module files for ${reference.nameWithoutPrefix}..." + println "Removing module files for ${reference}..." filesRemoved = storage.removeModule(reference) if( filesRemoved ) { println "Module files removed successfully" } else { - println "Module ${reference.nameWithoutPrefix} was not installed locally" + println "Module ${reference} was not installed locally" } } else { - println "Keeping module files for ${reference.nameWithoutPrefix} (due to -keep-files flag)" + println "Keeping module files for ${reference} (due to -keep-files flag)" } // Remove config entry unless -keep-config is set @@ -105,7 +103,7 @@ class CmdModuleRemove extends CmdBase { if( configRemoved ) { println "Module entry removed from configuration" } else { - println "Module ${reference.nameWithoutPrefix} was not configured in nextflow_spec.json" + println "Module ${reference} was not configured in nextflow_spec.json" } } else { println "Keeping module entry in nextflow_spec.json (due to -keep-config flag)" @@ -113,9 +111,9 @@ class CmdModuleRemove extends CmdBase { // Summary if( filesRemoved || configRemoved ) { - println "\nModule ${reference.nameWithoutPrefix} removal completed" + println "\nModule ${reference} removal completed" } else { - println "\nModule ${reference.nameWithoutPrefix} was not found" + println "\nModule ${reference} was not found" } } catch( AbortOperationException e ) { diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy index 4212ae0d2d..56f3cf18bf 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy @@ -61,15 +61,12 @@ class CmdModuleRun extends CmdRun { throw new AbortOperationException("Arguments not provided") } - // Parse module reference (first argument starting with @) - String moduleRef = '@' + args[0] - // Parse and validate module reference ModuleReference reference try { - reference = ModuleReference.parse(moduleRef) + reference = ModuleReference.parse(args[0]) } catch( Exception e ) { - throw new AbortOperationException("Invalid module reference: ${moduleRef}", e) + throw new AbortOperationException("Invalid module reference: ${args[0]}", e) } // Get config diff --git a/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy b/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy new file mode 100644 index 0000000000..e1db3366f0 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy @@ -0,0 +1,100 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.Global +import nextflow.NF +import nextflow.Session +import nextflow.config.ConfigBuilder +import nextflow.config.ModulesConfig +import nextflow.config.RegistryConfig +import nextflow.exception.IllegalModulePath +import nextflow.module.spi.RemoteModuleResolver +import nextflow.pipeline.PipelineSpec + +import java.nio.file.Path + +/** + * Default implementation of RemoteModuleResolver using the Nextflow module registry. + * + *

This implementation: + *

    + *
  • Checks for locally installed modules in the project's modules directory
  • + *
  • Downloads modules from the configured registry if not present
  • + *
  • Reads version constraints from nextflow_spec.json
  • + *
  • Uses the Session's registry configuration
  • + *
+ * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class DefaultRemoteModuleResolver implements RemoteModuleResolver { + + @Override + Path resolve(String moduleName, Path baseDir) { + + final modulesConfig = getModuleConfig(baseDir) + + final config = Global.config ?: new ConfigBuilder().setBaseDir(baseDir).build() + final registryConfig = config.navigate('registry') as RegistryConfig + + // Create module resolver + def resolver = new ModuleResolver(baseDir, modulesConfig, registryConfig) + + try { + log.debug "Resolving remote module: ${moduleName}" + + // Parse module reference + def reference = ModuleReference.parse(moduleName) + + // Resolve module (will auto-install if missing or version mismatch) + def mainFile = resolver.resolve(reference, null, true) + + log.debug "Module ${reference} resolved to ${mainFile}" + return mainFile + } catch (Exception e) { + throw new IllegalModulePath( + "Failed to resolve remote module ${moduleName}: ${e.message}", + e + ) + } + } + + @Override + int getPriority() { + return 0 // Default implementation has lowest priority + } + + private ModulesConfig getModuleConfig(Path baseDir) { + def specFile = new PipelineSpec(baseDir) + + if (!specFile.exists()) { + log.warn1("Remote module specified and 'nextflow_spec.json' not found") + return new ModulesConfig() + } + + def modules = specFile.getModules() + if (!modules || modules.isEmpty()) { + log.warn1("Remote module specified and no modules configured in 'nextflow_spec.json'") + return new ModulesConfig() + } + return new ModulesConfig(modules) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy b/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy index cb032d6c2a..88fddb02a6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy @@ -39,7 +39,7 @@ class InstalledModule { Path directory Path mainFile Path manifestFile - Path checksumFile + Path moduleInfoFile String installedVersion String expectedChecksum @@ -54,9 +54,9 @@ class InstalledModule { return ModuleIntegrity.CORRUPTED } - // Check if checksum file exists - if( !Files.exists(checksumFile) ) { - return ModuleIntegrity.MISSING_CHECKSUM + // Check if .module-info file exists + if( !Files.exists(moduleInfoFile) ) { + return ModuleIntegrity.NO_REMOTE_MODULE } try { @@ -71,7 +71,7 @@ class InstalledModule { return ModuleIntegrity.MODIFIED } } catch( Exception e ) { - log.warn "Failed to compute checksum for module ${reference.nameWithoutPrefix}: ${e.message}" + log.warn "Failed to compute checksum for module ${reference}: ${e.message}" return ModuleIntegrity.CORRUPTED } } @@ -84,6 +84,6 @@ class InstalledModule { enum ModuleIntegrity { VALID, // Checksum matches MODIFIED, // Checksum mismatch (local changes) - MISSING_CHECKSUM, // No .checksum file + NO_REMOTE_MODULE, // No .module-info file (local-only module, no registry origin) CORRUPTED // Missing required files } diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy index fc795f1b60..94235bc696 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy @@ -33,7 +33,7 @@ import java.security.MessageDigest class ModuleChecksum { public static final String CHECKSUM_ALGORITHM = "SHA-256" - public static final String CHECKSUM_FILE = ".checksum" + public static final String MODULE_INFO_FILE = ".module-info" /** * Compute the SHA-256 checksum of a module directory @@ -54,7 +54,7 @@ class ModuleChecksum { try( final walkStream = Files.walk(moduleDir) ) { walkStream .filter { Path path -> Files.isRegularFile(path) } - .filter { Path path -> !path.fileName.toString().equals(CHECKSUM_FILE) } + .filter { Path path -> !path.fileName.toString().equals(MODULE_INFO_FILE) } .sorted() .each { Path path -> files.add(path) } } @@ -85,28 +85,35 @@ class ModuleChecksum { } /** - * Save a checksum to the .checksum file in the module directory + * Save a checksum to the .module-info file in the module directory * * @param moduleDir The module directory path * @param checksum The checksum to save */ static void save(Path moduleDir, String checksum) { - def checksumFile = moduleDir.resolve(CHECKSUM_FILE) - Files.writeString(checksumFile, checksum) + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + def props = new Properties() + // If file exists loads to update current just checksum property + if( Files.exists( moduleInfoFile)) + moduleInfoFile.withInputStream { is -> props.load(is) } + props.setProperty('checksum', checksum) + moduleInfoFile.withOutputStream { os -> props.store(os, null) } } /** - * Load a checksum from the .checksum file in the module directory + * Load a checksum from the .module-info file in the module directory * * @param moduleDir The module directory path * @return The checksum, or null if file doesn't exist */ static String load(Path moduleDir) { - def checksumFile = moduleDir.resolve(CHECKSUM_FILE) - if( !Files.exists(checksumFile) ) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + if( !Files.exists(moduleInfoFile) ) { return null } - return checksumFile.text + def props = new Properties() + moduleInfoFile.withInputStream { is -> props.load(is) } + return props.getProperty('checksum') } /** diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy index 12bff1c6ac..23cbf7ba3d 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy @@ -34,7 +34,7 @@ class ModuleReference { // Pattern allows: optional @, scope with letters/digits/hyphens/dots/underscores, name segments separated by slashes (no trailing slash) // Scope: starts with letter/digit, followed by letters/digits/dots/underscores/hyphens // Name: one or more segments (each starting with letter, followed by letters/digits/underscores/hyphens), separated by slashes - private static final Pattern MODULE_NAME_PATTERN = ~/^@?([a-z0-9][a-z0-9._\-]*)\/([a-z][a-z0-9_\-]*(?:\/[a-z][a-z0-9_\-]*)*)$/ + private static final Pattern MODULE_NAME_PATTERN = ~/^([a-z0-9][a-z0-9._\-]*)\/([a-z][a-z0-9._\-]*(?:\/[a-z][a-z0-9._\-]*)*)$/ final String scope final String name @@ -43,7 +43,7 @@ class ModuleReference { ModuleReference(String scope, String name) { this.scope = scope this.name = name - this.fullName = "@${scope}/${name}" + this.fullName = "${scope}/${name}" } /** @@ -65,7 +65,7 @@ class ModuleReference { if( !matcher.matches() ) { throw new AbortOperationException( "Invalid module reference: '${source}'. " + - "Expected format: [@]scope/name where scope is lowercase alphanumeric with dots/underscores/hyphens " + + "Expected format: scope/name where scope is lowercase alphanumeric with dots/underscores/hyphens " + "and name is lowercase alphanumeric with underscores/hyphens, optionally with slash-separated segments" ) } @@ -73,15 +73,6 @@ class ModuleReference { return new ModuleReference(matcher.group(1), matcher.group(2)) } - /** - * Get the module name without the @ prefix - * - * @return Module name in format "scope/name" - */ - String getNameWithoutPrefix() { - return "${scope}/${name}" - } - @Override String toString() { return fullName diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy index fa1d827622..4b2fc225ec 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy @@ -56,10 +56,7 @@ class ModuleRegistryClient { } private String encodeName(String name) { - return URLEncoder.encode( - name.startsWith('@') ? name.substring(1) : name, - 'UTF-8' - ) + return URLEncoder.encode(name, 'UTF-8') } /** @@ -443,11 +440,7 @@ class ModuleRegistryClient { " }\n" ) } - try { - return publishModuleToRegistry(registryUrl, name, request, authToken) - } catch( Exception e ) { - throw new AbortOperationException("Failed to publish to ${registryUrl}", e) - } + return publishModuleToRegistry(registryUrl, name, request, authToken) } /** diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy index 11ad0f2335..a5a55c2fe8 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy @@ -69,25 +69,27 @@ class ModuleResolver { def integrity = installed.integrity if( integrity == ModuleIntegrity.CORRUPTED ) { throw new AbortOperationException( - "Module ${reference.nameWithoutPrefix} is corrupted (missing required files). " + + "Module ${reference} is corrupted (missing required files). " + "Please remove and reinstall." ) } if( integrity == ModuleIntegrity.MODIFIED ) { - log.warn "Module ${reference.nameWithoutPrefix} has local modifications (checksum mismatch)" + log.warn1 "Module ${reference} has local modifications (checksum mismatch)" + } else if( integrity == ModuleIntegrity.NO_REMOTE_MODULE ) { + log.warn1 "Module ${reference} has no registry origin (.module-info missing)" } // Check if version matches if( targetVersion && installed.installedVersion != targetVersion ) { if( autoInstall ) { - log.info "Upgrading module ${reference.nameWithoutPrefix} from ${installed.installedVersion} to ${targetVersion}" + log.info "Upgrading module ${reference} from ${installed.installedVersion} to ${targetVersion}" return installModule(reference, targetVersion) } else { throw new AbortOperationException( - "Module ${reference.nameWithoutPrefix} version mismatch: " + + "Module ${reference} version mismatch: " + "installed=${installed.installedVersion}, required=${targetVersion}. " + - "Run 'nextflow module install ${reference.nameWithoutPrefix}@${targetVersion}' to update." + "Run 'nextflow module install ${reference}@${targetVersion}' to update." ) } } @@ -101,17 +103,17 @@ class ModuleResolver { return installModule(reference, targetVersion) } else { throw new AbortOperationException( - "Module ${reference.nameWithoutPrefix} is not installed. " + - "Run 'nextflow module install ${reference.nameWithoutPrefix}' to install." + "Module ${reference} is not installed. " + + "Run 'nextflow module install ${reference}' to install." ) } } String resolveVersion(ModuleReference reference) { final version = modulesConfig.getVersion(reference.fullName) - ?: registryClient.fetchModule(reference.fullName).latest?.version + ?: registryClient.fetchModule(reference.fullName)?.latest?.version if( !version ) { - throw new AbortOperationException("Module ${reference.nameWithoutPrefix} has no published versions") + throw new AbortOperationException("Module ${reference} has no published versions") } return version } @@ -131,7 +133,7 @@ class ModuleResolver { if( storage.isInstalled(reference) ) { def installed = storage.getInstalledModule(reference) if( installed.installedVersion == version ) { - log.info "Module ${reference.nameWithoutPrefix}@${installed.installedVersion} is already installed (version $version)" + log.info "Module ${reference}@${installed.installedVersion} is already installed (version $version)" return installed.mainFile } @@ -139,14 +141,20 @@ class ModuleResolver { def integrity = installed.integrity if( integrity == ModuleIntegrity.MODIFIED && !force ) { throw new AbortOperationException( - "Module ${reference.nameWithoutPrefix} has local modifications. " + + "Module ${reference} has local modifications. " + + "Use --force to override, or save your changes first." + ) + } + if( integrity == ModuleIntegrity.NO_REMOTE_MODULE && !force ) { + throw new AbortOperationException( + " Folder 'modules/${reference}' already exists and is not a valid remote module. " + "Use --force to override, or save your changes first." ) } } - log.info "Installing module ${reference.nameWithoutPrefix}@${version}..." + log.info "Installing module ${reference}@${version}..." // Download module package to temporary location Path tempFile = Files.createTempFile("nf-module-", ".tgz") @@ -157,7 +165,7 @@ class ModuleResolver { // Install to modules directory (will compute directory checksum for future integrity checks) InstalledModule installed = storage.installModule(reference, version, tempFile) - log.info "Module ${reference.nameWithoutPrefix}@${version} installed successfully at ${installed.mainFile.parent}" + log.info "Module ${reference}@${version} installed successfully at ${installed.mainFile.parent}" return installed.mainFile } finally { diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy index 97802c711d..9cffaa0279 100644 --- a/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy @@ -70,7 +70,7 @@ class ModuleStorage { * @return The module directory path */ Path getModuleDir(ModuleReference reference) { - return modulesDir.resolve("@${reference.scope}").resolve(reference.name) + return modulesDir.resolve(reference.scope).resolve(reference.name) } /** @@ -101,7 +101,7 @@ class ModuleStorage { directory: moduleDir, mainFile: moduleDir.resolve(Const.DEFAULT_MAIN_FILE_NAME), manifestFile: moduleDir.resolve(MODULE_MANIFEST_FILE), - checksumFile: moduleDir.resolve(ModuleChecksum.CHECKSUM_FILE), + moduleInfoFile: moduleDir.resolve(ModuleChecksum.MODULE_INFO_FILE), ) // Load checksum if available @@ -111,7 +111,7 @@ class ModuleStorage { } /** - * List all installed modules + * List all installed modules by scanning for directories containing a .module-info marker file. * * @return List of InstalledModule objects */ @@ -122,63 +122,27 @@ class ModuleStorage { List modules = [] - // Iterate over scope directories - try( final scopeStream = Files.list(modulesDir) ){ - scopeStream.each { Path scopeDir -> - if (!Files.isDirectory(scopeDir)) return - - def scopeDirName = scopeDir.fileName.toString() - // Remove @ prefix from directory name to get scope - def scope = scopeDirName.startsWith('@') ? scopeDirName.substring(1) : scopeDirName - - // Recursively find all directories containing meta.yml under this scope - findModulesRecursive(scopeDir, scope, modules) - } - } - - return modules - } - - /** - * Recursively find modules in subdirectories - * @param dir Current directory to search - * @param scope Module scope - * @param modules List to accumulate found modules - */ - private void findModulesRecursive(Path dir, String scope, List modules) { - if (!Files.isDirectory(dir)) return - - // Check if current directory contains meta.yml (is a module) - if (Files.exists(dir.resolve(MODULE_MANIFEST_FILE))) { - // Calculate the module name from the path relative to scope directory - def scopeDir = dir.getParent() - while (scopeDir != null && !scopeDir.fileName.toString().equals('@' + scope)) { - scopeDir = scopeDir.getParent() - } - - if (scopeDir != null) { - def relativePath = scopeDir.relativize(dir).toString() - def name = relativePath.replace('\\', '/') // Normalize path separators - def reference = new ModuleReference(scope, name) - - def installed = getInstalledModule(reference) - if (installed) { - modules.add(installed) - } - } - } - - // Recursively search subdirectories - try (final subStream = Files.list(dir) ) { - subStream.each { Path subDir -> - if (Files.isDirectory(subDir)) { - findModulesRecursive(subDir, scope, modules) + try( final walkStream = Files.walk(modulesDir) ) { + walkStream + .filter { Path path -> Files.isDirectory(path) } + .filter { Path path -> Files.exists(path.resolve(ModuleChecksum.MODULE_INFO_FILE)) } + .each { Path moduleDir -> + try { + def rel = modulesDir.relativize(moduleDir) + if( rel.nameCount < 2 ) return // Need at least scope/name + def reference = ModuleReference.parse(rel.toString()) + def installed = getInstalledModule(reference) + if( installed ) modules.add(installed) + } catch(Exception e){ + // Catching exception to go on inspecting other valid folders + log.debug("Not a valid module reference - $e.message") } } - } catch (IOException e) { - log.warn "Failed to list directory ${dir}: ${e.message}" + log.warn "Failed to scan modules directory ${modulesDir}: ${e.message}" } + + return modules } /** @@ -214,7 +178,7 @@ class ModuleStorage { def checksum = ModuleChecksum.compute(moduleDir) ModuleChecksum.save(moduleDir, checksum) - log.debug "Installed module ${reference.nameWithoutPrefix}@${version} to ${moduleDir}" + log.debug "Installed module ${reference}@${version} to ${moduleDir}" return getInstalledModule(reference) } @@ -227,7 +191,7 @@ class ModuleStorage { log.warn "Failed to clean up after installation failure: ${cleanupError.message}" } } - throw new AbortOperationException("Failed to install module ${reference.nameWithoutPrefix}@${version}", e) + throw new AbortOperationException("Failed to install module ${reference}@${version}", e) } } @@ -246,7 +210,7 @@ class ModuleStorage { try { FileHelper.deletePath(moduleDir) - log.debug "Removed module: ${reference.nameWithoutPrefix}" + log.debug "Removed module: ${reference}" // Clean up empty scope directory def scopeDir = moduleDir.parent @@ -257,7 +221,7 @@ class ModuleStorage { return true } catch (Exception e) { - throw new AbortOperationException("Failed to remove module ${reference.nameWithoutPrefix}", e) + throw new AbortOperationException("Failed to remove module ${reference}", e) } } @@ -362,9 +326,9 @@ class ModuleStorage { * * @param moduleDir The module directory to bundle * @param targetFile The target bundle file path - * @return The created bundle file with its checksum + * @return The created bundle file checksum */ - Path createBundle(Path moduleDir, Path targetFile) { + static String createBundle(Path moduleDir, Path targetFile) { if (!Files.exists(moduleDir) || !Files.isDirectory(moduleDir)) { throw new AbortOperationException("Module directory not found: ${moduleDir}") } @@ -384,9 +348,9 @@ class ModuleStorage { } } } - - log.debug "Created module bundle: ${targetFile} (size: ${Files.size(targetFile)} bytes)" - return targetFile + final checksum = computeBundleChecksum(targetFile) + log.debug "Created module bundle: ${targetFile} (size: ${Files.size(targetFile)} bytes, checksum: $checksum)" + return checksum } catch (Exception e) { // Clean up partial file on failure @@ -408,12 +372,12 @@ class ModuleStorage { * @param sourceDir The source directory being archived * @param currentPath The current path being added */ - private void addToTarArchive(TarArchiveOutputStream tos, Path sourceDir, Path currentPath) { + private static void addToTarArchive(TarArchiveOutputStream tos, Path sourceDir, Path currentPath) { try ( def tarStream = Files.list(currentPath)) { tarStream.each { Path path -> - // Skip .checksum file when creating bundle - if (path.fileName.toString() == ModuleChecksum.CHECKSUM_FILE) { + // Skip .module-info file when creating bundle + if (path.fileName.toString() == ModuleChecksum.MODULE_INFO_FILE) { return } @@ -447,7 +411,7 @@ class ModuleStorage { * @param bundleFile The bundle file * @return The SHA-256 checksum as hex string */ - String computeBundleChecksum(Path bundleFile) { + static String computeBundleChecksum(Path bundleFile) { if (!Files.exists(bundleFile)) { throw new AbortOperationException("Bundle file not found: ${bundleFile}") } diff --git a/modules/nextflow/src/main/groovy/nextflow/pipeline/PipelineSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/pipeline/PipelineSpec.groovy index 766b71f62e..2e109cdc4f 100644 --- a/modules/nextflow/src/main/groovy/nextflow/pipeline/PipelineSpec.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/pipeline/PipelineSpec.groovy @@ -51,8 +51,8 @@ class PipelineSpec { * @param version The module version */ void addModuleEntry(String moduleName, String version) { - // Normalize module name (ensure it starts with @) - def normalizedName = moduleName.startsWith('@') ? moduleName : '@' + moduleName + // Normalize module name (strip leading @ if present) + def normalizedName = moduleName.startsWith('@') ? moduleName.substring(1) : moduleName def spec = readSpecFile() @@ -80,8 +80,6 @@ class PipelineSpec { * @return true if entry was removed, false if it didn't exist */ boolean removeModuleEntry(String moduleName) { - // Normalize module name (ensure it starts with @) - def normalizedName = moduleName.startsWith('@') ? moduleName : '@' + moduleName def spec = readSpecFile() @@ -89,10 +87,10 @@ class PipelineSpec { return false } final modules = spec.modules as Map - if( modules.remove(normalizedName) == null ) + if( modules.remove(moduleName) == null ) return false writeSpecFile(spec) - log.info "Removed ${normalizedName} from ${SPEC_FILE_NAME}" + log.info "Removed ${moduleName} from ${SPEC_FILE_NAME}" return true } /** @@ -145,4 +143,4 @@ class PipelineSpec { throw new RuntimeException("Failed to write spec file ${specFile}: ${e.message}", e) } } -} \ No newline at end of file +} diff --git a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy index b19f814745..531e6c4cc4 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy @@ -29,6 +29,8 @@ import nextflow.NF import nextflow.Session import nextflow.exception.IllegalModulePath import nextflow.exception.ScriptCompilationException +import nextflow.module.ModuleReference +import nextflow.module.spi.RemoteModuleResolverProvider import nextflow.plugin.Plugins import nextflow.plugin.extension.PluginExtensionProvider import nextflow.script.parser.v1.ScriptLoaderV1 @@ -162,14 +164,24 @@ class IncludeDef { @PackageScope Path resolveModulePath(include) { assert include - final result = include as Path if( result.isAbsolute() ) { if( result.scheme == 'file' ) return result throw new IllegalModulePath("Cannot resolve module path: ${result.toUriString()}") } + final str = include.toString() + if( str.startsWith('./') || str.startsWith('../') ) { + return getOwnerPath().resolveSibling(str).normalize() + } + // Not a local path — treat as remote module reference (scope/name) + return resolveRemoteModulePath(str) + } - return getOwnerPath().resolveSibling(include.toString()) + @PackageScope + Path resolveRemoteModulePath(String moduleName) { + // Use SPI to get the remote module resolver implementation + def resolver = RemoteModuleResolverProvider.getInstance() + return resolver.resolve(moduleName, session.baseDir) } @PackageScope @@ -206,9 +218,15 @@ class IncludeDef { throw new IllegalModulePath("Remote modules are not allowed -- Offending module: ${path.toUriString()}") final str = path.toString() - if( !str.startsWith('/') && !str.startsWith('./') && !str.startsWith('../') && !str.startsWith('plugin/') ) - throw new IllegalModulePath("Module path must start with / or ./ prefix -- Offending module: $str") + if( str.startsWith('/') || str.startsWith('./') || str.startsWith('../') || str.startsWith('plugin/') ) + return + // Otherwise must be a valid remote module reference in scope/name format + try { + ModuleReference.parse(str) + } catch( Exception e ) { + throw new IllegalModulePath("Module path must start with '/', './', '../' or 'plugin/' prefix, or be a valid remote module reference (scope/name) -- Offending module: $str") + } } @PackageScope diff --git a/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver b/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver new file mode 100644 index 0000000000..d12a870946 --- /dev/null +++ b/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver @@ -0,0 +1 @@ +nextflow.module.DefaultRemoteModuleResolver diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy index 7668fa7cf7..ba90e5d005 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy @@ -177,7 +177,7 @@ class CmdModuleInfoTest extends Specification { then: json.name == 'nf-core/fastqc' - json.fullName == '@nf-core/fastqc' + json.fullName == 'nf-core/fastqc' json.version == '1.0.0' json.description == 'FastQC quality control' json.authors == ['nf-core'] diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy index c04dad81bb..a2f088b544 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy @@ -20,6 +20,7 @@ import io.seqera.npr.api.schema.v1.Module import io.seqera.npr.api.schema.v1.ModuleRelease import nextflow.cli.Launcher import nextflow.exception.AbortOperationException +import nextflow.module.ModuleChecksum import nextflow.module.ModuleRegistryClient import nextflow.pipeline.PipelineSpec import org.apache.commons.compress.archivers.tar.TarArchiveEntry @@ -61,11 +62,11 @@ class CmdModuleInstallTest extends Specification { // Mock registry client def mockClient = Mock(ModuleRegistryClient) - mockClient.fetchModule('@nf-core/fastqc') >> new Module( - name: '@nf-core/fastqc', + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', latest: new ModuleRelease(version: '1.0.0') ) - mockClient.downloadModule('@nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> Files.write(dest, modulePackage) return dest } @@ -81,14 +82,14 @@ class CmdModuleInstallTest extends Specification { output.contains('1.0.0') and: - def moduleDir = tempDir.resolve('modules/@nf-core/fastqc') + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') Files.exists(moduleDir) Files.exists(moduleDir.resolve('main.nf')) Files.exists(moduleDir.resolve('meta.yml')) and: def spec = new PipelineSpec(tempDir) - spec.getModules().get('@nf-core/fastqc') == '1.0.0' + spec.getModules().get('nf-core/fastqc') == '1.0.0' } def 'should install module with specific version'() { @@ -122,14 +123,14 @@ class CmdModuleInstallTest extends Specification { and: def spec = new PipelineSpec(tempDir) - spec.getModules().get('@nf-core/fastqc') == '2.0.0' + spec.getModules().get('nf-core/fastqc') == '2.0.0' } def 'should update existing module with force flag'() { given: // Pre-install version 1.0.0 - def moduleDir = tempDir.resolve('modules/@nf-core/fastqc') + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') Files.createDirectories(moduleDir) moduleDir.resolve('main.nf').text = 'process OLD { }' moduleDir.resolve('meta.yml').text = """ @@ -139,7 +140,7 @@ class CmdModuleInstallTest extends Specification { """.stripIndent() def spec = new PipelineSpec(tempDir) - spec.addModuleEntry('@nf-core/fastqc', '1.0.0') + spec.addModuleEntry('nf-core/fastqc', '1.0.0') and: def cmd = new CmdModuleInstall() @@ -155,7 +156,7 @@ class CmdModuleInstallTest extends Specification { def modulePackage = createModulePackage('nf-core', 'fastqc', '2.0.0') def mockClient = Mock(ModuleRegistryClient) - mockClient.downloadModule('@nf-core/fastqc', '2.0.0', _) >> { String name, String version, Path dest -> + mockClient.downloadModule('nf-core/fastqc', '2.0.0', _) >> { String name, String version, Path dest -> Files.write(dest, modulePackage) return dest } @@ -171,7 +172,7 @@ class CmdModuleInstallTest extends Specification { and: def updatedSpec = new PipelineSpec(tempDir) - updatedSpec.getModules().get('@nf-core/fastqc') == '2.0.0' + updatedSpec.getModules().get('nf-core/fastqc') == '2.0.0' and: moduleDir.resolve('main.nf').text.contains('FASTQC') // New content @@ -180,7 +181,7 @@ class CmdModuleInstallTest extends Specification { def 'should fail when module already installed without force'() { given: // Pre-install the module - def moduleDir = tempDir.resolve('modules/@nf-core/fastqc') + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') Files.createDirectories(moduleDir) moduleDir.resolve('main.nf').text = 'process FASTQC { }' moduleDir.resolve('meta.yml').text = """ @@ -188,9 +189,9 @@ class CmdModuleInstallTest extends Specification { version: '1.0.0' description: Test module """.stripIndent() - moduleDir.resolve('.checksum').text = 'wrong-checksum' + ModuleChecksum.save(moduleDir, 'wrong-checksum') def spec = new PipelineSpec(tempDir) - spec.addModuleEntry('@nf-core/fastqc', '1.0.0') + spec.addModuleEntry('nf-core/fastqc', '1.0.0') and: def cmd = new CmdModuleInstall() @@ -202,8 +203,8 @@ class CmdModuleInstallTest extends Specification { cmd.root = tempDir def mockClient = Mock(ModuleRegistryClient) - mockClient.fetchModule('@nf-core/fastqc') >> new Module( - name: '@nf-core/fastqc', + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', latest: new ModuleRelease(version: '2.0.0') ) cmd.client = mockClient @@ -229,11 +230,11 @@ class CmdModuleInstallTest extends Specification { def modulePackage = createModulePackage('myorg', 'custom-module', '1.0.0') def mockClient = Mock(ModuleRegistryClient) - mockClient.fetchModule('@myorg/custom-module') >> new Module( - name: '@myorg/custom-module', + mockClient.fetchModule('myorg/custom-module') >> new Module( + name: 'myorg/custom-module', latest: new ModuleRelease(version: '1.0.0') ) - mockClient.downloadModule('@myorg/custom-module', '1.0.0', _) >> { String name, String version, Path dest -> + mockClient.downloadModule('myorg/custom-module', '1.0.0', _) >> { String name, String version, Path dest -> Files.write(dest, modulePackage) return dest } @@ -243,13 +244,13 @@ class CmdModuleInstallTest extends Specification { cmd.run() then: - def moduleDir = tempDir.resolve('modules/@myorg/custom-module') + def moduleDir = tempDir.resolve('modules/myorg/custom-module') Files.exists(moduleDir) Files.exists(moduleDir.resolve('main.nf')) and: def spec = new PipelineSpec(tempDir) - spec.getModules().get('@myorg/custom-module') == '1.0.0' + spec.getModules().get('myorg/custom-module') == '1.0.0' } def 'should create modules directory if it does not exist'() { @@ -265,11 +266,11 @@ class CmdModuleInstallTest extends Specification { def modulePackage = createModulePackage('nf-core', 'fastqc', '1.0.0') def mockClient = Mock(ModuleRegistryClient) - mockClient.fetchModule('@nf-core/fastqc') >> new Module( + mockClient.fetchModule('nf-core/fastqc') >> new Module( name: 'nf-core/fastqc', latest: new ModuleRelease(version: '1.0.0') ) - mockClient.downloadModule('@nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> Files.write(dest, modulePackage) return dest } @@ -280,8 +281,8 @@ class CmdModuleInstallTest extends Specification { then: Files.exists(tempDir.resolve('modules')) - Files.exists(tempDir.resolve('modules/@nf-core')) - Files.exists(tempDir.resolve('modules/@nf-core/fastqc')) + Files.exists(tempDir.resolve('modules/nf-core')) + Files.exists(tempDir.resolve('modules/nf-core/fastqc')) } def 'should create checksum file after installation'() { @@ -297,11 +298,11 @@ class CmdModuleInstallTest extends Specification { def modulePackage = createModulePackage('nf-core', 'fastqc', '1.0.0') def mockClient = Mock(ModuleRegistryClient) - mockClient.fetchModule('@nf-core/fastqc') >> new Module( + mockClient.fetchModule('nf-core/fastqc') >> new Module( name: 'nf-core/fastqc', latest: new ModuleRelease(version: '1.0.0') ) - mockClient.downloadModule('@nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> Files.write(dest, modulePackage) return dest } @@ -311,11 +312,11 @@ class CmdModuleInstallTest extends Specification { cmd.run() then: - def moduleDir = tempDir.resolve('modules/@nf-core/fastqc') - Files.exists(moduleDir.resolve('.checksum')) + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.exists(moduleDir.resolve('.module-info')) and: - def checksum = moduleDir.resolve('.checksum').text + def checksum = ModuleChecksum.load(moduleDir) checksum != null !checksum.isEmpty() } diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy index 8d745562a3..00bafc75b8 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy @@ -65,7 +65,7 @@ class CmdModuleListTest extends Specification { output.contains('1.0.0') output.contains('nf-core/multiqc') output.contains('2.1.0') - output.contains('OK') || output.contains('NO CHECKSUM') + output.contains('OK') } def 'should list installed modules with JSON output'() { @@ -177,9 +177,8 @@ class CmdModuleListTest extends Specification { description: Test module """.stripIndent() - // Create checksum - def checksum = ModuleChecksum.compute(moduleDir) - moduleDir.resolve('.checksum').text = checksum + // Create .module-info + ModuleChecksum.save(moduleDir, ModuleChecksum.compute(moduleDir)) return moduleDir } diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy index 0361b3a274..0b4972c3ef 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy @@ -51,7 +51,7 @@ class CmdModuleRemoveTest extends Specification { // Create spec file with module entry def specFile = new PipelineSpec(tempDir) - specFile.addModuleEntry('@nf-core/fastqc', '1.0.0') + specFile.addModuleEntry('nf-core/fastqc', '1.0.0') and: def cmd = new CmdModuleRemove() @@ -71,7 +71,7 @@ class CmdModuleRemoveTest extends Specification { and: def spec = new PipelineSpec(tempDir) - spec.getModules().get('@nf-core/fastqc') == null + spec.getModules().get('nf-core/fastqc') == null } def 'should keep config with -keep-config flag'() { @@ -82,7 +82,7 @@ class CmdModuleRemoveTest extends Specification { // Create spec file def specFile = new PipelineSpec(tempDir) - specFile.addModuleEntry('@nf-core/fastqc', '1.0.0') + specFile.addModuleEntry('nf-core/fastqc', '1.0.0') and: def cmd = new CmdModuleRemove() @@ -101,7 +101,7 @@ class CmdModuleRemoveTest extends Specification { and: def spec = new PipelineSpec(tempDir) - spec.getModules().get('@nf-core/fastqc') == '1.0.0' + spec.getModules().get('nf-core/fastqc') == '1.0.0' } def 'should keep files with -keep-files flag'() { @@ -112,7 +112,7 @@ class CmdModuleRemoveTest extends Specification { // Create spec file def specFile = new PipelineSpec(tempDir) - specFile.addModuleEntry('@nf-core/fastqc', '1.0.0') + specFile.addModuleEntry('nf-core/fastqc', '1.0.0') and: def cmd = new CmdModuleRemove() @@ -132,7 +132,7 @@ class CmdModuleRemoveTest extends Specification { and: def spec = new PipelineSpec(tempDir) - spec.getModules().get('@nf-core/fastqc') == null + spec.getModules().get('nf-core/fastqc') == null } def 'should fail when both keep flags are set'() { diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy index c3741d0fe1..d93f58be53 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy @@ -89,7 +89,7 @@ class CmdModuleRunTest extends Specification { def moduleRelease = new ModuleRelease() moduleRelease.version = '1.0.0' def module = new Module() - module.name = '@nf-core/test-module' + module.name = 'nf-core/test-module' module.latest = moduleRelease mockClient.fetchModule(_) >> module // Use wildcard to match any argument mockClient.downloadModule(_, _, _) >> { String name, String version, Path dest -> @@ -152,7 +152,7 @@ class CmdModuleRunTest extends Specification { def modulePackage = createModulePackage(moduleScript) def mockClient = Mock(ModuleRegistryClient) - mockClient.downloadModule('@nf-core/test-module', '2.0.0', _) >> { String name, String version, Path dest -> + mockClient.downloadModule('nf-core/test-module', '2.0.0', _) >> { String name, String version, Path dest -> Files.write(dest, modulePackage) return dest } diff --git a/modules/nextflow/src/test/groovy/nextflow/config/ModulesConfigTest.groovy b/modules/nextflow/src/test/groovy/nextflow/config/ModulesConfigTest.groovy index d4c28cbf6f..f320aa472d 100644 --- a/modules/nextflow/src/test/groovy/nextflow/config/ModulesConfigTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/config/ModulesConfigTest.groovy @@ -38,11 +38,11 @@ class ModulesConfigTest extends Specification { def config = new ModulesConfig() when: - config.setVersion('@nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/fastqc', '1.0.0') then: - config.getVersion('@nf-core/fastqc') == '1.0.0' - config.hasVersion('@nf-core/fastqc') + config.getVersion('nf-core/fastqc') == '1.0.0' + config.hasVersion('nf-core/fastqc') } def 'should return null for unconfigured module'() { @@ -50,33 +50,33 @@ class ModulesConfigTest extends Specification { def config = new ModulesConfig() when: - def version = config.getVersion('@nf-core/bwa') + def version = config.getVersion('nf-core/bwa') then: version == null - !config.hasVersion('@nf-core/bwa') + !config.hasVersion('nf-core/bwa') } def 'should override existing version'() { given: def config = new ModulesConfig() - config.setVersion('@nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/fastqc', '1.0.0') when: - config.setVersion('@nf-core/fastqc', '2.0.0') + config.setVersion('nf-core/fastqc', '2.0.0') then: - config.getVersion('@nf-core/fastqc') == '2.0.0' + config.getVersion('nf-core/fastqc') == '2.0.0' } def 'should return unmodifiable map from getModules'() { given: def config = new ModulesConfig() - config.setVersion('@nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/fastqc', '1.0.0') when: def modules = config.getAllModules() - modules.put('@nf-core/bwa', '2.0.0') + modules.put('nf-core/bwa', '2.0.0') then: thrown(UnsupportedOperationException) @@ -85,18 +85,18 @@ class ModulesConfigTest extends Specification { def 'should return all configured modules'() { given: def config = new ModulesConfig() - config.setVersion('@nf-core/fastqc', '1.0.0') - config.setVersion('@nf-core/bwa', '2.0.0') - config.setVersion('@myorg/custom', '0.5.0') + config.setVersion('nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/bwa', '2.0.0') + config.setVersion('myorg/custom', '0.5.0') when: def modules = config.getAllModules() then: modules.size() == 3 - modules['@nf-core/fastqc'] == '1.0.0' - modules['@nf-core/bwa'] == '2.0.0' - modules['@myorg/custom'] == '0.5.0' + modules['nf-core/fastqc'] == '1.0.0' + modules['nf-core/bwa'] == '2.0.0' + modules['myorg/custom'] == '0.5.0' } def 'should handle empty initialization'() { @@ -105,7 +105,7 @@ class ModulesConfigTest extends Specification { then: config.getAllModules().isEmpty() - !config.hasVersion('@nf-core/fastqc') + !config.hasVersion('nf-core/fastqc') } def 'should store multiple versions independently'() { @@ -113,14 +113,14 @@ class ModulesConfigTest extends Specification { def config = new ModulesConfig() when: - config.setVersion('@nf-core/fastqc', '1.0.0') - config.setVersion('@nf-core/bwa', '2.0.0') - config.setVersion('@myorg/custom', '0.5.0') + config.setVersion('nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/bwa', '2.0.0') + config.setVersion('myorg/custom', '0.5.0') then: - config.getVersion('@nf-core/fastqc') == '1.0.0' - config.getVersion('@nf-core/bwa') == '2.0.0' - config.getVersion('@myorg/custom') == '0.5.0' + config.getVersion('nf-core/fastqc') == '1.0.0' + config.getVersion('nf-core/bwa') == '2.0.0' + config.getVersion('myorg/custom') == '0.5.0' config.allModules.size() == 3 } @@ -129,13 +129,13 @@ class ModulesConfigTest extends Specification { def config = new ModulesConfig() when: - config.setVersion('@org-name/module-name', '1.0.0') - config.setVersion('@org_name/module_name', '2.0.0') + config.setVersion('org-name/module-name', '1.0.0') + config.setVersion('org_name/module_name', '2.0.0') config.setVersion('simple-module', '3.0.0') then: - config.getVersion('@org-name/module-name') == '1.0.0' - config.getVersion('@org_name/module_name') == '2.0.0' + config.getVersion('org-name/module-name') == '1.0.0' + config.getVersion('org_name/module_name') == '2.0.0' config.getVersion('simple-module') == '3.0.0' } @@ -144,41 +144,41 @@ class ModulesConfigTest extends Specification { def config = new ModulesConfig() when: - config.setVersion('@nf-core/fastqc', '1.0.0') - config.setVersion('@nf-core/bwa', 'v2.0.0') - config.setVersion('@nf-core/samtools', '1.0.0-beta') - config.setVersion('@nf-core/bowtie', '1.0.0-rc.1') + config.setVersion('nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/bwa', 'v2.0.0') + config.setVersion('nf-core/samtools', '1.0.0-beta') + config.setVersion('nf-core/bowtie', '1.0.0-rc.1') then: - config.getVersion('@nf-core/fastqc') == '1.0.0' - config.getVersion('@nf-core/bwa') == 'v2.0.0' - config.getVersion('@nf-core/samtools') == '1.0.0-beta' - config.getVersion('@nf-core/bowtie') == '1.0.0-rc.1' + config.getVersion('nf-core/fastqc') == '1.0.0' + config.getVersion('nf-core/bwa') == 'v2.0.0' + config.getVersion('nf-core/samtools') == '1.0.0-beta' + config.getVersion('nf-core/bowtie') == '1.0.0-rc.1' } def 'should check if multiple modules have versions'() { given: def config = new ModulesConfig() - config.setVersion('@nf-core/fastqc', '1.0.0') - config.setVersion('@nf-core/bwa', '2.0.0') + config.setVersion('nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/bwa', '2.0.0') expect: - config.hasVersion('@nf-core/fastqc') - config.hasVersion('@nf-core/bwa') - !config.hasVersion('@nf-core/samtools') + config.hasVersion('nf-core/fastqc') + config.hasVersion('nf-core/bwa') + !config.hasVersion('nf-core/samtools') } def 'should handle version updates'() { given: def config = new ModulesConfig() - config.setVersion('@nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/fastqc', '1.0.0') when: - config.setVersion('@nf-core/fastqc', '1.1.0') - config.setVersion('@nf-core/fastqc', '2.0.0') + config.setVersion('nf-core/fastqc', '1.1.0') + config.setVersion('nf-core/fastqc', '2.0.0') then: - config.getVersion('@nf-core/fastqc') == '2.0.0' + config.getVersion('nf-core/fastqc') == '2.0.0' } def 'should maintain separate versions for different modules'() { @@ -186,12 +186,12 @@ class ModulesConfigTest extends Specification { def config = new ModulesConfig() when: - config.setVersion('@nf-core/fastqc', '1.0.0') - config.setVersion('@nf-core/bwa', '2.0.0') + config.setVersion('nf-core/fastqc', '1.0.0') + config.setVersion('nf-core/bwa', '2.0.0') then: - config.getVersion('@nf-core/fastqc') == '1.0.0' - config.getVersion('@nf-core/bwa') == '2.0.0' - config.getVersion('@nf-core/fastqc') != config.getVersion('@nf-core/bwa') + config.getVersion('nf-core/fastqc') == '1.0.0' + config.getVersion('nf-core/bwa') == '2.0.0' + config.getVersion('nf-core/fastqc') != config.getVersion('nf-core/bwa') } } diff --git a/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy new file mode 100644 index 0000000000..bda7b7f5ee --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy @@ -0,0 +1,46 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.module.spi.RemoteModuleResolverProvider +import spock.lang.Specification + +/** + * Test for DefaultRemoteModuleResolver SPI implementation + * + * @author Jorge Ejarque + */ +class DefaultRemoteModuleResolverTest extends Specification { + + def 'should load resolver via SPI'() { + when: + def resolver = RemoteModuleResolverProvider.getInstance() + + then: + resolver != null + resolver.class.name == 'nextflow.module.DefaultRemoteModuleResolver' + resolver.priority == 0 + } + + def 'should return default priority'() { + given: + def resolver = new DefaultRemoteModuleResolver() + + expect: + resolver.getPriority() == 0 + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy index 850466ab48..e526dde81f 100644 --- a/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy @@ -60,7 +60,7 @@ class InstalledModuleTest extends Specification { directory: moduleDir, mainFile: mainFile, manifestFile: metaFile, - checksumFile: moduleDir.resolve('.checksum'), + moduleInfoFile: moduleDir.resolve('.module-info'), expectedChecksum: actualChecksum, installedVersion: "0.0.1" ) @@ -95,7 +95,7 @@ class InstalledModuleTest extends Specification { directory: moduleDir, mainFile: mainFile, manifestFile: metaFile, - checksumFile: moduleDir.resolve('.checksum'), + moduleInfoFile: moduleDir.resolve('.module-info'), expectedChecksum: originalChecksum, installedVersion: "0.0.1" ) @@ -118,15 +118,15 @@ class InstalledModuleTest extends Specification { def metaFile = moduleDir.resolve('meta.yml') metaFile.text = 'name: test/module\nversion: 0.0.1' - def checksumFile = moduleDir.resolve('.checksum') - checksumFile.text = 'some-checksum' + def moduleInfoFile = moduleDir.resolve('.module-info') + // CORRUPTED check happens before .module-info check, so content doesn't matter here def installed = new InstalledModule( reference: new ModuleReference('test', 'module'), directory: moduleDir, mainFile: mainFile, manifestFile: metaFile, - checksumFile: checksumFile, + moduleInfoFile: moduleInfoFile, expectedChecksum: 'some-checksum' ) @@ -137,7 +137,7 @@ class InstalledModuleTest extends Specification { integrity == ModuleIntegrity.CORRUPTED } - def 'should report MISSING_CHECKSUM when checksum file absent'() { + def 'should report NO_REMOTE_MODULE when .module-info file absent'() { given: def moduleDir = tempDir.resolve('module') Files.createDirectories(moduleDir) @@ -148,15 +148,15 @@ class InstalledModuleTest extends Specification { def metaFile = moduleDir.resolve('meta.yml') metaFile.text = 'name: test/module\nversion: 0.0.1' - def checksumFile = moduleDir.resolve('.checksum') - // Don't create checksum file + def moduleInfoFile = moduleDir.resolve('.module-info') + // Don't create .module-info file def installed = new InstalledModule( reference: new ModuleReference('test', 'module'), directory: moduleDir, mainFile: mainFile, manifestFile: metaFile, - checksumFile: checksumFile, + moduleInfoFile: moduleInfoFile, expectedChecksum: null ) @@ -164,7 +164,7 @@ class InstalledModuleTest extends Specification { def integrity = installed.getIntegrity() then: - integrity == ModuleIntegrity.MISSING_CHECKSUM + integrity == ModuleIntegrity.NO_REMOTE_MODULE } def 'should handle checksum computation failure gracefully'() { @@ -178,16 +178,16 @@ class InstalledModuleTest extends Specification { def metaFile = moduleDir.resolve('meta.yml') metaFile.text = 'name: test/module\nversion: 0.0.1' - // Create checksum file with a value that won't match the computed checksum - def checksumFile = moduleDir.resolve('.checksum') - checksumFile.text = 'expected-checksum-that-will-not-match' + // Create .module-info with a checksum that won't match the computed checksum + ModuleChecksum.save(moduleDir, 'expected-checksum-that-will-not-match') + def moduleInfoFile = moduleDir.resolve('.module-info') def installed = new InstalledModule( reference: new ModuleReference('test', 'module'), directory: moduleDir, mainFile: mainFile, manifestFile: metaFile, - checksumFile: checksumFile, + moduleInfoFile: moduleInfoFile, expectedChecksum: 'expected-checksum-that-will-not-match' ) diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy index 31c6600373..4bb9185681 100644 --- a/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy @@ -97,7 +97,7 @@ class ModuleChecksumTest extends Specification { checksum1 != checksum2 } - def 'should exclude .checksum file from computation'() { + def 'should exclude .module-info file from computation'() { given: def moduleDir = tempDir.resolve('module') Files.createDirectories(moduleDir) @@ -107,14 +107,14 @@ class ModuleChecksumTest extends Specification { // Compute initial checksum def checksum1 = ModuleChecksum.compute(moduleDir) - // Add .checksum file - moduleDir.resolve('.checksum').text = 'some-checksum-value' + // Add .module-info file + ModuleChecksum.save(moduleDir, 'some-checksum-value') // Compute checksum again def checksum2 = ModuleChecksum.compute(moduleDir) expect: - checksum1 == checksum2 // Should be the same, .checksum is ignored + checksum1 == checksum2 // Should be the same, .module-info is ignored } def 'should include subdirectories in checksum'() { @@ -139,7 +139,7 @@ class ModuleChecksumTest extends Specification { checksum1 != checksum2 // Checksums should differ } - def 'should save checksum to .checksum file'() { + def 'should save checksum to .module-info file'() { given: def moduleDir = tempDir.resolve('module') Files.createDirectories(moduleDir) @@ -149,17 +149,16 @@ class ModuleChecksumTest extends Specification { ModuleChecksum.save(moduleDir, checksumValue) then: - def checksumFile = moduleDir.resolve('.checksum') - Files.exists(checksumFile) - checksumFile.text.trim() == checksumValue + def moduleInfoFile = moduleDir.resolve('.module-info') + Files.exists(moduleInfoFile) + ModuleChecksum.load(moduleDir) == checksumValue } - def 'should load checksum from .checksum file'() { + def 'should load checksum from .module-info file'() { given: def moduleDir = tempDir.resolve('module') Files.createDirectories(moduleDir) - def checksumFile = moduleDir.resolve('.checksum') - checksumFile.text = 'abc123def456' + ModuleChecksum.save(moduleDir, 'abc123def456') when: def checksum = ModuleChecksum.load(moduleDir) diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy index 64b0613be8..b906b83c9f 100644 --- a/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy @@ -26,34 +26,32 @@ import spock.lang.Specification */ class ModuleReferenceTest extends Specification { - def 'should parse valid module reference with @'() { + def 'should parse valid module reference without @'() { when: - def ref = ModuleReference.parse('@nf-core/fastqc') + def ref = ModuleReference.parse('nf-core/fastqc') then: ref.scope == 'nf-core' ref.name == 'fastqc' - ref.fullName == '@nf-core/fastqc' + ref.fullName == 'nf-core/fastqc' } - def 'should parse valid module reference without @'() { + def 'should reject module reference with @ prefix'() { when: - def ref = ModuleReference.parse('nf-core/fastqc') + ModuleReference.parse('@nf-core/fastqc') then: - ref.scope == 'nf-core' - ref.name == 'fastqc' - ref.fullName == '@nf-core/fastqc' + thrown(AbortOperationException) } def 'should parse module reference with multiple slashes'() { when: - def ref = ModuleReference.parse('@myorg/samtools/view') + def ref = ModuleReference.parse('myorg/samtools/view') then: ref.scope == 'myorg' ref.name == 'samtools/view' - ref.fullName == '@myorg/samtools/view' + ref.fullName == 'myorg/samtools/view' } def 'should reject invalid module reference without scope'() { @@ -80,7 +78,7 @@ class ModuleReferenceTest extends Specification { thrown(AbortOperationException) } - def 'should reject module reference with only @'() { + def 'should reject bare @ character'() { when: ModuleReference.parse('@') @@ -90,15 +88,15 @@ class ModuleReferenceTest extends Specification { def 'should reject module reference with only scope'() { when: - ModuleReference.parse('@nf-core/') + ModuleReference.parse('nf-core/') then: thrown(AbortOperationException) } - def 'should handle module reference with trailing slash'() { + def 'should reject module reference with trailing slash'() { when: - ModuleReference.parse('@nf-core/fastqc/') + ModuleReference.parse('nf-core/fastqc/') then: thrown(AbortOperationException) @@ -111,12 +109,12 @@ class ModuleReferenceTest extends Specification { then: ref.scope == 'nf-core' ref.name == 'fastqc' - ref.fullName == '@nf-core/fastqc' + ref.fullName == 'nf-core/fastqc' } def 'should handle scope names with hyphens'() { when: - def ref = ModuleReference.parse('@my-org/my-module') + def ref = ModuleReference.parse('my-org/my-module') then: ref.scope == 'my-org' @@ -125,7 +123,7 @@ class ModuleReferenceTest extends Specification { def 'should handle scope names with underscores'() { when: - def ref = ModuleReference.parse('@my_org/my_module') + def ref = ModuleReference.parse('my_org/my_module') then: ref.scope == 'my_org' @@ -134,7 +132,7 @@ class ModuleReferenceTest extends Specification { def 'should handle module names with numbers'() { when: - def ref = ModuleReference.parse('@nf-core/bwa-mem2') + def ref = ModuleReference.parse('nf-core/bwa-mem2') then: ref.scope == 'nf-core' @@ -143,9 +141,9 @@ class ModuleReferenceTest extends Specification { def 'should implement equals correctly'() { given: - def ref1 = ModuleReference.parse('@nf-core/fastqc') - def ref2 = ModuleReference.parse('@nf-core/fastqc') - def ref3 = ModuleReference.parse('@nf-core/multiqc') + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') + def ref3 = ModuleReference.parse('nf-core/multiqc') expect: ref1 == ref2 @@ -154,8 +152,8 @@ class ModuleReferenceTest extends Specification { def 'should implement hashCode correctly'() { given: - def ref1 = ModuleReference.parse('@nf-core/fastqc') - def ref2 = ModuleReference.parse('@nf-core/fastqc') + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') expect: ref1.hashCode() == ref2.hashCode() @@ -163,17 +161,17 @@ class ModuleReferenceTest extends Specification { def 'should implement toString correctly'() { given: - def ref = ModuleReference.parse('@nf-core/fastqc') + def ref = ModuleReference.parse('nf-core/fastqc') expect: - ref.toString() == '@nf-core/fastqc' + ref.toString() == 'nf-core/fastqc' } def 'should be usable as map key'() { given: - def ref1 = ModuleReference.parse('@nf-core/fastqc') - def ref2 = ModuleReference.parse('@nf-core/fastqc') - def ref3 = ModuleReference.parse('@nf-core/multiqc') + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') + def ref3 = ModuleReference.parse('nf-core/multiqc') def map = [:] map[ref1] = 'value1' @@ -187,7 +185,7 @@ class ModuleReferenceTest extends Specification { def 'should handle org-style scopes'() { when: - def ref = ModuleReference.parse('@mycompany.io/custom-module') + def ref = ModuleReference.parse('mycompany.io/custom-module') then: ref.scope == 'mycompany.io' @@ -196,7 +194,7 @@ class ModuleReferenceTest extends Specification { def 'should reject module reference with spaces'() { when: - ModuleReference.parse('@nf-core/fast qc') + ModuleReference.parse('nf-core/fast qc') then: thrown(AbortOperationException) @@ -204,7 +202,7 @@ class ModuleReferenceTest extends Specification { def 'should reject module reference with special characters'() { when: - ModuleReference.parse('@nf-core/fastqc!') + ModuleReference.parse('nf-core/fastqc!') then: thrown(AbortOperationException) @@ -212,17 +210,17 @@ class ModuleReferenceTest extends Specification { def 'should handle deeply nested module names'() { when: - def ref = ModuleReference.parse('@nf-core/samtools/sort/parallel') + def ref = ModuleReference.parse('nf-core/samtools/sort/parallel') then: ref.scope == 'nf-core' ref.name == 'samtools/sort/parallel' - ref.fullName == '@nf-core/samtools/sort/parallel' + ref.fullName == 'nf-core/samtools/sort/parallel' } def 'should parse from string with leading/trailing whitespace'() { when: - def ref = ModuleReference.parse(' @nf-core/fastqc ') + def ref = ModuleReference.parse(' nf-core/fastqc ') then: ref.scope == 'nf-core' diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy index 8c145bb89c..ab959dfb57 100644 --- a/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy @@ -88,7 +88,7 @@ class ModuleResolverTest extends Specification { name: nf-core/fastqc version: 1.0.0 ''' - moduleDir.resolve('.checksum').text = 'wrong-checksum' + ModuleChecksum.save(moduleDir, 'wrong-checksum') when: def result = resolver.resolve(reference, null, false) @@ -103,7 +103,7 @@ class ModuleResolverTest extends Specification { def 'should throw exception when version mismatch without auto-install'() { given: - def modulesConfig = new ModulesConfig(['@nf-core/fastqc': '2.0.0']) + def modulesConfig = new ModulesConfig(['nf-core/fastqc': '2.0.0']) def resolver = new ModuleResolver(tempDir, modulesConfig, null) def reference = new ModuleReference('nf-core', 'fastqc') def storage = new ModuleStorage(tempDir) @@ -119,7 +119,7 @@ class ModuleResolverTest extends Specification { // Compute and save correct checksum def checksum = ModuleChecksum.compute(moduleDir) - moduleDir.resolve('.checksum').text = checksum + ModuleChecksum.save(moduleDir, checksum) when: resolver.resolve(reference, null, false) @@ -152,7 +152,7 @@ class ModuleResolverTest extends Specification { // Compute and save correct checksum def checksum = ModuleChecksum.compute(moduleDir) - moduleDir.resolve('.checksum').text = checksum + ModuleChecksum.save(moduleDir, checksum) when: def result = resolver.resolve(reference, '1.0.0', false) @@ -178,7 +178,7 @@ class ModuleResolverTest extends Specification { name: nf-core/fastqc version: 1.0.0 ''' - moduleDir.resolve('.checksum').text = 'wrong-checksum' + ModuleChecksum.save(moduleDir, 'wrong-checksum') when: resolver.installModule(reference, '2.0.0', false) diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy index dab9517ddc..258c4c354b 100644 --- a/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy @@ -50,7 +50,7 @@ class ModuleStorageTest extends Specification { def moduleDir = storage.getModuleDir(reference) then: - moduleDir == tempDir.resolve('modules/@nf-core/fastqc') + moduleDir == tempDir.resolve('modules/nf-core/fastqc') } def 'should check if module is installed'() { @@ -107,9 +107,9 @@ class ModuleStorageTest extends Specification { - fastqc '''.stripIndent() - // Create .checksum file - def checksumFile = moduleDir.resolve('.checksum') - checksumFile.text = 'abc123def456' + // Create .module-info file + ModuleChecksum.save(moduleDir, 'abc123def456') + def moduleInfoFile = moduleDir.resolve('.module-info') when: def installed = storage.getInstalledModule(reference) @@ -120,7 +120,7 @@ class ModuleStorageTest extends Specification { installed.directory == moduleDir installed.mainFile == mainFile installed.manifestFile == moduleDir.resolve('meta.yml') - installed.checksumFile == checksumFile + installed.moduleInfoFile == moduleInfoFile installed.expectedChecksum == 'abc123def456' installed.installedVersion == '1.0.0' } @@ -145,12 +145,12 @@ class ModuleStorageTest extends Specification { // Create meta.yml with version moduleDir.resolve('meta.yml').text = """ - name: ${ref.nameWithoutPrefix} + name: ${ref} version: 1.0.0 """.stripIndent() - // Create .checksum - moduleDir.resolve('.checksum').text = 'checksum' + // Create .module-info + ModuleChecksum.save(moduleDir, 'checksum') } when: @@ -158,7 +158,7 @@ class ModuleStorageTest extends Specification { then: installed.size() == 3 - installed*.reference.fullName.sort() == ['@myorg/custom', '@nf-core/fastqc', '@nf-core/multiqc'] + installed*.reference.fullName.sort() == ['myorg/custom', 'nf-core/fastqc', 'nf-core/multiqc'] } def 'should list nested modules recursively'() { @@ -182,14 +182,14 @@ class ModuleStorageTest extends Specification { // Create meta.yml with version moduleDir.resolve('meta.yml').text = """ - name: ${ref.nameWithoutPrefix} + name: ${ref} version: 1.0.0 description: Test module license: MIT """.stripIndent() - // Create .checksum - moduleDir.resolve('.checksum').text = 'checksum' + // Create .module-info + ModuleChecksum.save(moduleDir, 'checksum') } when: @@ -198,10 +198,10 @@ class ModuleStorageTest extends Specification { then: installed.size() == 4 installed*.reference.fullName.sort() == [ - '@myorg/tools/subtools/module', - '@nf-core/fastqc', - '@nf-core/gfatools/gfa2fa', - '@nf-core/gfatools/gfa2gfa' + 'myorg/tools/subtools/module', + 'nf-core/fastqc', + 'nf-core/gfatools/gfa2fa', + 'nf-core/gfatools/gfa2gfa' ] } @@ -234,7 +234,7 @@ class ModuleStorageTest extends Specification { installed.reference == reference installed.installedVersion == '1.0.0' Files.exists(installed.mainFile) - Files.exists(installed.checksumFile) + Files.exists(installed.moduleInfoFile) cleanup: packageFile?.delete() @@ -313,7 +313,7 @@ class ModuleStorageTest extends Specification { then: installed.expectedChecksum != null installed.expectedChecksum.length() > 0 - Files.exists(installed.checksumFile) + Files.exists(installed.moduleInfoFile) cleanup: packageFile?.delete() diff --git a/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy index b5e1b7425e..527cbbfbbd 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy @@ -45,8 +45,8 @@ class IncludeDefTest extends Specification { expect: include.resolveModulePath('/abs/foo.nf') == '/abs/foo.nf' as Path - include.resolveModulePath('module.nf') == '/some/path/module.nf' as Path - include.resolveModulePath('foo/bar.nf') == '/some/path/foo/bar.nf' as Path + include.resolveModulePath('./module.nf') == '/some/path/module.nf' as Path + include.resolveModulePath('./foo/bar.nf') == '/some/path/foo/bar.nf' as Path when: include.resolveModulePath('http://foo.com/bar') @@ -66,17 +66,17 @@ class IncludeDefTest extends Specification { include.getOwnerPath() >> script when: - def result = include.realModulePath( 'mod-x.nf') + def result = include.realModulePath( './mod-x.nf') then: result == module when: - result = include.realModulePath('mod-x') + result = include.realModulePath('./mod-x') then: result == module when: - include.realModulePath('xyz') + include.realModulePath('./xyz') then: thrown(NoSuchFileException) @@ -98,21 +98,21 @@ class IncludeDefTest extends Specification { // when the module name reference a directory that contains // a file named 'main.nf', it's considered a module 'bundle' when: - def result = include.realModulePath('foo') + def result = include.realModulePath('./foo') then: result == module when: - include.realModulePath('bar') + include.realModulePath('./bar') then: thrown(NoSuchFileException) when: folder.resolve('bar').mkdir() - include.realModulePath('bar') + include.realModulePath('./bar') then: def e = thrown(ScriptCompilationException) - e.message == "Include 'bar' does not provide any module script -- the following path should contain a 'main.nf' script: '${folder.resolve('bar')}'" + e.message == "Include './bar' does not provide any module script -- the following path should contain a 'main.nf' script: '${folder.resolve('bar')}'" } def 'should check valid path' () { @@ -137,6 +137,16 @@ class IncludeDefTest extends Specification { when: include.checkValidPath('this/dir') then: + noExceptionThrown() // valid remote module reference (scope/name) + + when: + include.checkValidPath('nf-core/fastqc') + then: + noExceptionThrown() // valid remote module reference + + when: + include.checkValidPath('invalid!') + then: thrown(IllegalModulePath) when: diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java new file mode 100644 index 0000000000..b438dbf7af --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java @@ -0,0 +1,45 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Fallback implementation of RemoteModuleResolver that is used when no other + * implementation is found via the SPI mechanism. + * + *

This implementation throws an exception with a helpful error message + * indicating that remote module resolution is not available. + * + * @author Jorge Ejarque + */ +public class FallbackRemoteModuleResolver implements RemoteModuleResolver { + + @Override + public Path resolve(String moduleName, Path baseDir) { + if (!Files.exists(baseDir.resolve(moduleName))) { + throw new IllegalStateException("Module '" + moduleName + "' not locally found at 'modules' folder - use 'nextflow install' to download module files"); + } + return baseDir.resolve(moduleName).resolve("main.nf"); + } + + @Override + public int getPriority() { + return Integer.MIN_VALUE; // Fallback has lowest possible priority + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java new file mode 100644 index 0000000000..5c6cf1fc9b --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java @@ -0,0 +1,68 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import java.nio.file.Path; + +/** + * Service Provider Interface for resolving remote modules referenced with '@scope/name' syntax. + * + *

Implementations should handle: + *

    + *
  • Checking if a module is already installed locally
  • + *
  • Downloading modules from a registry if not present
  • + *
  • Version resolution and validation
  • + *
+ * + *

The interface follows the Java SPI pattern. Implementations should be registered + * in META-INF/services/nextflow.module.spi.RemoteModuleResolver + * + * @author Jorge Ejarque + */ +public interface RemoteModuleResolver { + + /** + * Resolve a remote module reference (e.g., '@scope/name') to a local path. + * + *

This method should: + *

    + *
  1. Parse the module reference
  2. + *
  3. Check if the module is already installed locally
  4. + *
  5. Download and install the module if not present (auto-install)
  6. + *
  7. Validate version constraints if specified
  8. + *
+ * + * @param moduleName The module reference string (e.g., '@scope/name' or '@scope/name@version') + * @param baseDir The base directory for the project (used to locate the modules directory) + * @return Path to the resolved module's main.nf file + * @throws IllegalArgumentException if the module reference is invalid or resolution fails + */ + Path resolve(String moduleName, Path baseDir); + + /** + * Get the priority of this resolver. Higher priority resolvers are tried first. + * + *

Use this to allow custom implementations to override the default resolver. + * The default implementation should return 0. Custom implementations can return + * positive values to take precedence. + * + * @return Priority value (higher = tried first), default should be 0 + */ + default int getPriority() { + return 0; + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java new file mode 100644 index 0000000000..da1b2da9e7 --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java @@ -0,0 +1,92 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.ServiceLoader; + +/** + * Provider for accessing RemoteModuleResolver implementations via SPI. + * + *

This class uses the Java ServiceLoader mechanism to discover and load + * implementations of RemoteModuleResolver. It selects the implementation + * with the highest priority. + * + * @author Jorge Ejarque + */ +public class RemoteModuleResolverProvider { + + private static final Logger log = LoggerFactory.getLogger(RemoteModuleResolverProvider.class); + private static RemoteModuleResolver instance; + + /** + * Get the RemoteModuleResolver instance with the highest priority. + * + *

This method lazily loads and caches the resolver. It discovers all + * implementations via ServiceLoader and selects the one with the highest + * priority value. + * + *

If no implementations are found, returns the FallbackRemoteModuleResolver + * which throws an informative exception. + * + * @return The RemoteModuleResolver instance with highest priority + */ + public static synchronized RemoteModuleResolver getInstance() { + if (instance == null) { + instance = loadResolver(); + } + return instance; + } + + private static RemoteModuleResolver loadResolver() { + List resolvers = new ArrayList<>(); + ServiceLoader loader = ServiceLoader.load(RemoteModuleResolver.class); + + // Collect all available resolvers + for (RemoteModuleResolver resolver : loader) { + resolvers.add(resolver); + log.debug("Discovered RemoteModuleResolver: {} with priority {}", + resolver.getClass().getName(), resolver.getPriority()); + } + + // Sort by priority (highest first) + resolvers.sort(Comparator.comparingInt(RemoteModuleResolver::getPriority).reversed()); + + if (resolvers.isEmpty()) { + log.warn("No RemoteModuleResolver implementations found via SPI, using fallback"); + return new FallbackRemoteModuleResolver(); + } + + RemoteModuleResolver selected = resolvers.get(0); + log.debug("Selected RemoteModuleResolver: {} with priority {}", + selected.getClass().getName(), selected.getPriority()); + + return selected; + } + + /** + * Reset the cached instance. Used primarily for testing. + */ + public static synchronized void reset() { + instance = null; + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java index 5b76af4111..c52931142a 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java @@ -23,6 +23,7 @@ import java.util.Set; import java.util.function.Function; +import nextflow.module.spi.RemoteModuleResolverProvider; import nextflow.script.ast.IncludeNode; import nextflow.script.ast.ScriptNode; import org.codehaus.groovy.control.SourceUnit; @@ -72,8 +73,18 @@ private SourceUnit resolveInclude(IncludeNode node, SourceUnit sourceUnit, Funct var source = node.source.getText(); if( source.startsWith("plugin/") ) return null; - var uri = sourceUnit.getSource().getURI(); - var includeUri = getIncludeUri(uri, source); + + var parent = Path.of(sourceUnit.getSource().getURI()).getParent(); + + // Resolve remote module paths (scope/name format, not starting with local prefixes) + if( isRemoteModule(source) ) { + var modules = Path.of("./modules"); + var resolver = RemoteModuleResolverProvider.getInstance(); + resolver.resolve(source, modules.getParent()); + parent = modules; + } + + var includeUri = getIncludeUri(parent, source); if( compiler.getSource(includeUri) != null ) return null; if( !Files.exists(Path.of(includeUri)) ) @@ -86,8 +97,15 @@ private SourceUnit resolveInclude(IncludeNode node, SourceUnit sourceUnit, Funct return includeSource; } - private static URI getIncludeUri(URI uri, String source) { - Path includePath = Path.of(uri).getParent().resolve(source); + static boolean isRemoteModule(String source) { + if( source.startsWith("/") || source.startsWith("./") || source.startsWith("../") ) + return false; + // Must match scope/name pattern: scope is lowercase alphanumeric with dots/underscores/hyphens + return source.matches("^[a-z0-9][a-z0-9._\\-]*/[a-z][a-z0-9._\\-]*(/[a-z][a-z0-9._\\-]*)*$"); + } + + private static URI getIncludeUri(Path parent, String source) { + Path includePath = parent.resolve(source); if( Files.isDirectory(includePath) ) includePath = includePath.resolve("main.nf"); else if( !source.endsWith(".nf") ) diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java index 7dc9929bbe..b8bf711d93 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java @@ -83,7 +83,10 @@ public void visitInclude(IncludeNode node) { setPlaceholderTargets(node); return; } - var includeUri = getIncludeUri(uri, source); + + var isRemoteModule = ModuleResolver.isRemoteModule(source); + var parent = isRemoteModule ? Path.of("modules") : Path.of(uri).getParent(); + var includeUri = getIncludeUri(parent, source); if( !isIncludeStale(node, includeUri) ) return; changed = true; @@ -121,8 +124,8 @@ private static void setPlaceholderTargets(IncludeNode node) { } } - private static URI getIncludeUri(URI uri, String source) { - Path includePath = Path.of(uri).getParent().resolve(source); + private static URI getIncludeUri(Path parent, String source) { + Path includePath = parent.resolve(source); if( Files.isDirectory(includePath) ) includePath = includePath.resolve("main.nf"); else if( !source.endsWith(".nf") ) diff --git a/settings.gradle b/settings.gradle index fe58cb245b..22284225f8 100644 --- a/settings.gradle +++ b/settings.gradle @@ -50,3 +50,4 @@ include 'plugins:nf-k8s' include 'plugins:nf-seqera' //includeBuild '../sched' +//includeBuild('../plugin-registry') diff --git a/specs/251117-module-system/data-model.md b/specs/251117-module-system/data-model.md index 448509e52b..0d3ef4cf8a 100644 --- a/specs/251117-module-system/data-model.md +++ b/specs/251117-module-system/data-model.md @@ -164,7 +164,29 @@ registry { --- -### 5. PipelineSpec +### 5. DefaultRemoteModuleResolver (SPI) + +Bridges the DSL parser to the module resolution runtime. Class: `nextflow.module.DefaultRemoteModuleResolver`. + +```groovy +// Implements: nextflow.module.spi.RemoteModuleResolver (nf-lang) +class DefaultRemoteModuleResolver implements RemoteModuleResolver { + int getPriority() { return 0 } // Can be overridden by plugins with higher priority + + Path resolve(String moduleName, Path baseDir) { + // 1. Parse ModuleReference from "@scope/name" + // 2. Read version constraints from nextflow_spec.json / ModulesConfig + // 3. Call ModuleResolver.installModule(reference, version, autoInstall=true) + // 4. Return path to modules/@scope/name/main.nf + } +} +``` + +The SPI is loaded via Java `ServiceLoader` by `RemoteModuleResolverProvider` (in `nf-lang`), which selects the highest-priority implementation available. + +--- + +### 6. PipelineSpec Reads and writes `nextflow_spec.json` in the project root. Class: `nextflow.pipeline.PipelineSpec`. diff --git a/specs/251117-module-system/plan.md b/specs/251117-module-system/plan.md index 03ecb5cfff..e9d11d552f 100644 --- a/specs/251117-module-system/plan.md +++ b/specs/251117-module-system/plan.md @@ -77,17 +77,24 @@ modules/nextflow/src/main/groovy/nextflow/ │ └── RegistryConfig.groovy # registry{} config scope (fields: url, apiKey) ├── module/ │ ├── ModuleReference.groovy # @scope/name parser -│ ├── ModuleResolver.groovy # Core resolution logic +│ ├── ModuleResolver.groovy # Core resolution logic (version/integrity/install) │ ├── ModuleStorage.groovy # Local filesystem operations │ ├── ModuleRegistryClient.groovy # HTTP registry client │ ├── ModuleChecksum.groovy # SHA-256 integrity verification │ ├── ModuleSpec.groovy # Module manifest (meta.yaml) entity -│ └── InstalledModule.groovy # Installed module entity +│ ├── InstalledModule.groovy # Installed module entity +│ └── DefaultRemoteModuleResolver.groovy # SPI impl: bridges DSL parser → ModuleResolver └── pipeline/ └── PipelineSpec.groovy # nextflow_spec.json read/write modules/nf-lang/src/main/java/nextflow/script/ -└── ResolveIncludeVisitor.java # MODIFY: Add @scope/name detection +└── control/ResolveIncludeVisitor.java # MODIFIED: Delegates @scope/name to SPI resolver + +modules/nf-lang/src/main/java/nextflow/module/spi/ +├── RemoteModuleResolver.java # SPI interface (extensible by plugins) +├── RemoteModuleResolverProvider.java # ServiceLoader wrapper (singleton) +└── FallbackRemoteModuleResolver.java # Error fallback when no impl found + modules/nextflow/src/test/groovy/nextflow/ ├── cli/module/ @@ -105,7 +112,25 @@ tests/modules/ └── [other integration tests] ``` -**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang`. No new plugins required. +**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang` via SPI. No new plugins required. + +## Architecture Notes + +### Remote Module Inclusion — SPI Pattern + +The DSL parser (`ResolveIncludeVisitor`) detects the `@` prefix in `include` statements and delegates resolution to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader`. This keeps `nf-lang` decoupled from the runtime module resolution logic: + +``` +include { X } from '@nf-core/fastqc' + ↓ +ResolveIncludeVisitor (nf-lang) + source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(...) + ↓ +DefaultRemoteModuleResolver (nextflow module) + auto-installs via ModuleResolver if missing → returns Path to main.nf +``` + +The `RemoteModuleResolver` interface in `nf-lang` can be overridden by plugins with a higher priority value. ## Complexity Tracking diff --git a/specs/251117-module-system/research.md b/specs/251117-module-system/research.md index 40cb783e2b..d08005fc89 100644 --- a/specs/251117-module-system/research.md +++ b/specs/251117-module-system/research.md @@ -58,33 +58,40 @@ class CmdModule extends CmdBase implements UsageAware { **Research Question**: How to extend `include` statement parsing for registry modules? -**Decision**: Extend ResolveIncludeVisitor to detect `@` prefix and delegate to ModuleResolver +**Decision**: Extend `ResolveIncludeVisitor` to detect `@` prefix and delegate to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader` **Rationale**: -- IncludeNode already captures source path as string -- Detection: `source.startsWith('@')` distinguishes registry vs local paths -- Resolution happens at parse time (after plugin resolution) per ADR -- Preserves existing local file include behavior +- Keeps `nf-lang` decoupled from runtime module resolution (`nf-lang` has no dependency on `nextflow` module) +- SPI pattern allows plugins or custom implementations to override the default resolver +- Detection: `source.startsWith('@')` distinguishes registry vs local paths — preserves existing include behavior +- Resolution at parse time (after plugin resolution) per ADR -**Reference Implementation**: +**Implemented Architecture**: ``` -Location: modules/nf-lang/src/main/java/nextflow/script/ResolveIncludeVisitor.java -Extension Point: visitInclude() method -Pattern: - 1. Check if source starts with '@' - 2. If yes: call ModuleResolver.resolve(source, configuredVersion) - 3. ModuleResolver returns absolute path to modules/@scope/name/main.nf - 4. Continue with standard include processing +include { X } from '@scope/name' + ↓ +ResolveIncludeVisitor.visitInclude() [nf-lang] + source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(source, baseDir) + ↓ +RemoteModuleResolverProvider [nf-lang] + Java ServiceLoader discovers implementations; picks highest priority + ↓ +DefaultRemoteModuleResolver [nextflow module] + Calls ModuleResolver.installModule(reference, version, autoInstall=true) + Returns Path to modules/@scope/name/main.nf ``` **Key Files**: -- `IncludeNode.java` - AST representation -- `IncludeEntryNode.java` - Individual entries -- `ResolveIncludeVisitor.java` - Visitor for resolution +- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java` — SPI interface +- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java` — ServiceLoader singleton +- `modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java` — error fallback +- `modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java` — MODIFIED +- `modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy` — default impl **Alternatives Considered**: -- New ANTLR grammar token for `@`: Rejected - unnecessary parser complexity -- Dot file marker for local modules: Deferred to Open Questions in ADR +- New ANTLR grammar token for `@`: Rejected — unnecessary parser complexity +- Direct dependency from nf-lang to nextflow module: Rejected — circular dependency risk; SPI decouples cleanly +- Dot file marker for local modules: Deferred in ADR; current impl uses `@` for registry, `.`/`/` for local --- @@ -326,7 +333,7 @@ class ModuleChecksum { | Area | Decision | Key Reference | |------|----------|---------------| | CLI | JCommander subcommands; each extends CmdBase (ModuleRun extends CmdRun) | CmdModule.groovy | -| DSL Parser | Extend ResolveIncludeVisitor for `@scope/name` — pending | ResolveIncludeVisitor.java | +| DSL Parser | SPI pattern — ResolveIncludeVisitor delegates to RemoteModuleResolver; DefaultRemoteModuleResolver bridges to ModuleResolver | ResolveIncludeVisitor.java, RemoteModuleResolver.java | | Config | ModulesConfig + RegistryConfig (ConfigScope) | FusionConfig.groovy, ConfigScope.java | | Registry HTTP | ModuleRegistryClient using HxClient + npr-api models | HttpPluginRepository.groovy | | Authentication | `NXF_REGISTRY_TOKEN` env var or `registry.apiKey` config field (primary registry only) | RegistryConfig.groovy | @@ -342,4 +349,4 @@ class ModuleChecksum { 1. **Local vs managed module distinction**: Resolved — `@` prefix for registry modules only; local paths start with `.` or `/` 2. **Tool arguments**: Removed from ADR — not in scope 3. **Module version location**: Resolved — `nextflow_spec.json` (auto-managed by `module install`); `modules {}` block in `nextflow.config` supported as alternative -4. **DSL parser `@scope/name` include**: Pending (T017) \ No newline at end of file +4. **DSL parser `@scope/name` include**: ✅ Resolved — SPI pattern implemented (T017a-d) \ No newline at end of file