Skip to content

Fix remote modules when base dir is not the current working dir#6932

Merged
bentsherman merged 6 commits intomasterfrom
fix-module-system-basedir
Mar 17, 2026
Merged

Fix remote modules when base dir is not the current working dir#6932
bentsherman merged 6 commits intomasterfrom
fix-module-system-basedir

Conversation

@jorgee
Copy link
Copy Markdown
Contributor

@jorgee jorgee commented Mar 17, 2026

This pull request refactors how remote module resolution works at parsing and compile time. The RemoteModuleResolver interface is updated to remove the need to explicitly pass a base directory.
At pipeline execution modules are resolved according to the basedir. Otherwise, it uses "./modules" as resolution base dir. This fixes the problems when invoking remote pipelines with modules or executing a pipeline with a different path.

A pipeline including remote modules is created to test this behaviour.

In the main branch, it uses remote modules without pinned versions. You can run with the following command

./launch.sh run jorgee/nf-modules-test

The first time you run this pipeline, it will install the modules in the version assets folder (execution basedir)

It also contains the pinned-versions branch where the remote module is pinned (commited files). So, modules shouldn't be installed at execution in this case

./launch.sh run jorgee/nf-modules-test -r pinned-versions

Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee jorgee requested a review from a team as a code owner March 17, 2026 12:07
@netlify
Copy link
Copy Markdown

netlify Bot commented Mar 17, 2026

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit b46e3ff
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69b9c6ec861e840008f401af

@jorgee jorgee changed the title Fix execution with remote modules when basedir is different current working dir Fix execution with remote modules when basedir is not the current working dir Mar 17, 2026
@jorgee jorgee requested a review from pditommaso March 17, 2026 12:11
Copy link
Copy Markdown
Member

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Good change — removing baseDir from the SPI is the right call since the old nf-lang caller was passing a wrong value (Path.of("./modules").getParent() resolves to .). The callers in ModuleResolver.java and ResolveIncludeVisitor.java are nicely simplified.

1. FallbackRemoteModuleResolver — relative path

Path.of("modules") is relative to CWD, which is fragile. This is acceptable for a fallback, but worth a comment explaining the intent. Also, consider using Path.of("modules").toAbsolutePath() for better diagnostics if it fails.

2. DefaultRemoteModuleResolver — null session fallback

def baseDir = Global.session?.baseDir ?: Path.of('.')

When Global.session is null (e.g. nextflow module install before session creation, or testing), this falls back to a relative Path.of('.'). Consider Path.of('.').toAbsolutePath() to at least make the resolved path absolute for better error messages.

3. ResolveIncludeVisitor — unhandled exception from resolver

ResolveIncludeVisitor now calls RemoteModuleResolverProvider.getInstance().resolve(source) which can throw IllegalStateException from the fallback resolver when the module isn't locally installed. This exception doesn't appear to be caught — it could crash the language server / IDE integration. Consider catching it and reporting as a diagnostic error instead.

4. Naming simplification

Consider renaming for clarity:

  • RemoteModuleResolverProviderRemoteModuleResolver (it's the provider/factory, the current name is redundant)
  • FallbackRemoteModuleResolverDefaultModuleResolver (simpler, clearer role)
  • ModuleResolver (in nf-lang) → keep as-is or clarify its relationship to the SPI

5. Log message verbosity

log.info "Module ${reference}@${version} installed successfully at ${installed.mainFile.parent.toAbsolutePath().toString()}"

The .toString() is redundant in GString (Groovy calls it automatically). Also, the chain installed.mainFile.parent.toAbsolutePath().toString() is quite long inline. Consider a small helper or local variable:

def installDir = installed.mainFile.parent.toAbsolutePath()
log.info "Module ${reference}@${version} installed successfully at ${installDir}"

Or a utility like Path.toUriString() if this pattern repeats elsewhere.

Comment thread modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy Outdated
@bentsherman
Copy link
Copy Markdown
Member

Thanks Jorge. Let me look at this with the linter and language server. We might need to be able to pass in the projectDir (baseDir) in those cases

jorgee and others added 2 commits March 17, 2026 15:02
Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
Copy link
Copy Markdown
Contributor Author

jorgee commented Mar 17, 2026

Addressed review comment. I skipped the suggested renaming as it collides with other existing classes

Signed-off-by: jorgee <jorge.ejarque@seqera.io>
pditommaso and others added 2 commits March 17, 2026 20:46
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman bentsherman changed the title Fix execution with remote modules when basedir is not the current working dir Fix remote modules when base dir is not the current working dir Mar 17, 2026
@bentsherman bentsherman merged commit ca1e625 into master Mar 17, 2026
25 checks passed
@bentsherman bentsherman deleted the fix-module-system-basedir branch March 17, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants