diff --git a/adr/20251114-module-system.md b/adr/20251114-module-system.md new file mode 100644 index 0000000000..f819b81068 --- /dev/null +++ b/adr/20251114-module-system.md @@ -0,0 +1,897 @@ +# Module System for Nextflow + +- Authors: Paolo Di Tommaso +- Status: approved +- Date: 2025-01-06 +- Tags: modules, dsl, registry, versioning, architecture +- Version: 2.7 + +## Updates + +### Version 2.7 (2026-03-09) +- **Renamed `.checksum` to `.module-info`**: Leaves room for additional properties in the future +- **Removed `@` prefix from module scopes**: Local modules are distinguished from remote modules by presence/absence of `./` prefix +- **Removed version pinning from config**: Installed module versions are now inferred from the `meta.yml` of each module in the `modules/` directory instead of being declared in `nextflow.config` + +### Version 2.6 (2026-01-28) +- **Removed module parameters**: Module parameters specification moved to separate spec document. + +### Version 2.5 (2026-01-23) +- **Module parameters**: Replaced structured tool arguments with general module parameters defined in `meta.yml` +- **Simplified tools section**: Removed `args` property from tools; tool arguments now configured via module parameters +- **Simplified `requires` block**: Removed `plugins`, `modules`, and `subworkflows` sub-properties; `requires` now only contains `nextflow` version constraint +- **Process modules focus**: Removed sub-workflow references; spec is now focused on process modules only + +### Version 2.4 (2026-01-15) +- **Removed transitive dependency resolution**: Module dependencies are explicit only; no automatic transitive resolution +- **Removed `freeze` command**: No longer needed without transitive dependency management +- **Simplified model**: Each module explicitly declares its dependencies in `nextflow.config` + +### Version 2.3 (2026-01-15) +- **Resolution Rules table**: Added clear table specifying behavior for each combination of local state and declared version +- **Local modification protection**: Locally modified modules (checksum mismatch) are NOT overridden unless `-force` flag is used +- **Simplified storage model**: Single version per module locally (`modules/@scope/name/` without version in path) +- **`.checksum` file**: Registry checksum cached locally for fast integrity verification without network calls + +### Version 2.2 (2025-01-06) — *Superseded by v2.5* +- **Structured tool arguments**: Added `args` property to `tools` section for type-safe argument configuration +- **New implicit variables**: `tools..args.` returns formatted flag+value; `tools..args` returns all args concatenated +- **Deprecation**: All `ext.*` custom directives (e.g., `ext.args`, `ext.args2`, `ext.args3`, `ext.prefix`, `ext.suffix`) deprecated in favor of structured tool arguments +- *Note: Tool arguments replaced by module parameters in v2.5* + +### Version 2.1 (2024-12-11) +- **Unified dependencies**: Consolidated `components`, `dependencies`, and `requires` into single `requires` field +- **Unified version syntax**: `[scope/]name[@constraint]` format across plugins and modules +- **Deprecation**: `components` field deprecated (use top-level `modules` instead) + +## Context and Problem Statement + +Nextflow supports local script inclusion via `include` directive but lacks standardized mechanisms for package management, versioning, and distribution of reusable process definitions. This limits code reuse and reproducibility across the ecosystem. + +Discussion/request goes back to at least 2019, see GitHub issues [#1376](https://github.com/nextflow-io/nextflow/issues/1376), [#1463](https://github.com/nextflow-io/nextflow/issues/1463) and [#4122](https://github.com/nextflow-io/nextflow/issues/4112). + +## Decision + +Implement a module system with four core capabilities: + +1. **Remote module inclusion** via registry +2. **Semantic versioning** with dependency resolution +3. **Unified Nextflow Registry** (rebrand existing Nextflow registry) +4. **First-class CLI support** (install, publish, search, list, remove, run) + +## Core Capabilities + +### 1. Remote Module Inclusion + +**DSL Syntax**: +```groovy +// Include from registry (scoped module name without `./` prefix) +include { BWA_ALIGN } from 'nf-core/bwa-align' + +// Existing file-based includes remain supported +include { MY_PROCESS } from './modules/my-process.nf' +``` + +**Module Naming**: Scoped modules `scope/name` (e.g., `nf-core/salmon`, `myorg/custom`). Local paths supported for backwards compatibility. No nested paths with the module are allowed - each module must have a `main.nf` as the entry point. + +**Version Resolution**: Installed module versions are inferred from the `meta.yml` of each module in the `modules/` directory. If a module is not present locally, the latest available version is downloaded from the registry. + +**Resolution Order**: +1. Check local `modules/scope/name/` exists +2. Verify integrity against `.module-info` file +3. Apply resolution rules (see below) + +**Resolution Rules**: + +| Local State | Action | +|-------------|--------| +| Missing | Download latest from registry | +| Exists, checksum valid | Use local module (version from `meta.yml`) | +| Exists, checksum mismatch | **Warn**: locally modified, will NOT replace unless `-force` is used | + +**Key Behaviors**: +- **Local modification**: When the local module content was manually changed (checksum mismatch with `.module-info`), Nextflow warns and does NOT override to prevent accidental loss of local changes +- **Force flag**: Use `-force` with `nextflow module install` to override locally modified modules + +**Resolution Timing**: Modules resolved at workflow parse time (after plugin resolution at startup). + +**Local Storage**: Downloaded modules stored in `modules/scope/name/` directory in project root (not global cache). Each module must contain a `main.nf` file as the required entry point. It is intended that module source code will be committed to the pipeline git repository. + +### 2. Semantic Versioning and Configuration + +**Version Format**: MAJOR.MINOR.PATCH +- **MAJOR**: Breaking changes to process signatures, inputs, or outputs +- **MINOR**: New processes, backward-compatible enhancements +- **PATCH**: Bug fixes, documentation updates + +**Registry Configuration** (`nextflow.config`): +```groovy +registry { + url = 'https://registry.nextflow.io' // Default registry + + // allow the use of multiple registry url for resolving module + // across custom registries, e.g. + // url = [ 'https://custom.registry.com', 'https://registry.nextflow.io' ] + + auth { + 'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}' + 'npm.myorg.com' = '${MYORG_TOKEN}' + } +} +``` + +**Module Spec** (`meta.yml`): +```yaml +name: nf-core/bwa-align +version: 1.2.4 # This module's version + +requires: + nextflow: ">=24.04.0" +``` + +**Version Constraints** (unified `name@constraint` syntax): +- `name`: Any version (latest) +- `name@1.2.3`: Exact version +- `name@>=1.2.3`: Greater or equal +- `name@>=1.2.3,<2.0.0`: Range (comma-separated) + +**Version Notation Consistency**: + +Modules use the same version constraint syntax already supported by both `nextflowVersion` and plugins: + +| Notation | Meaning | nextflowVersion | Plugins | Modules | +| :---- | :---- | :---- | :---- | :---- | +| 1.2.3 | Exact version | ✓ | ✓ | ✓ | +| >=1.2.3 | Greater or equal | ✓ | ✓ | ✓ | +| <=1.2.3 | Less or equal | ✓ | ✓ | ✓ | +| >1.2.3 | Greater than | ✓ | ✓ | ✓ | +| <1.2.3 | Less than | ✓ | ✓ | ✓ | +| >=1.2, <2.0 | Range (comma) | ✓ | ✓ | ✓ | +| !=1.2.3 | Not equal | ✓ | - | - | +| 1.2+ | >=1.2.x <2.0 | ✓ | - | - | +| 1.2.+ | >=1.2.0 <1.3.0 | ✓ | - | - | +| ~1.2.3 | >=1.2.3 <1.3.0 | - | ✓ | - | + +Using comparison operators (`>=`, `<`) with comma-separated ranges provides the same expressive power as +npm-style `^` and `~` notation while maintaining consistency with existing Nextflow version constraint syntax. +This avoids introducing new notation that would require additional parser support. + +**Module Resolution**: + +Installed module versions are inferred from the `meta.yml` file for each module in the `modules/` directory. + +### 3. Unified Nextflow Registry + +**Architecture Decision**: Extend existing Nextflow registry at `registry.nextflow.io` to host both plugins and modules. + +**Current Plugin API** (reference: https://registry.nextflow.io/openapi/): +``` +GET /api/v1/plugins # List/search plugins +GET /api/v1/plugins/{pluginId} # Get plugin + all releases +GET /api/v1/plugins/{pluginId}/{version} # Get specific release +GET /api/v1/plugins/{pluginId}/{version}/download/{fileName} # Download artifact +POST /api/v1/plugins/release # Create draft release +POST /api/v1/plugins/release/{releaseId}/upload # Upload artifact +``` + +**Module API** (reference: https://github.com/seqeralabs/plugin-registry/pull/266): +``` +GET /api/modules?query= # Search modules (semantic search) +GET /api/modules/{name} # Get module + latest release +GET /api/modules/{name}/releases # List all releases +GET /api/modules/{name}/{version} # Get specific release +GET /api/modules/{name}/{version}/download # Download module bundle +POST /api/modules/{name} # Publish module version (authenticated) +``` + +Note: The `{name}` parameter includes the namespace prefix (e.g., "nf-core/fastqc"). + +**Registry URL**: `registry.nextflow.io` + +**Artifact Types**: +- **Plugins**: JAR files with JSON metadata, resolved at startup +- **Modules**: Source archives (.nf + meta.yml), resolved at parse time + +**Benefits**: +- Reuses existing infrastructure (HTTP service, S3 storage, authentication) +- Consistent API patterns for both artifact types +- Operational simplicity (one service vs. two) +- Internal module API already partially implemented + +### 4. First-Class CLI Support + +**Commands**: +```bash +nextflow module run scope/name # Run a module directly without a wrapper script +nextflow module search # Search registry +nextflow module install scope/name # Install a module +nextflow module list # Show installed vs configured +nextflow module remove scope/name # Remove from config + local cache +nextflow module publish scope/name # Publish to registry (requires api key) +``` + +**General Notes**: +- All commands respect the `registry.url` configuration for custom registries + +#### `nextflow module run scope/name` + +Run a module directly without requiring a wrapper workflow script. This command enables standalone execution of any module by automatically mapping command-line arguments to the module's process inputs. If the module is not available locally, it is automatically installed before execution. + +**Arguments**: +- `scope/name`: Module identifier to run (required) + +**Options**: +- `-version `: Run a specific version (default: latest or configured version) +- `-- `: Map value to the corresponding module process input channel +- All standard `nextflow run` options (e.g., `-profile`, `-work-dir`, `-resume`, etc.) + +**Behavior**: +1. Checks if module is installed locally; if not, downloads from registry +2. Parses the module's `main.nf` to identify the main process and its input declarations +3. Validates command-line arguments against the process input declarations +4. Generates an implicit workflow that wires CLI arguments to process inputs +5. Executes the workflow using standard Nextflow runtime + +**Input Mapping**: +- Named arguments (`--reads`, `--reference`) are mapped to corresponding process inputs +- File paths are automatically converted to files for process file inputs +- Multiple values can be provided for inputs expecting collections +- Required inputs without defaults must be provided; optional inputs use declared defaults + +**Example**: +```bash +# Run BWA alignment module with input files +nextflow module run nf-core/bwa-align \ + --reads 'samples/*_{1,2}.fastq.gz' \ + --reference genome.fa + +# Run a specific version with Nextflow options +nextflow module run nf-core/fastqc -version 1.0.0 \ + --input 'data/*.fastq.gz' \ + -profile docker \ + -resume + +# Run with work directory and output specification +nextflow module run nf-core/salmon \ + --reads reads.fq \ + --index salmon_index \ + -work-dir /tmp/work \ + -output-dir results/ +``` + +--- + +#### `nextflow module search ` + +Search the Nextflow registry for available modules matching the specified query. The search operates against module names, descriptions, tags, and author information. Results are displayed with module name, latest version, description, and download statistics. + +**Arguments**: +- ``: Search term (required) - matches against module metadata + +**Options**: +- `-limit `: Maximum number of results to return (default: 10) +- `-json`: Output results in JSON format for programmatic use + +**Example**: +```bash +nextflow module search bwa +nextflow module search "alignment" -limit 50 +``` + +--- + +#### `nextflow module install ` + +Download and install a module to the local `modules/` directory. + +**Arguments**: +- ``: Module identifier. + +**Options**: +- `-version `: Install a specific version (default: latest) +- `-force`: Overwrite any local changes + +**Behavior**: +1. If `-version` not specified, queries registry for the latest available version +2. Checks if local module exists and verifies integrity against `.module-info` file +3. If local module is unmodified and version differs: replaces with requested version +4. If local module was modified (checksum mismatch): warns and aborts unless `-force` is used +5. Downloads the module archive from the registry +6. Extracts to `modules/scope/name/` directory +7. Stores `.module-info` file from registry's X-Checksum response header + +**Example**: +```bash +nextflow module install nf-core/bwa-align # Install specific module (latest) +nextflow module install nf-core/salmon -version 1.2.0 +``` + +--- + +#### `nextflow module list` + +Display the status of all modules, comparing what is configured in `nextflow.config` against what is actually installed in the `modules/` directory. + +**Options**: +- `-json`: Output in JSON format +- `-outdated`: Only show modules with available updates + +**Output columns**: +- Module name (`scope/name`) +- Installed version (from `modules/scope/name/meta.yml`) +- Latest available version (from registry) +- Status indicator (up-to-date, outdated, missing) + +**Example**: +```bash +nextflow module list +nextflow module list -outdated +``` + +--- + +#### `nextflow module remove scope/name` + +Remove a module from the local `modules/` directory. + +**Arguments**: +- `scope/name`: Module identifier to remove (required) + +**Options**: +- `-keep-files`: Remove `.module-info` file but keep local files + +**Behavior**: +1. Removes the module directory from `modules/scope/name/` + +**Example**: +```bash +nextflow module remove nf-core/bwa-align +nextflow module remove myorg/custom -keep-files +``` + +--- + +#### `nextflow module publish scope/name` + +Publish a module to the Nextflow registry, making it available for others to install. Requires authentication via API key and appropriate permissions for the target scope. + +**Arguments**: +- `scope/name`: Module identifier to publish (required) + +**Options**: +- `-registry `: Target registry URL (default: `registry.nextflow.io`) +- `-tag `: Additional tags for discoverability +- `-dry-run`: Validate without publishing + +**Behavior**: +1. Validates `meta.yml` schema and required fields (name, version, description) +2. Verifies that `main.nf` exists and is valid Nextflow syntax +3. Verifies that `README.md` documentation is present +4. Authenticates with registry using configured credentials +5. Creates a release draft and uploads the module archive +6. Publishes the release, making it available for installation + +**Requirements**: +- Valid `meta.yml` with name, version, and description +- `main.nf` entry point file +- `README.md` documentation +- Authentication token configured in `registry.auth` or `NXF_REGISTRY_TOKEN` +- Write permission for the target scope + +**Example**: +```bash +nextflow module publish myorg/my-process +nextflow module publish myorg/my-process -dry-run +``` + +## Module Structure + +**Directory Layout**: +Everything within the module directory should be uploaded. Module bundle should not exceed 1MB (uncompressed). Typically this is expected to look something like this: +``` +my-module/ +├── main.nf # Required: entry point for module +├── meta.yml # Required: Module spec (version, metadata, I/O specs) +├── README.md # Required: Module description +└── tests/ # Optional tests +``` + +**Module Spec extension** (`meta.yml`): +```yaml +name: nf-core/bwa-align +version: 1.2.4 # This module's version +description: Align reads using BWA-MEM +authors: + - nf-core community +license: MIT + +requires: + nextflow: ">=24.04.0" +``` + +**Local Storage Structure**: +``` +project-root/ +├── nextflow.config +├── main.nf +└── modules/ # Local module cache + ├── nf-core/ + │ ├── bwa-align/ + │ │ ├── .module-info # Cached registry checksum + │ │ ├── meta.yml + │ │ └── main.nf # Required entry point + │ └── samtools/view/ + │ ├── .module-info + │ ├── meta.yml + │ └── main.nf # Required entry point + └── myorg/ + └── custom-process/ + ├── .module-info + ├── meta.yml + └── main.nf # Required entry point +``` + +**Module Integrity Verification**: +- On install: `.module-info` file created from registry's X-Checksum response header +- On run: Local module checksum compared against `.module-info` file +- If match: Proceed without network call +- If mismatch: Report warning (module may have been locally modified) + +## Implementation Strategy + +**Phase 1**: Module schema, local module loading, validation tools + +**Phase 2**: Extend Nextflow registry for modules, implement caching, add `install` and `search` commands + +**Phase 3**: Extend DSL parser for `from module` syntax + +**Phase 4**: Implement `publish` command with authentication and `run` command + +**Phase 5**: Advanced features (search UI, language server integration, ontology validation) + +## Technical Details + +**Module Resolution Flow**: +1. Parse `include` statements → extract module names (e.g., `nf-core/bwa-align`) +2. For each module: + a. Check local `modules/scope/name/` exists + - If exists → read installed version from `modules/scope/name/meta.yml` + - If missing → download latest version from registry + b. Verify local module integrity against `.module-info` file + - Checksum mismatch → warn and do NOT override (local changes detected) +3. On download: store module to `modules/scope/name/` with `.module-info` file +4. Read `meta.yml` file: Validates Nextflow requirement → Fail if not fulfilled +5. Parse module's `main.nf` file → make processes available + +**Security**: +- SHA-256 checksum verification on download (stored in `.module-info` file) +- Integrity verification on run (local checksum vs `.module-info` file) +- Authentication required for publishing +- Support for private registries + +**Integration with Plugin System**: +- Both plugins and modules query same registry +- Single authentication system +- Separate cache locations: `$NXF_HOME/plugins/` (global) vs `modules/` (per-project) + +## Comparison: Plugins vs. Modules + +| Aspect | Plugins | Modules | +|--------|---------|---------| +| Purpose | Extend runtime | Reusable processes | +| Format | JAR files | Source code (.nf) | +| Resolution | Startup | Parse time | +| Metadata | JSON spec | YAML spec | +| Naming | `nf-amazon` | `nf-core/salmon` | +| Cache Location | `$NXF_HOME/plugins/` | `modules/scope/name/` | +| Version Config | `plugins {}` in config | `meta.yml` in `modules/` directory | +| Registry Path | `/api/v1/plugins/` | `/api/modules/{name}` | + +## Rationale + +**Why unified registry?** +- Reuses battle-tested infrastructure (HTTP API, S3, auth) +- Single discovery experience for ecosystem +- Lower operational overhead +- Type-specific handling maintains separation of concerns + +**Why infer versions from `meta.yml` instead of pinning in a separate file?** +- Simple: install a version once and it is captured in the module files +- Reproducibility via committing the `modules/` directory (including `meta.yml`) to the project git repository +- Reduces configuration burden: no need to keep config in sync with installed state + +**Why parse-time resolution?** +- Modules are source code, not compiled artifacts +- Allows inspection/modification for reproducibility +- Enables dependency analysis before execution + +**Why scoped modules?** +- Organization namespacing prevents name collisions (`nf-core/salmon` vs `myorg/salmon`) +- Clear ownership and provenance of modules +- Supports private registries per scope +- Industry-standard pattern (NPM, Terraform, others) +- Enables ecosystem organization by maintainer/organization + +**Why semantic versioning?** +- Clear compatibility guarantees +- Industry standard (npm, cargo, Go modules) + +## Consequences + +**Positive**: +- Enables ecosystem-wide code reuse +- Reproducible workflows via committing the `modules/` directory (including `meta.yml`) to the project git repository +- Centralized discovery and distribution via unified registry +- Minimal operational overhead (single registry for both plugins and modules) +- Module scoping enables organization namespaces and private registries +- Local `modules/` directory provides project isolation +- No version duplication: installed `meta.yml` is the single source of truth +- Simple module structure: each module has single `main.nf` entry point + +**Negative**: +- Registry becomes critical infrastructure (requires HA setup) +- Type-specific handling adds registry complexity +- Parse-time resolution adds latency to workflow startup +- Local `modules/` directory duplicates storage across projects (unlike global cache) + +**Neutral**: +- Modules and plugins conceptually distinct but share infrastructure +- Different resolution timing supported by same API + +## Links + +- Related: [Plugin Spec ADR](20250922-plugin-spec.md) +- Inspired by: [Go Modules](https://go.dev/ref/mod), [npm](https://docs.npmjs.com), [Cargo](https://doc.rust-lang.org/cargo/) +- Related: [nf-core modules](https://nf-co.re/modules) + +--- + +## Appendix A: Module Schema Specification + +This appendix defines the JSON schema for module `meta.yml` files. The schema maintains backward compatibility with existing nf-core module metadata patterns while supporting the new Nextflow module system features. + +**Schema File:** [module-spec-schema.json](module-spec-schema.json) +**Published URL:** `https://registry.nextflow.io/schemas/module-spec/v1.0.0` + +### Field Reference + +#### Core Fields (Existing nf-core Pattern) + +These fields are already widely adopted in the nf-core community and remain fully supported: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | Module identifier | +| `description` | string | Yes | Brief description of module functionality | +| `keywords` | array[string] | Recommended | Discovery and categorization keywords | +| `authors` | array[string] | Recommended | Original authors (GitHub handles) | +| `maintainers` | array[string] | Recommended | Current maintainers | +| `tools` | array[object] | Conditional | Software tools wrapped by the module | +| `input` | array/object | Recommended | Input channel specifications | +| `output` | object/array | Recommended | Output channel specifications | + +#### Extension Fields (Nextflow Module System) + +These fields extend the schema to support the new Nextflow module system: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `version` | string | Registry | Semantic version (MAJOR.MINOR.PATCH) | +| `license` | string | Registry | SPDX license identifier for module code | +| `requires` | object | Optional | Runtime requirements | +| `requires.nextflow` | string | Optional | Nextflow version constraint | + +### Detailed Field Specifications + +#### `name` + +The module name must be a fully qualified scoped identifier in `scope/name` format: + +```yaml +name: nf-core/fastqc +name: nf-core/bwa-mem +name: myorg/custom-aligner +``` + +**Naming Rules:** +- Format: `scope/name` (e.g., `nf-core/salmon`, `myorg/custom`) +- Scope: lowercase alphanumeric with hyphens (organization/owner identifier) +- Name: lowercase alphanumeric with underscores/hyphens (module identifier) +- Pattern: `^[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` + +#### `version` + +Semantic version following [SemVer 2.0.0](https://semver.org/): + +```yaml +version: "1.0.0" +version: "2.3.1" +version: "1.0.0-beta.1" +``` + +**Version Semantics:** +- **MAJOR:** Breaking changes to process signatures, inputs, or outputs +- **MINOR:** New processes, backward-compatible enhancements +- **PATCH:** Bug fixes, documentation updates + +**Requirement:** Mandatory for registry-published modules (scoped names in `scope/name` format). + +#### `requires` + +Specifies runtime requirements for the module. + +```yaml +requires: + nextflow: ">=24.04.0" +``` + +**`requires.nextflow`** - Nextflow version constraint: +```yaml +requires: + nextflow: ">=24.04.0" # minimum version + nextflow: ">=24.04.0,<25.0.0" # version range +``` + +#### `tools` + +Documents the software tools wrapped by the module: + +```yaml +tools: + - bwa: + description: BWA aligner + homepage: http://bio-bwa.sourceforge.net/ + license: ["GPL-3.0-or-later"] + identifier: biotools:bwa +``` + +**Tool Properties:** + +| Property | Required | Description | +|----------|----------|-------------| +| `description` | Yes | Tool description | +| `homepage` | One of these | Tool homepage URL | +| `documentation` | One of these | Documentation URL | +| `tool_dev_url` | One of these | Development/source URL | +| `doi` | One of these | Publication DOI | +| `arxiv` | No | arXiv identifier | +| `license` | Recommended | SPDX license(s) | +| `identifier` | Recommended | bio.tools identifier | +| `manual` | No | User manual URL | + +#### `input` and `output` + +The schema supports both nf-core patterns to ensure backward compatibility: + +**Module Pattern (Tuple-based):** +```yaml +input: + - - meta: + type: map + description: Sample metadata + - reads: + type: file + description: Input FastQ files + ontologies: + - edam: "http://edamontology.org/format_1930" + - - index: + type: directory + description: Reference index + +output: + bam: + - - meta: + type: map + description: Sample metadata + - "*.bam": + type: file + description: Aligned BAM file + pattern: "*.bam" + versions: + - versions.yml: + type: file + description: Software versions +``` + +**Channel Element Properties:** + +| Property | Type | Description | +|----------|------|-------------| +| `type` | string | Data type: `map`, `file`, `directory`, `string`, `integer`, `float`, `boolean`, `list`, `val` | +| `description` | string | Human-readable description | +| `pattern` | string | File glob pattern or value pattern | +| `optional` | boolean | Whether input is optional (default: false) | +| `default` | any | Default value if not provided | +| `enum` | array | List of allowed values | +| `ontologies` | array | EDAM or other ontology annotations | + +### Migration Guide + +#### From nf-core Module to Registry Module + +**Before (nf-core local):** +```yaml +name: bwa_mem +description: Align reads using BWA-MEM +keywords: + - alignment + - bwa +tools: + - bwa: + description: BWA software + homepage: http://bio-bwa.sourceforge.net/ + license: ["GPL-3.0-or-later"] + identifier: biotools:bwa +authors: + - "@drpatelh" +maintainers: + - "@drpatelh" +input: + # ... existing input spec +output: + # ... existing output spec +``` + +**After (Registry-ready):** +```yaml +name: nf-core/bwa-mem # Added scope prefix +version: "1.0.0" # Added version +description: Align reads using BWA-MEM +keywords: + - alignment + - bwa +license: MIT # Added module license +requires: # Added requirements + nextflow: ">=24.04.0" +tools: + - bwa: + description: BWA software + homepage: http://bio-bwa.sourceforge.net/ + license: ["GPL-3.0-or-later"] + identifier: biotools:bwa +authors: + - "@drpatelh" +maintainers: + - "@drpatelh" +input: + # ... unchanged +output: + # ... unchanged +``` + +#### Schema Validation + +Use the schema reference in your `meta.yml`: + +```yaml +# yaml-language-server: $schema=https://registry.nextflow.io/schemas/module-spec/v1.0.0 + +name: nf-core/my-module +version: "1.0.0" +# ... +``` + +### Compatibility Matrix + +| Feature | nf-core Current | Nextflow Module System | +|---------|-----------------|------------------------| +| Simple names | Yes | Yes (local only) | +| Scoped names | No | Yes (registry) | +| Version field | No | Yes (required for registry) | +| `tools` section | Yes | Yes | +| `components` | Yes | Deprecated | +| `requires` | No | Yes (Nextflow version constraint) | +| I/O specifications | Yes | Yes | +| Ontologies | Yes | Yes | + +### Unsupported nf-core Attributes + +The following attributes from the nf-core meta schema are **not supported** in the Nextflow module system: + +| Attribute | Reason | Alternative | +|-----------|--------|-------------| +| `extra_args` | Not adopted in practice by nf-core modules | To be defined | +| `components` | No longer supported | Module dependencies are managed via `nextflow.config` | + +### Complete Examples + +#### Minimal nf-core Module + +```yaml +name: fastqc +description: Run FastQC on sequenced reads +keywords: + - quality control + - qc + - fastq +tools: + - fastqc: + description: FastQC quality metrics + homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/ + license: ["GPL-2.0-only"] + identifier: biotools:fastqc +authors: + - "@drpatelh" +maintainers: + - "@drpatelh" +output: + html: + - "*.html": + type: file + description: FastQC HTML report + versions: + - versions.yml: + type: file + description: Software versions +``` + +#### Full Registry Module + +```yaml +name: nf-core/bwa-align +version: "1.2.4" +description: Align reads to reference genome using BWA-MEM algorithm +keywords: + - alignment + - mapping + - bwa + - bam + - fastq +license: MIT + +requires: + nextflow: ">=24.04.0" + +tools: + - bwa: + description: | + BWA is a software package for mapping DNA sequences + against a large reference genome. + homepage: http://bio-bwa.sourceforge.net/ + documentation: https://bio-bwa.sourceforge.net/bwa.shtml + doi: 10.1093/bioinformatics/btp324 + license: ["GPL-3.0-or-later"] + identifier: biotools:bwa + +authors: + - "@nf-core" +maintainers: + - "@drpatelh" + - "@maxulysse" + +input: + - - meta: + type: map + description: Sample metadata map (e.g., [ id:'sample1', single_end:false ]) + - reads: + type: file + description: Input FastQ files + ontologies: + - edam: "http://edamontology.org/format_1930" + - - meta2: + type: map + description: Reference metadata + - index: + type: directory + description: BWA index directory + ontologies: + - edam: "http://edamontology.org/data_3210" + +output: + bam: + - - meta: + type: map + description: Sample metadata + - "*.bam": + type: file + description: Aligned BAM file + pattern: "*.bam" + ontologies: + - edam: "http://edamontology.org/format_2572" + versions: + - versions.yml: + type: file + description: Software versions + pattern: "versions.yml" +``` + diff --git a/adr/module-spec-schema.json b/adr/module-spec-schema.json new file mode 100644 index 0000000000..25f77a0bfc --- /dev/null +++ b/adr/module-spec-schema.json @@ -0,0 +1,467 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://raw.githubusercontent.com/nextflow-io/schemas/main/module/v1/schema.json", + "title": "Nextflow Module Schema", + "description": "Schema for Nextflow module meta.yml files, supporting both nf-core community patterns and the Nextflow module system", + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "Module name. Can be a simple identifier (e.g., 'fastqc', 'bwa_mem') for local/nf-core modules, or a fully qualified scoped name (e.g., 'nf-core/fastqc', 'myorg/custom') for registry modules.", + "examples": ["fastqc", "bwa_mem", "nf-core/fastqc", "myorg/salmon-quant"], + "pattern": "^([a-z0-9][a-z0-9-]*/)?[a-z][a-z0-9_-]*$" + }, + "version": { + "type": "string", + "description": "Semantic version of the module (MAJOR.MINOR.PATCH). Required for registry publication", + "pattern": "^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(-[0-9A-Za-z-]+(\\.[0-9A-Za-z-]+)*)?(\\+[0-9A-Za-z-]+(\\.[0-9A-Za-z-]+)*)?$", + "examples": ["1.0.0", "2.1.3", "1.0.0-beta.1"] + }, + "description": { + "type": "string", + "description": "Brief description of what the module does", + "minLength": 10, + "maxLength": 500 + }, + "keywords": { + "type": "array", + "description": "Keywords for discovery and categorization", + "items": { + "type": "string", + "minLength": 2 + }, + "minItems": 1, + "uniqueItems": true + }, + "license": { + "type": "string", + "description": "SPDX license identifier for the module code itself", + "examples": ["MIT", "Apache-2.0", "GPL-3.0-or-later"] + }, + "authors": { + "type": "array", + "description": "Original authors of the module (GitHub handles preferred)", + "items": { + "type": "string", + "pattern": "^@?[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?$" + }, + "minItems": 1 + }, + "maintainers": { + "type": "array", + "description": "Current maintainers of the module (GitHub handles preferred)", + "items": { + "type": "string", + "pattern": "^@?[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?$" + } + }, + "requires": { + "type": "object", + "description": "Runtime requirements for the module", + "properties": { + "nextflow": { + "type": "string", + "description": "Nextflow version constraint using comparison operators", + "examples": [">=24.04.0", ">=24.04.0,<25.0.0"], + "pattern": "^[<>=!]+[0-9]+\\.[0-9]+\\.[0-9]+(-[a-zA-Z0-9]+)?(,\\s*[<>=!]+[0-9]+\\.[0-9]+\\.[0-9]+(-[a-zA-Z0-9]+)?)*$" + } + }, + "additionalProperties": false + }, + "input": { + "type": "array", + "description": "Inputs of the module", + "items": { + "$ref": "#/$defs/structuredParameter" + } + }, + "params": { + "type": "array", + "description": "Module parameter specifications", + "items": { + "$ref": "#/$defs/paramSpec" + } + }, + "output": { + "type": "array", + "description": "Outputs of the module", + "items": { + "$ref": "#/$defs/structuredParameter" + } + }, + "topics": { + "type": "array", + "description": "Topics of the module", + "items": { + "$ref": "#/$defs/structuredParameter" + } + }, + "tools": { + "type": "array", + "description": "Software tools wrapped by this module with their metadata", + "items": { + "type": "object", + "minProperties": 1, + "maxProperties": 1, + "patternProperties": { + "^[a-zA-Z][a-zA-Z0-9_-]*$": { + "$ref": "#/$defs/toolSpec" + } + } + } + } + }, + "required": ["name", "description"], + "$defs": { + "toolSpec": { + "type": "object", + "description": "Specification for a software tool used by the module", + "properties": { + "description": { + "type": "string", + "description": "Description of the tool and its purpose" + }, + "homepage": { + "type": "string", + "format": "uri", + "description": "Tool's homepage URL", + "pattern": "^https?://.*$" + }, + "documentation": { + "type": "string", + "format": "uri", + "description": "Documentation URL", + "pattern": "^(https?|ftp)://.*$" + }, + "tool_dev_url": { + "type": "string", + "format": "uri", + "description": "Development/source code URL", + "pattern": "^https?://.*$" + }, + "doi": { + "description": "Digital Object Identifier for the tool's publication", + "oneOf": [ + { + "type": "string", + "pattern": "^10\\.\\d{4,9}/[^,]+$" + }, + { + "type": "string", + "const": "no DOI available" + } + ] + }, + "arxiv": { + "type": "string", + "description": "arXiv identifier", + "pattern": "^arXiv:\\d{4}\\.\\d{4,5}(v\\d+)?$" + }, + "licence": { + "type": "array", + "description": "SPDX license identifier(s) for the tool", + "items": { + "type": "string" + }, + "minItems": 1, + "uniqueItems": true + }, + "identifier": { + "description": "bio.tools identifier or empty string", + "oneOf": [ + { + "type": "string", + "pattern": "^biotools:[a-zA-Z0-9_-]+$" + }, + { + "type": "string", + "maxLength": 0 + } + ] + }, + "manual": { + "type": "string", + "format": "uri", + "description": "Manual/user guide URL" + } + }, + "required": ["description"], + "anyOf": [ + { + "required": ["homepage"] + }, + { + "required": ["documentation"] + }, + { + "required": ["tool_dev_url"] + }, + { + "required": ["doi"] + } + ] + }, + "structuredParameter": { + "type": "array", + "items": { + "oneOf": [ + { + "$ref": "#/$defs/paramSpec" + }, + { + "type": "array", + "items": { + "$ref": "#/$defs/paramSpec" + } + } + ] + } + }, + "paramSpec": { + "type": "object", + "description": "Specification for a module parameter", + "properties": { + "name": { + "type": "string", + "description": "Parameter identifier", + "pattern": "^[a-zA-Z_][a-zA-Z0-9_]*$" + }, + "type": { + "type": "string", + "description": "Data type of the parameter value", + "enum": [ + "boolean", + "float", + "integer", + "string", + "list", + "map", + "file", + "directory" + ] + }, + "description": { + "type": "string", + "description": "Human-readable description of the parameter" + }, + "pattern": { + "type": "string", + "description": "Glob pattern for file/directory parameters" + }, + "optional": { + "type": "boolean", + "description": "Whether this parameter is optional", + "default": false + }, + "enum": { + "type": "array", + "description": "List of allowed values", + "uniqueItems": true + }, + "ontologies": { + "type": "array", + "description": "Ontology annotations (e.g., EDAM)", + "items": { + "type": "object", + "patternProperties": { + "^[a-zA-Z]+$": { + "type": "string", + "format": "uri", + "description": "Ontology URI" + } + } + }, + "uniqueItems": true + } + }, + "required": ["name", "type", "description"] + } + }, + "allOf": [ + { + "if": { + "properties": { + "name": { + "pattern": "^[a-z0-9][a-z0-9-]*/" + } + }, + "required": ["name"] + }, + "then": { + "required": ["name", "description", "version"], + "properties": { + "version": { + "description": "Version is required for scoped/registry modules (scope/name format)" + } + } + } + } + ], + "examples": [ + { + "name": "fastqc", + "description": "Run FastQC on sequenced reads", + "keywords": ["quality control", "qc", "adapters", "fastq"], + "tools": [ + { + "fastqc": { + "description": "FastQC gives general quality metrics about your reads.", + "homepage": "https://www.bioinformatics.babraham.ac.uk/projects/fastqc/", + "documentation": "https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/", + "licence": ["GPL-2.0-only"], + "identifier": "biotools:fastqc" + } + } + ], + "input": [ + [ + { + "meta": { + "type": "map", + "description": "Groovy Map containing sample information" + } + }, + { + "reads": { + "type": "file", + "description": "Input FastQ files", + "ontologies": [] + } + } + ] + ], + "output": { + "html": [ + [ + { + "meta": { + "type": "map", + "description": "Sample information" + } + }, + { + "*.html": { + "type": "file", + "description": "FastQC report", + "pattern": "*_{fastqc.html}", + "ontologies": [] + } + } + ] + ], + "versions": [ + { + "versions.yml": { + "type": "file", + "description": "File containing software versions", + "pattern": "versions.yml" + } + } + ] + }, + "authors": ["@drpatelh", "@ewels"], + "maintainers": ["@drpatelh", "@ewels"] + }, + { + "name": "nf-core/bwa-align", + "version": "1.2.4", + "description": "Align reads using BWA-MEM algorithm", + "keywords": ["alignment", "bwa", "mapping", "fastq", "bam"], + "license": "MIT", + "authors": ["@nf-core"], + "maintainers": ["@nf-core"], + "requires": { + "nextflow": ">=24.04.0" + }, + "tools": [ + { + "bwa": { + "description": "BWA aligner", + "homepage": "http://bio-bwa.sourceforge.net/", + "licence": ["GPL-3.0-or-later"] + } + }, + { + "samtools": { + "description": "SAMtools", + "homepage": "http://www.htslib.org/", + "licence": ["MIT"] + } + } + ], + "params": [ + { + "name": "batch_size", + "type": "integer", + "description": "Process INT input bases in each batch", + "example": 100000000 + }, + { + "name": "use_soft_clipping", + "type": "boolean", + "description": "Use soft clipping for supplementary alignments" + }, + { + "name": "output_format", + "type": "string", + "description": "Output format (sam, bam, or cram)" + } + ], + "input": [ + [ + { + "meta": { + "type": "map", + "description": "Sample metadata map" + } + }, + { + "reads": { + "type": "file", + "description": "Input FastQ files", + "ontologies": [ + { "edam": "http://edamontology.org/format_1930" } + ] + } + } + ], + { + "index": { + "type": "directory", + "description": "BWA index directory" + } + } + ], + "output": { + "bam": [ + [ + { + "meta": { + "type": "map", + "description": "Sample metadata" + } + }, + { + "*.bam": { + "type": "file", + "description": "Aligned BAM file", + "pattern": "*.bam", + "ontologies": [ + { "edam": "http://edamontology.org/format_2572" } + ] + } + } + ] + ], + "versions": [ + { + "versions.yml": { + "type": "file", + "description": "Software versions" + } + } + ] + } + } + ] +} \ No newline at end of file diff --git a/docs/cli.md b/docs/cli.md index 607a041756..b09e03c4de 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -262,7 +262,137 @@ $ nextflow secrets set AWS_ACCESS_KEY_ID $ nextflow secrets delete AWS_ACCESS_KEY_ID ``` -See {ref}`cli-secrets` for more information. +See {ref}`cli-secrets` for more information. + +## Module management + +:::{versionadded} 26.04.0 +::: + +The `module` command enables working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and sharing improvements with the community. + +Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use. + +### Searching for modules + +The `module search` command queries the module registry to discover available modules by keyword or name. + +Use this to find modules for specific tasks, explore available tools, or discover community modules. + +```console +$ nextflow module search alignment +$ nextflow module search "quality control" -limit 10 +$ nextflow module search bwa -output json +``` + +Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for JSON-formatted output. + +See {ref}`cli-module-search` for more information. + +### Installing modules + +The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL. + +Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions. + +```console +$ nextflow module install nf-core/fastqc +$ nextflow module install nf-core/fastqc -version 1.0.0 +``` + +The installed module will be available in `modules/nf-core/fastqc`. + +Use the `-force` flag to reinstall a module even if local modifications exist. + +See {ref}`cli-module-install` for more information. + +### Listing modules + +The `module list` command displays all modules currently installed in your project, showing their versions and integrity status. + +Use this to review installed modules, check module versions, or detect local modifications. + +```console +$ nextflow module list +$ nextflow module list -output json +``` + +The output shows each module's name, installed version, and whether it has been modified locally. Use `-o json` for JSON-formatted output. + +See {ref}`cli-module-list` for more information. + +### Viewing module information + +The `module info` command displays detailed metadata and usage information for a specific module from the registry. + +Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module. + +```console +$ nextflow module info nf-core/fastqc +$ nextflow module info nf-core/fastqc -version 1.0.0 +$ nextflow module info nf-core/fastqc -output json +``` + +The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access. + +See {ref}`cli-module-info` for more information. + +### Running modules directly + +The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing. + +Use this to quickly run a module, test module functionality, or execute one-off data processing tasks. + +```console +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 +``` + +The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.): + +```console +$ nextflow module run nf-core/salmon \ + --reads reads.fq \ + --index salmon_index \ + -profile docker \ + -resume +``` + +Process inputs can be specified like params on the command line. For example, `--reads reads.fq` corresponds to the `reads` input in the `nf-core/salmon` module. Run `nextflow module info nf-core/salmon` to see the available params for the module. + +See {ref}`cli-module-run` for more information. + +### Removing modules + +The `module remove` command deletes modules from your project, removing local files and configuration entries. + +Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline. + +```console +$ nextflow module remove nf-core/fastqc +$ nextflow module remove nf-core/fastqc -keep-files +``` + +By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files. + +See {ref}`cli-module-remove` for more information. + +### Publishing modules + +The `module publish` command uploads modules to a registry, making them available for others to install and use. + +Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization. + +```console +$ nextflow module publish myorg/my-module +$ nextflow module publish myorg/my-module -dry-run +``` + +Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. The module must include `main.nf`, `meta.yml`, and `README.md` files. + +Use `-dry-run` to validate your module structure without uploading. + +See {ref}`cli-module-publish` for more information. ## Configuration and validation @@ -384,7 +514,7 @@ Use this to understand input/output relationships between tasks, trace data flow $ nextflow lineage ``` -See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information. +See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information. ## Seqera Platform diff --git a/docs/module.md b/docs/module.md index 73d92fe4e2..eb5361d9fd 100644 --- a/docs/module.md +++ b/docs/module.md @@ -279,8 +279,164 @@ This feature requires the use of a local or shared file system for the pipeline ## Sharing modules -Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. While Nextflow does not provide an explicit mechanism for sharing modules, there are several ways to do it: +Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules: +- Use the Nextflow module registry (recommended, see below) - Simply copy the module files into your pipeline repository - Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy - Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community + +(module-registry)= + +## Registry-based modules + +:::{versionadded} 26.04.0 +::: + +Nextflow provides a module registry that enables you to install, publish, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow language. + +### Installing modules from a registry + +Use the `module install` command to download modules from a registry: + +```console +$ nextflow module install nf-core/fastqc +$ nextflow module install nf-core/fastqc -version 1.0.0 +``` + +Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path: + +```nextflow +include { FASTQC } from 'nf-core/fastqc' + +workflow { + reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') + FASTQC(reads) +} +``` + +### Running modules directly + +For ad-hoc tasks or testing, you can run a module directly without creating a workflow: + +```console +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +``` + +This command accepts all standard `nextflow run` options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed. + +### Discovering modules + +Search for available modules using the `module search` command: + +```console +$ nextflow module search alignment +$ nextflow module search "quality control" -limit 10 +``` + +List installed modules in your project: + +```console +$ nextflow module list +``` + +### Module checksum verification + +Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites: + +```console +$ nextflow module install nf-core/fastqc -version 1.1.0 +Warning: Module nf-core/fastqc has local modifications. Use -force to override. +``` + +Use the `-force` flag to override local modifications when needed. + +### Removing modules + +Use the `module remove` command to uninstall a module: + +```console +$ nextflow module remove nf-core/fastqc +``` + +By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour: + +- `-keep-files`: Remove the `.module-info` file created at install but keep the rest of files +- `-force`: Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications + +### Viewing module information + +Use the `module info` command to display metadata and a usage template for a module: + +```console +$ nextflow module info nf-core/fastqc +$ nextflow module info nf-core/fastqc -version 1.0.0 +``` + +The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-o json` to get machine-readable output. + +### Publishing modules + +To share your own modules, use the `module publish` command: + +```console +$ nextflow module publish myorg/my-module +``` + +The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. + +Your module directory must include: + +- `main.nf`: The module entry point +- `meta.yml`: Module spec (name, description, version, etc.) +- `README.md`: Module documentation + +Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration: + +```groovy +registry { + apiKey = 'YOUR_REGISTRY_TOKEN' +} +``` + +Use `-dry-run` to validate your module structure without uploading: + +```console +$ nextflow module publish myorg/my-module -dry-run +``` + +### Registry configuration + +By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries: + +```groovy +registry { + url = [ + 'https://private.registry.myorg.com', + 'https://registry.nextflow.io' + ] + apiKey = '${MYORG_TOKEN}' +} +``` + +Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry. + +### Module directory structure + +Registry modules follow a standard directory structure: + +``` +modules/ +└── scope/ + └── module-name/ + ├── .module-info # Integrity checksum (generated automatically) + ├── README.md # Documentation (required for publishing) + ├── main.nf # Module script (required) + ├── meta.yml # Module spec (required for publishing) + ├── resources/ # Optional: module binaries and resources + └── templates/ # Optional: process templates +``` + +The `modules/` directory should be committed to your Git repository to ensure reproducibility. + +See the {ref}`cli-page` documentation for more information about module commands. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 23f7ed1855..74be2fdd20 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1125,6 +1125,211 @@ $ nextflow log tiny_leavitt -F 'process =~ /split_letters/' work/1f/f1ea9158fb23b53d5083953121d6b6 ``` +(cli-module)= + +### `module` + +:::{versionadded} 26.04.0 +::: + +Manage Nextflow modules. + +**Usage** + +```console +$ nextflow module [options] +``` + +**Description** + +The `module` command provides a comprehensive system for managing registry-based modules. It enables installing modules from registries, running them directly, searching for available modules, and publishing your own modules to a registry. + +**Subcommands** + +(cli-module-info)= + +`info [options] [scope/name]` + +: Display detailed information about a module from the registry. +: Shows module name, version, description, and other metadata, as well as example usage. +: The following options are available: + + `-version` + : Specify the module version to query (e.g., `1.0.0`). If not specified, displays information for the latest version. + + `-o, -output` (`text`) + : Output mode for info results. Options: `text` (default), `json`. + +: **Examples:** + + ```console + # Display information for latest version + $ nextflow module info nf-core/fastqc + + # Display information for specific version + $ nextflow module info nf-core/fastqc -version 1.0.0 + + # Get results as JSON + $ nextflow module info nf-core/fastqc -output json + ``` + +(cli-module-install)= + +`install [options] [scope/name]` + +: Install a module from the registry into your project. +: Downloaded modules are stored in the `modules/` directory. +: The `.module-info` file is created in the module directory to store additional information of the installed module. +: The following options are available: + + `-version` + : Specify the module version to install (e.g., `1.0.0`). If not specified, installs the latest version. + + `-force` + : Force reinstall even if the module exists locally with modifications. Without this flag, Nextflow prevents overwriting locally modified modules. + +: **Examples:** + + ```console + # Install latest version + $ nextflow module install nf-core/fastqc + + # Install specific version + $ nextflow module install nf-core/fastqc -version 1.0.0 + + # Force reinstall over local modifications + $ nextflow module install nf-core/fastqc -force + ``` + +(cli-module-list)= + +`list [options]` + +: List all modules currently installed in your project. +: Shows each module's name, version, and integrity status (whether it has been modified locally). +: The following options are available: + + `-o, -output` (`table`) + : Output mode for list results. Options: `table` (default), `json`. + +: **Examples:** + + ```console + # Display installed modules in formatted table + $ nextflow module list + + # Output as JSON + $ nextflow module list -output 'json' + ``` + +(cli-module-publish)= + +`publish [options] [scope/name | path]` + +: Publish a module to the registry, making it available for others to install. +: The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. +: Requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. +: The module directory must contain `main.nf`, `meta.yml`, and `README.md`. +: The following options are available: + + `-dry-run` + : Validate the module structure and metadata without uploading to the registry. Useful for testing before publishing. + + `-registry` + : Specify the registry to publish the module (default: `https://registry.nextflow.io`) + +: **Examples:** + + ```console + # Validate module structure without publishing + $ nextflow module publish myorg/my-module -dry-run + + # Publish to nextflow registry + $ export NXF_REGISTRY_TOKEN=your-token + $ nextflow module publish myorg/my-module + + # Publish to a custom registry + $ export NXF_REGISTRY_TOKEN=your-token + $ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' + ``` + +(cli-module-remove)= + +`remove [options] [scope/name]` + +: Remove a module from your project. +: By default, removes both local files and configuration entries. Use options to control what gets removed. +: The following options are available: + + `-force` + : Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications. + + `-keep-files` + : Remove the `.module-info` but keep local files in the `modules/` directory. + +: **Examples:** + + ```console + # Remove module completely + $ nextflow module remove nf-core/fastqc + + # Remove from config but keep local files + $ nextflow module remove nf-core/fastqc -keep-files + ``` + +(cli-module-run)= + +`run [options] [scope/name] [-- ]` + +: Execute a module directly from the registry without creating a wrapper workflow. +: Automatically downloads the module if not already installed. Accepts all standard Nextflow run options. +: The `module run` command extends the `run` command and accepts all its options, including `-profile`, `-resume`, `-c`, etc. Command-line params (i.e., `--`) are inferred from the module's declared inputs. +: The following additional options are available: + + `-version` + : Specify the module version to run (e.g., `1.0.0`). If not specified, uses the latest version. + +: **Examples:** + + ```console + # Run module with inputs + $ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' + + # Run specific version with Nextflow options + $ nextflow module run nf-core/fastqc \ + --input 'data/*.fastq.gz' \ + -version 1.0.0 \ + -profile docker \ + -resume + ``` + +(cli-module-search)= + +`search [options] [query]` + +: Search for modules in the registry by keyword or name. +: Returns modules matching the query with their names, versions, descriptions, and download statistics. +: The following options are available: + + `-limit` + : Maximum number of results to return (default: varies by registry). + + `-o, -output` (`simple`) + : Output mode for search results. Options: `simple` (default), `json`. + +: **Examples:** + + ```console + # Search for alignment-related modules + $ nextflow module search alignment + + # Search with limited results + $ nextflow module search "quality control" -limit 10 + + # Get results as JSON + $ nextflow module search bwa -output json + ``` + (cli-plugin)= ### `plugin` @@ -1171,7 +1376,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo : Update all downloaded projects. `-d, -deep` -: :::{deprecated} 25.12.0-edge. +: :::{deprecated} 25.12.0-edge Ignored for new multi-revision asset management strategy. Still used in legacy assets. ::: : Create a shallow clone of the specified depth. diff --git a/modules/nextflow/build.gradle b/modules/nextflow/build.gradle index 2c083183aa..c4323f909b 100644 --- a/modules/nextflow/build.gradle +++ b/modules/nextflow/build.gradle @@ -70,9 +70,12 @@ dependencies { api 'dev.failsafe:failsafe:3.1.0' api 'io.seqera:lib-trace:0.1.0' api 'com.fasterxml.woodstox:woodstox-core:7.1.1' + api 'org.apache.commons:commons-compress:1.27.1' // For tar.gz extraction + api 'io.seqera:npr-api:0.21.4-SNAPSHOT' testImplementation 'org.subethamail:subethasmtp:3.1.7' testImplementation (project(':nf-lineage')) + testImplementation 'org.wiremock:wiremock:3.13.1' // test configuration testFixturesApi ("org.apache.groovy:groovy-test:4.0.30") { exclude group: 'org.apache.groovy' } testFixturesApi ("org.objenesis:objenesis:3.4") diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdBase.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdBase.groovy index 1b115bda15..aef79a7a4e 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdBase.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdBase.groovy @@ -26,9 +26,14 @@ import com.beust.jcommander.Parameter abstract class CmdBase implements Runnable { private Launcher launcher + private List unknownOptions abstract String getName() + protected List getUnknownOptions(){ return this.unknownOptions } + + void setUnknownOptions(List options){ this.unknownOptions = options } + Launcher getLauncher() { launcher } void setLauncher( Launcher value ) { this.launcher = value } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdModule.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdModule.groovy new file mode 100644 index 0000000000..241c8e631f --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdModule.groovy @@ -0,0 +1,152 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli + +import com.beust.jcommander.JCommander +import com.beust.jcommander.Parameter +import com.beust.jcommander.ParameterException +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.cli.module.CmdModuleInfo +import nextflow.cli.module.CmdModuleInstall +import nextflow.cli.module.CmdModuleList +import nextflow.cli.module.CmdModulePublish +import nextflow.cli.module.CmdModuleRemove +import nextflow.cli.module.CmdModuleRun +import nextflow.cli.module.CmdModuleSearch +import nextflow.exception.AbortOperationException + +/** + * Implements `module` command + * + * @author Jorge Ejarque + */ +@CompileStatic +@Slf4j +@Parameters(commandDescription = "Manage Nextflow modules") +class CmdModule extends CmdBase implements UsageAware { + + static final public String NAME = 'module' + + private JCommander jCommander + + static final List commands = new ArrayList<>() + + static { + commands << new CmdModuleInstall() + commands << new CmdModuleRun() + commands << new CmdModuleList() + commands << new CmdModuleRemove() + commands << new CmdModuleSearch() + commands << new CmdModuleInfo() + commands << new CmdModulePublish() + } + + protected JCommander commander() { + if( !this.jCommander ) { + this.jCommander = new JCommander(this) + this.jCommander.setProgramName('nextflow module') + // Register all subcommands + commands.each { cmd -> + cmd.launcher = this.launcher + this.jCommander.addCommand(cmd.getName(), cmd, new String[0]) + } + } + return jCommander + } + + @Parameter + List args + + @Override + String getName() { + return NAME + } + + @Override + void run() { + + + try { + if( !args ) { + usage() + return + } + final jc = commander() + final moduleArgs = args + unknownOptions + jc.parse(moduleArgs as String[]) + + final parsedCommand = jc.getParsedCommand() + if( !parsedCommand ) { + jc.usage() + return + } + + // Get the parsed subcommand instance + final subcommand = jc.getCommands() + .get(parsedCommand) + .getObjects()[0] as CmdBase + + // Execute with fields already populated by JCommander + subcommand.run() + + } catch( ParameterException e ) { + throw new AbortOperationException("${e.getMessage()} -- Check the available commands and options and syntax with 'nextflow module -h'") + } + } + + private CmdBase findCmd(String name) { + commands.find { it.name == name } + } + + /** + * Print the command usage help + */ + @Override + void usage() { + usage(args) + } + + /** + * Print the command usage help + * + * @param args The arguments as entered by the user + */ + @Override + void usage(List args) { + def result = [] + if( !args ) { + result << 'Usage: nextflow module [options]' + result << '' + result << 'Commands:' + commands.each { + def description = it.getClass().getAnnotation(Parameters)?.commandDescription() + result << " ${it.name.padRight(12)}${description}" + } + result << '' + println result.join('\n').toString() + } else { + final sub = findCmd(args[0]) + if( sub ) { + commander().usage(args[0]) + } else { + throw new AbortOperationException("Unknown module sub-command: ${args[0]}") + } + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy index 9a5cd860bc..a4f581e9d8 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/Launcher.groovy @@ -111,7 +111,8 @@ class Launcher { new CmdPlugin(), new CmdInspect(), new CmdLint(), - new CmdLineage() + new CmdLineage(), + new CmdModule() ] if(SecretsLoader.isEnabled()) @@ -129,6 +130,9 @@ class Launcher { jcommander.addCommand(cmd.name, cmd, aliases(cmd)) } jcommander.setProgramName( APP_NAME ) + + //Allow unknown options for module command + jcommander.getCommands().get(CmdModule.NAME)?.setAcceptUnknownOptions(true) } private static final String[] EMPTY = new String[0] @@ -154,6 +158,11 @@ class Launcher { jcommander.parse( normalizedArgs as String[] ) fullVersion = '-version' in normalizedArgs command = allCommands.find { it.name == jcommander.getParsedCommand() } + //Attach unknown options to command in case of needed + if (command) { + final unknownOptions = jcommander.commands.get(jcommander.getParsedCommand())?.getUnknownOptions() ?: [] + command.setUnknownOptions(unknownOptions) + } // whether is running a daemon daemonMode = command instanceof CmdNode // set the log file name diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy new file mode 100644 index 0000000000..26a438a682 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInfo.groovy @@ -0,0 +1,342 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.IParameterValidator +import com.beust.jcommander.Parameter +import com.beust.jcommander.ParameterException +import com.beust.jcommander.Parameters +import groovy.json.JsonOutput +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.npr.api.schema.v1.ModuleChannel +import io.seqera.npr.api.schema.v1.ModuleChannelItem +import io.seqera.npr.api.schema.v1.ModuleMetadata +import io.seqera.npr.api.schema.v1.ModuleRelease +import io.seqera.npr.api.schema.v1.ModuleTool +import nextflow.cli.CmdBase +import nextflow.config.ConfigBuilder +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.util.TestOnly + +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module info subcommand - displays module metadata and usage template + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "Show module information and usage template") +class CmdModuleInfo extends CmdBase { + + @Parameter(names = ["-version"], description = "Module version") + String version + + @Parameter( + names = ['-o', '-output'], + description = 'Output mode for reporting search results: text, json', + validateWith = OutputModeValidator + ) + String output = 'text' + + static class OutputModeValidator implements IParameterValidator { + + private static final List MODES = List.of('text', 'json') + + @Override + void validate(String name, String value) { + if( !MODES.contains(value) ) + throw new ParameterException("Output mode must be one of $MODES (found: $value)") + } + } + + @Parameter(description = "[scope/name]", required = true) + List args + + @TestOnly + protected Path root + + @TestOnly + protected ModuleRegistryClient client + + @Override + String getName() { + return 'info' + } + + @Override + void run() { + if( !args || args.size() != 1 ) { + throw new AbortOperationException("Incorrect number of arguments") + } + + def reference = ModuleReference.parse(args[0]) + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(baseDir) + .build() + final registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + + // Fetch full metadata from registry to get input/output parameters + def registryClient = this.client ?: new ModuleRegistryClient(registryConfig) + ModuleRelease release = null + + try { + if( version ) { + release = registryClient.fetchRelease(reference.fullName, version) + + } else { + release = registryClient.fetchModule(reference.fullName).latest + } + } catch( Exception e ) { + log.warn "Failed to fetch metadata from registry: ${e.message}" + } + if( !release ) { + throw new AbortOperationException("No release information available for ${reference}") + } + if( !release.metadata ) { + log.info("No metadata found for $reference ${release.version ? "($release.version)" : ''}") + } + def moduleUrl = buildModuleUrl(registryConfig.url, reference, release.version) + if( !output || output == 'text' ) { + printFormattedInfo(reference, release, moduleUrl) + } else if( output == 'json' ) { + printJsonInfo(reference, release, moduleUrl) + } else { + throw new AbortOperationException("Not implemented output mode $output)") + } + } + + private void printFormattedInfo(ModuleReference reference, ModuleRelease release, String moduleUrl) { + ModuleMetadata metadata = release.metadata + println "" + println "Module: ${reference}" + println "Version: ${release.version}" + println "URL: ${moduleUrl}" + println "Description: ${metadata.description ?: release.description ?: 'N/A'}" + + if( metadata.authors ) { + println "Authors: ${metadata.authors.join(', ')}" + } + + if( metadata.maintainers ) { + println "Maintainers: ${metadata.maintainers.join(', ')}" + } + + if( metadata.keywords ) { + println "Keywords: ${metadata.keywords.join(', ')}" + } + + printToolsInfo(metadata?.tools ?: []) + + printInputsInfo(metadata.input ?: []) + + printOutputsInfo(metadata.output ?: [:]) + + // Generate and display usage template + println "" + println "Usage Template:" + println "-" * 80 + println generateUsageTemplate(reference, metadata).join(" \\\n ") + println "" + } + + private void printOutputsInfo(Map outputs) { + if( outputs ) { + println "" + println "Output:" + outputs.each { name, output -> + println "- ${name} ${output.tuple ? '(tuple)' : ''}" + displayChannel("\t", output) + } + } + } + + private void printInputsInfo(List inputs) { + if( inputs ) { + println "" + println "Input:" + inputs.each { input -> + if( input.tuple ) { + println "- (tuple)" + displayChannel("\t", input) + } else + displayChannel("", input) + } + } + } + + private void printToolsInfo(List toolsList) { + if( toolsList ) { + println "" + println "Tools:" + toolsList.each { tool -> + println " - ${tool.name}${tool.version ? ' v' + tool.version : ''}" + if( tool.homepage ) { + println " Homepage: ${tool.homepage}" + } + } + } + } + + private void displayChannel(String prefix, ModuleChannel channel) { + channel.items.each { ModuleChannelItem item -> + println "${prefix}- ${item.name}${item.type ? ' (' + item.type + ')' : ''}" + if( item.description ) { + println "${prefix}\t${item.description.replaceAll(/\R/, ' ')}" + } + if( item.pattern ) { + println "${prefix}\tPattern: ${item.pattern}" + } + } + } + + private List generateUsageTemplate(ModuleReference reference, ModuleMetadata metadata) { + def template = new ArrayList() + template.add("nextflow module run ${reference}".toString()) + if( version ) + template.add(" -version $version".toString()) + + def inputs = metadata?.input ?: [] + inputs.each { input -> + input.items.each { ModuleChannelItem item -> + template.add(reference.scope == 'nf-core' + ? inferNfCoreParam(item.name, item.type) + : inferNormalParam(item.name, item.type)) + + } + } + if( reference.scope == 'nf-core' ) { + template.add('--outdir ') + } + return template + } + + private static String inferNfCoreParam(String paramName, String type) { + if( type?.equalsIgnoreCase("map") && paramName.equalsIgnoreCase("meta") ) { + return "--${paramName}.id " + } + if( type?.equalsIgnoreCase("file") || type?.equalsIgnoreCase("path") ) { + return "--${paramName} ${inferBioFilePlaceholder(paramName)}" + } + return inferNormalParam(paramName, type) + } + + private static String inferBioFilePlaceholder(String paramName) { + final String lower = paramName.toLowerCase() + if( lower.contains("fasta") ) return "" + if( lower.contains("bam") ) return "" + if( lower.contains("fastq") || lower.equals("reads") ) return "" + if( lower.contains("vcf") ) return "" + if( lower.contains("ref") ) return "" + if( lower.contains("bed") ) return "" + if( lower.contains("gff") || lower.contains("gtf") ) return "" + + return "<${paramName.toUpperCase().replaceAll(/[^A-Z0-9]/, '_')}_PATH>" + } + + private static String inferNormalParam(String paramName, String type) { + final paramPlaceholder = paramName.toUpperCase().replaceAll(/[^A-Z0-9]/, '_') + if( type?.equalsIgnoreCase("map") ) { + return "--${paramName}. <${paramPlaceholder}_KEY_VALUE>" + } + if( type?.equalsIgnoreCase("file") || type?.equalsIgnoreCase("path") ) { + return "--${paramName} <${paramPlaceholder}_PATH>" + } + return "--${paramName} <${paramPlaceholder}>" + } + + private static String buildModuleUrl(String registryUrl, ModuleReference reference, String version) { + // Strip /api suffix to get the base UI URL + def baseUrl = registryUrl.endsWith('/api') ? registryUrl[0..-5] : registryUrl + def encodedName = URLEncoder.encode(reference.name, 'UTF-8') + return "${baseUrl}/admin/modules/${reference.scope}/${encodedName}@${version}" + } + + private void printJsonInfo(ModuleReference reference, ModuleRelease release, String moduleUrl) { + def metadata = release?.metadata + def info = [ + name : reference.toString(), + fullName : reference.fullName, + version : release.version, + url : moduleUrl, + description: metadata.description ?: release.description, + authors : metadata.authors, + keywords : metadata.keywords, + ] + + def toolsList = metadata.tools ?: [] + if( toolsList ) { + info.tools = toolsList.collect { tool -> + return [ + name : tool.name, + version : tool.version, + homepage : tool.homepage, + documentation: tool.documentation + ] + } + } + + def inputs = metadata.input ?: [] + if( inputs ) { + info.input = inputs.collect { input -> + return [ + tuple: input.tuple, + items: input.items?.collect { item -> + [ + name : item.name, + type : item.type, + description: item.description, + pattern : item.pattern + ] + } + ] + } + } + + def outputs = metadata.output ?: [:] + if( outputs ) { + info.output = outputs.collectEntries { name, output -> + return [name, [ + tuple: output.tuple, + items: output.items?.collect { item -> + [ + name : item.name, + type : item.type, + description: item.description, + pattern : item.pattern + ] + } + ]] + } + } + + info.usageTemplate = generateUsageTemplate(reference, metadata).join(" ") + + println JsonOutput.prettyPrint(JsonOutput.toJson(info)) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy new file mode 100644 index 0000000000..0a260c716f --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleInstall.groovy @@ -0,0 +1,99 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.Parameter +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.cli.CmdBase +import nextflow.config.ConfigBuilder + +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleResolver + +import nextflow.util.TestOnly + +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module install subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@Parameters(commandDescription = "Install a module from the registry") +@CompileStatic +class CmdModuleInstall extends CmdBase { + + @Parameter(names = ["-version"], description = "Module version") + String version + + @Parameter(names = ["-force"], description = "Force reinstall even if already installed", arity = 0) + boolean force = false + + @Parameter(description = "[scope/name]", required = true) + List args + + @TestOnly + protected Path root + + @TestOnly + protected ModuleRegistryClient client + + @Override + String getName() { + return 'install' + } + + @Override + void run() { + if( !args || args.size() != 1 ) { + throw new AbortOperationException("Incorrect number of arguments") + } + + def reference = ModuleReference.parse(args[0]) + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(baseDir) + .build() + final registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + // Create resolver and install + def resolver = new ModuleResolver(baseDir, client ?: new ModuleRegistryClient(registryConfig)) + + try { + def installedMainFile = resolver.installModule(reference, version, force) + def installedVersion = version ?: resolver.resolveVersion(reference) + + println "Module ${reference}@${installedVersion} installed and configured successfully" + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Installation failed: ${e.message}", e) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy new file mode 100644 index 0000000000..a9ecf87aca --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleList.groovy @@ -0,0 +1,148 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.IParameterValidator +import com.beust.jcommander.Parameter +import com.beust.jcommander.ParameterException +import com.beust.jcommander.Parameters +import groovy.json.JsonOutput +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.cli.CmdBase +import nextflow.exception.AbortOperationException +import nextflow.module.InstalledModule +import nextflow.module.ModuleIntegrity +import nextflow.module.ModuleStorage +import nextflow.util.TestOnly + +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module list subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "List all installed modules") +class CmdModuleList extends CmdBase { + + @Parameter( + names = ['-o', '-output'], + description = 'Output mode for reporting search results: table, json', + validateWith = OutputModeValidator + ) + String output = 'table' + + static class OutputModeValidator implements IParameterValidator { + + private static final List MODES = List.of('table', 'json') + + @Override + void validate(String name, String value) { + if( !MODES.contains(value) ) + throw new ParameterException("Output mode must be one of $MODES (found: $value)") + } + } + + @TestOnly + protected Path root + + @Override + String getName() { + return 'list' + } + + @Override + void run() { + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + + + // Create resolver and list modules + def storage = new ModuleStorage(baseDir) + + try { + def installed = storage.listInstalled() + + if( installed.isEmpty() ) { + println "No modules installed" + return + } + + if( !output || output == 'table' ) { + printFormattedList(installed) + } else if( output == 'json' ) { + printJsonList(installed) + } else { + throw new AbortOperationException("Not implemented output mode $output)") + } + + } + catch( Exception e ) { + log.error("Failed to list modules", e) + throw new AbortOperationException("List failed: ${e.message}", e) + } + } + + private void printFormattedList(List installed) { + println "" + println "Installed modules:" + println "" + println "Module".padRight(40) + "Version".padRight(15) + "Status" + println("-" * 70) + + installed.each { module -> + def status = getStatusString(module.integrity) + println "${module.reference.toString().padRight(40)}${(module.installedVersion ?: 'unknown').padRight(15)}${status}" + } + println "" + } + + private void printJsonList(List installed) { + def modules = installed.collect { module -> + [ + name : module.reference.toString(), + version : module.installedVersion ?: 'unknown', + integrity: module.integrity.toString(), + directory: module.directory.toString(), + registry : module.registryUrl ?: 'unknown' + ] + } + + // Simple JSON output (could use groovy.json.JsonOutput for better formatting) + println JsonOutput.prettyPrint(JsonOutput.toJson(modules: modules)) + } + + private String getStatusString(ModuleIntegrity integrity) { + switch( integrity ) { + case ModuleIntegrity.VALID: + return 'OK' + case ModuleIntegrity.MODIFIED: + return 'MODIFIED' + case ModuleIntegrity.NO_REMOTE_MODULE: + return 'LOCAL' + case ModuleIntegrity.CORRUPTED: + return 'CORRUPTED' + default: + return 'UNKNOWN' + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy new file mode 100644 index 0000000000..60f59649bb --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModulePublish.groovy @@ -0,0 +1,268 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.Parameter +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.Const +import nextflow.cli.CmdBase +import nextflow.config.ConfigBuilder +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleChecksum +import nextflow.module.ModuleInfo +import nextflow.module.ModuleSpec +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleStorage +import nextflow.util.TestOnly + +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module publish subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "Publish a module to the registry") +class CmdModulePublish extends CmdBase { + + @Parameter(names = ["-dry-run"], description = "Validate without uploading", arity=0) + boolean dryRun = false + + @Parameter(names = ["-registry"], description = "Target registry URL.") + String registryUrl + + @Parameter(description = "Module directory path or scope/name") + List args + + @TestOnly + protected Path root + + @TestOnly + protected ModuleRegistryClient client + + //Flag if publish is invoked from a scope/name. In this case we should create/update the .module-info with the correct checksum + private boolean useModuleReference = false + + @Override + String getName() { + return 'publish' + } + + @Override + void run() { + if (!args || args.size() != 1) { + throw new AbortOperationException("Incorrect number of arguments") + } + + Path moduleDir = determineModuleDir(args[0]) + + log.info "Publishing module from: ${moduleDir}" + + // Step 1: Validate module structure + def validationErrors = validateModuleStructure(moduleDir) + if (!validationErrors.isEmpty()) { + throw new AbortOperationException( + "Module validation failed:\n" + validationErrors.collect { " - ${it}" }.join('\n') + ) + } + + // Step 2: Load and validate spec + def manifestPath = moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE) + def spec = ModuleSpec.load(manifestPath) + + def manifestErrors = spec.validate() + if (!manifestErrors.isEmpty()) { + throw new AbortOperationException( + "Module spec validation failed:\n" + manifestErrors.collect { " - ${it}" }.join('\n') + ) + } + + log.info "Module validated: ${spec.name}@${spec.version}" + + if (dryRun) { + printDryRunInfo(spec) + return + } + + // Step 3: Get authentication token + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(moduleDir) + .build() + + def registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + publishModule(moduleDir, registryConfig, spec) + + } + + private void publishModule(Path moduleDir, RegistryConfig registryConfig, ModuleSpec spec){ + log.info "Creating module bundle..." + def tempBundleFile = Files.createTempFile("nf-module-publish-", ".tar.gz") + + try { + def checksum = ModuleStorage.createBundle(moduleDir, tempBundleFile) + log.info "Bundle checksum: ${checksum}" + + // Read bundle content as bytes + def bundleBytes = Files.readAllBytes(tempBundleFile) + + // Create publish request as a map (npr-api will serialize it) + def request = [ + version: spec.version, + bundle: bundleBytes + ] + + // Publish to registry + final registry = registryUrl ?: registryConfig.url + log.info "Publishing module to registry: ${registryUrl ?: registryConfig.url}" + def registryClient = new ModuleRegistryClient(registryConfig) + def response = registryClient.publishModule(spec.name, request, registry) + + if (useModuleReference) { + // If publish is performed using the module reference we should create/update the .module-info with the correct checksum + try { + ModuleInfo.save(moduleDir, [checksum: ModuleChecksum.compute(moduleDir), registryUrl: registry] ) + }catch (Exception e){ + log.warn("Unable to save the checksum - ${e.message}") + } + } + println "✓ Module published successfully!" + println "" + println "Module details:" + println " Name: ${spec.name}" + println " Version: ${spec.version}" + println " DownloadUrl: ${response.downloadUrl}" + + println "" + println "Others can now install this module using:" + println " nextflow module install ${spec.name}" + + } finally { + // Clean up temporary bundle file + if (Files.exists(tempBundleFile)) { + try { + Files.delete(tempBundleFile) + } catch (Exception e) { + log.warn "Failed to clean up temporary bundle file: ${e.message}" + } + } + } + } + + private void printDryRunInfo(ModuleSpec spec) { + println "✓ Module structure is valid" + println "" + println "Module details:" + println " Name: ${spec.name}" + println " Version: ${spec.version}" + println " Description: ${spec.description}" + println " License: ${spec.license}" + if( spec.authors ) { + println " Authors: ${spec.authors.join(', ')}" + } + if( spec.keywords ) { + println " Keywords: ${spec.keywords.join(', ')}" + } + if( spec.requires ) { + println " Requires:" + spec.requires.each { name, version -> + println " - ${name}: ${version}" + } + } + println "" + println "Dry run complete. Module is ready to publish." + println "Run without --dry-run to publish to the registry." + } + + /** + * Validate that the module directory has the required structure + * + * @param moduleDir The module directory path + * @return List of validation error messages (empty if valid) + */ + private List validateModuleStructure(Path moduleDir) { + List errors = [] + + if (!Files.exists(moduleDir) || !Files.isDirectory(moduleDir)) { + errors << "Module directory does not exist: ${moduleDir}".toString() + return errors + } + + // Check for required files + def mainNf = moduleDir.resolve(Const.DEFAULT_MAIN_FILE_NAME) + if (!Files.exists(mainNf)) { + errors << "Missing required file: $Const.DEFAULT_MAIN_FILE_NAME".toString() + } + + def metaYaml = moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE) + if (!Files.exists(metaYaml)) { + errors << "Missing required file: $ModuleStorage.MODULE_MANIFEST_FILE".toString() + } + + def readme = moduleDir.resolve(ModuleStorage.MODULE_README_FILE) + if (!Files.exists(readme)) { + errors << "Missing required file: $ModuleStorage.MODULE_README_FILE".toString() + } + + // Check bundle size (1MB uncompressed limit) + try (final sizeStream = Files.walk(moduleDir)){ + long totalSize = sizeStream + .filter { Files.isRegularFile(it) } + .mapToLong { Files.size(it) } + .sum() + + def maxSize = 1024 * 1024 // 1MB in bytes + if (totalSize > maxSize) { + def sizeMB = totalSize / (1024 * 1024) + errors << "Module size exceeds 1MB limit (current: ${String.format('%.2f', sizeMB)}MB)".toString() + } + } catch (Exception e) { + log.warn "Failed to check module size: ${e.message}" + } + + return errors + } + /** + * Determine if the specified module is a local path or a reference + * @param module + * @return + */ + private Path determineModuleDir(String module) { + //If local path exists return this path as module dir + if (Paths.get(module).exists()){ + return Paths.get(module).toAbsolutePath().normalize() + } + + final ref = ModuleReference.parse(module) + final localStorage = new ModuleStorage(root ?: Paths.get('.').toAbsolutePath().normalize()) + + if (!localStorage.isInstalled(ref)){ + throw new AbortOperationException("No module diretory found for $module") + } + useModuleReference = true + return localStorage.getModuleDir(ref) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy new file mode 100644 index 0000000000..cfcea1b1f4 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRemove.groovy @@ -0,0 +1,106 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.Parameter +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.cli.CmdBase +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleInfo +import nextflow.module.ModuleReference +import nextflow.module.ModuleStorage + +import nextflow.util.TestOnly + +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module remove subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "Remove an installed module") +class CmdModuleRemove extends CmdBase { + + @Parameter(description = "", required = true) + List args + + @Parameter(names = ["-keep-files"], description = "Remove only .module-info keeping the local files", arity = 0) + boolean keepFiles = false + + @Parameter(names = ["-force"], description = "Force remove", arity = 0) + boolean force = false + + @TestOnly + protected Path root + + @Override + String getName() { + return 'remove' + } + + @Override + void run() { + if( !args || args.size() != 1 ) { + throw new AbortOperationException("Incorrect number of arguments") + } + if( keepFiles && force ) { + throw new AbortOperationException("Cannot use both -keep-files and -force options") + } + + def reference = ModuleReference.parse(args[0]) + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + + // Create resolver and spec file manager + def storage = new ModuleStorage(baseDir) + + try { + def filesRemoved = false + + // Remove local files unless -keep-files is set + if( !keepFiles ) { + filesRemoved = storage.removeModule(reference, force) + if( filesRemoved ) { + println "Module ${reference} files removed successfully" + } else { + println "Module ${reference} not found locally" + } + } else { + println "Keeping module files for ${reference} (-keep-files flag)" + final moduleInfo = storage.getModuleInfo(reference) + if( Files.exists(moduleInfo) ) { + Files.delete(moduleInfo) + } + } + + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to remove module $reference: ${e.message}", e) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy new file mode 100644 index 0000000000..d09bbd0ee4 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleRun.groovy @@ -0,0 +1,88 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.Parameter +import com.beust.jcommander.Parameters +import groovy.transform.CompileStatic +import nextflow.cli.CmdRun +import nextflow.config.ConfigBuilder + +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleResolver + +import nextflow.util.TestOnly + +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Module run subcommand + * + * @author Jorge Ejarque + */ +@CompileStatic +@Parameters(commandDescription = "Run a module directly from the registry") +class CmdModuleRun extends CmdRun { + @Parameter(names = ["-version"], description = "Module version") + String version + + @TestOnly + protected Path root + + @TestOnly + protected ModuleRegistryClient client + + @Override + String getName() { + return 'run' + } + + @Override + void run() { + if( !args ) { + throw new AbortOperationException("Arguments not provided") + } + + // Parse and validate module reference + ModuleReference reference + try { + reference = ModuleReference.parse(args[0]) + } catch( Exception e ) { + throw new AbortOperationException("Invalid module reference: ${args[0]}", e) + } + + // Get config + def baseDir = root ?: Paths.get('.').toAbsolutePath().normalize() + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(baseDir) + .build() + + def registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + def resolver = new ModuleResolver(baseDir, client ?: new ModuleRegistryClient(registryConfig)) + Path moduleFile = resolver.installModule(reference, version) + if( moduleFile ) { + args[0] = moduleFile.toAbsolutePath().toString() + super.run() + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleSearch.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleSearch.groovy new file mode 100644 index 0000000000..2b55de24d8 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/cli/module/CmdModuleSearch.groovy @@ -0,0 +1,157 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import com.beust.jcommander.IParameterValidator +import com.beust.jcommander.Parameter +import com.beust.jcommander.ParameterException +import com.beust.jcommander.Parameters +import groovy.json.JsonOutput +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.npr.api.schema.v1.ModuleSearchResult +import io.seqera.npr.api.schema.v1.SearchModulesResponse +import nextflow.cli.CmdBase +import nextflow.config.ConfigBuilder +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleRegistryClient +import nextflow.util.TestOnly + +import java.nio.file.Paths + +/** + * Module search subcommand + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@Parameters(commandDescription = "Search for modules in the registry") +class CmdModuleSearch extends CmdBase { + + @Parameter(names = ["-limit"], description = "Maximum number of results") + int limit = 20 + + @Parameter( + names = ['-o', '-output'], + description = 'Output mode for reporting search results: simple, json', + validateWith = OutputModeValidator + ) + String output = 'simple' + + static class OutputModeValidator implements IParameterValidator { + + private static final List MODES = List.of('simple', 'json') + + @Override + void validate(String name, String value) { + if( !MODES.contains(value) ) + throw new ParameterException("Output mode must be one of $MODES (found: $value)") + } + } + + @Parameter(description = "", required = true) + List args + + @TestOnly + protected ModuleRegistryClient client + + @Override + String getName() { + return 'search' + } + + @Override + void run() { + if( !args || args.size() != 1 ) { + throw new AbortOperationException("Unexpected number of parameters") + } + String query = args[0] + + // Get config + def baseDir = Paths.get('.').toAbsolutePath().normalize() + def config = new ConfigBuilder() + .setOptions(launcher.options) + .setBaseDir(baseDir) + .build() + + final registryConfig = config.navigate('registry') as RegistryConfig ?: new RegistryConfig() + + // Create client to search + final client = this.client ?: new ModuleRegistryClient(registryConfig) + + try { + println "Searching for '${query}'..." + final results = client.search(query, limit) + + if( !results || results.totalResults == 0 || !results.results || results.results.isEmpty() ) { + println "No modules found" + return + } + + if( !output || output == 'simple' ) { + printFormattedResults(results) + } else if( output == 'json' ) { + printJsonResults(results) + } else { + throw new AbortOperationException("Not implemented output mode $output)") + } + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + log.error("Failed to search modules", e) + throw new AbortOperationException("Search failed: ${e.message}", e) + } + } + + private void printFormattedResults(SearchModulesResponse response) { + println "" + println "Found ${response.totalResults} module(s):" + println "" + + response.results.each { ModuleSearchResult result -> + println " ${result.name}" + if( result.description ) { + println " Description: ${result.description}" + } + println "" + } + } + + private void printJsonResults(SearchModulesResponse response) { + final modules = response.results.collect { ModuleSearchResult result -> + [ + name : result.name, + repositoryPath: result.repositoryPath, + description : result.description, + relevanceScore: result.relevanceScore, + keywords : result.keywords, + tools : result.tools, + revoked : result.revoked + ] + } + + println JsonOutput.prettyPrint(JsonOutput.toJson( + query: response.query, + totalResults: response.totalResults, + results: modules + )) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy b/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy new file mode 100644 index 0000000000..82e17d735e --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy @@ -0,0 +1,76 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.Global +import nextflow.config.ConfigBuilder + +import nextflow.config.RegistryConfig +import nextflow.exception.IllegalModulePath +import nextflow.module.spi.RemoteModuleResolver + +import java.nio.file.Path + +/** + * Default implementation of RemoteModuleResolver using the Nextflow module registry. + * + *

This implementation: + *

    + *
  • Checks for locally installed modules in the project's modules directory
    • + *
  • Downloads modules from the configured registry if not present
    • + *
  • Reads version constraints from nextflow_spec.json
    • + *
  • Uses the Session's registry configuration
    • + *
+ * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class DefaultRemoteModuleResolver implements RemoteModuleResolver { + + @Override + Path resolve(String moduleName, Path baseDir) { + + final config = Global.config ?: new ConfigBuilder().setBaseDir(baseDir).build() + final registryConfig = config.navigate('registry') as RegistryConfig + + // Create module resolver + def resolver = new ModuleResolver(baseDir, registryConfig) + + try { + log.debug "Resolving remote module: ${moduleName}" + + // Parse module reference + def reference = ModuleReference.parse(moduleName) + + // Resolve module (will auto-install if missing or version mismatch) + def mainFile = resolver.resolve(reference, null, true) + + log.debug "Module ${reference} resolved to ${mainFile}" + return mainFile + } catch (Exception e) { + throw new IllegalModulePath("Failed to resolve remote module ${moduleName}: ${e.message}", e) + } + } + + @Override + int getPriority() { + return 0 // Default implementation has lowest priority + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy b/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy new file mode 100644 index 0000000000..e951df7c59 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/InstalledModule.groovy @@ -0,0 +1,90 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.transform.ToString +import groovy.util.logging.Slf4j +import io.seqera.npr.api.schema.v1.ModuleMetadata +import org.yaml.snakeyaml.Yaml + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Represents a module installed in the local modules/ directory + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +@ToString(includeNames = true) +class InstalledModule { + + ModuleReference reference + Path directory + Path mainFile + Path manifestFile + Path moduleInfoFile + String installedVersion + String expectedChecksum + String registryUrl + + /** + * Get the integrity status of this installed module + * + * @return ModuleIntegrity status + */ + ModuleIntegrity getIntegrity() { + // Check if main.nf exists + if( !Files.exists(mainFile) || !Files.exists(manifestFile) ) { + return ModuleIntegrity.CORRUPTED + } + + // Check if .module-info file exists + if( !Files.exists(moduleInfoFile) ) { + return ModuleIntegrity.NO_REMOTE_MODULE + } + + try { + // Compute actual checksum + def actualChecksum = ModuleChecksum.compute(directory) + + // Compare with expected + if( actualChecksum == expectedChecksum ) { + return ModuleIntegrity.VALID + } else { + log.debug("Actual: $actualChecksum, expected: $expectedChecksum") + return ModuleIntegrity.MODIFIED + } + } catch( Exception e ) { + log.warn "Failed to compute checksum for module ${reference}: ${e.message}" + return ModuleIntegrity.CORRUPTED + } + } +} + +/** + * Module integrity status + */ +@CompileStatic +enum ModuleIntegrity { + VALID, // Checksum matches + MODIFIED, // Checksum mismatch (local changes) + NO_REMOTE_MODULE, // No .module-info file (local-only module, no registry origin) + CORRUPTED // Missing required files +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy new file mode 100644 index 0000000000..0693b3155c --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleChecksum.groovy @@ -0,0 +1,164 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j + +import java.nio.file.Files +import java.nio.file.Path +import java.security.MessageDigest + +import static nextflow.module.ModuleInfo.MODULE_INFO_FILE + +/** + * Utility class for computing SHA-256 checksums of module directories + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleChecksum { + + public static final String CHECKSUM_ALGORITHM = "SHA-256" + + /** + * Compute the SHA-256 checksum of a module directory + * + * @param moduleDir The module directory path + * @return The hex-encoded SHA-256 checksum + */ + static String compute(Path moduleDir) { + if( !Files.exists(moduleDir) || !Files.isDirectory(moduleDir) ) { + throw new IllegalArgumentException("Module directory does not exist or is not a directory: ${moduleDir}") + } + + try { + def digest = MessageDigest.getInstance(CHECKSUM_ALGORITHM) + + // Collect all files in sorted order for consistent checksums + List files = [] + try( final walkStream = Files.walk(moduleDir) ) { + walkStream + .filter { Path path -> Files.isRegularFile(path) } + .filter { Path path -> !path.fileName.toString().equals(MODULE_INFO_FILE) } + .sorted() + .each { Path path -> files.add(path) } + } + + // Compute checksum over all file contents + byte[] buf = new byte[8192] + for( Path file : files ) { + // Include relative path in checksum for directory structure integrity + def relativePath = moduleDir.relativize(file).toString() + digest.update(relativePath.bytes) + + // Include file contents via streaming to avoid loading large files into memory + Files.newInputStream(file).withCloseable { is -> + int n + while( (n = is.read(buf)) != -1 ) { + digest.update(buf, 0, n) + } + } + } + + def hashBytes = digest.digest() + return bytesToHex(hashBytes) + } + catch( Exception e ) { + log.error("Failed to compute checksum for module directory: ${moduleDir}", e) + throw new RuntimeException("Failed to compute module checksum", e) + } + } + + /** + * Save a checksum to the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @param checksum The checksum to save + */ + static void save(Path moduleDir, String checksum) { + ModuleInfo.save(moduleDir,'checksum', checksum) + } + + /** + * Load a checksum from the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @return The checksum, or null if file doesn't exist + */ + static String load(Path moduleDir) { + ModuleInfo.load(moduleDir, 'checksum') + } + + /** + * Verify that a module directory matches the expected checksum + * + * @param moduleDir The module directory path + * @param expectedChecksum The expected checksum + * @return true if checksums match, false otherwise + */ + static boolean verify(Path moduleDir, String expectedChecksum) { + def actualChecksum = compute(moduleDir) + return actualChecksum == expectedChecksum + } + + /** + * Compute the checksum of a single file + * + * @param file The file path + * @param type checksum algorithm (sha-256 if not provided) + * @return The hex-encoded checksum + */ + static String computeFile(Path file, String type = CHECKSUM_ALGORITHM) { + if( !Files.exists(file) || !Files.isRegularFile(file) ) { + throw new IllegalArgumentException("File does not exist or is not a regular file: ${file}") + } + + try { + final digest = MessageDigest.getInstance(type) + final byte[] buf = new byte[8192] + Files.newInputStream(file).withCloseable { is -> + int n + while( (n = is.read(buf)) != -1 ) { + digest.update(buf, 0, n) + } + } + return bytesToHex(digest.digest()) + } + catch( Exception e ) { + log.error("Failed to compute checksum for file: ${file}", e) + throw new RuntimeException("Failed to compute file checksum", e) + } + } + + /** + * Convert byte array to hex string + * + * @param bytes The byte array + * @return Hex-encoded string + */ + private static String bytesToHex(byte[] bytes) { + def hexChars = new char[bytes.length * 2] + for( int i = 0; i < bytes.length; i++ ) { + int v = bytes[i] & 0xFF + hexChars[i * 2] = Character.forDigit(v >>> 4, 16) + hexChars[i * 2 + 1] = Character.forDigit(v & 0x0F, 16) + } + return new String(hexChars) + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleInfo.groovy new file mode 100644 index 0000000000..a0fd26511f --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleInfo.groovy @@ -0,0 +1,109 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Utility class for managing .module-info + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleInfo { + + public static final String MODULE_INFO_FILE = ".module-info" + + + /** + * Save a property to the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @param property The property to save + * @param value The property value to save + */ + static void save(Path moduleDir, String property, String value) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + def props = new Properties() + // If file exists loads to update current just checksum property + if( Files.exists( moduleInfoFile)) + moduleInfoFile.withInputStream { is -> props.load(is) } + props.setProperty(property, value) + moduleInfoFile.withOutputStream { os -> props.store(os, null) } + } + + /** + * Save a property to the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @param properties Map with properties to save + */ + static void save(Path moduleDir, Map properties) { + if( properties ) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + def props = new Properties() + // If file exists loads to update current just checksum property + if( Files.exists( moduleInfoFile ) ) + moduleInfoFile.withInputStream { is -> props.load(is) } + + for( final property : properties.entrySet() ) { + props.setProperty(property.key, property.value) + } + moduleInfoFile.withOutputStream { os -> props.store(os, null) } + } + } + + /** + * Return the value of property from the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @param moduleDir The module directory path + * @return The checksum, or null if file doesn't exist + */ + static String load(Path moduleDir, String property) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + if( !Files.exists(moduleInfoFile) ) { + log.debug("Module file $moduleInfoFile not found") + return null + } + def props = new Properties() + moduleInfoFile.withInputStream { is -> props.load(is) } + return props.getProperty(property) + } + + /** + * Load all properties from the .module-info file in the module directory + * + * @param moduleDir The module directory path + * @return The checksum, or null if file doesn't exist + */ + static Map load(Path moduleDir) { + def moduleInfoFile = moduleDir.resolve(MODULE_INFO_FILE) + if( !Files.exists(moduleInfoFile) ) { + log.debug("Module file $moduleInfoFile not found") + return [:] + } + def props = new Properties() + moduleInfoFile.withInputStream { is -> props.load(is) } + return props as Map + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy new file mode 100644 index 0000000000..9cb0225e72 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleReference.groovy @@ -0,0 +1,80 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.transform.EqualsAndHashCode +import nextflow.exception.AbortOperationException + +import java.util.regex.Pattern + +/** + * Represents a reference to a module in DSL include statements + * + * @author Jorge Ejarqe + */ +@CompileStatic +@EqualsAndHashCode +class ModuleReference { + + // Pattern allows: scope with letters/digits/hyphens/dots/underscores, name segments separated by slashes (no trailing slash) + // Scope: starts with letter/digit, followed by letters/digits/dots/underscores/hyphens + // Name: one or more segments (each starting with letter, followed by letters/digits/underscores/hyphens), separated by slashes + private static final Pattern MODULE_NAME_PATTERN = ~/^([a-z0-9][a-z0-9._\-]*)\/([a-z][a-z0-9._\-]*(?:\/[a-z][a-z0-9._\-]*)*)$/ + + final String scope + final String name + final String fullName + + ModuleReference(String scope, String name) { + this.scope = scope + this.name = name + this.fullName = "${scope}/${name}" + } + + /** + * Parse a module reference from a string as "scope/name" + * + * @param source The module reference string + * @return A ModuleReference object + * @throws AbortOperationException if the format is invalid + */ + static ModuleReference parse(String source) { + if( !source ) { + throw new AbortOperationException("Module reference cannot be empty") + } + + // Trim whitespace + source = source.trim() + + def matcher = MODULE_NAME_PATTERN.matcher(source) + if( !matcher.matches() ) { + throw new AbortOperationException( + "Invalid module reference: '${source}'. " + + "Expected format: scope/name where scope is lowercase alphanumeric with dots/underscores/hyphens " + + "and name is lowercase alphanumeric with underscores/hyphens, optionally with slash-separated segments" + ) + } + + return new ModuleReference(matcher.group(1), matcher.group(2)) + } + + @Override + String toString() { + return fullName + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy new file mode 100644 index 0000000000..0528b17609 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy @@ -0,0 +1,494 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import com.google.gson.Gson +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.http.HxClient +import io.seqera.npr.api.schema.v1.Module +import io.seqera.npr.api.schema.v1.ModuleRelease +import io.seqera.npr.api.schema.v1.PublishModuleResponse +import io.seqera.npr.api.schema.v1.SearchModulesResponse +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.serde.gson.GsonEncoder +import nextflow.util.RetryConfig + +import java.net.http.HttpClient +import java.net.http.HttpRequest +import java.net.http.HttpResponse +import java.nio.file.Files +import java.nio.file.Path + +/** + * REST API client for Nextflow module registry using npr-api models + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleRegistryClient { + + private final RegistryConfig config + private final HxClient httpClient + + ModuleRegistryClient(RegistryConfig config) { + this.config = config ?: new RegistryConfig() + this.httpClient = HxClient.newBuilder() + .retryConfig(RetryConfig.config()) + .followRedirects(HttpClient.Redirect.NORMAL) + .build() + } + + private String encodeName(String name) { + return URLEncoder.encode(name, 'UTF-8') + } + + /** + * Fetch module metadata from the registry + * + * @param name The module name (e.g., "nf-core/fastqc") + * @return Module object with metadata + */ + Module fetchModule(String name) { + def registryUrls = config.allUrls + + Exception lastError = null + for( String registryUrl : registryUrls ) { + log.debug "Trying to fetch from $registryUrl" + try { + return fetchModuleFromRegistry(registryUrl, name) + } catch( Exception e ) { + log.debug "Failed to fetch module from ${registryUrl}: ${e.message}" + lastError = e + } + } + + throw new AbortOperationException( + "Unable to fetch module ${name} from any configured registry", + lastError + ) + } + + /** + * Fetch module from a specific registry URL + */ + private Module fetchModuleFromRegistry(String registryUrl, String name) { + def endpoint = "${registryUrl}/v1/modules/${encodeName(name)}" + def uri = URI.create(endpoint) + + def requestBuilder = HttpRequest.newBuilder() + .uri(uri) + .GET() + + // Add authentication if available + log.debug "Getting auth from: ${registryUrl}" + def token = config.getApiKey() + if( token ) { + requestBuilder.header("Authorization", "Bearer ${token}") + } + log.debug "Building request: ${registryUrl}" + def request = requestBuilder.build() + + try { + log.debug "Fetching module from: ${uri}" + def response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()) + def body = response.body() + + log.debug "Registry request: ${response.uri()}\n- code: ${response.statusCode()}\n- body: ${body}" + + if( response.statusCode() == 404 ) { + throw new AbortOperationException("Module not found: ${name}") + } + + if( response.statusCode() != 200 ) { + throw new AbortOperationException( + "Invalid response from registry: ${uri}\n" + + "- http status: ${response.statusCode()}\n" + + "- response: ${body}" + ) + } + + // Parse response using npr-api Module model + def encoder = new GsonEncoder() {} + return encoder.decode(body) + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to fetch module from: ${uri}", e) + } + } + + /** + * Fetch specific module version/release + * + * @param name The module name + * @param version The version string + * @return ModuleRelease object from npr-api + */ + ModuleRelease fetchRelease(String name, String version) { + def registryUrls = config.allUrls + + Exception lastError = null + for( String registryUrl : registryUrls ) { + try { + return fetchReleaseFromRegistry(registryUrl, name, version) + } catch( Exception e ) { + log.debug "Failed to fetch release from ${registryUrl}: ${e.message}" + lastError = e + } + } + + throw new AbortOperationException( + "Unable to fetch module ${name}@${version} from any configured registry", + lastError + ) + } + + /** + * Fetch release from a specific registry URL + */ + private ModuleRelease fetchReleaseFromRegistry(String registryUrl, String name, String version) { + def endpoint = "${registryUrl}/v1/modules/${encodeName(name)}/${version}" + def uri = URI.create(endpoint) + + def requestBuilder = HttpRequest.newBuilder() + .uri(uri) + .GET() + + def token = config.getApiKey() + if( token ) { + requestBuilder.header("Authorization", "Bearer ${token}") + } + + def request = requestBuilder.build() + + try { + log.debug "Fetching module release from: ${uri}" + def response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()) + def body = response.body() + + if( response.statusCode() == 404 ) { + throw new AbortOperationException("Module version not found: ${name}@${version}") + } + + if( response.statusCode() != 200 ) { + throw new AbortOperationException( + "Invalid response from registry: ${uri}\n" + + "- http status: ${response.statusCode()}\n" + + "- response: ${body}" + ) + } + + // Parse response using npr-api ModuleRelease model + return new GsonEncoder() {}.decode(body) + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to fetch module release from: ${uri}", e) + } + } + + /** + * Download a module bundle from the registry + * + * @param name The module name + * @param version The module version + * @param targetPath The target path to download to + * @return URL of the repository used to downloaded file path + */ + String downloadModule(String name, String version, Path targetPath) { + def registryUrls = config.allUrls + if( targetPath.exists() ) { + targetPath.delete() + } + Exception lastError = null + for( String registryUrl : registryUrls ) { + try { + return downloadModuleFromRegistry(registryUrl, name, version, targetPath) + } catch( Exception e ) { + log.debug "Failed to download from ${registryUrl}: ${e.message}" + lastError = e + } + } + + throw new AbortOperationException( + "Unable to download module ${name}@${version} from any configured registry", + lastError + ) + } + + /** + * Download module from a specific registry URL + */ + private String downloadModuleFromRegistry(String registryUrl, String name, String version, Path targetPath) { + def endpoint = "${registryUrl}/v1/modules/${encodeName(name)}/${version}/download" + def uri = URI.create(endpoint) + + def requestBuilder = HttpRequest.newBuilder() + .uri(uri) + .GET() + + def token = config.getApiKey() + if( token ) { + requestBuilder.header("Authorization", "Bearer ${token}") + } + + def request = requestBuilder.build() + + try { + log.debug "Downloading module from: ${uri}" + def response = httpClient.send(request, HttpResponse.BodyHandlers.ofInputStream()) + + if( response.statusCode() == 404 ) { + throw new AbortOperationException("Module bundle not found: ${name}@${version}") + } + + if( response.statusCode() != 200 ) { + throw new AbortOperationException( + "Invalid response from registry: ${uri}\n" + + "- http status: ${response.statusCode()}" + ) + } + + // Create parent directories if needed + if( targetPath.parent ) { + Files.createDirectories(targetPath.parent) + } + + // Write response body to file + Files.copy(response.body(), targetPath) + log.debug "Downloaded module to: ${targetPath}" + + validateDownloadIntegrity(response, uri, targetPath, name, version) + + return registryUrl + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to download module from: ${uri}", e) + } + } + + private void validateDownloadIntegrity(HttpResponse response, uri, Path targetPath, String name, String version) { + def checksumType = ModuleChecksum.CHECKSUM_ALGORITHM + def checksum = getChecksumFromHeaders(response) + + if( !checksum ) { + log.warn "No X-Checksum or Docker-Content-Digest header found in response from ${uri}" + return + } + + // Check if checksum has a digest format including algorithm: "sha256:abc123..." + def parts = checksum.split(':', 2) + if( parts.length == 2 ) { + checksumType = parts[0].toLowerCase() + checksum = parts[1] + } + log.debug "Using checksum: ${checksumType}:${checksum}" + + def actualChecksum = ModuleChecksum.computeFile(targetPath, checksumType) + if( actualChecksum != checksum ) { + // Clean up downloaded file + Files.delete(targetPath) + throw new AbortOperationException( + "Downloaded module checksum mismatch for ${name}@${version}:\n" + + "- expected (${checksumType}): ${checksum}\n" + + "- actual: ${actualChecksum}\n" + + "The download may be corrupted or tampered with." + ) + } + log.debug "Checksum validated successfully: ${checksumType}:${checksum}" + } + + private String getChecksumFromHeaders(HttpResponse response) { + // Get X-Checksum from response headers + def checksum = getChecksumFromHeader(response) + if( checksum ) { + return checksum + } + // If not look if it is a previous redirected response header + Optional> prev = response.previousResponse() + while( prev.isPresent() ) { + HttpResponse r = prev.get(); + checksum = getChecksumFromHeader(r) + if( checksum ) { + return checksum + } + prev = r.previousResponse(); + } + return null + } + + private String getChecksumFromHeader(HttpResponse response) { + def checksum = response.headers().firstValue("X-NF-Module-Checksum").orElse(null) + if( !checksum ) { + checksum = response.headers().firstValue("Docker-Content-Digest").orElse(null) + } + return checksum + } + + + /** + * Search for modules in the registry + * + * @param query The search query + * @param limit Maximum number of results (default: 20) + * @return SearchModulesResponse with results + */ + SearchModulesResponse search(String query, int limit = 20) { + def registryUrls = config.allUrls + + Exception lastError = null + for( String registryUrl : registryUrls ) { + try { + return searchInRegistry(registryUrl, query, limit) + } catch( Exception e ) { + log.debug "Failed to search in ${registryUrl}: ${e.message}" + lastError = e + } + } + + throw new AbortOperationException( + "Unable to search modules in any configured registry", + lastError + ) + } + + /** + * Search in a specific registry + */ + private SearchModulesResponse searchInRegistry(String registryUrl, String query, int limit) { + def endpoint = "${registryUrl}/v1/modules?query=${URLEncoder.encode(query, 'UTF-8')}&limit=${limit}" + def uri = URI.create(endpoint) + + def requestBuilder = HttpRequest.newBuilder() + .uri(uri) + .GET() + + def token = config.getApiKey() + if( token ) { + requestBuilder.header("Authorization", "Bearer ${token}") + } + + def request = requestBuilder.build() + + try { + log.debug "Searching modules: ${uri}" + def response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()) + def body = response.body() + + if( response.statusCode() != 200 ) { + throw new AbortOperationException( + "Invalid response from registry: ${uri}\n" + + "- http status: ${response.statusCode()}\n" + + "- response: ${body}" + ) + } + + // Parse response using npr-api SearchModulesResponse model + def encoder = new GsonEncoder() {} + return encoder.decode(body) + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to search modules in: ${uri}", e) + } + } + + /** + * Publish a module to the registry (authenticated) + * + * @param name The module name + * @param request The publish request from npr-api + * @param authToken The authentication token + * @return PublishModuleResponse from npr-api + */ + PublishModuleResponse publishModule(String name, def request, String registry = null) { + final registryUrl = registry ?: config.url + final authToken = config.apiKey + + if( !authToken ) { + throw new AbortOperationException( + "Authentication required to publish modules.\n" + + "Please set 'NXF_REGISTRY_TOKEN' environment variable or configure 'registry.apiKey' in nextflow.config:\n\n" + + " registry {\n" + + " apiKey = 'YOUR_REGISTRY_TOKEN'\n" + + " }\n" + ) + } + return publishModuleToRegistry(registryUrl, name, request, authToken) + } + + /** + * Publish module to a specific registry + */ + private PublishModuleResponse publishModuleToRegistry( + String registryUrl, + String name, + def request, + String authToken) { + + String endpoint = "${registryUrl}/v1/modules/${encodeName(name)}".toString() + URI uri = URI.create(endpoint) + + // Serialize request to JSON + def gson = new Gson() + String requestBody = gson.toJson(request) + + HttpRequest httpRequest = HttpRequest.newBuilder() + .uri(uri) + .header("Content-Type", "application/json") + .header("Authorization", "Bearer ${authToken}".toString()) + .POST(HttpRequest.BodyPublishers.ofString(requestBody)) + .build() + + try { + log.debug "Publishing module to: ${uri}" + log.trace "Request: \n\t${httpRequest}\n\theaders: ${httpRequest.headers()}\n\tbody: ${requestBody}" + + HttpResponse response = httpClient.send(httpRequest, HttpResponse.BodyHandlers.ofString()) + String body = response.body() + + if( response.statusCode() != 201 ) { + throw new AbortOperationException( + "Failed to publish module: ${uri}\n" + + "- http status: ${response.statusCode()}\n" + + "- response: ${body}" + ) + } + + // Parse response using npr-api PublishModuleResponse model + return new GsonEncoder() {}.decode(body) + } + catch( AbortOperationException e ) { + throw e + } + catch( Exception e ) { + throw new AbortOperationException("Failed to publish module to: ${uri}", e) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy new file mode 100644 index 0000000000..2c96a89f85 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleResolver.groovy @@ -0,0 +1,174 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j + +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Core module resolution logic that coordinates registry, storage, and version management + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleResolver { + + private final ModuleRegistryClient registryClient + private final ModuleStorage storage + + ModuleResolver(Path baseDir, ModuleRegistryClient registryClient) { + this.registryClient = registryClient + this.storage = new ModuleStorage(baseDir) + } + + ModuleResolver(Path baseDir, RegistryConfig registryConfig = null) { + this(baseDir, new ModuleRegistryClient(registryConfig ?: new RegistryConfig())) + + } + + /** + * Resolve a module reference to an installed module path + * + * @param reference The module reference + * @param version Optional specific version (null = use config or latest) + * @param autoInstall Whether to auto-install if not present (default: false) + * @return Path to the module's main.nf file + */ + Path resolve(ModuleReference reference, String version = null, boolean autoInstall = false) { + + // Check if module is already installed + def installed = storage.getInstalledModule(reference) + + if( installed ) { + // Check integrity + def integrity = installed.integrity + if( integrity == ModuleIntegrity.CORRUPTED ) { + throw new AbortOperationException( + "Module ${reference} is corrupted (missing required files). " + + "Please remove and reinstall." + ) + } + + if( integrity == ModuleIntegrity.MODIFIED ) { + log.warn1 "Module ${reference} has local modifications (checksum mismatch)" + } else if( integrity == ModuleIntegrity.NO_REMOTE_MODULE ) { + log.warn1 "Module ${reference} has no registry origin (.module-info missing)" + } + + // Check if version matches + if( version && installed.installedVersion != version ) { + if( autoInstall ) { + log.info "Upgrading module ${reference} from ${installed.installedVersion} to ${version}" + return installModule(reference, version) + } else { + throw new AbortOperationException( + "Module ${reference} version mismatch: " + + "installed=${installed.installedVersion}, required=${version}. " + + "Run 'nextflow module install ${reference}@${version}' to update." + ) + } + } + + // Module is installed and version matches + return installed.mainFile + } + + // Module not installed + if( autoInstall ) { + return installModule(reference, version) + } else { + throw new AbortOperationException( + "Module ${reference} is not installed. " + + "Run 'nextflow module install ${reference}' to install." + ) + } + } + + String resolveVersion(ModuleReference reference) { + final version = registryClient.fetchModule(reference.fullName)?.latest?.version + if( !version ) { + throw new AbortOperationException("Module ${reference} has no published versions") + } + return version + } + + /** + * Install or update a module + * + * @param reference The module reference + * @param version Optional specific version (null = latest) + * @param force Force reinstall even if already installed + * @return Path to the installed module's main.nf file + */ + Path installModule(ModuleReference reference, String version = null, boolean force = false) { + if( !version ) + version = resolveVersion(reference) + // Check if already installed + if( storage.isInstalled(reference) ) { + def installed = storage.getInstalledModule(reference) + if( installed.installedVersion == version ) { + log.debug "Module ${reference}@${installed.installedVersion} is already installed (version $version)" + return installed.mainFile + } + + // No desired version, check for local modifications + def integrity = installed.integrity + if( integrity == ModuleIntegrity.MODIFIED && !force ) { + throw new AbortOperationException( + "Module ${reference} has local modifications. " + + "Use '-force' to override, or save your changes first." + ) + } + if( integrity == ModuleIntegrity.NO_REMOTE_MODULE && !force ) { + throw new AbortOperationException( + "Folder 'modules/${reference}' already exists and is not a valid remote module. " + + "Use '-force' to override, or save your changes first." + ) + } + } + + + log.info "Installing module ${reference}@${version}..." + + // Download module package to temporary location + Path tempFile = Files.createTempFile("nf-module-", ".tgz") + try { + // Download and validate integrity using server checksum + def downloadUrl = registryClient.downloadModule(reference.fullName, version, tempFile) + + // Install to modules directory (will compute directory checksum for future integrity checks) + InstalledModule installed = storage.installModule(reference, version, tempFile, downloadUrl) + + log.info "Module ${reference}@${version} installed successfully at ${installed.mainFile.parent}" + return installed.mainFile + } + finally { + // Clean up temporary file + if( Files.exists(tempFile) ) { + Files.delete(tempFile) + } + } + } + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy new file mode 100644 index 0000000000..73ec7d96af --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleSpec.groovy @@ -0,0 +1,117 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.exception.AbortOperationException +import org.yaml.snakeyaml.Yaml + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Represents a module spec (meta.yml) with validation + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleSpec { + + String name + String version + String description + List authors + String license + List keywords + Map requires + + /** + * Load a module spec from a meta.yml file + * + * @param metaYamlPath Path to meta.yml + * @return ModuleSpec instance + */ + static ModuleSpec load(Path metaYamlPath) { + if( !Files.exists(metaYamlPath) ) { + throw new AbortOperationException("Module spec not found: ${metaYamlPath}") + } + + try { + def yaml = new Yaml() + def data = yaml.load(Files.newInputStream(metaYamlPath)) as Map + + def spec = new ModuleSpec() + spec.name = data.name as String + spec.version = data.version as String + spec.description = data.description as String + spec.authors = data.authors as List ?: [] + spec.license = data.license as String + spec.keywords = data.keywords as List ?: [] + spec.requires = data.requires as Map ?: [:] + + return spec + } + catch( Exception e ) { + throw new AbortOperationException("Failed to parse module spec: ${metaYamlPath}", e) + } + } + + /** + * Validate the module spec for required fields + * + * @return List of validation errors (empty if valid) + */ + List validate() { + List errors = [] + + if( !name ) { + errors << "Missing required field: name" + } + if( !version ) { + errors << "Missing required field: version" + } + if( !description ) { + errors << "Missing required field: description" + } + if( !license ) { + errors << "Missing required field: license" + } + + // Validate version format (semantic versioning) + if( version && !version.matches(/^\d+\.\d+\.\d+(-[\w.-]+)?$/) ) { + errors << "Invalid version format: ${version} (expected semantic versioning, e.g., 1.0.0)".toString() + } + + // Validate name format (scope/name or scope/path/to/name for nested modules) + if( name && !name.matches(/^[a-zA-Z0-9._-]+\/[a-zA-Z0-9_-]+(?:\/[a-zA-Z0-9_-]+)*$/) ) { + errors << "Invalid module name format: ${name} (expected scope/name or scope/path/to/name, e.g., nf-core/fastqc or nf-core/gfatools/gfa2fa)".toString() + } + + return errors + } + + /** + * Check if the spec is valid + * + * @return true if valid, false otherwise + */ + boolean isValid() { + return validate().isEmpty() + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy new file mode 100644 index 0000000000..2e1b90c1c1 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/module/ModuleStorage.groovy @@ -0,0 +1,461 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.Const +import nextflow.exception.AbortOperationException +import nextflow.file.FileHelper +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveInputStream +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream +import org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream +import org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream + +import java.nio.file.FileVisitResult +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.SimpleFileVisitor +import java.nio.file.attribute.BasicFileAttributes +import java.util.stream.Stream +import java.util.zip.ZipEntry +import java.util.zip.ZipInputStream + +import static nextflow.module.ModuleInfo.MODULE_INFO_FILE + +/** + * Manages local filesystem storage for modules + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class ModuleStorage { + public static final String MODULE_MANIFEST_FILE = "meta.yml" + public static final String MODULE_README_FILE = "README.md" + private final Path modulesDir + + /** + * Create a ModuleStorage instance + * + * @param baseDir The base directory (usually project root) + */ + ModuleStorage(Path baseDir) { + this.modulesDir = baseDir.resolve('modules') + } + + /** + * Get the modules directory path + * + * @return The modules directory + */ + Path getModulesDir() { + return modulesDir + } + + /** + * Get the directory path for a specific module + * + * @param reference The module reference + * @return The module directory path + */ + Path getModuleDir(ModuleReference reference) { + return modulesDir.resolve(reference.scope).resolve(reference.name) + } + + /** + * Get the module info path for a specific module + * + * @param reference The module reference + * @return The module info path + */ + Path getModuleInfo(ModuleReference reference) { + return modulesDir.resolve(reference.scope).resolve(reference.name).resolve(MODULE_INFO_FILE) + } + + /** + * Check if a module is installed locally + * + * @param reference The module reference + * @return true if the module directory exists + */ + boolean isInstalled(ModuleReference reference) { + def moduleDir = getModuleDir(reference) + return Files.exists(moduleDir) && Files.isDirectory(moduleDir) + } + + /** + * Get an installed module + * + * @param reference The module reference + * @return InstalledModule object, or null if not installed + */ + InstalledModule getInstalledModule(ModuleReference reference) { + def moduleDir = getModuleDir(reference) + if (!Files.exists(moduleDir) || !Files.isDirectory(moduleDir)) { + return null + } + + def installed = new InstalledModule( + reference: reference, + directory: moduleDir, + mainFile: moduleDir.resolve(Const.DEFAULT_MAIN_FILE_NAME), + manifestFile: moduleDir.resolve(MODULE_MANIFEST_FILE), + moduleInfoFile: moduleDir.resolve(MODULE_INFO_FILE), + ) + + // Load checksum if available + Map infoProps = ModuleInfo.load(moduleDir) + installed.expectedChecksum = infoProps?.checksum + installed.registryUrl = infoProps?.registryUrl + installed.installedVersion = ModuleSpec.load(installed.manifestFile).version + return installed + } + + /** + * List all installed modules by scanning for directories containing a .module-info marker file. + * + * @return List of InstalledModule objects + */ + List listInstalled() { + if (!Files.exists(modulesDir) || !Files.isDirectory(modulesDir)) { + return [] + } + + List modules = [] + + try { + Files.walkFileTree(modulesDir, new SimpleFileVisitor() { + @Override + FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) { + if( dir == modulesDir ) + return FileVisitResult.CONTINUE + if( !Files.exists(dir.resolve(MODULE_INFO_FILE)) ) + return FileVisitResult.CONTINUE + try { + final rel = modulesDir.relativize(dir) + if( rel.nameCount >= 2 ) { + final reference = ModuleReference.parse(rel.toString()) + final installed = getInstalledModule(reference) + if( installed ) modules.add(installed) + } + } catch(Exception e) { + // Catching exception to continue inspecting other valid folders + log.debug("Not a valid module reference: $e.message") + } + return FileVisitResult.SKIP_SUBTREE + } + }) + } catch (IOException e) { + log.warn "Failed to scan modules directory ${modulesDir}: ${e.message}" + } + + return modules + } + + /** + * Install a module from a downloaded package file + * + * @param reference The module reference + * @param version The module version + * @param packageFile The downloaded package file (zip or tar.gz) + * @return The InstalledModule object + */ + InstalledModule installModule(ModuleReference reference, String version, Path packageFile, String downloadUrl) { + def moduleDir = getModuleDir(reference) + + try { + // Remove existing installation if present + if (Files.exists(moduleDir)) { + log.debug "Removing existing module installation: ${moduleDir}" + FileHelper.deletePath(moduleDir) + } + + // Create module directory + Files.createDirectories(moduleDir) + + // Extract package - detect format by file extension + if (packageFile.toString().endsWith('.tgz') || packageFile.toString().endsWith('.tar.gz')) { + extractTarGz(packageFile, moduleDir) + } else { + extractZip(packageFile, moduleDir) + } + + // Compute and save checksum of extracted directory contents + // This checksum is used to detect local modifications + def checksum = ModuleChecksum.compute(moduleDir) + ModuleInfo.save(moduleDir, [checksum: checksum, registryUrl: downloadUrl]) + + log.debug "Installed module ${reference}@${version} to ${moduleDir}" + + return getInstalledModule(reference) + } + catch (Exception e) { + // Clean up on failure + if (Files.exists(moduleDir)) { + try { + FileHelper.deletePath(moduleDir) + } catch (Exception cleanupError) { + log.warn "Failed to clean up after installation failure: ${cleanupError.message}" + } + } + throw new AbortOperationException("Failed to install module ${reference}@${version}", e) + } + } + + /** + * Remove an installed module + * + * @param reference The module reference + * @param force Force local module folder + * @return true if module was removed, false if not installed + */ + boolean removeModule(ModuleReference reference, boolean force) { + final installed = getInstalledModule(reference) + if( !installed ) + return + final integrity = installed.integrity + if( integrity == ModuleIntegrity.NO_REMOTE_MODULE && !force ) { + throw new AbortOperationException( + "Folder 'modules/${reference}' already exists and is not a valid remote module ($MODULE_INFO_FILE missing). " + + "Use '-force' to remove, or save your changes first.") + } + + if( integrity == ModuleIntegrity.MODIFIED && !force ) { + throw new AbortOperationException( + "Module ${reference} has local modifications. " + + "Use '-force' to remove, or save your changes first." + ) + } + + try { + FileHelper.deletePath(installed.directory) + log.debug "Removed module: ${reference}" + + // Clean up empty scope directory + def scopeDir = installed.directory.parent + if (Files.exists(scopeDir) && isEmpty(scopeDir)) { + Files.delete(scopeDir) + } + + return true + } + catch (Exception e) { + throw new AbortOperationException("Failed to remove module ${reference}", e) + } + } + + /** + * Extract a zip file to a target directory + * + * @param zipFile The zip file path + * @param targetDir The target directory + */ + private void extractZip(Path zipFile, Path targetDir) { + Files.newInputStream(zipFile).withCloseable { fis -> + new ZipInputStream(fis).withCloseable { zis -> + ZipEntry entry + while ((entry = zis.nextEntry) != null) { + def targetPath = targetDir.resolve(entry.name) + + // Security check: prevent zip slip + if (!targetPath.normalize().startsWith(targetDir.normalize())) { + throw new AbortOperationException("Invalid zip entry: ${entry.name}") + } + + if (entry.directory) { + Files.createDirectories(targetPath) + } else { + // Create parent directories if needed + if (targetPath.parent) { + Files.createDirectories(targetPath.parent) + } + + // Write file + Files.copy(zis, targetPath) + } + + zis.closeEntry() + } + } + } + } + + /** + * Extract a tar.gz file to a target directory + * + * @param tarGzFile The tar.gz file path + * @param targetDir The target directory + */ + private void extractTarGz(Path tarGzFile, Path targetDir) { + Files.newInputStream(tarGzFile).withCloseable { fis -> + new GzipCompressorInputStream(fis).withCloseable { gzis -> + new TarArchiveInputStream(gzis).withCloseable { tis -> + TarArchiveEntry entry + while ((entry = tis.nextTarEntry) != null) { + def targetPath = targetDir.resolve(entry.name) + + // Security check: prevent tar slip + if (!targetPath.normalize().startsWith(targetDir.normalize())) { + throw new AbortOperationException("Invalid tar entry: ${entry.name}") + } + + if (entry.directory) { + Files.createDirectories(targetPath) + } else { + // Create parent directories if needed + if (targetPath.parent) { + Files.createDirectories(targetPath.parent) + } + + // Write file + Files.copy(tis, targetPath) + } + } + } + } + } + } + + /** + * Check if a directory is empty + * + * @param dir The directory to check + * @return true if directory is empty + */ + private boolean isEmpty(Path dir) { + if (!Files.exists(dir) || !Files.isDirectory(dir)) { + return true + } + + try { + Stream entries = Files.list(dir) + try { + return !entries.findFirst().isPresent() + } finally { + entries.close() + } + } catch (Exception e) { + log.warn "Failed to check if directory is empty: ${dir}", e + return false + } + } + + /** + * Create a module bundle (tar.gz) from a module directory for publishing + * + * @param moduleDir The module directory to bundle + * @param targetFile The target bundle file path + * @return The created bundle file checksum + */ + static String createBundle(Path moduleDir, Path targetFile) { + if (!Files.exists(moduleDir) || !Files.isDirectory(moduleDir)) { + throw new AbortOperationException("Module directory not found: ${moduleDir}") + } + + try { + // Create parent directories if needed + if (targetFile.parent) { + Files.createDirectories(targetFile.parent) + } + + // Create tar.gz bundle + Files.newOutputStream(targetFile).withCloseable { fos -> + new GzipCompressorOutputStream(fos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add all files in module directory to bundle + addToTarArchive(tos, moduleDir, moduleDir) + } + } + } + final checksum = computeBundleChecksum(targetFile) + log.debug "Created module bundle: ${targetFile} (size: ${Files.size(targetFile)} bytes, checksum: $checksum)" + return checksum + } + catch (Exception e) { + // Clean up partial file on failure + if (Files.exists(targetFile)) { + try { + Files.delete(targetFile) + } catch (Exception cleanupError) { + log.warn "Failed to clean up after bundle creation failure: ${cleanupError.message}" + } + } + throw new AbortOperationException("Failed to create module bundle", e) + } + } + + /** + * Add files to tar archive recursively + * + * @param tos The tar archive output stream + * @param sourceDir The source directory being archived + * @param currentPath The current path being added + */ + private static void addToTarArchive(TarArchiveOutputStream tos, Path sourceDir, Path currentPath) { + + try ( def tarStream = Files.list(currentPath)) { + tarStream.each { Path path -> + // Skip .module-info file when creating bundle + if (path.fileName.toString() == MODULE_INFO_FILE) { + return + } + + def relativePath = sourceDir.relativize(path).toString() + + if (Files.isDirectory(path)) { + // Add directory entry + def entry = new TarArchiveEntry(path.toFile(), "${relativePath}/") + tos.putArchiveEntry(entry) + tos.closeArchiveEntry() + + // Recursively add directory contents + addToTarArchive(tos, sourceDir, path) + } else { + // Add file entry + def entry = new TarArchiveEntry(path.toFile(), relativePath) + entry.setSize(Files.size(path)) + tos.putArchiveEntry(entry) + + // Copy file content + Files.copy(path, tos) + tos.closeArchiveEntry() + } + } + } + } + + /** + * Compute the checksum of a module bundle + * + * @param bundleFile The bundle file + * @return The SHA-256 checksum as hex string + */ + static String computeBundleChecksum(Path bundleFile) { + if (!Files.exists(bundleFile)) { + throw new AbortOperationException("Bundle file not found: ${bundleFile}") + } + + try { + return ModuleChecksum.computeFile(bundleFile) + } + catch (Exception e) { + throw new AbortOperationException("Failed to compute bundle checksum", e) + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy index b19f814745..531e6c4cc4 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy @@ -29,6 +29,8 @@ import nextflow.NF import nextflow.Session import nextflow.exception.IllegalModulePath import nextflow.exception.ScriptCompilationException +import nextflow.module.ModuleReference +import nextflow.module.spi.RemoteModuleResolverProvider import nextflow.plugin.Plugins import nextflow.plugin.extension.PluginExtensionProvider import nextflow.script.parser.v1.ScriptLoaderV1 @@ -162,14 +164,24 @@ class IncludeDef { @PackageScope Path resolveModulePath(include) { assert include - final result = include as Path if( result.isAbsolute() ) { if( result.scheme == 'file' ) return result throw new IllegalModulePath("Cannot resolve module path: ${result.toUriString()}") } + final str = include.toString() + if( str.startsWith('./') || str.startsWith('../') ) { + return getOwnerPath().resolveSibling(str).normalize() + } + // Not a local path — treat as remote module reference (scope/name) + return resolveRemoteModulePath(str) + } - return getOwnerPath().resolveSibling(include.toString()) + @PackageScope + Path resolveRemoteModulePath(String moduleName) { + // Use SPI to get the remote module resolver implementation + def resolver = RemoteModuleResolverProvider.getInstance() + return resolver.resolve(moduleName, session.baseDir) } @PackageScope @@ -206,9 +218,15 @@ class IncludeDef { throw new IllegalModulePath("Remote modules are not allowed -- Offending module: ${path.toUriString()}") final str = path.toString() - if( !str.startsWith('/') && !str.startsWith('./') && !str.startsWith('../') && !str.startsWith('plugin/') ) - throw new IllegalModulePath("Module path must start with / or ./ prefix -- Offending module: $str") + if( str.startsWith('/') || str.startsWith('./') || str.startsWith('../') || str.startsWith('plugin/') ) + return + // Otherwise must be a valid remote module reference in scope/name format + try { + ModuleReference.parse(str) + } catch( Exception e ) { + throw new IllegalModulePath("Module path must start with '/', './', '../' or 'plugin/' prefix, or be a valid remote module reference (scope/name) -- Offending module: $str") + } } @PackageScope diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy index 221b05f8df..356f262f7d 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ProcessEntryHandler.groovy @@ -19,8 +19,10 @@ package nextflow.script import java.nio.file.Path import groovy.transform.CompileStatic import groovy.util.logging.Slf4j +import groovyx.gpars.dataflow.DataflowVariable import nextflow.Session import nextflow.Nextflow +import nextflow.extension.DumpHelper import nextflow.script.params.EnvInParam import nextflow.script.params.FileInParam import nextflow.script.params.InParam @@ -85,9 +87,11 @@ class ProcessEntryHandler { final workflowExecutionClosure = { -> // Get input parameter values and execute the process final inputArgs = getProcessArguments(processDef) - final processResult = script.invokeMethod(processName, inputArgs as Object[]) - - return processResult + final output = meta.getProcess(processName).run(inputArgs as Object[]) as ChannelOut + session.addIgniter { + printOutput(processName, output) + } + return output } // Create the body definition with execution logic @@ -98,6 +102,48 @@ class ProcessEntryHandler { return new WorkflowDef(script, workflowBody) } + /** + * Prints the process outputs. + * + * @param processName + * @param output + */ + private void printOutput(String processName, ChannelOut output) { + if( output.isEmpty() ) { + log.debug("Process ${processName} does not declare any outputs") + return + } + + Object result = null + + // print process output directly if it is a single expression + if( output.size() == 1 && (output.getNames().isEmpty() || output.getNames().first() == '$out') ) { + result = (output[0] as DataflowVariable).get() + } + + // otherwise, construct map of process emits + else { + if( output.size() != output.getNames().size() ) + log.warn("Process ${processName} is missing emit names for one or more outputs -- unnamed outputs will be omitted") + + // compute reverse lookup of emit names + final reverseLookup = new HashMap(output.size()) + for( final name : output.getNames() ) + reverseLookup.put(output.getProperty(name), name) + + // combine process emits into map + final combinedOutputs = new LinkedHashMap(output.size()) + for( final ch : output ) { + final name = reverseLookup.get(ch) + combinedOutputs.put(name, (ch as DataflowVariable).get()) + } + + result = combinedOutputs + } + + println DumpHelper.prettyPrintJson(result) + } + /** * Gets the input arguments for a process by parsing input parameter structures * and mapping them from session.params, supporting dot notation for complex inputs. diff --git a/modules/nextflow/src/main/resources/META-INF/extensions.idx b/modules/nextflow/src/main/resources/META-INF/extensions.idx index 7250b10d76..7a5a21cc99 100644 --- a/modules/nextflow/src/main/resources/META-INF/extensions.idx +++ b/modules/nextflow/src/main/resources/META-INF/extensions.idx @@ -18,6 +18,7 @@ nextflow.cache.DefaultCacheFactory nextflow.conda.CondaConfig nextflow.config.ConfigMap nextflow.config.Manifest +nextflow.config.RegistryConfig nextflow.config.WorkflowConfig nextflow.container.ApptainerConfig nextflow.container.CharliecloudConfig diff --git a/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver b/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver new file mode 100644 index 0000000000..d12a870946 --- /dev/null +++ b/modules/nextflow/src/main/resources/META-INF/services/nextflow.module.spi.RemoteModuleResolver @@ -0,0 +1 @@ +nextflow.module.DefaultRemoteModuleResolver diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy new file mode 100644 index 0000000000..ba90e5d005 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInfoTest.groovy @@ -0,0 +1,820 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import groovy.json.JsonSlurper +import io.seqera.npr.api.schema.v1.Module +import io.seqera.npr.api.schema.v1.ModuleChannel +import io.seqera.npr.api.schema.v1.ModuleChannelItem +import io.seqera.npr.api.schema.v1.ModuleMetadata +import io.seqera.npr.api.schema.v1.ModuleRelease +import io.seqera.npr.api.schema.v1.ModuleTool +import nextflow.cli.Launcher +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleRegistryClient +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Path + +/** + * Tests for CmdModuleInfo command + * + * @author Jorge Ejarque + */ +class CmdModuleInfoTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + def 'should display module info in formatted output'() { + given: + def metadata = new ModuleMetadata( + description: 'FastQC quality control analysis', + authors: ['nf-core', 'community'], + keywords: ['quality-control', 'fastqc', 'reads'] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + description: 'FastQC module', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module:') + output.contains('nf-core/fastqc') + output.contains('Version:') + output.contains('1.0.0') + output.contains('Description:') + output.contains('FastQC quality control analysis') + output.contains('Authors:') + output.contains('nf-core, community') + output.contains('Keywords:') + output.contains('quality-control, fastqc, reads') + output.contains('Usage Template:') + } + + def 'should display module info with specific version'() { + given: + def metadata = new ModuleMetadata( + description: 'FastQC quality control' + ) + + and: + def release = new ModuleRelease( + version: '0.9.0', + description: 'FastQC module', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.version = '0.9.0' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockClient = Mock(ModuleRegistryClient) { + fetchRelease(_, _) >> release + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Version:') + output.contains('0.9.0') + } + + def 'should display module info in JSON format'() { + given: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + authors: ['nf-core'], + keywords: ['quality-control'] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + description: 'FastQC module', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.output = 'json' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + // Extract JSON part (skip debug/log lines) + def lines = output.readLines() + def jsonStart = lines.findIndexOf { it.trim().startsWith('{') } + def jsonText = lines[jsonStart..-1].join('\n') + def json = new JsonSlurper().parseText(jsonText) + + then: + json.name == 'nf-core/fastqc' + json.fullName == 'nf-core/fastqc' + json.version == '1.0.0' + json.description == 'FastQC quality control' + json.authors == ['nf-core'] + json.keywords == ['quality-control'] + json.usageTemplate != null + } + + def 'should display module info with tools'() { + given: + def tool = new ModuleTool( + name: 'fastqc', + version: '0.12.1', + homepage: URI.create('https://www.bioinformatics.babraham.ac.uk/projects/fastqc/'), + documentation: URI.create('https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/') + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + tools: [tool] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Tools:') + output.contains('fastqc v0.12.1') + output.contains('Homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/') + } + + def 'should display module info with inputs'() { + given: + def inputItem1 = new ModuleChannelItem( + name: 'reads', + type: 'file', + description: 'Input FASTQ files', + pattern: '*.fastq.gz' + ) + def inputItem2 = new ModuleChannelItem( + name: 'meta', + type: 'map', + description: 'Sample metadata' + ) + + and: + def input = new ModuleChannel( + tuple: true, + items: [inputItem1, inputItem2] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [input] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Input:') + output.contains('(tuple)') + output.contains('reads (file)') + output.contains('Input FASTQ files') + output.contains('Pattern: *.fastq.gz') + output.contains('meta (map)') + output.contains('Sample metadata') + } + + def 'should display module info with outputs'() { + given: + def outputItem = new ModuleChannelItem( + name: 'html', + type: 'file', + description: 'FastQC HTML report', + pattern: '*_fastqc.html' + ) + + and: + def outputChannel = new ModuleChannel( + tuple: false, + items: [outputItem] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + output: ['html': outputChannel] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Output:') + output.contains('html') + output.contains('html (file)') + output.contains('FastQC HTML report') + output.contains('Pattern: *_fastqc.html') + } + + def 'should generate usage template with inputs'() { + given: + def inputItem1 = new ModuleChannelItem( + name: 'reads', + type: 'file' + ) + def inputItem2 = new ModuleChannelItem( + name: 'sample-id', + type: 'val' + ) + + and: + def input1 = new ModuleChannel( + tuple: false, + items: [inputItem1] + ) + def input2 = new ModuleChannel( + tuple: false, + items: [inputItem2] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [input1, input2] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Usage Template:') + output.contains('nextflow module run nf-core/fastqc') + output.contains('--reads ') + output.contains('--sample-id ') + } + + def 'should generate usage template with version'() { + given: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [] + ) + + and: + def release = new ModuleRelease( + version: '2.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.version = '2.0.0' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockClient = Mock(ModuleRegistryClient) { + fetchRelease(_, _) >> release + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Usage Template:') + output.contains('nextflow module run nf-core/fastqc') + output.contains('-version 2.0.0') + } + + def 'should generate usage template with map inputs'() { + given: + def inputItem = new ModuleChannelItem( + name: 'params', + type: 'map' + ) + + and: + def input = new ModuleChannel( + tuple: false, + items: [inputItem] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [input] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Usage Template:') + output.contains('--params. ') + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleInfo() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = [] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with multiple arguments'() { + given: + def cmd = new CmdModuleInfo() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['module1', 'module2'] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should display minimal info when metadata is sparse'() { + given: + def metadata = new ModuleMetadata( + description: null, + authors: null, + keywords: null + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + description: 'Module description', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module:') + output.contains('nf-core/fastqc') + output.contains('Version:') + output.contains('1.0.0') + output.contains('Description:') + output.contains('Module description') + } + + def 'should display N/A when no description is available'() { + given: + def metadata = new ModuleMetadata( + description: null + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + description: null, + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Description: N/A') + } + + def 'should display module with tuple outputs'() { + given: + def outputItem1 = new ModuleChannelItem( + name: 'html', + type: 'file', + description: 'HTML report' + ) + def outputItem2 = new ModuleChannelItem( + name: 'zip', + type: 'file', + description: 'ZIP archive' + ) + + and: + def outputChannel = new ModuleChannel( + tuple: true, + items: [outputItem1, outputItem2] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + output: ['qc': outputChannel] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Output:') + output.contains('qc (tuple)') + output.contains('html (file)') + output.contains('HTML report') + output.contains('zip (file)') + output.contains('ZIP archive') + output.contains(')') + } + + def 'should include all tool information in JSON output'() { + given: + def tool = new ModuleTool( + name: 'fastqc', + version: '0.12.1', + homepage: URI.create('https://example.com'), + documentation: URI.create('https://docs.example.com') + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + tools: [tool] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.output = 'json' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + // Extract JSON part (skip debug/log lines) + def lines = output.readLines() + def jsonStart = lines.findIndexOf { it.trim().startsWith('{') } + def jsonText = lines[jsonStart..-1].join('\n') + def json = new JsonSlurper().parseText(jsonText) + + then: + json.tools.size() == 1 + json.tools[0].name == 'fastqc' + json.tools[0].version == '0.12.1' + json.tools[0].homepage.scheme == 'https' + json.tools[0].homepage.host == 'example.com' + json.tools[0].documentation.scheme == 'https' + json.tools[0].documentation.host == 'docs.example.com' + } + + def 'should include input/output information in JSON output'() { + given: + def inputItem = new ModuleChannelItem( + name: 'reads', + type: 'file', + description: 'Input reads', + pattern: '*.fastq.gz' + ) + def inputChannel = new ModuleChannel( + tuple: true, + items: [inputItem] + ) + + and: + def outputItem = new ModuleChannelItem( + name: 'html', + type: 'file', + description: 'HTML report', + pattern: '*.html' + ) + def outputChannel = new ModuleChannel( + tuple: false, + items: [outputItem] + ) + + and: + def metadata = new ModuleMetadata( + description: 'FastQC quality control', + input: [inputChannel], + output: ['html': outputChannel] + ) + + and: + def release = new ModuleRelease( + version: '1.0.0', + metadata: metadata + ) + + and: + def cmd = new CmdModuleInfo() + cmd.args = ['nf-core/fastqc'] + cmd.output = 'json' + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.root = tempDir + + and: + def mockModule = Stub(Module) { + getLatest() >> release + } + def mockClient = Mock(ModuleRegistryClient) { + fetchModule(_) >> mockModule + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + // Extract JSON part (skip debug/log lines) + def lines = output.readLines() + def jsonStart = lines.findIndexOf { it.trim().startsWith('{') } + def jsonText = lines[jsonStart..-1].join('\n') + def json = new JsonSlurper().parseText(jsonText) + + then: + json.input.size() == 1 + json.input[0].tuple == true + json.input[0].items[0].name == 'reads' + json.input[0].items[0].type == 'file' + json.input[0].items[0].description == 'Input reads' + json.input[0].items[0].pattern == '*.fastq.gz' + + and: + json.output.html.tuple == false + json.output.html.items[0].name == 'html' + json.output.html.items[0].type == 'file' + json.output.html.items[0].description == 'HTML report' + json.output.html.items[0].pattern == '*.html' + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy new file mode 100644 index 0000000000..6f3e9d08e9 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleInstallTest.groovy @@ -0,0 +1,404 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import io.seqera.npr.api.schema.v1.Module +import io.seqera.npr.api.schema.v1.ModuleRelease +import nextflow.cli.Launcher +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleChecksum +import nextflow.module.ModuleInfo +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleStorage +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Files +import java.nio.file.Path +import java.util.zip.GZIPOutputStream + +/** + * Tests for CmdModuleInstall command + * + * @author Jorge Ejarque + */ +class CmdModuleInstallTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + def 'should install module with latest version'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + and: + // Create mock module package + def modulePackage = createModulePackage('nf-core', 'fastqc', '1.0.0') + + // Mock registry client + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', + latest: new ModuleRelease(version: '1.0.0') + ) + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Installing') + output.contains('nf-core/fastqc') + output.contains('1.0.0') + + and: + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + Files.exists(moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE)) + Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should install module with specific version'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.version = '2.0.0' + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('nf-core', 'fastqc', '2.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.downloadModule(_, _, _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return 'http://registry.com' + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Installing') + output.contains('nf-core/fastqc') + output.contains('2.0.0') + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + Files.exists(moduleDir.resolve(ModuleStorage.MODULE_MANIFEST_FILE)) + Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + ModuleInfo.load(moduleDir, 'registryUrl') == "http://registry.com" + + } + + def 'should update existing module with force flag'() { + given: + // Pre-install version 1.0.0 + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process OLD { }' + moduleDir.resolve('meta.yml').text = """ + name: nf-core/fastqc + version: '1.0.0' + description: Test module + """.stripIndent() + + and: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.version = '2.0.0' + cmd.force = true + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('nf-core', 'fastqc', '2.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.downloadModule('nf-core/fastqc', '2.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return 'registry' + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Installing') + output.contains('2.0.0') + + and: + moduleDir.resolve('main.nf').text.contains('FASTQC') // New content + } + + def 'should fail when module already installed without force'() { + given: + // Pre-install the module + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process FASTQC { }' + moduleDir.resolve('meta.yml').text = """ + name: nf-core/fastqc + version: '1.0.0' + description: Test module + """.stripIndent() + ModuleChecksum.save(moduleDir, 'wrong-checksum') + + and: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.version = '2.0.0' + cmd.root = tempDir + + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', + latest: new ModuleRelease(version: '2.0.0') + ) + cmd.client = mockClient + + when: + cmd.run() + + then: + def e = thrown(AbortOperationException) + e.message.contains('already installed') || e.message.contains('-force') + } + + def 'should handle module with scope in name'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['myorg/custom-module'] + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('myorg', 'custom-module', '1.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('myorg/custom-module') >> new Module( + name: 'myorg/custom-module', + latest: new ModuleRelease(version: '1.0.0') + ) + mockClient.downloadModule('myorg/custom-module', '1.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + cmd.client = mockClient + + when: + cmd.run() + + then: + def moduleDir = tempDir.resolve('modules/myorg/custom-module') + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + } + + def 'should create modules directory if it does not exist'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('nf-core', 'fastqc', '1.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', + latest: new ModuleRelease(version: '1.0.0') + ) + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + cmd.client = mockClient + + when: + cmd.run() + + then: + Files.exists(tempDir.resolve('modules')) + Files.exists(tempDir.resolve('modules/nf-core')) + Files.exists(tempDir.resolve('modules/nf-core/fastqc')) + } + + def 'should create checksum file after installation'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + and: + def modulePackage = createModulePackage('nf-core', 'fastqc', '1.0.0') + + def mockClient = Mock(ModuleRegistryClient) + mockClient.fetchModule('nf-core/fastqc') >> new Module( + name: 'nf-core/fastqc', + latest: new ModuleRelease(version: '1.0.0') + ) + mockClient.downloadModule('nf-core/fastqc', '1.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + cmd.client = mockClient + + when: + cmd.run() + + then: + def moduleDir = tempDir.resolve('modules/nf-core/fastqc') + Files.exists(moduleDir.resolve('.module-info')) + + and: + def checksum = ModuleChecksum.load(moduleDir) + checksum != null + !checksum.isEmpty() + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = [] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with too many arguments'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['nf-core/fastqc', 'extra-arg'] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with invalid module reference'() { + given: + def cmd = new CmdModuleInstall() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['invalid-module-name'] // Missing scope + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + // Helper method to create a module package (tar.gz) + private byte[] createModulePackage(String scope, String name, String version) { + def baos = new ByteArrayOutputStream() + + // Create tar.gz with module files + def mainNfContent = """ + process ${name.toUpperCase().replaceAll('-', '_')} { + input: + path reads + + output: + path "*.html" + + script: + \"\"\" + echo "Running ${name}" + \"\"\" + } + """.stripIndent() + new GZIPOutputStream(baos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add main.nf + addTarEntry(tos, 'main.nf', mainNfContent.bytes) + + // Add meta.yml + def metaContent = """ + name: ${scope}/${name} + version: ${version} + description: Test module + """.stripIndent() + addTarEntry(tos, 'meta.yml', metaContent.bytes) + } + } + + return baos.toByteArray() + } + + private void addTarEntry(TarArchiveOutputStream tos, String name, byte[] content) { + def entry = new TarArchiveEntry(name) + entry.setSize(content.length) + tos.putArchiveEntry(entry) + tos.write(content) + tos.closeArchiveEntry() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy new file mode 100644 index 0000000000..dace2c8ba0 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleListTest.groovy @@ -0,0 +1,186 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import groovy.json.JsonSlurper +import nextflow.module.ModuleChecksum +import nextflow.module.ModuleInfo +import nextflow.module.ModuleReference +import nextflow.module.ModuleStorage +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Tests for CmdModuleList command + * + * @author Jorge Ejarque + */ +class CmdModuleListTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + // No setup needed - using root field directly + + def 'should list installed modules with formatted output'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create test modules + createTestModule(storage, 'nf-core', 'fastqc', '1.0.0') + createTestModule(storage, 'nf-core', 'multiqc', '2.1.0') + + and: + def cmd = new CmdModuleList() + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Installed modules:') + output.contains('nf-core/fastqc') + output.contains('1.0.0') + output.contains('nf-core/multiqc') + output.contains('2.1.0') + output.contains('OK') + } + + def 'should list installed modules with JSON output'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create test module + createTestModule(storage, 'nf-core', 'fastqc', '1.5.0') + + and: + def cmd = new CmdModuleList() + cmd.output = 'json' + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + def json = new JsonSlurper().parseText(output) + + then: + json.modules != null + json.modules.size() == 1 + json.modules[0].name == 'nf-core/fastqc' + json.modules[0].version == '1.5.0' + json.modules[0].integrity != null + } + + def 'should handle no installed modules'() { + given: + def cmd = new CmdModuleList() + cmd.root = tempDir // Use test directory with no modules + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('No modules installed') + } + + def 'should show modified status for locally modified modules'() { + given: + def storage = new ModuleStorage(tempDir) + def moduleDir = createTestModule(storage, 'nf-core', 'fastqc', '1.0.0') + + // Modify the module to trigger checksum mismatch + moduleDir.resolve('main.nf').text = 'process MODIFIED { }' + + and: + def cmd = new CmdModuleList() + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('nf-core/fastqc') + output.contains('MODIFIED') + } + + def 'should list multiple modules sorted by name'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create modules in random order + createTestModule(storage, 'nf-core', 'samtools', '1.0.0') + createTestModule(storage, 'nf-core', 'fastqc', '1.0.0') + createTestModule(storage, 'myorg', 'custom', '2.0.0') + + and: + def cmd = new CmdModuleList() + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('fastqc') + output.contains('samtools') + output.contains('myorg/custom') + } + + private Path createTestModule(ModuleStorage storage, String scope, String name, String version) { + def moduleDir = storage.getModuleDir(new ModuleReference(scope, name)) + Files.createDirectories(moduleDir) + + // Create main.nf + moduleDir.resolve('main.nf').text = """ + process ${name.toUpperCase()} { + input: + path reads + + output: + path "*.html" + + script: + \"\"\" + echo "test" + \"\"\" + } + """.stripIndent() + + // Create meta.yml + moduleDir.resolve('meta.yml').text = """ + name: ${scope}/${name} + version: ${version} + description: Test module + """.stripIndent() + + // Create .module-info + ModuleInfo.save(moduleDir, [checksum: ModuleChecksum.compute(moduleDir), registryUrl: 'http://registry.com']) + + return moduleDir + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy new file mode 100644 index 0000000000..55d4d9379d --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModulePublishTest.groovy @@ -0,0 +1,143 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import nextflow.cli.Launcher +import spock.lang.Specification +import spock.lang.TempDir + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Tests for CmdModulePublish command + * + * @author Jorge Ejarque + */ +class CmdModulePublishTest extends Specification { + + @TempDir + Path tempDir + + def 'should validate module structure' () { + given: + def moduleDir = tempDir.resolve('my-module') + Files.createDirectories(moduleDir) + + // Create required files + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('README.md').text = '# Test Module' + moduleDir.resolve('meta.yml').text = ''' +name: test/module +version: 1.0.0 +description: Test module +license: MIT +''' + + and: + def cmd = new CmdModulePublish() + cmd.dryRun = true + cmd.args = [moduleDir.toString()] + + when: + def errors = cmd.invokeMethod('validateModuleStructure', moduleDir) + + then: + errors.isEmpty() + } + + def 'should detect missing required files' () { + given: + def moduleDir = tempDir.resolve('my-module') + Files.createDirectories(moduleDir) + + // Only create main.nf, missing meta.yml and README.md + moduleDir.resolve('main.nf').text = 'process TEST { }' + + and: + def cmd = new CmdModulePublish() + + when: + def errors = cmd.invokeMethod('validateModuleStructure', moduleDir) + + then: + errors.size() == 2 + errors.any { it.contains('meta.yml') } + errors.any { it.contains('README.md') } + } + + def 'should detect oversized module' () { + given: + def moduleDir = tempDir.resolve('my-module') + Files.createDirectories(moduleDir) + + // Create required files + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('README.md').text = '# Test Module' + moduleDir.resolve('meta.yml').text = ''' +name: test/module +version: 1.0.0 +description: Test module +license: MIT +''' + + // Create a large file (>1MB) + def largeFile = moduleDir.resolve('large-file.txt') + def content = 'x' * (1024 * 1024 + 1000) // 1MB + 1000 bytes + Files.writeString(largeFile, content) + + and: + def cmd = new CmdModulePublish() + + when: + def errors = cmd.invokeMethod('validateModuleStructure', moduleDir) + + then: + errors.size() == 1 + errors[0].contains('1MB limit') + } + + def 'should succeed in dry-run mode without authentication' () { + given: + def moduleDir = tempDir.resolve('my-module') + Files.createDirectories(moduleDir) + + // Create required files + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('README.md').text = '# Test Module' + moduleDir.resolve('meta.yml').text = ''' +name: test/module +version: 1.0.0 +description: Test module +license: MIT +''' + + and: + def cmd = new CmdModulePublish() + def launcher = new Launcher() + launcher.options = [:] + cmd.launcher = launcher + cmd.args = [moduleDir.toString()] + cmd.dryRun = true + + when: + cmd.run() + + then: + noExceptionThrown() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy new file mode 100644 index 0000000000..bdc306bdb5 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRemoveTest.groovy @@ -0,0 +1,263 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleChecksum +import nextflow.module.ModuleInfo +import nextflow.module.ModuleReference +import nextflow.module.ModuleStorage +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Tests for CmdModuleRemove command + * + * @author Jorge Ejarque + */ +class CmdModuleRemoveTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + // No setup needed - using root field directly + + def 'should remove all module files'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference, true) + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module nf-core/fastqc files removed successfully') + !Files.exists(moduleDir) + } + + def 'should keep files with -keep-files flag'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference, true) + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.keepFiles = true + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Keeping module files for nf-core/fastqc ') + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + !Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should fail when both keep-files and force are set'() { + given: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.force = true + cmd.keepFiles = true + cmd.root = tempDir + + when: + cmd.run() + + then: + def e = thrown(AbortOperationException) + e.message.contains('Cannot use both -keep-files and -force options') + } + + def 'should fail to remove module without .module-info when force not set'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + createTestModule(storage, reference) // no .module-info created + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + when: + cmd.run() + + then: + def e = thrown(AbortOperationException) + e.message.contains('.module-info missing') + } + + def 'should force remove module without .module-info'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference) // no .module-info created + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.force = true + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module nf-core/fastqc files removed successfully') + !Files.exists(moduleDir) + } + + def 'should fail to remove modified module when force not set'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference, true) + // Modify a file to cause checksum mismatch + moduleDir.resolve('main.nf').text = 'process MODIFIED { }' + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.root = tempDir + + when: + cmd.run() + + then: + def e = thrown(AbortOperationException) + e.message.contains('local modifications') + } + + def 'should force remove modified module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = createTestModule(storage, reference, true) + // Modify a file to cause checksum mismatch + moduleDir.resolve('main.nf').text = 'process MODIFIED { }' + + and: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc'] + cmd.force = true + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module nf-core/fastqc files removed successfully') + !Files.exists(moduleDir) + } + + def 'should handle removing non-existent module'() { + given: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/nonexistent'] + cmd.root = tempDir + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Module nf-core/nonexistent not found locally') + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleRemove() + cmd.args = [] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with too many arguments'() { + given: + def cmd = new CmdModuleRemove() + cmd.args = ['nf-core/fastqc', 'extra-arg'] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + private Path createTestModule(ModuleStorage storage, ModuleReference reference, boolean withModuleInfo = false) { + def moduleDir = storage.getModuleDir(reference) + Files.createDirectories(moduleDir) + + // Create main.nf + moduleDir.resolve('main.nf').text = ''' + process FASTQC { + input: + path reads + + output: + path "*.html" + + script: + """ + fastqc ${reads} + """ + } + '''.stripIndent() + + // Create meta.yml + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + description: FastQC quality control + '''.stripIndent() + + if( withModuleInfo ) { + ModuleChecksum.save(moduleDir, ModuleChecksum.compute(moduleDir)) + } + + return moduleDir + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy new file mode 100644 index 0000000000..9edcbd1600 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleRunTest.groovy @@ -0,0 +1,255 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import io.seqera.npr.api.schema.v1.Module +import io.seqera.npr.api.schema.v1.ModuleRelease +import nextflow.cli.CliOptions +import nextflow.cli.Launcher +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleReference +import nextflow.module.ModuleRegistryClient +import nextflow.module.ModuleStorage +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream +import org.junit.Rule +import spock.lang.Specification +import spock.lang.TempDir +import test.OutputCapture + +import java.nio.file.Files +import java.nio.file.Path +import java.util.regex.Pattern +import java.util.zip.GZIPOutputStream + +/** + * Tests for CmdModuleRun command + * + * @author Jorge Ejarque + */ +class CmdModuleRunTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + @TempDir + Path tempDir + + def 'should run module and create output file'() { + given: + // Create a simple module script that creates a file + def moduleScript = ''' + process CREATE_FILE { + output: + path "test_output.txt" + + script: + """ + echo "Module executed successfully" > test_output.txt + """ + } + '''.stripIndent() + + and: + // Create module directory structure + def storage = new ModuleStorage(tempDir) + def moduleRef = new ModuleReference('nf-core', 'test-module') + def moduleDir = storage.getModuleDir(moduleRef) + Files.createDirectories(moduleDir) + + // Write main.nf + moduleDir.resolve('main.nf').text = moduleScript + + // Write meta.yml + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/test-module + version: 1.0.0 + description: Test module that creates a file + '''.stripIndent() + + and: + // Create mock module package + def modulePackage = createModulePackage(moduleScript) + + // Mock registry client + def mockClient = Stub(ModuleRegistryClient) + def moduleRelease = new ModuleRelease() + moduleRelease.version = '1.0.0' + def module = new Module() + module.name = 'nf-core/test-module' + module.latest = moduleRelease + mockClient.fetchModule(_) >> module // Use wildcard to match any argument + mockClient.downloadModule(_, _, _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + def escapedPath = Pattern.quote(tempDir.toString()) + def pattern = ~/"${escapedPath}\/.+\/test_output\.txt"/ + + and: + def cmd = new CmdModuleRun() + def opts = new CliOptions() + opts.setQuiet(true) + cmd.launcher = Mock(Launcher) { + getOptions() >> opts + getCliString() >> "nextflow module run nf-core/test-module" + } + cmd.args = ['nf-core/test-module'] + cmd.root = tempDir + cmd.workDir = tempDir.toString() + cmd.client = mockClient + + when: + cmd.run() + def stdout = capture + .toString() + .readLines()// remove the log part + .findResults { line -> !line.contains('DEBUG') ? line : null } + .findResults { line -> !line.contains('INFO') ? line : null }.join(" ") + + then: + assert (stdout =~ pattern).find() + and: + // Verify module was installed + Files.exists(moduleDir) + Files.exists(moduleDir.resolve('main.nf')) + + } + + def 'should run module with specific version'() { + given: + def moduleScript = ''' + process CREATE_FILE_V2 { + output: + path "test_output_v2.txt", emit: output_path + + script: + """ + echo "Module version 2.0.0 executed successfully" > test_output_v2.txt + """ + } + ''' + + and: + def storage = new ModuleStorage(tempDir) + def moduleRef = new ModuleReference('nf-core', 'test-module') + def moduleDir = storage.getModuleDir(moduleRef) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = moduleScript + moduleDir.resolve('meta.yml').text = 'name: nf-core/test-module\nversion: 2.0.0' + def escapedPath = Pattern.quote(tempDir.toString()) + def pattern = ~/"output_path": "${escapedPath}\/.+\/test_output_v2\.txt"/ + + and: + def modulePackage = createModulePackage(moduleScript) + + def mockClient = Mock(ModuleRegistryClient) + mockClient.downloadModule('nf-core/test-module', '2.0.0', _) >> { String name, String version, Path dest -> + Files.write(dest, modulePackage) + return dest + } + + and: + def cmd = new CmdModuleRun() + def opts = new CliOptions() + opts.setQuiet(true) + cmd.launcher = Mock(Launcher) { + getOptions() >> opts + getCliString() >> "nextflow module run nf-core/test-module" + } + cmd.args = ['nf-core/test-module'] + cmd.version = '2.0.0' + cmd.root = tempDir + cmd.workDir = tempDir.toString() + cmd.client = mockClient + + when: + cmd.run() + + then: + def stdout = capture + .toString() + .readLines()// remove the log part + .findResults { line -> !line.contains('DEBUG') ? line : null } + .findResults { line -> !line.contains('INFO') ? line : null } + .findResults { line -> !line.contains('plugin') ? line : null }.join(" ") + assert (stdout =~ pattern).find() + + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleRun() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = [] + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should fail with invalid module reference'() { + given: + def cmd = new CmdModuleRun() + cmd.launcher = Mock(Launcher) { + getOptions() >> null + } + cmd.args = ['invalid-module'] // Missing scope + cmd.root = tempDir + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + // Helper method to create a module package (tar.gz) + private byte[] createModulePackage(String mainNfContent) { + def baos = new ByteArrayOutputStream() + + new GZIPOutputStream(baos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add main.nf + addTarEntry(tos, 'main.nf', mainNfContent.bytes) + + // Add meta.yml + def metaContent = ''' + name: test-module + version: 1.0.0 + description: Test module + '''.stripIndent() + addTarEntry(tos, 'meta.yml', metaContent.bytes) + } + } + + return baos.toByteArray() + } + + private void addTarEntry(TarArchiveOutputStream tos, String name, byte[] content) { + def entry = new TarArchiveEntry(name) + entry.setSize(content.length) + tos.putArchiveEntry(entry) + tos.write(content) + tos.closeArchiveEntry() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleSearchTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleSearchTest.groovy new file mode 100644 index 0000000000..155a3ae994 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/cli/module/CmdModuleSearchTest.groovy @@ -0,0 +1,228 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.cli.module + +import groovy.json.JsonSlurper +import io.seqera.npr.api.schema.v1.ModuleSearchResult +import io.seqera.npr.api.schema.v1.SearchModulesResponse +import nextflow.exception.AbortOperationException +import nextflow.module.ModuleRegistryClient +import nextflow.cli.Launcher +import org.junit.Rule +import spock.lang.Specification +import test.OutputCapture + + +/** + * Tests for CmdModuleSearch command + * + * @author Jorge Ejarque + */ +class CmdModuleSearchTest extends Specification { + + @Rule + OutputCapture capture = new OutputCapture() + + // No setup needed - using client field for mocking + + def 'should search and display results in formatted output'() { + given: + def result1 = new ModuleSearchResult( + name: 'nf-core/fastqc', + repositoryPath: 'nf-core/modules', + description: 'FastQC quality control', + relevanceScore: 0.95, + keywords: ['quality-control', 'fastqc'], + tools: ['fastqc'], + revoked: false + ) + def result2 = new ModuleSearchResult( + name: 'nf-core/multiqc', + repositoryPath: 'nf-core/modules', + description: 'MultiQC reporting', + relevanceScore: 0.85, + keywords: ['quality-control', 'reporting'], + tools: ['multiqc'], + revoked: false + ) + + and: + def cmd = new CmdModuleSearch() + cmd.args = ['quality'] + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.limit = 20 + + and: + def response = new SearchModulesResponse( + query: 'quality', + totalResults: 2, + results: [result1, result2] + ) + assert response.results + // Mock the registry client directly using the test field + def mockClient = Mock(ModuleRegistryClient) { + search(_, _) >> response + } + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('Searching for') + output.contains('quality') + output.contains('nf-core/fastqc') + output.contains('FastQC quality control') + output.contains('nf-core/multiqc') + output.contains('MultiQC reporting') + + } + + def 'should search and display results in JSON output'() { + given: + def result1 = new ModuleSearchResult( + name: 'nf-core/fastqc', + description: 'FastQC quality control', + relevanceScore: 0.95, + keywords: ['quality-control'], + tools: ['fastqc'], + revoked: false + ) + + and: + def cmd = new CmdModuleSearch() + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.args = ['fastqc'] + cmd.limit = 10 + cmd.output = 'json' + + and: + // Mock the registry client + def mockClient = Mock(ModuleRegistryClient) + mockClient.search('fastqc', 10) >> new SearchModulesResponse( + query: 'fastqc', + totalResults: 1, + results: [result1] + ) + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString().readLines() + .findResults { line -> !line.contains('DEBUG') ? line : null } + .findResults { line -> !line.contains('INFO') ? line : null } + .findResults { line -> !line.contains('Searching for') ? line : null }.join("\n") + def json = new JsonSlurper().parseText(output) + + then: + json.query == 'fastqc' + json.totalResults == 1 + json.results.size() == 1 + json.results[0].name == 'nf-core/fastqc' + json.results[0].description == 'FastQC quality control' + + } + + def 'should handle no search results'() { + given: + def cmd = new CmdModuleSearch() + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.args = ['nonexistent-module'] + cmd.limit = 20 + + and: + // Mock empty results + def mockClient = Mock(ModuleRegistryClient) + mockClient.search('nonexistent-module', 20) >> new SearchModulesResponse( + query: 'nonexistent-module', + totalResults: 0, + results: [] + ) + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('No modules found') + + } + + def 'should fail with no arguments'() { + given: + def cmd = new CmdModuleSearch() + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.args = [] + + when: + cmd.run() + + then: + thrown(AbortOperationException) + } + + def 'should handle search with custom limit'() { + given: + def results = (1..5).collect { i -> + new ModuleSearchResult( + name: "nf-core/module${i}", + description: "Module ${i}", + relevanceScore: 0.9 - (i * 0.1), + keywords: ['test'], + tools: ["tool${i}"], + revoked: false + ) + } + + and: + def cmd = new CmdModuleSearch() + cmd.launcher = Mock(Launcher){ + getOptions() >> null + } + cmd.args = ['test'] + cmd.limit = 5 + + and: + // Mock the client with 5 results + def mockClient = Mock(ModuleRegistryClient) + mockClient.search('test', 5) >> new SearchModulesResponse( + query: 'test', + totalResults: 5, + results: results + ) + cmd.client = mockClient + + when: + cmd.run() + def output = capture.toString() + + then: + output.contains('nf-core/module1') + output.contains('nf-core/module5') + (1..5).every { i -> output.contains("Module ${i}") } + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy new file mode 100644 index 0000000000..bda7b7f5ee --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/DefaultRemoteModuleResolverTest.groovy @@ -0,0 +1,46 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.module.spi.RemoteModuleResolverProvider +import spock.lang.Specification + +/** + * Test for DefaultRemoteModuleResolver SPI implementation + * + * @author Jorge Ejarque + */ +class DefaultRemoteModuleResolverTest extends Specification { + + def 'should load resolver via SPI'() { + when: + def resolver = RemoteModuleResolverProvider.getInstance() + + then: + resolver != null + resolver.class.name == 'nextflow.module.DefaultRemoteModuleResolver' + resolver.priority == 0 + } + + def 'should return default priority'() { + given: + def resolver = new DefaultRemoteModuleResolver() + + expect: + resolver.getPriority() == 0 + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy new file mode 100644 index 0000000000..e526dde81f --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/InstalledModuleTest.groovy @@ -0,0 +1,201 @@ +/* + * Copyright 2013-2024, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import java.nio.file.Files +import java.nio.file.Path + +import spock.lang.Specification + +/** + * Test suite for InstalledModule + * + * @author Jorge Ejarque + */ +class InstalledModuleTest extends Specification { + + Path tempDir + + def setup() { + tempDir = Files.createTempDirectory('nf-installed-module-test-') + } + + def cleanup() { + tempDir?.deleteDir() + } + + def 'should report VALID integrity when checksum matches'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + // Compute actual checksum + def actualChecksum = ModuleChecksum.compute(moduleDir) + + // Save checksum + ModuleChecksum.save(moduleDir, actualChecksum) + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleDir.resolve('.module-info'), + expectedChecksum: actualChecksum, + installedVersion: "0.0.1" + ) + + when: + def integrity = installed.getIntegrity() + + then: + integrity == ModuleIntegrity.VALID + } + + def 'should report MODIFIED integrity when checksum differs'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + // Compute initial checksum + def originalChecksum = ModuleChecksum.compute(moduleDir) + ModuleChecksum.save(moduleDir, originalChecksum) + + // Modify the file + mainFile.text = 'process TEST { println "modified" }' + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleDir.resolve('.module-info'), + expectedChecksum: originalChecksum, + installedVersion: "0.0.1" + ) + + when: + def integrity = installed.getIntegrity() + + then: + integrity == ModuleIntegrity.MODIFIED + } + + def 'should report CORRUPTED integrity when main.nf is missing'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + // Don't create main.nf + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + def moduleInfoFile = moduleDir.resolve('.module-info') + // CORRUPTED check happens before .module-info check, so content doesn't matter here + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleInfoFile, + expectedChecksum: 'some-checksum' + ) + + when: + def integrity = installed.getIntegrity() + + then: + integrity == ModuleIntegrity.CORRUPTED + } + + def 'should report NO_REMOTE_MODULE when .module-info file absent'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + def moduleInfoFile = moduleDir.resolve('.module-info') + // Don't create .module-info file + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleInfoFile, + expectedChecksum: null + ) + + when: + def integrity = installed.getIntegrity() + + then: + integrity == ModuleIntegrity.NO_REMOTE_MODULE + } + + def 'should handle checksum computation failure gracefully'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = 'name: test/module\nversion: 0.0.1' + + // Create .module-info with a checksum that won't match the computed checksum + ModuleChecksum.save(moduleDir, 'expected-checksum-that-will-not-match') + def moduleInfoFile = moduleDir.resolve('.module-info') + + def installed = new InstalledModule( + reference: new ModuleReference('test', 'module'), + directory: moduleDir, + mainFile: mainFile, + manifestFile: metaFile, + moduleInfoFile: moduleInfoFile, + expectedChecksum: 'expected-checksum-that-will-not-match' + ) + + when: + def integrity = installed.getIntegrity() + + then: + // Should handle gracefully - since checksum won't match, it should report as MODIFIED + integrity in [ModuleIntegrity.VALID, ModuleIntegrity.MODIFIED, ModuleIntegrity.CORRUPTED] + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy new file mode 100644 index 0000000000..4bb9185681 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleChecksumTest.groovy @@ -0,0 +1,414 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import java.nio.file.Files +import java.nio.file.Path + +import spock.lang.Specification + +/** + * Test suite for ModuleChecksum + * + * @author Jorge Ejarque + */ +class ModuleChecksumTest extends Specification { + + Path tempDir + + def setup() { + tempDir = Files.createTempDirectory('nf-checksum-test-') + } + + def cleanup() { + tempDir?.deleteDir() + } + + def 'should compute checksum for directory'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + // Create test files + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: test\nversion: 1.0.0' + moduleDir.resolve('README.md').text = '# Test Module' + + when: + def checksum = ModuleChecksum.compute(moduleDir) + + then: + checksum != null + checksum.length() == 64 // SHA-256 produces 64 hex characters + checksum ==~ /^[a-f0-9]{64}$/ + } + + def 'should produce consistent checksums for same content'() { + given: + def moduleDir1 = tempDir.resolve('module1') + def moduleDir2 = tempDir.resolve('module2') + Files.createDirectories(moduleDir1) + Files.createDirectories(moduleDir2) + + // Create identical content in both directories + ['main.nf', 'meta.yml', 'README.md'].each { filename -> + moduleDir1.resolve(filename).text = "content of ${filename}" + moduleDir2.resolve(filename).text = "content of ${filename}" + } + + when: + def checksum1 = ModuleChecksum.compute(moduleDir1) + def checksum2 = ModuleChecksum.compute(moduleDir2) + + then: + checksum1 == checksum2 + } + + def 'should produce different checksums for different content'() { + given: + def moduleDir1 = tempDir.resolve('module1') + def moduleDir2 = tempDir.resolve('module2') + Files.createDirectories(moduleDir1) + Files.createDirectories(moduleDir2) + + // Create different content + moduleDir1.resolve('main.nf').text = 'process TEST1 { }' + moduleDir2.resolve('main.nf').text = 'process TEST2 { }' + + when: + def checksum1 = ModuleChecksum.compute(moduleDir1) + def checksum2 = ModuleChecksum.compute(moduleDir2) + + then: + checksum1 != checksum2 + } + + def 'should exclude .module-info file from computation'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + moduleDir.resolve('main.nf').text = 'process TEST { }' + + // Compute initial checksum + def checksum1 = ModuleChecksum.compute(moduleDir) + + // Add .module-info file + ModuleChecksum.save(moduleDir, 'some-checksum-value') + + // Compute checksum again + def checksum2 = ModuleChecksum.compute(moduleDir) + + expect: + checksum1 == checksum2 // Should be the same, .module-info is ignored + } + + def 'should include subdirectories in checksum'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + moduleDir.resolve('main.nf').text = 'process TEST { }' + + // Compute checksum without subdirectory + def checksum1 = ModuleChecksum.compute(moduleDir) + + // Add subdirectory with file + def subDir = moduleDir.resolve('templates') + Files.createDirectories(subDir) + subDir.resolve('script.sh').text = '#!/bin/bash\necho "test"' + + // Compute checksum with subdirectory + def checksum2 = ModuleChecksum.compute(moduleDir) + + expect: + checksum1 != checksum2 // Checksums should differ + } + + def 'should save checksum to .module-info file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + def checksumValue = 'abc123def456' + + when: + ModuleChecksum.save(moduleDir, checksumValue) + + then: + def moduleInfoFile = moduleDir.resolve('.module-info') + Files.exists(moduleInfoFile) + ModuleChecksum.load(moduleDir) == checksumValue + } + + def 'should load checksum from .module-info file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleChecksum.save(moduleDir, 'abc123def456') + + when: + def checksum = ModuleChecksum.load(moduleDir) + + then: + checksum == 'abc123def456' + } + + def 'should return null when loading non-existent checksum file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + def checksum = ModuleChecksum.load(moduleDir) + + then: + checksum == null + } + + def 'should handle empty directory'() { + given: + def moduleDir = tempDir.resolve('empty-module') + Files.createDirectories(moduleDir) + + when: + def checksum = ModuleChecksum.compute(moduleDir) + + then: + checksum != null + checksum.length() == 64 + } + + def 'should handle files with special characters in names'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + // Create files with special characters (but valid on filesystem) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('file with spaces.txt').text = 'content' + moduleDir.resolve('file-with-dashes.txt').text = 'content' + + when: + def checksum = ModuleChecksum.compute(moduleDir) + + then: + checksum != null + checksum.length() == 64 + } + + def 'should handle binary files'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + // Create text and binary files + moduleDir.resolve('main.nf').text = 'process TEST { }' + def binaryFile = moduleDir.resolve('data.bin') + binaryFile.bytes = [0x00, 0x01, 0x02, 0xFF] as byte[] + + when: + def checksum = ModuleChecksum.compute(moduleDir) + + then: + checksum != null + checksum.length() == 64 + } + + def 'should sort files consistently for checksum computation'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + // Create files in arbitrary order + moduleDir.resolve('zzz.nf').text = 'content' + moduleDir.resolve('aaa.nf').text = 'content' + moduleDir.resolve('mmm.nf').text = 'content' + + def checksum1 = ModuleChecksum.compute(moduleDir) + + // Create another directory with files in different order + def moduleDir2 = tempDir.resolve('module2') + Files.createDirectories(moduleDir2) + moduleDir2.resolve('aaa.nf').text = 'content' + moduleDir2.resolve('mmm.nf').text = 'content' + moduleDir2.resolve('zzz.nf').text = 'content' + + def checksum2 = ModuleChecksum.compute(moduleDir2) + + expect: + checksum1 == checksum2 // Order shouldn't matter + } + + // Tests for computeFile() method + + def 'should compute checksum for a single file with default algorithm'() { + given: + def testFile = tempDir.resolve('test.txt') + testFile.text = 'Hello, World!' + + when: + def checksum = ModuleChecksum.computeFile(testFile) + + then: + checksum != null + checksum.length() == 64 // SHA-256 produces 64 hex characters + checksum ==~ /^[a-f0-9]{64}$/ + } + + def 'should produce consistent checksums for same file content'() { + given: + def file1 = tempDir.resolve('file1.txt') + def file2 = tempDir.resolve('file2.txt') + def content = 'Same content in both files' + file1.text = content + file2.text = content + + when: + def checksum1 = ModuleChecksum.computeFile(file1) + def checksum2 = ModuleChecksum.computeFile(file2) + + then: + checksum1 == checksum2 + } + + def 'should produce different checksums for different file content'() { + given: + def file1 = tempDir.resolve('file1.txt') + def file2 = tempDir.resolve('file2.txt') + file1.text = 'Content A' + file2.text = 'Content B' + + when: + def checksum1 = ModuleChecksum.computeFile(file1) + def checksum2 = ModuleChecksum.computeFile(file2) + + then: + checksum1 != checksum2 + } + + def 'should compute checksum for empty file'() { + given: + def emptyFile = tempDir.resolve('empty.txt') + emptyFile.text = '' + + when: + def checksum = ModuleChecksum.computeFile(emptyFile) + + then: + checksum != null + checksum.length() == 64 + // SHA-256 of empty string + checksum == 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' + } + + def 'should compute checksum for binary file'() { + given: + def binaryFile = tempDir.resolve('binary.dat') + binaryFile.bytes = [0x00, 0xFF, 0x42, 0xAB, 0xCD, 0xEF] as byte[] + + when: + def checksum = ModuleChecksum.computeFile(binaryFile) + + then: + checksum != null + checksum.length() == 64 + checksum ==~ /^[a-f0-9]{64}$/ + } + + def 'should support different hash algorithms'() { + given: + def testFile = tempDir.resolve('test.txt') + testFile.text = 'Test content for different algorithms' + + when: + def sha256 = ModuleChecksum.computeFile(testFile, 'SHA-256') + def sha512 = ModuleChecksum.computeFile(testFile, 'SHA-512') + + then: + sha256 != null + sha512 != null + sha256.length() == 64 // SHA-256: 256 bits = 64 hex chars + sha512.length() == 128 // SHA-512: 512 bits = 128 hex chars + sha256 != sha512 + } + + def 'should handle case-insensitive algorithm names'() { + given: + def testFile = tempDir.resolve('test.txt') + testFile.text = 'Test content' + + when: + def checksum1 = ModuleChecksum.computeFile(testFile, 'sha-256') + def checksum2 = ModuleChecksum.computeFile(testFile, 'SHA-256') + def checksum3 = ModuleChecksum.computeFile(testFile, 'Sha-256') + + then: + checksum1 == checksum2 + checksum2 == checksum3 + } + + def 'should throw exception for non-existent file'() { + given: + def nonExistentFile = tempDir.resolve('does-not-exist.txt') + + when: + ModuleChecksum.computeFile(nonExistentFile) + + then: + thrown(IllegalArgumentException) + } + + def 'should throw exception for directory instead of file'() { + given: + def directory = tempDir.resolve('subdir') + Files.createDirectories(directory) + + when: + ModuleChecksum.computeFile(directory) + + then: + thrown(IllegalArgumentException) + } + + def 'should compute known SHA-256 checksum correctly'() { + given: + def testFile = tempDir.resolve('known.txt') + testFile.text = 'abc' + + when: + def checksum = ModuleChecksum.computeFile(testFile) + + then: + // Known SHA-256 hash of "abc" + checksum == 'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad' + } + + def 'should handle large file checksum computation'() { + given: + def largeFile = tempDir.resolve('large.txt') + // Create a file with ~1MB of data + def content = 'x' * 1024 * 1024 + largeFile.text = content + + when: + def checksum = ModuleChecksum.computeFile(largeFile) + + then: + checksum != null + checksum.length() == 64 + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleInfoTest.groovy new file mode 100644 index 0000000000..0df03532e7 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleInfoTest.groovy @@ -0,0 +1,183 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import java.nio.file.Files +import java.nio.file.Path + +import spock.lang.Specification + +/** + * Test suite for ModuleInfo + * + * @author Jorge Ejarque + */ +class ModuleInfoTest extends Specification { + + Path tempDir + + def setup() { + tempDir = Files.createTempDirectory('nf-module-info-test-') + } + + def cleanup() { + tempDir?.deleteDir() + } + + def 'should save and load a single property'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + ModuleInfo.save(moduleDir, 'checksum', 'abc123') + + then: + ModuleInfo.load(moduleDir, 'checksum') == 'abc123' + } + + def 'should create .module-info file on save'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + ModuleInfo.save(moduleDir, 'version', '1.0.0') + + then: + Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should update existing property without affecting others'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleInfo.save(moduleDir, 'checksum', 'original') + ModuleInfo.save(moduleDir, 'registryUrl', 'http://registry.com') + + when: + ModuleInfo.save(moduleDir, 'checksum', 'updated') + + then: + ModuleInfo.load(moduleDir, 'checksum') == 'updated' + ModuleInfo.load(moduleDir, 'registryUrl') == 'http://registry.com' + } + + def 'should return null when loading property from non-existent file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + def result = ModuleInfo.load(moduleDir, 'checksum') + + then: + result == null + } + + def 'should return null when loading non-existent property'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleInfo.save(moduleDir, 'registryUrl', 'http://registry.com') + + when: + def result = ModuleInfo.load(moduleDir, 'missing-property') + + then: + result == null + } + + def 'should save and load multiple properties via map'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + def props = [checksum: 'abc123', registryUrl: 'http://registry.com', author: 'test'] + + when: + ModuleInfo.save(moduleDir, props) + + then: + ModuleInfo.load(moduleDir, 'checksum') == 'abc123' + ModuleInfo.load(moduleDir, 'registryUrl') == 'http://registry.com' + ModuleInfo.load(moduleDir, 'author') == 'test' + } + + def 'should merge map properties with existing ones'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleInfo.save(moduleDir, 'existing', 'value') + + when: + ModuleInfo.save(moduleDir, [newprop: 'newval']) + + then: + ModuleInfo.load(moduleDir, 'existing') == 'value' + ModuleInfo.load(moduleDir, 'newprop') == 'newval' + } + + def 'should do nothing when saving null map'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + ModuleInfo.save(moduleDir, (Map) null) + + then: + !Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should do nothing when saving empty map'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + ModuleInfo.save(moduleDir, [:]) + + then: + !Files.exists(moduleDir.resolve(ModuleInfo.MODULE_INFO_FILE)) + } + + def 'should load all properties as map'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + ModuleInfo.save(moduleDir, [checksum: 'abc123', registryUrl: 'http://registry.com']) + + when: + def result = ModuleInfo.load(moduleDir) + + then: + result['checksum'] == 'abc123' + result['registryUrl'] == 'http://registry.com' + } + + def 'should return empty map when loading all from non-existent file'() { + given: + def moduleDir = tempDir.resolve('module') + Files.createDirectories(moduleDir) + + when: + def result = ModuleInfo.load(moduleDir) + + then: + result == [:] + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy new file mode 100644 index 0000000000..b906b83c9f --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleReferenceTest.groovy @@ -0,0 +1,229 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.exception.AbortOperationException +import spock.lang.Specification + +/** + * Test suite for ModuleReference + * + * @author Jorge Ejarque + */ +class ModuleReferenceTest extends Specification { + + def 'should parse valid module reference without @'() { + when: + def ref = ModuleReference.parse('nf-core/fastqc') + + then: + ref.scope == 'nf-core' + ref.name == 'fastqc' + ref.fullName == 'nf-core/fastqc' + } + + def 'should reject module reference with @ prefix'() { + when: + ModuleReference.parse('@nf-core/fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should parse module reference with multiple slashes'() { + when: + def ref = ModuleReference.parse('myorg/samtools/view') + + then: + ref.scope == 'myorg' + ref.name == 'samtools/view' + ref.fullName == 'myorg/samtools/view' + } + + def 'should reject invalid module reference without scope'() { + when: + ModuleReference.parse('fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should reject empty module reference'() { + when: + ModuleReference.parse('') + + then: + thrown(AbortOperationException) + } + + def 'should reject null module reference'() { + when: + ModuleReference.parse(null) + + then: + thrown(AbortOperationException) + } + + def 'should reject bare @ character'() { + when: + ModuleReference.parse('@') + + then: + thrown(AbortOperationException) + } + + def 'should reject module reference with only scope'() { + when: + ModuleReference.parse('nf-core/') + + then: + thrown(AbortOperationException) + } + + def 'should reject module reference with trailing slash'() { + when: + ModuleReference.parse('nf-core/fastqc/') + + then: + thrown(AbortOperationException) + } + + def 'should create module reference from components'() { + when: + def ref = new ModuleReference('nf-core', 'fastqc') + + then: + ref.scope == 'nf-core' + ref.name == 'fastqc' + ref.fullName == 'nf-core/fastqc' + } + + def 'should handle scope names with hyphens'() { + when: + def ref = ModuleReference.parse('my-org/my-module') + + then: + ref.scope == 'my-org' + ref.name == 'my-module' + } + + def 'should handle scope names with underscores'() { + when: + def ref = ModuleReference.parse('my_org/my_module') + + then: + ref.scope == 'my_org' + ref.name == 'my_module' + } + + def 'should handle module names with numbers'() { + when: + def ref = ModuleReference.parse('nf-core/bwa-mem2') + + then: + ref.scope == 'nf-core' + ref.name == 'bwa-mem2' + } + + def 'should implement equals correctly'() { + given: + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') + def ref3 = ModuleReference.parse('nf-core/multiqc') + + expect: + ref1 == ref2 + ref1 != ref3 + } + + def 'should implement hashCode correctly'() { + given: + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') + + expect: + ref1.hashCode() == ref2.hashCode() + } + + def 'should implement toString correctly'() { + given: + def ref = ModuleReference.parse('nf-core/fastqc') + + expect: + ref.toString() == 'nf-core/fastqc' + } + + def 'should be usable as map key'() { + given: + def ref1 = ModuleReference.parse('nf-core/fastqc') + def ref2 = ModuleReference.parse('nf-core/fastqc') + def ref3 = ModuleReference.parse('nf-core/multiqc') + + def map = [:] + map[ref1] = 'value1' + map[ref3] = 'value3' + + expect: + map[ref2] == 'value1' // ref2 equals ref1, should get same value + map[ref3] == 'value3' + map.size() == 2 + } + + def 'should handle org-style scopes'() { + when: + def ref = ModuleReference.parse('mycompany.io/custom-module') + + then: + ref.scope == 'mycompany.io' + ref.name == 'custom-module' + } + + def 'should reject module reference with spaces'() { + when: + ModuleReference.parse('nf-core/fast qc') + + then: + thrown(AbortOperationException) + } + + def 'should reject module reference with special characters'() { + when: + ModuleReference.parse('nf-core/fastqc!') + + then: + thrown(AbortOperationException) + } + + def 'should handle deeply nested module names'() { + when: + def ref = ModuleReference.parse('nf-core/samtools/sort/parallel') + + then: + ref.scope == 'nf-core' + ref.name == 'samtools/sort/parallel' + ref.fullName == 'nf-core/samtools/sort/parallel' + } + + def 'should parse from string with leading/trailing whitespace'() { + when: + def ref = ModuleReference.parse(' nf-core/fastqc ') + + then: + ref.scope == 'nf-core' + ref.name == 'fastqc' + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleRegistryClientTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleRegistryClientTest.groovy new file mode 100644 index 0000000000..66fbee0ff6 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleRegistryClientTest.groovy @@ -0,0 +1,435 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import com.github.tomakehurst.wiremock.WireMockServer +import com.github.tomakehurst.wiremock.client.WireMock +import groovy.json.JsonOutput +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream +import spock.lang.Specification +import spock.lang.TempDir + +import java.nio.file.Files +import java.nio.file.Path +import java.util.zip.GZIPOutputStream + +import static com.github.tomakehurst.wiremock.client.WireMock.* +import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig + +/** + * Integration tests for ModuleRegistryClient using WireMock + * + * @author Jorge Ejarque + */ +class ModuleRegistryClientTest extends Specification { + + @TempDir + Path tempDir + + WireMockServer wireMock + String url + static final String MODULES_API_PATH = "/api/v1/modules" + def setup() { + wireMock = new WireMockServer(wireMockConfig().dynamicPort()) + wireMock.start() + WireMock.configureFor("localhost", wireMock.port()) + url = "http://localhost:${wireMock.port()}/api" + } + + def cleanup() { + wireMock?.stop() + } + + def 'should fetch module metadata from registry'() { + given: + def moduleResponse = [ + name: 'nf-core/fastqc', + description: 'FastQC quality control', + latest: [ + version: '1.1.0', + createdAt: '2024-02-01T00:00:00Z' + ] + ] + + // Note: nf-core/fastqc is URL-encoded as nf-core%2Ffastqc + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson(moduleResponse)))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + def result = client.fetchModule('nf-core/fastqc') + + then: + result != null + result.name == 'nf-core/fastqc' + result.latest != null + result.latest.version == '1.1.0' + + and: 'verify request was made' + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc'))) + } + + def 'should search modules in registry'() { + given: + def searchResponse = [ + query: 'fastqc', + totalResults: 2, + results: [ + [ + name: 'nf-core/fastqc', + description: 'FastQC quality control', + relevanceScore: 0.95, + keywords: ['quality-control'], + tools: ['fastqc'], + revoked: false + ], + [ + name: 'other/fastqc', + description: 'Another FastQC module', + relevanceScore: 0.75, + keywords: ['qc'], + tools: ['fastqc'], + revoked: false + ] + ] + ] + + stubFor(get(urlPathEqualTo(MODULES_API_PATH)) + .withQueryParam('query', equalTo('fastqc')) + .withQueryParam('limit', equalTo('10')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson(searchResponse)))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + def result = client.search('fastqc', 10) + + then: + result != null + result.query == 'fastqc' + result.totalResults == 2 + result.results.size() == 2 + result.results[0].name == 'nf-core/fastqc' + // Use closeTo for Float/BigDecimal comparison + Math.abs(result.results[0].relevanceScore - 0.95) < 0.001 + + and: 'verify query parameters' + verify(getRequestedFor(urlPathEqualTo(MODULES_API_PATH)) + .withQueryParam('query', equalTo('fastqc')) + .withQueryParam('limit', equalTo('10'))) + } + + def 'should download module package from registry'() { + given: + def modulePackage = createTestModulePackage() + def expectedChecksum = "${computeSha256(modulePackage)}" + + // Note: URL-encoded path + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc/1.0.0/download')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/gzip') + .withHeader('X-NF-Module-Checksum', "sha256:${expectedChecksum}") + .withBody(modulePackage))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + def destFile = tempDir.resolve('module.tgz') + + when: + def result = client.downloadModule('nf-core/fastqc', '1.0.0', destFile) + + then: + result == url + Files.exists(destFile) + Files.size(destFile) == modulePackage.length + + and: + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH +'/nf-core%2Ffastqc/1.0.0/download'))) + } + + def 'should successfully fetch module without authentication'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson([name: 'nf-core/fastqc', latest: [version: '1.0.0']])))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + def result = client.fetchModule('nf-core/fastqc') + + then: + result != null + result.name == 'nf-core/fastqc' + + and: 'verify request was made' + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc'))) + } + + def 'should handle 404 not found error'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Fnonexistent')) + .willReturn(aResponse() + .withStatus(404) + .withBody('Module not found'))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/nonexistent') + + then: + def ex = thrown(AbortOperationException) + ex.message.contains('Unable to fetch module') || ex.message.contains('Module not found') + + and: + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Fnonexistent'))) + } + + def 'should handle 500 server error'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(500) + .withBody('Internal Server Error'))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should send user agent header'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson([name: 'nf-core/fastqc', latest: [version: '1.0.0'], releases: []])))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/fastqc') + + then: + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .withHeader('User-Agent', matching('.*'))) + } + + def 'should handle empty search results'() { + given: + stubFor(get(urlPathEqualTo(MODULES_API_PATH)) + .withQueryParam('query', equalTo('nonexistent')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson([ + query: 'nonexistent', + totalResults: 0, + results: [] + ])))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + def result = client.search('nonexistent', 10) + + then: + result != null + result.totalResults == 0 + result.results.isEmpty() + } + + def 'should respect custom search limit'() { + given: + stubFor(get(urlPathEqualTo(MODULES_API_PATH)) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody(JsonOutput.toJson([ + query: 'test', + totalResults: 0, + results: [] + ])))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.search('test', 25) + + then: + verify(getRequestedFor(urlPathEqualTo(MODULES_API_PATH )) + .withQueryParam('limit', equalTo('25'))) + } + + def 'should handle malformed JSON response'() { + given: + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/json') + .withBody('not valid json {]'))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should verify download includes checksum header'() { + given: + def modulePackage = createTestModulePackage() + def checksum = "sha256:${computeSha256(modulePackage)}" + + stubFor(get(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc/1.0.0/download')) + .willReturn(aResponse() + .withStatus(200) + .withHeader('Content-Type', 'application/gzip') + .withHeader('X-NF-Module-Checksum', checksum) + .withHeader('Docker-Content-Digest', checksum) + .withBody(modulePackage))) + + and: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + def destFile = tempDir.resolve('module.tgz') + + when: + def result = client.downloadModule('nf-core/fastqc', '1.0.0', destFile) + + then: + result == url + Files.exists(destFile) + + and: 'verify checksum header was present' + verify(getRequestedFor(urlEqualTo(MODULES_API_PATH + '/nf-core%2Ffastqc/1.0.0/download'))) + } + + def 'should handle network errors gracefully'() { + given: + // Stub will not be set up, causing connection refused + def config = new RegistryConfig([url: "http://localhost:9999"]) // Invalid port + def client = new ModuleRegistryClient(config) + + when: + client.fetchModule('nf-core/fastqc') + + then: + thrown(AbortOperationException) + } + + def 'should require authentication for publish'() { + given: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + def publishRequest = [name: 'nf-core/mymodule', version: '1.0.0'] + + when: + client.publishModule('nf-core/mymodule', publishRequest) + + then: + def ex = thrown(AbortOperationException) + ex.message.contains('Authentication required') + } + + def 'should handle publish failure with no auth token'() { + given: + def config = new RegistryConfig([url: url]) + def client = new ModuleRegistryClient(config) + def publishRequest = [name: 'nf-core/mymodule', version: '1.0.0'] + + when: + client.publishModule('nf-core/mymodule', publishRequest) + + then: + def ex = thrown(AbortOperationException) + ex.message.contains('Authentication required') + } + + // Helper methods + + private byte[] createTestModulePackage() { + def baos = new ByteArrayOutputStream() + + new GZIPOutputStream(baos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add main.nf + def mainContent = 'process TEST { script: "echo test" }' + addTarEntry(tos, 'main.nf', mainContent.bytes) + + // Add meta.yml + def metaContent = 'name: test\nversion: 1.0.0' + addTarEntry(tos, 'meta.yml', metaContent.bytes) + } + } + + return baos.toByteArray() + } + + private void addTarEntry(TarArchiveOutputStream tos, String name, byte[] content) { + def entry = new TarArchiveEntry(name) + entry.setSize(content.length) + tos.putArchiveEntry(entry) + tos.write(content) + tos.closeArchiveEntry() + } + + private String computeSha256(byte[] data) { + def digest = java.security.MessageDigest.getInstance('SHA-256') + def hash = digest.digest(data) + return hash.encodeHex().toString() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy new file mode 100644 index 0000000000..2c87bdde8e --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleResolverTest.groovy @@ -0,0 +1,193 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.config.RegistryConfig +import nextflow.exception.AbortOperationException +import nextflow.file.FileHelper +import spock.lang.Specification +import spock.lang.TempDir + +import java.nio.file.Files +import java.nio.file.Path + +/** + * Tests for ModuleResolver + * + * @author Jorge Ejarque + */ +class ModuleResolverTest extends Specification { + + @TempDir + Path tempDir + + def 'should throw exception when resolving non-installed module without auto-install'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + + when: + resolver.resolve(reference, null, false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('not installed') + e.message.contains('nextflow module install') + } + + def 'should throw exception when installed module is corrupted'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create corrupted module (directory exists but no main.nf) + Files.createDirectories(moduleDir) + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + + when: + resolver.resolve(reference, null, false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('corrupted') + + cleanup: + FileHelper.deletePath(moduleDir) + } + + def 'should warn about locally modified module'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create module with mismatched checksum + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + ModuleChecksum.save(moduleDir, 'wrong-checksum') + + when: + def result = resolver.resolve(reference, null, false) + + then: + result != null + result == moduleDir.resolve('main.nf') + + cleanup: + FileHelper.deletePath(moduleDir) + } + + def 'should throw exception when version mismatch without auto-install'() { + given: + def resolver = new ModuleResolver(tempDir, new RegistryConfig()) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create module with different version + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + + // Compute and save correct checksum + def checksum = ModuleChecksum.compute(moduleDir) + ModuleChecksum.save(moduleDir, checksum) + + when: + resolver.resolve(reference, '2.0.0', false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('version mismatch') + e.message.contains('installed=1.0.0') + e.message.contains('required=2.0.0') + + cleanup: + if( moduleDir ) FileHelper.deletePath(moduleDir) + } + + def 'should resolve installed module with matching version'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create valid module + Files.createDirectories(moduleDir) + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + + // Compute and save correct checksum + def checksum = ModuleChecksum.compute(moduleDir) + ModuleChecksum.save(moduleDir, checksum) + + when: + def result = resolver.resolve(reference, '1.0.0', false) + + then: + result == mainFile + + cleanup: + FileHelper.deletePath(moduleDir) + } + + def 'should throw exception when trying to update a module with local modifications without force'() { + given: + def resolver = new ModuleResolver(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def storage = new ModuleStorage(tempDir) + def moduleDir = storage.getModuleDir(reference) + + // Create module with wrong checksum (simulating local modifications) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + ''' + ModuleChecksum.save(moduleDir, 'wrong-checksum') + + when: + resolver.installModule(reference, '2.0.0', false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('local modifications') + e.message.contains('-force') + + cleanup: + FileHelper.deletePath(moduleDir) + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy new file mode 100644 index 0000000000..17aa247679 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleSpecTest.groovy @@ -0,0 +1,165 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import nextflow.exception.AbortOperationException +import spock.lang.Specification +import spock.lang.TempDir + +import java.nio.file.Path + +/** + * Tests for ModuleSpec + * + * @author Paolo Di Tommaso + */ +class ModuleSpecTest extends Specification { + + @TempDir + Path tempDir + + def 'should load valid spec' () { + given: + def metaYaml = tempDir.resolve('meta.yml') + metaYaml.text = ''' +name: nf-core/fastqc +version: 1.0.0 +description: FastQC quality control +authors: + - John Doe +license: MIT +keywords: + - quality-control + - fastq +requires: + nextflow: ">=24.04.0" +''' + + when: + def spec = ModuleSpec.load(metaYaml) + + then: + spec.name == 'nf-core/fastqc' + spec.version == '1.0.0' + spec.description == 'FastQC quality control' + spec.authors == ['John Doe'] + spec.license == 'MIT' + spec.keywords == ['quality-control', 'fastq'] + spec.requires == ['nextflow': '>=24.04.0'] + } + + def 'should fail to load non-existent spec' () { + given: + def metaYaml = tempDir.resolve('meta.yml') + + when: + ModuleSpec.load(metaYaml) + + then: + thrown(AbortOperationException) + } + + def 'should validate complete spec' () { + given: + def spec = new ModuleSpec( + name: 'nf-core/fastqc', + version: '1.0.0', + description: 'FastQC quality control', + license: 'MIT' + ) + + when: + def errors = spec.validate() + + then: + errors.isEmpty() + spec.isValid() + } + + def 'should detect missing required fields' () { + given: + def spec = new ModuleSpec( + name: 'nf-core/fastqc' + // missing version, description, license + ) + + when: + def errors = spec.validate() + + then: + errors.size() == 3 + errors.any { it.contains('version') } + errors.any { it.contains('description') } + errors.any { it.contains('license') } + !spec.isValid() + } + + def 'should validate version format' () { + given: + def spec = new ModuleSpec( + name: 'nf-core/fastqc', + version: version, + description: 'Test', + license: 'MIT' + ) + + when: + def errors = spec.validate() + + then: + errors.isEmpty() == valid + + where: + version | valid + '1.0.0' | true + '1.0.0-alpha' | true + '1.0.0-beta.1' | true + '1.0' | false + 'v1.0.0' | false + '1.0.0.0' | false + } + + def 'should validate module name format' () { + given: + def spec = new ModuleSpec( + name: name, + version: '1.0.0', + description: 'Test', + license: 'MIT' + ) + + when: + def errors = spec.validate() + + then: + errors.isEmpty() == valid + + where: + name | valid + 'nf-core/fastqc' | true + 'myorg/my-module' | true + 'org_1/tool_2' | true + 'nf-core/gfatools/gfa2fa' | true // nested module path + 'myorg/tools/sub/module' | true // deeply nested + 'org.name/tool/sub' | true // dot in scope + 'fastqc' | false + '@nf-core/fastqc' | false + 'nf-core/fast qc' | false + 'nf-core/' | false // trailing slash + '/nf-core/fastqc' | false // leading slash + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy b/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy new file mode 100644 index 0000000000..9abf819b21 --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/module/ModuleStorageTest.groovy @@ -0,0 +1,532 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module + +import java.nio.file.Files +import java.nio.file.Path +import java.util.zip.GZIPOutputStream +import org.apache.commons.compress.archivers.tar.TarArchiveEntry +import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream + +import nextflow.exception.AbortOperationException + +import spock.lang.Specification + +/** + * Test suite for ModuleStorage + * + * @author Jorge Ejarque + */ +class ModuleStorageTest extends Specification { + + Path tempDir + + def setup() { + tempDir = Files.createTempDirectory('nf-module-storage-test-') + } + + def cleanup() { + tempDir?.deleteDir() + } + + def 'should get module directory path'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + + when: + def moduleDir = storage.getModuleDir(reference) + + then: + moduleDir == tempDir.resolve('modules/nf-core/fastqc') + } + + def 'should check if module is installed'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + when: + def installed = storage.isInstalled(reference) + + then: + !installed + + when: + Files.createDirectories(moduleDir) + installed = storage.isInstalled(reference) + + then: + installed + } + + def 'should return null for non-existent module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'nonexistent') + + when: + def installed = storage.getInstalledModule(reference) + + then: + installed == null + } + + def 'should get installed module with metadata'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + Files.createDirectories(moduleDir) + + // Create main.nf + def mainFile = moduleDir.resolve('main.nf') + mainFile.text = 'process FASTQC { }' + + // Create meta.yml with version + def metaFile = moduleDir.resolve('meta.yml') + metaFile.text = ''' + name: nf-core/fastqc + version: 1.0.0 + description: FastQC quality control + keywords: + - quality-control + - fastqc + '''.stripIndent() + + // Create .module-info file + ModuleChecksum.save(moduleDir, 'abc123def456') + def moduleInfoFile = moduleDir.resolve('.module-info') + + when: + def installed = storage.getInstalledModule(reference) + + then: + installed != null + installed.reference == reference + installed.directory == moduleDir + installed.mainFile == mainFile + installed.manifestFile == moduleDir.resolve('meta.yml') + installed.moduleInfoFile == moduleInfoFile + installed.expectedChecksum == 'abc123def456' + installed.installedVersion == '1.0.0' + } + + def 'should list all installed modules'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create multiple modules + def modules = [ + new ModuleReference('nf-core', 'fastqc'), + new ModuleReference('nf-core', 'multiqc'), + new ModuleReference('myorg', 'custom') + ] + + modules.each { ref -> + def moduleDir = storage.getModuleDir(ref) + Files.createDirectories(moduleDir) + + // Create main.nf + moduleDir.resolve('main.nf').text = 'process TEST { }' + + // Create meta.yml with version + moduleDir.resolve('meta.yml').text = """ + name: ${ref} + version: 1.0.0 + """.stripIndent() + + // Create .module-info + ModuleChecksum.save(moduleDir, 'checksum') + } + + when: + def installed = storage.listInstalled() + + then: + installed.size() == 3 + installed*.reference.fullName.sort() == ['myorg/custom', 'nf-core/fastqc', 'nf-core/multiqc'] + } + + def 'should list nested modules recursively'() { + given: + def storage = new ModuleStorage(tempDir) + + // Create modules with nested paths + def modules = [ + new ModuleReference('nf-core', 'fastqc'), + new ModuleReference('nf-core', 'gfatools/gfa2fa'), + new ModuleReference('nf-core', 'gfatools/gfa2gfa'), + new ModuleReference('myorg', 'tools/subtools/module') + ] + + modules.each { ref -> + def moduleDir = storage.getModuleDir(ref) + Files.createDirectories(moduleDir) + + // Create main.nf + moduleDir.resolve('main.nf').text = 'process TEST { }' + + // Create meta.yml with version + moduleDir.resolve('meta.yml').text = """ + name: ${ref} + version: 1.0.0 + description: Test module + license: MIT + """.stripIndent() + + // Create .module-info + ModuleChecksum.save(moduleDir, 'checksum') + } + + when: + def installed = storage.listInstalled() + + then: + installed.size() == 4 + installed*.reference.fullName.sort() == [ + 'myorg/tools/subtools/module', + 'nf-core/fastqc', + 'nf-core/gfatools/gfa2fa', + 'nf-core/gfatools/gfa2gfa' + ] + } + + def 'should ignore directories without MODULE_INFO_FILE and not descend into module subdirectories'() { + given: + def storage = new ModuleStorage(tempDir) + + // A valid module with subdirectories (bin/, src/) + def moduleDir = storage.getModuleDir(new ModuleReference('nf-core', 'fastqc')) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process FASTQC { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + ModuleChecksum.save(moduleDir, 'checksum') + // Subdirectories inside the module — must NOT be listed as separate modules + def binDir = moduleDir.resolve('bin') + Files.createDirectories(binDir) + binDir.resolve('fastqc.sh').text = '#!/bin/bash' + def srcDir = moduleDir.resolve('src/main') + Files.createDirectories(srcDir) + srcDir.resolve('helper.groovy').text = 'def foo() {}' + + // A plain directory tree with no MODULE_INFO_FILE anywhere — must be ignored entirely + def orphanDir = tempDir.resolve('modules/other-org/tool/deep/nested') + Files.createDirectories(orphanDir) + orphanDir.resolve('somefile.txt').text = 'not a module' + + when: + def installed = storage.listInstalled() + + then: + installed.size() == 1 + installed[0].reference.fullName == 'nf-core/fastqc' + } + + def 'should return empty list when no modules installed'() { + given: + def storage = new ModuleStorage(tempDir) + + when: + def installed = storage.listInstalled() + + then: + installed.isEmpty() + } + + def 'should install module from gzip package'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def version = '1.0.0' + def url = "http://registry.com" + + // Create a gzip package file + def packageFile = Files.createTempFile('module-', '.tgz') + createTestPackage(packageFile) + + when: + def installed = storage.installModule(reference, version, packageFile, url) + + then: + installed != null + installed.reference == reference + installed.installedVersion == '1.0.0' + Files.exists(installed.mainFile) + Files.exists(installed.moduleInfoFile) + Files.exists(installed.directory) + installed.registryUrl == url + installed.expectedChecksum == ModuleChecksum.compute(installed.directory) + + cleanup: + packageFile?.delete() + } + + def 'should replace existing module on install'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + def url = "http://registry.com" + + // Create existing installation + Files.createDirectories(moduleDir) + def oldFile = moduleDir.resolve('old-file.txt') + oldFile.text = 'old content' + + // Create package + def packageFile = Files.createTempFile('module-', '.tgz') + createTestPackage(packageFile) + + when: + def installed = storage.installModule(reference, '2.0.0', packageFile, url) + + then: + installed != null + !Files.exists(oldFile) // Old file should be removed + Files.exists(installed.mainFile) + + cleanup: + packageFile?.delete() + } + + def 'should remove installed module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create valid module with correct checksum + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + ModuleChecksum.save(moduleDir, ModuleChecksum.compute(moduleDir)) + + expect: + Files.exists(moduleDir) + + when: + def removed = storage.removeModule(reference, false) + + then: + removed + !Files.exists(moduleDir) + } + + def 'should return false when removing non-existent module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'nonexistent') + + when: + def removed = storage.removeModule(reference, false) + + then: + !removed + } + + def 'should throw when removing module without .module-info and force is false'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create module WITHOUT .module-info (but with required files for NO_REMOTE_MODULE integrity) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + + when: + storage.removeModule(reference, false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('.module-info missing') + Files.exists(moduleDir) + } + + def 'should force remove module without .module-info'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create module WITHOUT .module-info (but with required files for NO_REMOTE_MODULE integrity) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + + expect: + Files.exists(moduleDir) + + when: + def removed = storage.removeModule(reference, true) + + then: + removed + !Files.exists(moduleDir) + } + + def 'should throw when removing modified module and force is false'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create module with mismatched checksum (simulates local modification) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + ModuleChecksum.save(moduleDir, 'stale-checksum-from-install') + // Now modify a file to cause checksum mismatch + moduleDir.resolve('main.nf').text = 'process TEST_MODIFIED { }' + + when: + storage.removeModule(reference, false) + + then: + def e = thrown(AbortOperationException) + e.message.contains('local modifications') + Files.exists(moduleDir) + } + + def 'should force remove modified module'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def moduleDir = storage.getModuleDir(reference) + + // Create module with mismatched checksum (simulates local modification) + Files.createDirectories(moduleDir) + moduleDir.resolve('main.nf').text = 'process TEST { }' + moduleDir.resolve('meta.yml').text = 'name: nf-core/fastqc\nversion: 1.0.0\n' + ModuleChecksum.save(moduleDir, 'stale-checksum-from-install') + moduleDir.resolve('main.nf').text = 'process TEST_MODIFIED { }' + + when: + def removed = storage.removeModule(reference, true) + + then: + removed + !Files.exists(moduleDir) + } + + def 'should compute and save checksum on install'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def packageFile = Files.createTempFile('module-', '.tgz') + createTestPackage(packageFile) + def url = "http://registry.com" + + when: + def installed = storage.installModule(reference, '1.0.0', packageFile, url) + + then: + installed.expectedChecksum != null + installed.expectedChecksum.length() > 0 + Files.exists(installed.moduleInfoFile) + + cleanup: + packageFile?.delete() + } + + def 'should handle installation failure gracefully'() { + given: + def storage = new ModuleStorage(tempDir) + def reference = new ModuleReference('nf-core', 'fastqc') + def invalidPackage = Files.createTempFile('invalid-', '.tgz') + invalidPackage.text = 'not a valid gzip file' + + when: + storage.installModule(reference, '1.0.0', invalidPackage, null) + + then: + thrown(Exception) + + and: + !storage.isInstalled(reference) // Should not leave partial installation + + cleanup: + invalidPackage?.delete() + } + + /** + * Helper method to create a test package file + */ + private void createTestPackage(Path packageFile) { + // Create a temporary directory with module content + def tempModuleDir = Files.createTempDirectory('temp-module-') + + // Create main.nf + tempModuleDir.resolve('main.nf').text = ''' + process FASTQC { + input: + path reads + + output: + path "*.html" + + script: + """ + fastqc ${reads} + """ + } + '''.stripIndent() + + // Create meta.yml + tempModuleDir.resolve('meta.yml').text = ''' + name: nf-core/fastqc + version: 1.0.0 + description: FastQC quality control + keywords: + - quality-control + - fastqc + '''.stripIndent() + + // Create README + tempModuleDir.resolve('README.md').text = '# FastQC Module' + + // Create tar.gz archive using Java libraries + Files.newOutputStream(packageFile).withCloseable { fos -> + new GZIPOutputStream(fos).withCloseable { gzos -> + new TarArchiveOutputStream(gzos).withCloseable { tos -> + // Add all files from tempModuleDir + Files.walk(tempModuleDir).each { Path path -> + if (Files.isRegularFile(path)) { + // Get relative path + def relativePath = tempModuleDir.relativize(path).toString() + + // Create tar entry + def entry = new TarArchiveEntry(path.toFile(), relativePath) + tos.putArchiveEntry(entry) + + // Write file content + Files.copy(path, tos) + + tos.closeArchiveEntry() + } + } + } + } + } + + // Cleanup temp directory + tempModuleDir.deleteDir() + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy index b5e1b7425e..527cbbfbbd 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/IncludeDefTest.groovy @@ -45,8 +45,8 @@ class IncludeDefTest extends Specification { expect: include.resolveModulePath('/abs/foo.nf') == '/abs/foo.nf' as Path - include.resolveModulePath('module.nf') == '/some/path/module.nf' as Path - include.resolveModulePath('foo/bar.nf') == '/some/path/foo/bar.nf' as Path + include.resolveModulePath('./module.nf') == '/some/path/module.nf' as Path + include.resolveModulePath('./foo/bar.nf') == '/some/path/foo/bar.nf' as Path when: include.resolveModulePath('http://foo.com/bar') @@ -66,17 +66,17 @@ class IncludeDefTest extends Specification { include.getOwnerPath() >> script when: - def result = include.realModulePath( 'mod-x.nf') + def result = include.realModulePath( './mod-x.nf') then: result == module when: - result = include.realModulePath('mod-x') + result = include.realModulePath('./mod-x') then: result == module when: - include.realModulePath('xyz') + include.realModulePath('./xyz') then: thrown(NoSuchFileException) @@ -98,21 +98,21 @@ class IncludeDefTest extends Specification { // when the module name reference a directory that contains // a file named 'main.nf', it's considered a module 'bundle' when: - def result = include.realModulePath('foo') + def result = include.realModulePath('./foo') then: result == module when: - include.realModulePath('bar') + include.realModulePath('./bar') then: thrown(NoSuchFileException) when: folder.resolve('bar').mkdir() - include.realModulePath('bar') + include.realModulePath('./bar') then: def e = thrown(ScriptCompilationException) - e.message == "Include 'bar' does not provide any module script -- the following path should contain a 'main.nf' script: '${folder.resolve('bar')}'" + e.message == "Include './bar' does not provide any module script -- the following path should contain a 'main.nf' script: '${folder.resolve('bar')}'" } def 'should check valid path' () { @@ -137,6 +137,16 @@ class IncludeDefTest extends Specification { when: include.checkValidPath('this/dir') then: + noExceptionThrown() // valid remote module reference (scope/name) + + when: + include.checkValidPath('nf-core/fastqc') + then: + noExceptionThrown() // valid remote module reference + + when: + include.checkValidPath('invalid!') + then: thrown(IllegalModulePath) when: diff --git a/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy b/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy index 16b45f87d3..c79ed822d1 100644 --- a/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy +++ b/modules/nextflow/src/testFixtures/groovy/test/TestHelper.groovy @@ -15,13 +15,17 @@ */ package test + import java.nio.file.Files import java.nio.file.Path import java.util.zip.GZIPInputStream import com.google.common.jimfs.Configuration import com.google.common.jimfs.Jimfs +import com.google.common.jimfs.JimfsPath import groovy.transform.Memoized +import nextflow.util.KryoHelper +import nextflow.util.PathSerializer /** * * @author Paolo Di Tommaso @@ -53,6 +57,13 @@ class TestHelper { static private fs = Jimfs.newFileSystem(Configuration.unix()); + static { + // Some tests failed after a Guava update (Guava 33.4+) when using Jimfs. + // Adding a default serializer for JimfsPaths to prevent Kryo's FieldSerializer from recursing into internal + // fields that may contain non-serializable objects such as lambdas + KryoHelper.kryo().addDefaultSerializer(JimfsPath.class, PathSerializer) + } + static Path createInMemTempDir() { Path tmp = fs.getPath("/tmp"); tmp.mkdir() diff --git a/modules/nf-commons/build.gradle b/modules/nf-commons/build.gradle index 958c789478..8ea4b12220 100644 --- a/modules/nf-commons/build.gradle +++ b/modules/nf-commons/build.gradle @@ -38,7 +38,7 @@ dependencies { api 'io.seqera:lib-retry:2.0.0' // patch gson dependency required by pf4j api 'com.google.code.gson:gson:2.13.1' - api 'io.seqera:npr-api:0.6.1' + api 'io.seqera:npr-api:0.21.4-SNAPSHOT' /* testImplementation inherited from top gradle build file */ testImplementation(testFixtures(project(":nextflow"))) diff --git a/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy b/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy new file mode 100644 index 0000000000..f43c0f131d --- /dev/null +++ b/modules/nf-commons/src/main/nextflow/config/RegistryConfig.groovy @@ -0,0 +1,91 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config + +import groovy.transform.CompileStatic +import nextflow.SysEnv +import nextflow.config.spec.ConfigOption +import nextflow.config.spec.ConfigScope +import nextflow.config.spec.ScopeName +import nextflow.script.dsl.Description + +/** + * Configuration scope for module registry settings + * + * @author Jorge Ejarque + */ +@ScopeName("registry") +@Description(""" + The `registry` scope provides configuration for the Nextflow module registry. + This includes the registry URL(s) and API key for authentication. +""") +@CompileStatic +class RegistryConfig implements ConfigScope { + + public static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' + + @ConfigOption + @Description("Registry URL or list of registry URLs in priority order (primary URL first)") + private final Collection url + + @ConfigOption + @Description("API key for authenticating with the primary registry") + private final String apiKey + + /* required by extension point -- do not remove */ + RegistryConfig() { + this.url = [DEFAULT_REGISTRY_URL] + this.apiKey = null + } + + RegistryConfig(Map opts) { + final urlObject = opts.url ?: [DEFAULT_REGISTRY_URL] + if (urlObject instanceof Collection) + this.url = urlObject as Collection + else + this.url = [urlObject.toString()] + this.apiKey = opts.apiKey as String + } + + /** + * Get the primary (first) registry URL + * + * @return The primary registry URL + */ + String getUrl() { + return this.url ? url[0] as String : DEFAULT_REGISTRY_URL + } + + /** + * Get all registry URLs (primary first, fallbacks after) + * + * @return Collection of registry URLs + */ + Collection getAllUrls() { + return this.url ?: [DEFAULT_REGISTRY_URL] + } + + /** + * Get the API key for the primary registry. + * Authentication is only supported for the primary registry. + * + * @return The API key, or 'NXF_REGISTRY_TOKEN' env value if not configured + */ + String getApiKey() { + return apiKey ?: SysEnv.get('NXF_REGISTRY_TOKEN') + } +} diff --git a/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy b/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy index d066f51640..0313b9d0e4 100644 --- a/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy +++ b/modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy @@ -25,7 +25,7 @@ import groovy.transform.CompileStatic import groovy.util.logging.Slf4j import io.seqera.http.HxClient import io.seqera.npr.api.schema.v1.ListDependenciesResponse -import io.seqera.npr.api.schema.v1.Plugin +import io.seqera.npr.api.schema.v1.PluginDependency import nextflow.BuildInfo import nextflow.util.RetryConfig import org.pf4j.PluginRuntimeException @@ -168,11 +168,11 @@ class HttpPluginRepository implements PrefetchUpdateRepository { } try { final ListDependenciesResponse decoded = encoder.decode(body) - if( decoded.plugins == null ) { + if( !decoded.plugins ) { throw new PluginRuntimeException("Failed to download plugin metadata: Failed to parse response body") } final result = new HashMap() - for( Plugin plugin : decoded.plugins ) { + for( PluginDependency plugin : decoded.plugins ) { if( plugin.releases ) { final pluginInfo = mapToPluginInfo(plugin) result.put(plugin.id, pluginInfo) @@ -195,7 +195,7 @@ class HttpPluginRepository implements PrefetchUpdateRepository { * @param plugin The Plugin object from the repository API response * @return A PluginInfo object compatible with pf4j's update repository interface */ - static protected PluginInfo mapToPluginInfo(Plugin plugin) { + static protected PluginInfo mapToPluginInfo(PluginDependency plugin) { assert plugin.releases, "Plugin releases cannot be empty" final pluginInfo = new PluginInfo() diff --git a/modules/nf-commons/src/test/nextflow/config/RegistryConfigTest.groovy b/modules/nf-commons/src/test/nextflow/config/RegistryConfigTest.groovy new file mode 100644 index 0000000000..26c727d960 --- /dev/null +++ b/modules/nf-commons/src/test/nextflow/config/RegistryConfigTest.groovy @@ -0,0 +1,107 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.config + +import nextflow.SysEnv +import spock.lang.Specification + +/** + * Tests for RegistryConfig + * + * @author Jorge Ejarque + */ +class RegistryConfigTest extends Specification { + + def 'should create config with default values'() { + when: + def config = new RegistryConfig() + + then: + config.url == RegistryConfig.DEFAULT_REGISTRY_URL + config.allUrls == [RegistryConfig.DEFAULT_REGISTRY_URL] + config.apiKey == null + } + + def 'should initialize with custom URL as string'() { + when: + def config = new RegistryConfig([url: 'https://custom.registry.com']) + + then: + config.url == 'https://custom.registry.com' + config.allUrls == ['https://custom.registry.com'] + } + + def 'should initialize with URL as list'() { + when: + def config = new RegistryConfig([url: ['https://primary.registry.com', 'https://fallback.registry.com']]) + + then: + config.url == 'https://primary.registry.com' + config.allUrls == ['https://primary.registry.com', 'https://fallback.registry.com'] + } + + def 'should use default URL when none provided'() { + when: + def config = new RegistryConfig([:]) + + then: + config.url == RegistryConfig.DEFAULT_REGISTRY_URL + config.allUrls == [RegistryConfig.DEFAULT_REGISTRY_URL] + config.apiKey == null + } + + def 'should initialize with apiKey'() { + when: + def config = new RegistryConfig([url: 'https://registry.com', apiKey: 'token123']) + + then: + config.apiKey == 'token123' + } + + + + def 'should fall back to NXF_REGISTRY_TOKEN env var when apiKey not set'() { + given: + SysEnv.push([NXF_REGISTRY_TOKEN: 'env_var_token']) + def config = new RegistryConfig([url: 'https://registry.com']) + + expect: + // Without env var set, returns null + config.apiKey == 'env_var_token' + + cleanup: + SysEnv.pop() + } + + def 'should preserve order of URLs in list'() { + when: + def config = new RegistryConfig([url: ['https://first.com', 'https://second.com', 'https://third.com']]) + + then: + config.allUrls == ['https://first.com', 'https://second.com', 'https://third.com'] + config.url == 'https://first.com' + } + + def 'should handle empty list gracefully'() { + when: + def config = new RegistryConfig([url: []]) + + then: + config.allUrls == [RegistryConfig.DEFAULT_REGISTRY_URL] + config.url == RegistryConfig.DEFAULT_REGISTRY_URL + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java new file mode 100644 index 0000000000..b438dbf7af --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java @@ -0,0 +1,45 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Fallback implementation of RemoteModuleResolver that is used when no other + * implementation is found via the SPI mechanism. + * + *

This implementation throws an exception with a helpful error message + * indicating that remote module resolution is not available. + * + * @author Jorge Ejarque + */ +public class FallbackRemoteModuleResolver implements RemoteModuleResolver { + + @Override + public Path resolve(String moduleName, Path baseDir) { + if (!Files.exists(baseDir.resolve(moduleName))) { + throw new IllegalStateException("Module '" + moduleName + "' not locally found at 'modules' folder - use 'nextflow install' to download module files"); + } + return baseDir.resolve(moduleName).resolve("main.nf"); + } + + @Override + public int getPriority() { + return Integer.MIN_VALUE; // Fallback has lowest possible priority + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java new file mode 100644 index 0000000000..5c6cf1fc9b --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java @@ -0,0 +1,68 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import java.nio.file.Path; + +/** + * Service Provider Interface for resolving remote modules referenced with '@scope/name' syntax. + * + *

Implementations should handle: + *

    + *
  • Checking if a module is already installed locally
    • + *
  • Downloading modules from a registry if not present
    • + *
  • Version resolution and validation
    • + *
+ * + *

The interface follows the Java SPI pattern. Implementations should be registered + * in META-INF/services/nextflow.module.spi.RemoteModuleResolver + * + * @author Jorge Ejarque + */ +public interface RemoteModuleResolver { + + /** + * Resolve a remote module reference (e.g., '@scope/name') to a local path. + * + *

This method should: + *

    + *
  1. Parse the module reference
    2. + *
  2. Check if the module is already installed locally
    3. + *
  3. Download and install the module if not present (auto-install)
    4. + *
  4. Validate version constraints if specified
    5. + *
+ * + * @param moduleName The module reference string (e.g., '@scope/name' or '@scope/name@version') + * @param baseDir The base directory for the project (used to locate the modules directory) + * @return Path to the resolved module's main.nf file + * @throws IllegalArgumentException if the module reference is invalid or resolution fails + */ + Path resolve(String moduleName, Path baseDir); + + /** + * Get the priority of this resolver. Higher priority resolvers are tried first. + * + *

Use this to allow custom implementations to override the default resolver. + * The default implementation should return 0. Custom implementations can return + * positive values to take precedence. + * + * @return Priority value (higher = tried first), default should be 0 + */ + default int getPriority() { + return 0; + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java new file mode 100644 index 0000000000..da1b2da9e7 --- /dev/null +++ b/modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java @@ -0,0 +1,92 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.module.spi; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.ServiceLoader; + +/** + * Provider for accessing RemoteModuleResolver implementations via SPI. + * + *

This class uses the Java ServiceLoader mechanism to discover and load + * implementations of RemoteModuleResolver. It selects the implementation + * with the highest priority. + * + * @author Jorge Ejarque + */ +public class RemoteModuleResolverProvider { + + private static final Logger log = LoggerFactory.getLogger(RemoteModuleResolverProvider.class); + private static RemoteModuleResolver instance; + + /** + * Get the RemoteModuleResolver instance with the highest priority. + * + *

This method lazily loads and caches the resolver. It discovers all + * implementations via ServiceLoader and selects the one with the highest + * priority value. + * + *

If no implementations are found, returns the FallbackRemoteModuleResolver + * which throws an informative exception. + * + * @return The RemoteModuleResolver instance with highest priority + */ + public static synchronized RemoteModuleResolver getInstance() { + if (instance == null) { + instance = loadResolver(); + } + return instance; + } + + private static RemoteModuleResolver loadResolver() { + List resolvers = new ArrayList<>(); + ServiceLoader loader = ServiceLoader.load(RemoteModuleResolver.class); + + // Collect all available resolvers + for (RemoteModuleResolver resolver : loader) { + resolvers.add(resolver); + log.debug("Discovered RemoteModuleResolver: {} with priority {}", + resolver.getClass().getName(), resolver.getPriority()); + } + + // Sort by priority (highest first) + resolvers.sort(Comparator.comparingInt(RemoteModuleResolver::getPriority).reversed()); + + if (resolvers.isEmpty()) { + log.warn("No RemoteModuleResolver implementations found via SPI, using fallback"); + return new FallbackRemoteModuleResolver(); + } + + RemoteModuleResolver selected = resolvers.get(0); + log.debug("Selected RemoteModuleResolver: {} with priority {}", + selected.getClass().getName(), selected.getPriority()); + + return selected; + } + + /** + * Reset the cached instance. Used primarily for testing. + */ + public static synchronized void reset() { + instance = null; + } +} \ No newline at end of file diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java b/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java index 5b76af4111..c52931142a 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ModuleResolver.java @@ -23,6 +23,7 @@ import java.util.Set; import java.util.function.Function; +import nextflow.module.spi.RemoteModuleResolverProvider; import nextflow.script.ast.IncludeNode; import nextflow.script.ast.ScriptNode; import org.codehaus.groovy.control.SourceUnit; @@ -72,8 +73,18 @@ private SourceUnit resolveInclude(IncludeNode node, SourceUnit sourceUnit, Funct var source = node.source.getText(); if( source.startsWith("plugin/") ) return null; - var uri = sourceUnit.getSource().getURI(); - var includeUri = getIncludeUri(uri, source); + + var parent = Path.of(sourceUnit.getSource().getURI()).getParent(); + + // Resolve remote module paths (scope/name format, not starting with local prefixes) + if( isRemoteModule(source) ) { + var modules = Path.of("./modules"); + var resolver = RemoteModuleResolverProvider.getInstance(); + resolver.resolve(source, modules.getParent()); + parent = modules; + } + + var includeUri = getIncludeUri(parent, source); if( compiler.getSource(includeUri) != null ) return null; if( !Files.exists(Path.of(includeUri)) ) @@ -86,8 +97,15 @@ private SourceUnit resolveInclude(IncludeNode node, SourceUnit sourceUnit, Funct return includeSource; } - private static URI getIncludeUri(URI uri, String source) { - Path includePath = Path.of(uri).getParent().resolve(source); + static boolean isRemoteModule(String source) { + if( source.startsWith("/") || source.startsWith("./") || source.startsWith("../") ) + return false; + // Must match scope/name pattern: scope is lowercase alphanumeric with dots/underscores/hyphens + return source.matches("^[a-z0-9][a-z0-9._\\-]*/[a-z][a-z0-9._\\-]*(/[a-z][a-z0-9._\\-]*)*$"); + } + + private static URI getIncludeUri(Path parent, String source) { + Path includePath = parent.resolve(source); if( Files.isDirectory(includePath) ) includePath = includePath.resolve("main.nf"); else if( !source.endsWith(".nf") ) diff --git a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java index 7b8c9bbd0d..11ad336a74 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java +++ b/modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java @@ -85,7 +85,10 @@ public void visitInclude(IncludeNode node) { setPlaceholderTargets(node); return; } - var includeUri = getIncludeUri(uri, source); + + var isRemoteModule = ModuleResolver.isRemoteModule(source); + var parent = isRemoteModule ? Path.of("modules") : Path.of(uri).getParent(); + var includeUri = getIncludeUri(parent, source); if( !isIncludeStale(node, includeUri) ) return; changed = true; @@ -123,8 +126,8 @@ private static void setPlaceholderTargets(IncludeNode node) { } } - private static URI getIncludeUri(URI uri, String source) { - Path includePath = Path.of(uri).getParent().resolve(source); + private static URI getIncludeUri(Path parent, String source) { + Path includePath = parent.resolve(source); if( Files.isDirectory(includePath) ) includePath = includePath.resolve("main.nf"); else if( !source.endsWith(".nf") ) diff --git a/settings.gradle b/settings.gradle index fe58cb245b..bb90d2b520 100644 --- a/settings.gradle +++ b/settings.gradle @@ -49,4 +49,5 @@ include 'plugins:nf-cloudcache' include 'plugins:nf-k8s' include 'plugins:nf-seqera' +//includeBuild('../plugin-registry') //includeBuild '../sched' diff --git a/specs/251117-module-system/checklists/requirements.md b/specs/251117-module-system/checklists/requirements.md new file mode 100644 index 0000000000..fe90e53ec3 --- /dev/null +++ b/specs/251117-module-system/checklists/requirements.md @@ -0,0 +1,39 @@ +# Specification Quality Checklist: Nextflow Module System Client + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-01-15 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Specification is complete and ready for `/speckit.plan` +- All 8 user stories have clear acceptance scenarios +- 29 functional requirements defined across 6 categories +- 8 success criteria defined with measurable outcomes +- Edge cases documented for error handling scenarios +- Registry backend is explicitly out of scope (assumed implemented) \ No newline at end of file diff --git a/specs/251117-module-system/contracts/registry-api.yaml b/specs/251117-module-system/contracts/registry-api.yaml new file mode 100644 index 0000000000..2541e12c02 --- /dev/null +++ b/specs/251117-module-system/contracts/registry-api.yaml @@ -0,0 +1,388 @@ +openapi: 3.0.3 +info: + title: Nextflow Module Registry API + description: | + API specification for the Nextflow Module Registry. + This documents the endpoints that the Nextflow module system client consumes. + The registry backend is assumed to be already implemented at registry.nextflow.io. + version: 1.0.0 + contact: + name: Nextflow Team + url: https://nextflow.io + +servers: + - url: https://registry.nextflow.io/api + description: Production registry + +security: + - BearerAuth: [] + +paths: + /modules: + get: + operationId: searchModules + summary: Search modules + description: Search for modules by query text (semantic search across name, description, keywords) + tags: + - Modules + parameters: + - name: query + in: query + required: true + description: Search query text + schema: + type: string + example: "alignment" + - name: limit + in: query + required: false + description: Maximum number of results + schema: + type: integer + default: 10 + maximum: 100 + responses: + '200': + description: Search results + content: + application/json: + schema: + type: object + properties: + modules: + type: array + items: + $ref: '#/components/schemas/ModuleSummary' + total: + type: integer + description: Total matching modules + '400': + description: Invalid query parameters + + /modules/{name}: + get: + operationId: getModule + summary: Get module details + description: Get module metadata including latest release information + tags: + - Modules + parameters: + - name: name + in: path + required: true + description: Module name including scope (e.g., nf-core/fastqc) + schema: + type: string + example: "nf-core/fastqc" + responses: + '200': + description: Module details + content: + application/json: + schema: + $ref: '#/components/schemas/ModuleDetails' + '404': + description: Module not found + + post: + operationId: publishModule + summary: Publish module version + description: Upload and publish a new module version (requires authentication) + tags: + - Modules + security: + - BearerAuth: [] + parameters: + - name: name + in: path + required: true + description: Module name including scope + schema: + type: string + requestBody: + required: true + content: + multipart/form-data: + schema: + type: object + required: + - bundle + properties: + bundle: + type: string + format: binary + description: Module bundle (tar.gz archive) + tags: + type: array + items: + type: string + description: Additional tags for discoverability + responses: + '201': + description: Module published successfully + content: + application/json: + schema: + $ref: '#/components/schemas/PublishResult' + '400': + description: Invalid module bundle or manifest + content: + application/json: + schema: + $ref: '#/components/schemas/ValidationError' + '401': + description: Authentication required + '403': + description: Insufficient permissions for scope + + /modules/{name}/releases: + get: + operationId: listReleases + summary: List module releases + description: Get all available versions of a module + tags: + - Modules + parameters: + - name: name + in: path + required: true + schema: + type: string + responses: + '200': + description: List of releases + content: + application/json: + schema: + type: object + properties: + releases: + type: array + items: + $ref: '#/components/schemas/ReleaseInfo' + '404': + description: Module not found + + /modules/{name}/{version}: + get: + operationId: getRelease + summary: Get specific release + description: Get metadata for a specific module version + tags: + - Modules + parameters: + - name: name + in: path + required: true + schema: + type: string + - name: version + in: path + required: true + description: Semantic version (e.g., 1.0.0) + schema: + type: string + example: "1.0.0" + responses: + '200': + description: Release details + content: + application/json: + schema: + $ref: '#/components/schemas/ReleaseInfo' + '404': + description: Module or version not found + + /modules/{name}/{version}/download: + get: + operationId: downloadModule + summary: Download module bundle + description: Download the module source archive for a specific version + tags: + - Modules + parameters: + - name: name + in: path + required: true + schema: + type: string + - name: version + in: path + required: true + schema: + type: string + responses: + '200': + description: Module bundle + headers: + X-Checksum: + description: SHA-256 checksum of the bundle + schema: + type: string + example: "sha256:abc123..." + Content-Disposition: + description: Suggested filename + schema: + type: string + example: "attachment; filename=nf-core-fastqc-1.0.0.tar.gz" + content: + application/gzip: + schema: + type: string + format: binary + '404': + description: Module or version not found + +components: + securitySchemes: + BearerAuth: + type: http + scheme: bearer + description: | + Authentication token. Can be provided via: + - NXF_REGISTRY_TOKEN environment variable + - registry.auth config block + + schemas: + ModuleSummary: + type: object + required: + - name + - latestVersion + - description + properties: + name: + type: string + description: Full module name with scope + example: "nf-core/fastqc" + latestVersion: + type: string + description: Latest available version + example: "1.2.0" + description: + type: string + description: Short module description + example: "Run FastQC on sequenced reads" + downloadCount: + type: integer + description: Total download count + example: 15420 + keywords: + type: array + items: + type: string + example: ["quality control", "fastq"] + + ModuleDetails: + type: object + required: + - name + - latestVersion + - description + properties: + name: + type: string + example: "nf-core/fastqc" + latestVersion: + type: string + example: "1.2.0" + description: + type: string + authors: + type: array + items: + type: string + example: ["@drpatelh"] + maintainers: + type: array + items: + type: string + license: + type: string + example: "MIT" + keywords: + type: array + items: + type: string + downloadCount: + type: integer + createdAt: + type: string + format: date-time + updatedAt: + type: string + format: date-time + latestRelease: + $ref: '#/components/schemas/ReleaseInfo' + + ReleaseInfo: + type: object + required: + - version + - checksum + - publishedAt + properties: + version: + type: string + description: Semantic version + example: "1.2.0" + checksum: + type: string + description: SHA-256 checksum of bundle + example: "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + publishedAt: + type: string + format: date-time + size: + type: integer + description: Bundle size in bytes + example: 45678 + requires: + type: object + properties: + nextflow: + type: string + example: ">=24.04.0" + plugins: + type: array + items: + type: string + modules: + type: array + items: + type: string + + PublishResult: + type: object + properties: + name: + type: string + version: + type: string + checksum: + type: string + publishedAt: + type: string + format: date-time + downloadUrl: + type: string + format: uri + + ValidationError: + type: object + properties: + code: + type: string + enum: + - INVALID_MANIFEST + - MISSING_MAIN_NF + - MISSING_README + - INVALID_VERSION + - BUNDLE_TOO_LARGE + - DUPLICATE_VERSION + message: + type: string + details: + type: array + items: + type: string \ No newline at end of file diff --git a/specs/251117-module-system/data-model.md b/specs/251117-module-system/data-model.md new file mode 100644 index 0000000000..0d3ef4cf8a --- /dev/null +++ b/specs/251117-module-system/data-model.md @@ -0,0 +1,272 @@ +# Data Model: Nextflow Module System Client + +**Date**: 2026-01-19 +**Feature**: 251117-module-system +**Last Updated**: 2026-02-27 (reflects final implementation) + +## Overview + +This document defines the data entities, their attributes, relationships, and state transitions for the Nextflow module system client implementation. + +--- + +## Entity Definitions + +### 1. ModuleReference + +Represents a reference to a module in DSL `include` statements. + +```groovy +@CompileStatic +class ModuleReference { + String scope // e.g., "nf-core" + String name // e.g., "fastqc" + String fullName // e.g., "@nf-core/fastqc" + + static ModuleReference parse(String source) { + // Parses "@scope/name" format + } + + boolean isRegistryModule() { + return fullName.startsWith('@') + } +} +``` + +**Validation Rules**: +- `scope`: lowercase alphanumeric with hyphens, pattern `[a-z0-9][a-z0-9-]*` +- `name`: lowercase alphanumeric with underscores/hyphens, pattern `[a-z][a-z0-9_-]*` +- `fullName`: must match `^@[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` + +--- + +### 2. ModuleSpec + +Parsed representation of `meta.yaml` file. Class: `nextflow.module.ModuleSpec`. + +```groovy +@CompileStatic +class ModuleSpec { + String name // e.g., "nf-core/fastqc" (without @) + String version // e.g., "1.0.0" + String description // Module description + List keywords // Discovery keywords + List authors // GitHub handles + String license // SPDX identifier + Map requires // dependency -> version constraint + + static ModuleSpec load(Path metaYamlPath) { ... } + List validate() { ... } // Returns list of validation errors + boolean isValid() { ... } +} +``` + +**Validation Rules**: +- `name`: Must match `scope/name` or `scope/path/to/name` pattern +- `version`: Must be valid SemVer (`MAJOR.MINOR.PATCH[-prerelease]`) +- `description`, `license`: Required fields (validate() reports missing) + +**Note**: Tool/argument definitions were removed from the ADR and are not part of `ModuleSpec`. + +--- + +### 3. InstalledModule + +Represents a module in the local `modules/` directory. + +```groovy +@CompileStatic +class InstalledModule { + ModuleReference reference + Path directory // e.g., /project/modules/@nf-core/fastqc + Path mainFile // e.g., /project/modules/@nf-core/fastqc/main.nf + Path manifestFile // e.g., /project/modules/@nf-core/fastqc/meta.yaml + Path checksumFile // e.g., /project/modules/@nf-core/fastqc/.checksum + String installedVersion + String expectedChecksum + + ModuleIntegrity getIntegrity() { + // Compute and compare checksum + } +} + +enum ModuleIntegrity { + VALID, // Checksum matches + MODIFIED, // Checksum mismatch (local changes) + MISSING_CHECKSUM, // No .checksum file + CORRUPTED // Missing required files +} +``` + +**State Transitions**: +``` +[NOT_INSTALLED] --install--> [VALID] +[VALID] --user edits--> [MODIFIED] +[MODIFIED] --install -force--> [VALID] +[VALID] --version change in config--> [VALID] (replaced) +[MODIFIED] --version change in config--> [MODIFIED] (blocked, warn) +``` + +--- + +### 4. ModulesConfig and RegistryConfig + +Modules configuration loaded from `nextflow_spec.json` ( or the `modules {}` block in `nextflow.config` as alternative). Registry settings from the `registry {}` block in `nextflow.config`. + +```groovy +@ScopeName("modules") +@CompileStatic +class ModulesConfig implements ConfigScope { + Map modules = [:] // module fullName -> version + + String getVersion(String moduleName) { ... } + boolean hasVersion(String moduleName) { ... } +} + +@ScopeName("registry") +@CompileStatic +class RegistryConfig implements ConfigScope { + static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' + + Collection url // Registry URL(s) in priority order + String apiKey // API key (falls back to NXF_REGISTRY_TOKEN env var) + + String getUrl() // Returns primary (first) URL + Collection getAllUrls() + String getApiKey() // Returns apiKey or NXF_REGISTRY_TOKEN +} +``` + +**Config Syntax**: +```nextflow +// nextflow_spec.json (current approach) +{ + "modules": { + "@nf-core/fastqc": "1.0.0", + "@nf-core/bwa-align": "1.2.0" + } +} + +// nextflow.config (alternative not currently used) +modules { + '@nf-core/fastqc' = '1.0.0' + '@nf-core/bwa-align' = '1.2.0' +} + +registry { + url = [ + 'https://private.registry.myorg.com', + 'https://registry.nextflow.io/api' + ] + apiKey = '${MYORG_TOKEN}' // Only applied to the primary registry +} +``` + +--- + +### 5. DefaultRemoteModuleResolver (SPI) + +Bridges the DSL parser to the module resolution runtime. Class: `nextflow.module.DefaultRemoteModuleResolver`. + +```groovy +// Implements: nextflow.module.spi.RemoteModuleResolver (nf-lang) +class DefaultRemoteModuleResolver implements RemoteModuleResolver { + int getPriority() { return 0 } // Can be overridden by plugins with higher priority + + Path resolve(String moduleName, Path baseDir) { + // 1. Parse ModuleReference from "@scope/name" + // 2. Read version constraints from nextflow_spec.json / ModulesConfig + // 3. Call ModuleResolver.installModule(reference, version, autoInstall=true) + // 4. Return path to modules/@scope/name/main.nf + } +} +``` + +The SPI is loaded via Java `ServiceLoader` by `RemoteModuleResolverProvider` (in `nf-lang`), which selects the highest-priority implementation available. + +--- + +### 6. PipelineSpec + +Reads and writes `nextflow_spec.json` in the project root. Class: `nextflow.pipeline.PipelineSpec`. + +```groovy +class PipelineSpec { + PipelineSpec(Path baseDir) + Map getModules() + void addModuleEntry(String name, String version) + boolean removeModuleEntry(String name) +} +``` + +--- + +### 6. ModuleResolutionResult + +Result of module resolution process. + +```groovy +@CompileStatic +class ModuleResolutionResult { + ModuleReference reference + Path resolvedPath // Absolute path to main.nf + ResolutionAction action + String message // Warning/info message if any +} + +enum ResolutionAction { + USE_LOCAL, // Used existing local module + DOWNLOADED, // Downloaded from registry + REPLACED, // Replaced local with different version + BLOCKED_MODIFIED, // Local modified, not replaced (warning issued) + FAILED // Resolution failed (error) +} +``` + +--- + +## Relationships + +``` +PipelineSpec (1) -----> (*) ModuleReference (nextflow_spec.json) +ModulesConfig (1) -----> (*) ModuleReference (nextflow.config alternative) +RegistryConfig (1) -----> (*) Registry URLs + +ModuleReference (1) -----> (0..1) InstalledModule + | + v (via registry) +ModuleSpec (1) <----- InstalledModule (from meta.yaml) +``` + +--- + +## Storage Layout + +``` +project-root/ +├── nextflow.config # registry{} block; optional modules{} block +├── nextflow_spec.json # auto-managed module version pins +├── main.nf # include { X } from '@scope/name' +└── modules/ + └── @scope/ + └── name/ + ├── .checksum # SHA-256 from registry (download integrity) + ├── main.nf # Entry point (required) + ├── meta.yaml # Manifest (required for publishing) + ├── README.md # Documentation (required for publishing) + └── [other files] # Supporting files +``` + +--- + +## Validation Summary + +| Entity | Field | Validation | +|--------|-------|------------| +| ModuleReference | fullName | Pattern: `^@[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` | +| ModuleSpec | name | Pattern: `scope/name` or `scope/path/to/name` | +| ModuleSpec | version | SemVer: `MAJOR.MINOR.PATCH[-prerelease]` | +| ModuleSpec | description, license | Required (non-empty) | +| InstalledModule | directory | Must contain main.nf | +| ModulesConfig | modules keys | Must be valid module fullName | +| RegistryConfig | url | Valid HTTPS URL | \ No newline at end of file diff --git a/specs/251117-module-system/plan.md b/specs/251117-module-system/plan.md new file mode 100644 index 0000000000..e9d11d552f --- /dev/null +++ b/specs/251117-module-system/plan.md @@ -0,0 +1,137 @@ +# Implementation Plan: Nextflow Module System Client + +**Branch**: `251117-module-system` | **Date**: 2026-01-19 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `/specs/251117-module-system/spec.md` + +## Summary + +Implement client-side module system for Nextflow enabling pipeline developers to include remote modules from the Nextflow registry using `@scope/name` syntax, manage versions via `nextflow.config`, configure module parameters via `meta.yaml`, and use CLI commands (install, search, list, remove, publish, run). Implementation extends existing DSL parser, config parser, and follows plugin system patterns for registry communication and authentication. + +## Technical Context + +**Language/Version**: Groovy 4.0.29 (targeting Java 17 runtime, Java 21 toolchain for development) +**Primary Dependencies**: +- Existing Nextflow DSL parser (nf-lang module, ANTLR) +- Existing config parser (ConfigBuilder, ConfigParser) +- Existing HTTP client (HxClient from io.seqera.http) +- Existing plugin authentication infrastructure +- Existing npr-api (registry data models and schema validation) +**Storage**: Local filesystem (`modules/@scope/name/` per-project, `.checksum` files) +**Testing**: Spock Framework for unit tests, integration tests in `tests/` directory +**Target Platform**: JVM 17+ (same as Nextflow core) +**Project Type**: Multi-module Gradle project extension (core modules + CLI) +**Performance Goals**: Module resolution adds <2 seconds to workflow startup when cached locally (SC-002) +**Constraints**: +- Module bundle size limit: 1MB uncompressed (enforced by registry) +- Backward compatibility: Must not break existing `include` statements +- Offline operation: Must work with locally cached modules +**Scale/Scope**: Ecosystem-wide module distribution; typical project: 5-20 modules + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +| Principle | Status | Evidence | +|-----------|--------|----------| +| I. Modular Architecture | PASS | Module system client belongs in `modules/nextflow` (core CLI) with potential shared utilities in `nf-commons` | +| II. Test-Driven Quality | PASS | Unit tests (Spock), integration tests planned, smoke test support | +| III. Dataflow Programming Model | PASS | Modules are process definitions; include resolution at parse time preserves dataflow semantics | +| IV. Apache 2.0 License | PASS | All new code will include Apache 2.0 headers | +| V. DCO Sign-off | PASS | All commits will use `git commit -s` | +| VI. Semantic Versioning | PASS | Modules use SemVer; plugin-compatible version constraint syntax | +| VII. Groovy Idioms | PASS | Follow existing patterns from CmdPlugin, ConfigBuilder, HttpPluginRepository | + +**Gate Status**: PASS - No violations requiring justification + +## Project Structure + +### Documentation (this feature) + +```text +specs/251117-module-system/ +├── plan.md # This file +├── spec.md # Feature specification +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +├── contracts/ # Phase 1 output (API contracts) +└── tasks.md # Phase 2 output (from /speckit.tasks) +``` + +### Source Code (repository root) + +```text +modules/nextflow/src/main/groovy/nextflow/ +├── cli/ +│ ├── CmdModule.groovy # Main module command (uses JCommander) +│ └── module/ +│ ├── ModuleInstall.groovy # Install subcommand (extends CmdBase) +│ ├── ModuleRun.groovy # Run subcommand (extends CmdRun) +│ ├── ModuleList.groovy # List subcommand (extends CmdBase) +│ ├── ModuleRemove.groovy # Remove subcommand (extends CmdBase) +│ ├── ModuleSearch.groovy # Search subcommand (extends CmdBase) +│ ├── ModuleInfo.groovy # Info subcommand (extends CmdBase) +│ └── ModulePublish.groovy # Publish subcommand (extends CmdBase) +├── config/ +│ ├── ModulesConfig.groovy # modules{} config scope +│ └── RegistryConfig.groovy # registry{} config scope (fields: url, apiKey) +├── module/ +│ ├── ModuleReference.groovy # @scope/name parser +│ ├── ModuleResolver.groovy # Core resolution logic (version/integrity/install) +│ ├── ModuleStorage.groovy # Local filesystem operations +│ ├── ModuleRegistryClient.groovy # HTTP registry client +│ ├── ModuleChecksum.groovy # SHA-256 integrity verification +│ ├── ModuleSpec.groovy # Module manifest (meta.yaml) entity +│ ├── InstalledModule.groovy # Installed module entity +│ └── DefaultRemoteModuleResolver.groovy # SPI impl: bridges DSL parser → ModuleResolver +└── pipeline/ + └── PipelineSpec.groovy # nextflow_spec.json read/write + +modules/nf-lang/src/main/java/nextflow/script/ +└── control/ResolveIncludeVisitor.java # MODIFIED: Delegates @scope/name to SPI resolver + +modules/nf-lang/src/main/java/nextflow/module/spi/ +├── RemoteModuleResolver.java # SPI interface (extensible by plugins) +├── RemoteModuleResolverProvider.java # ServiceLoader wrapper (singleton) +└── FallbackRemoteModuleResolver.java # Error fallback when no impl found + + +modules/nextflow/src/test/groovy/nextflow/ +├── cli/module/ +│ ├── ModuleInstallTest.groovy +│ ├── ModuleRunTest.groovy +│ └── [other subcommand tests] +└── module/ + ├── ModuleResolverTest.groovy + ├── ModuleStorageTest.groovy + └── [other module tests] + +tests/modules/ +├── install-module.nf # Integration tests +├── run-module.nf +└── [other integration tests] +``` + +**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang` via SPI. No new plugins required. + +## Architecture Notes + +### Remote Module Inclusion — SPI Pattern + +The DSL parser (`ResolveIncludeVisitor`) detects the `@` prefix in `include` statements and delegates resolution to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader`. This keeps `nf-lang` decoupled from the runtime module resolution logic: + +``` +include { X } from '@nf-core/fastqc' + ↓ +ResolveIncludeVisitor (nf-lang) + source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(...) + ↓ +DefaultRemoteModuleResolver (nextflow module) + auto-installs via ModuleResolver if missing → returns Path to main.nf +``` + +The `RemoteModuleResolver` interface in `nf-lang` can be overridden by plugins with a higher priority value. + +## Complexity Tracking + +No constitution violations requiring justification. \ No newline at end of file diff --git a/specs/251117-module-system/quickstart.md b/specs/251117-module-system/quickstart.md new file mode 100644 index 0000000000..71e356e534 --- /dev/null +++ b/specs/251117-module-system/quickstart.md @@ -0,0 +1,274 @@ +# Quickstart: Nextflow Module System + +This guide covers the essential workflows for using the Nextflow module system. + +## Prerequisites + +- Nextflow 25.x or later (with module system support) +- Network connectivity for initial module downloads +- Optional: `NXF_REGISTRY_TOKEN` for publishing + +--- + +## 1. Install and Use a Module + +### Install a module + +```bash +# Install latest version +nextflow module install nf-core/fastqc + +# Install specific version +nextflow module install nf-core/fastqc -version 1.0.0 +``` + +This downloads the module to `modules/@nf-core/fastqc/` and updates `nextflow_spec.json` with the installed version. + +### Use in your workflow + +```groovy +// main.nf +include { FASTQC } from '@nf-core/fastqc' + +workflow { + reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') + FASTQC(reads) +} +``` + +### Run your workflow + +```bash +nextflow run main.nf +``` + +--- + +## 2. Run a Module Directly + +Execute a module without writing a wrapper workflow: + +```bash +# Basic usage +nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' + +# Run specific version +nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 + +# With Nextflow options +nextflow module run nf-core/salmon \ + --reads reads.fq \ + --index salmon_index \ + -profile docker \ + -resume +``` + +## 3. View Module Information + +```bash +# Show module metadata and a generated usage template +nextflow module info nf-core/fastqc + +# Show a specific version +nextflow module info nf-core/fastqc -version 1.0.0 + +# JSON output for scripting +nextflow module info nf-core/fastqc -json +``` + +--- + +## 4. Manage Module Versions + +### Version tracking + +Module versions are automatically recorded in `nextflow_spec.json` by `nextflow module install`. You can also pin versions manually: + +```json +// nextflow_spec.json +{ + "modules": { + "@nf-core/fastqc": "1.0.0", + "@nf-core/bwa-align": "1.2.0" + } +} +``` + +Alternatively, declare versions in `nextflow.config` (not currently used): + +```nextflow +modules { + '@nf-core/fastqc' = '1.0.0' + '@nf-core/bwa-align' = '1.2.0' +} +``` + +### Check module status + +```bash +# List all modules +nextflow module list + +# Output: +# MODULE CONFIGURED INSTALLED LATEST STATUS +# @nf-core/fastqc 1.0.0 1.0.0 1.2.0 outdated +# @nf-core/bwa-align 1.2.0 1.2.0 1.2.0 up-to-date +# @nf-core/samtools 2.1.0 - 2.1.0 missing +``` + +### Update a module + +Change the version in `nextflow_spec.json` (or `nextflow.config`), then run your workflow. Nextflow automatically downloads the new version. + +--- + +## 5. Search for Modules + +```bash +# Search by keyword +nextflow module search alignment + +# Limit results +nextflow module search "quality control" -limit 5 + +# JSON output for scripting +nextflow module search bwa -json +``` + +--- + +## 6. Work with Private Registries + +### Configure authentication + +```nextflow +// nextflow.config +registry { + // Multiple registries (tried in order) + url = [ + 'https://private.registry.myorg.com', + 'https://registry.nextflow.io/api' + ] + apiKey = 'MYORG_TOKEN' // Applied to the primary (first) registry only +} +``` + +### Or use environment variable + +```bash +export NXF_REGISTRY_TOKEN=your-token-here +nextflow module install nf-core/fastqc +``` + +--- + +## 7. Publish a Module + +### Prepare your module + +``` +my-module/ +├── main.nf # Required: entry point +├── meta.yaml # Required for registry +├── README.md # Required for registry +└── tests/ # Recommended +``` + +### Validate before publishing + +```bash +nextflow module publish myorg/my-module -dry-run +``` + +### Publish to registry + +```bash +export NXF_REGISTRY_TOKEN=your-token +nextflow module publish myorg/my-module +``` + +--- + +## 8. Handle Local Modifications + +If you modify a module locally (for debugging), Nextflow protects your changes: + +```bash +# This warns and does NOT override your changes +nextflow module install nf-core/fastqc -version 1.1.0 +# Warning: Module @nf-core/fastqc has local modifications. Use -force to override. + +# Force replacement if needed +nextflow module install nf-core/fastqc -version 1.1.0 -force +``` + +--- + +## 9. Remove a Module + +```bash +# Remove module and config entry +nextflow module remove nf-core/fastqc + +# Keep config entry (just delete local files) +nextflow module remove nf-core/fastqc -keep-config + +# Keep local files (just remove from config) +nextflow module remove nf-core/fastqc -keep-files +``` + +--- + +## Common Patterns + +### Offline operation + +Modules are cached locally in `modules/`. Once installed, workflows run without network access. + +### Git integration + +The `modules/` directory is intended to be committed to your git repository: + +```bash +git add modules/ +git commit -m "Add module dependencies" +``` + +--- + +## Troubleshooting + +### Module not found + +```bash +# Check if module exists in registry +nextflow module search exact-module-name + +# Verify spelling and scope +# Correct: @nf-core/fastqc +# Wrong: @nfcore/fastqc, nf-core/fastqc (without @) +``` + +### Authentication errors + +```bash +# Verify token is set +echo $NXF_REGISTRY_TOKEN + +# Check registry config +grep -A5 'registry' nextflow.config +``` + +### Version conflicts + +If two modules require incompatible versions of a dependency: +- Nextflow selects the highest compatible version automatically +- If no compatible version exists, an error lists the conflicts + +### Checksum warnings + +``` +Warning: Module @nf-core/fastqc has local modifications +``` + +This means the local module content differs from the registry version. Your changes are preserved. Use `-force` only if you want to discard local changes. \ No newline at end of file diff --git a/specs/251117-module-system/research.md b/specs/251117-module-system/research.md new file mode 100644 index 0000000000..d08005fc89 --- /dev/null +++ b/specs/251117-module-system/research.md @@ -0,0 +1,352 @@ +# Research: Nextflow Module System Client + +**Date**: 2026-01-19 +**Feature**: 251117-module-system + +## Overview + +This document captures technical research and decisions for implementing the Nextflow module system client. All NEEDS CLARIFICATION items from Technical Context have been resolved through codebase exploration. + +--- + +## 1. CLI Command Structure + +**Research Question**: How should `nextflow module` CLI commands be implemented? + +**Decision**: JCommander native subcommands — each subcommand extends `CmdBase` directly; no trait needed + +**Rationale**: +- JCommander's subcommand support handles parameter parsing automatically per subcommand +- Each subcommand (install, run, list, remove, search, info, publish) is a separate class extending CmdBase +- `ModuleRun` extends `CmdRun` to reuse pipeline execution logic (PR #6381) +- No custom `ModuleSubCmd` trait needed; cleaner architecture +- `CmdModule` is registered in `Launcher` alongside all other top-level commands + +**Implemented Pattern**: +```groovy +@Parameters(commandDescription = "Manage Nextflow modules") +class CmdModule extends CmdBase implements UsageAware { + static final List commands = [] + + static { + commands << new ModuleInstall() // extends CmdBase + commands << new ModuleRun() // extends CmdRun + commands << new ModuleList() // extends CmdBase + commands << new ModuleRemove() // extends CmdBase + commands << new ModuleSearch() // extends CmdBase + commands << new ModuleInfo() // extends CmdBase + commands << new ModulePublish() // extends CmdBase + } + + void run() { + final jc = commander() // JCommander with all subcommands registered + jc.parse(args as String[]) + final subcommand = jc.getCommands().get(jc.getParsedCommand()).getObjects()[0] + subcommand.run() + } +} +``` + +**Alternatives Considered**: +- CmdFs trait pattern: Considered initially; replaced by JCommander native subcommands — simpler and avoids custom parsing +- Separate top-level Cmd classes (CmdModuleInstall, etc.): Rejected — too many entry points +- Plugin-based CLI extension: Rejected — module system is core functionality, not optional + +--- + +## 2. DSL Parser Extension for @scope/name + +**Research Question**: How to extend `include` statement parsing for registry modules? + +**Decision**: Extend `ResolveIncludeVisitor` to detect `@` prefix and delegate to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader` + +**Rationale**: +- Keeps `nf-lang` decoupled from runtime module resolution (`nf-lang` has no dependency on `nextflow` module) +- SPI pattern allows plugins or custom implementations to override the default resolver +- Detection: `source.startsWith('@')` distinguishes registry vs local paths — preserves existing include behavior +- Resolution at parse time (after plugin resolution) per ADR + +**Implemented Architecture**: +``` +include { X } from '@scope/name' + ↓ +ResolveIncludeVisitor.visitInclude() [nf-lang] + source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(source, baseDir) + ↓ +RemoteModuleResolverProvider [nf-lang] + Java ServiceLoader discovers implementations; picks highest priority + ↓ +DefaultRemoteModuleResolver [nextflow module] + Calls ModuleResolver.installModule(reference, version, autoInstall=true) + Returns Path to modules/@scope/name/main.nf +``` + +**Key Files**: +- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java` — SPI interface +- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java` — ServiceLoader singleton +- `modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java` — error fallback +- `modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java` — MODIFIED +- `modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy` — default impl + +**Alternatives Considered**: +- New ANTLR grammar token for `@`: Rejected — unnecessary parser complexity +- Direct dependency from nf-lang to nextflow module: Rejected — circular dependency risk; SPI decouples cleanly +- Dot file marker for local modules: Deferred in ADR; current impl uses `@` for registry, `.`/`/` for local + +--- + +## 3. Config Parsing for modules{} and registry{} Blocks + +**Research Question**: How to add new config DSL blocks? + +**Decision**: Create ModulesConfig and RegistryConfig classes implementing ConfigScope interface + +**Rationale**: +- ConfigScope is an ExtensionPoint (pf4j) that ConfigBuilder automatically discovers +- Classes implementing ConfigScope and annotated with @ScopeName are automatically parsed +- No need to modify ConfigBuilder or create custom DSL parsers +- Pattern used throughout Nextflow: FusionConfig, CondaConfig, DockerConfig, etc. +- Provides type safety via @CompileStatic and validation via @ConfigOption + +**Reference Implementation**: +``` +Location: modules/nextflow/src/main/groovy/nextflow/fusion/FusionConfig.groovy +Pattern: + @ScopeName("modules") + @Description("Module version declarations") + @CompileStatic + class ModulesConfig implements ConfigScope { + @ConfigOption + @Description("Module version mappings") + final Map modules = [:] + + ModulesConfig() {} + + ModulesConfig(Map opts) { + // Parse from config map + } + } +``` + +**ConfigScope Interface**: +``` +Location: modules/nf-lang/src/main/java/nextflow/config/spec/ConfigScope.java +public interface ConfigScope extends ExtensionPoint {} +``` + +**RegistryConfig Pattern**: +```groovy +@ScopeName("registry") +@Description("Module registry configuration") +@CompileStatic +class RegistryConfig implements ConfigScope { + static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api' + + @ConfigOption + final Collection url // One or more URLs in priority order + + @ConfigOption + final String apiKey // API key; falls back to NXF_REGISTRY_TOKEN env var + + RegistryConfig() { + url = [DEFAULT_REGISTRY_URL] + apiKey = null + } + + RegistryConfig(Map opts) { + url = opts.url ?: [DEFAULT_REGISTRY_URL] + apiKey = opts.apiKey as String + } + + String getUrl() { url ? url[0] : DEFAULT_REGISTRY_URL } + Collection getAllUrls() { url ?: [DEFAULT_REGISTRY_URL] } + String getApiKey() { apiKey ?: SysEnv.get('NXF_REGISTRY_TOKEN') } +} +``` + +**Integration Point**: ConfigBuilder automatically discovers and parses ConfigScope implementations via ExtensionPoint mechanism + +**Alternatives Considered**: +- Custom DSL parsers (ModulesDsl/RegistryDsl): Rejected - unnecessary complexity, ConfigScope pattern handles this automatically +- JSON/YAML config file: Rejected - inconsistent with Nextflow config style +- Dedicated pipeline.yaml: Deferred per ADR Open Questions + +--- + +## 4. Registry HTTP Communication + +**Research Question**: How to communicate with module registry API? + +**Decision**: Create HttpModuleRepository following HttpPluginRepository pattern + +**Rationale**: +- HttpPluginRepository provides robust HTTP client with retry logic +- Uses HxClient from io.seqera.http (already a dependency) +- Handles authentication headers consistently +- Supports connection pooling and timeout configuration + +**Reference Implementation**: +``` +Location: modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy +Pattern: + class HttpModuleRepository { + private final URI url + private final HxClient httpClient + private final String authToken + + ModuleInfo getModule(String name, String version) + List search(String query, int limit) + Path download(String name, String version, Path target) + void publish(String name, Path bundle) + } +``` + +**API Endpoints** (from ADR): +``` +GET /api/modules?query= # Search +GET /api/modules/{name} # Get module + latest release +GET /api/modules/{name}/releases # List all releases +GET /api/modules/{name}/{version} # Get specific release +GET /api/modules/{name}/{version}/download # Download bundle +POST /api/modules/{name} # Publish (authenticated) +``` + +**Alternatives Considered**: +- Direct HttpClient usage: Rejected - loses retry, pooling benefits +- gRPC protocol: Rejected - registry already uses REST + +--- + +## 5. Authentication Patterns + +**Research Question**: How to handle registry authentication? + +**Decision**: Support `NXF_REGISTRY_TOKEN` env var + `registry.apiKey` config field + +**Rationale**: +- Environment variable provides CI/CD compatibility +- `apiKey` config field allows explicit token configuration +- Authentication is only applied to the primary (first) registry URL +- Bearer token in Authorization header (standard HTTP auth) + +**Implementation**: +``` +RegistryConfig.getApiKey() returns: + 1. registry.apiKey config value if set + 2. NXF_REGISTRY_TOKEN environment variable as fallback + 3. null if neither is set (unauthenticated requests) +``` + +**Config Syntax**: +```nextflow +registry { + apiKey = '${NXF_REGISTRY_TOKEN}' +} +``` + +**Alternatives Considered**: +- Per-registry token map (`auth {}` block): Was in initial design; simplified to single `apiKey` since only the primary registry uses authentication +- Secrets file (~/.nextflow/secrets.json): Possible future enhancement +- OAuth flow: Rejected for CLI — token-based simpler + +--- + +## 6. Checksum Verification + +**Research Question**: How to implement module integrity verification? + +**Decision**: SHA-256 checksum stored in `.checksum` file, verified on every run + +**Rationale**: +- SHA-256 is industry standard, already used for plugin verification +- `.checksum` file stores registry-provided checksum (from X-Checksum header) +- Local checksum computed on-demand and compared +- Mismatch indicates local modification (warn, don't override) + +**Implementation Pattern**: +```groovy +class ModuleChecksum { + static final String ALGORITHM = 'SHA-256' + + static String compute(Path moduleDir) { + // Hash all files in module directory + // Exclude .checksum itself + // Return hex-encoded SHA-256 + } + + static boolean verify(Path moduleDir) { + def expected = moduleDir.resolve('.checksum').text.trim() + def actual = compute(moduleDir) + return expected == actual + } + + static void save(Path moduleDir, String checksum) { + moduleDir.resolve('.checksum').text = checksum + } +} +``` + +**Checksum Scope**: Covers all files in module directory (main.nf, meta.yaml, README.md, etc.) + +**Alternatives Considered**: +- Per-file checksums: Rejected - adds complexity, single checksum sufficient +- MD5: Rejected - SHA-256 more secure + +--- + +## 7. Version Constraint Syntax + +**Research Question**: What version constraint syntax to use for module dependencies? + +**Decision**: Reuse existing Nextflow plugin version constraint syntax + +**Rationale**: +- Already implemented and tested in plugin system +- Users familiar with existing `nextflowVersion` syntax +- Supports ranges, comparisons, exact versions +- No new parser code needed + +**Supported Syntax**: +| Notation | Meaning | Example | +|----------|---------|---------| +| `1.2.3` | Exact version | `@nf-core/fastqc@1.0.0` | +| `>=1.2.3` | Greater or equal | `@nf-core/fastqc@>=1.0.0` | +| `<=1.2.3` | Less or equal | `@nf-core/fastqc@<=2.0.0` | +| `>=1.2.0,<2.0.0` | Range | `@nf-core/samtools@>=1.0.0,<2.0.0` | + +**Reference**: Version parsing code exists in plugin system; reuse VersionNumber class + +**Alternatives Considered**: +- NPM-style `^` and `~`: Rejected - inconsistent with existing Nextflow patterns +- Always latest: Rejected - breaks reproducibility + +--- + +## 8. Tool Arguments Implementation + +> **⚠️ REMOVED FROM ADR** — The tool arguments feature (`tools..args` in meta.yaml and process config) was removed from the module system ADR. It is not implemented and not planned in the current scope. The `meta.yaml` format used in the actual implementation (`ModuleSpec`) does not include tool/argument definitions. + +--- + +## Summary of Key Decisions + +| Area | Decision | Key Reference | +|------|----------|---------------| +| CLI | JCommander subcommands; each extends CmdBase (ModuleRun extends CmdRun) | CmdModule.groovy | +| DSL Parser | SPI pattern — ResolveIncludeVisitor delegates to RemoteModuleResolver; DefaultRemoteModuleResolver bridges to ModuleResolver | ResolveIncludeVisitor.java, RemoteModuleResolver.java | +| Config | ModulesConfig + RegistryConfig (ConfigScope) | FusionConfig.groovy, ConfigScope.java | +| Registry HTTP | ModuleRegistryClient using HxClient + npr-api models | HttpPluginRepository.groovy | +| Authentication | `NXF_REGISTRY_TOKEN` env var or `registry.apiKey` config field (primary registry only) | RegistryConfig.groovy | +| Checksums | SHA-256/SHA-512, `.checksum` file, download integrity via X-Checksum header | ModuleChecksum.groovy | +| Version Storage | `nextflow_spec.json` (auto-managed); `modules {}` in nextflow.config (manual alternative) | PipelineSpec.groovy | +| Version Syntax | Plugin-compatible constraints | VersionNumber class | +| Tool Args | ~~Implicit variable, parse-time validation~~ — **Removed from ADR** | N/A | + +--- + +## Open Items (Deferred) + +1. **Local vs managed module distinction**: Resolved — `@` prefix for registry modules only; local paths start with `.` or `/` +2. **Tool arguments**: Removed from ADR — not in scope +3. **Module version location**: Resolved — `nextflow_spec.json` (auto-managed by `module install`); `modules {}` block in `nextflow.config` supported as alternative +4. **DSL parser `@scope/name` include**: ✅ Resolved — SPI pattern implemented (T017a-d) \ No newline at end of file diff --git a/specs/251117-module-system/spec.md b/specs/251117-module-system/spec.md new file mode 100644 index 0000000000..fba59f7923 --- /dev/null +++ b/specs/251117-module-system/spec.md @@ -0,0 +1,261 @@ +# Feature Specification: Nextflow Module System Client + +**Feature Branch**: `251117-module-system` +**Created**: 2026-01-15 +**Status**: Draft +**Input**: User description: "Implement Nextflow module system client based on ADR 20251114-module-system.md. Focus on client-side implementation only - CLI commands, DSL parser extensions, dependency resolution, and local storage. Registry backend is assumed to be already implemented." + +## Overview + +This specification covers the **Nextflow client-side implementation** of the module system, enabling pipeline developers to: +- Include remote modules from the Nextflow registry using `@scope/name` syntax +- Manage module versions through `nextflow.config` +- Use CLI commands to install, search, list, remove, publish, and run modules +- Configure module parameters through structured `meta.yaml` definitions + +**Out of Scope**: Registry backend implementation (assumed already available at `registry.nextflow.io`) + +## User Scenarios & Testing + +### User Story 1 - Install and Use Registry Module (Priority: P1) + +A pipeline developer wants to use a pre-built module from the Nextflow registry in their workflow without manually downloading or managing module files. + +**Why this priority**: This is the core value proposition - enabling code reuse from the ecosystem. Without this, the module system provides no benefit. + +**Independent Test**: Can be fully tested by running `nextflow module install nf-core/fastqc` and then executing a workflow that includes the module. Delivers immediate value by enabling module consumption. + +**Acceptance Scenarios**: + +1. **Given** a new Nextflow project with no modules installed, **When** user runs `nextflow module install nf-core/fastqc`, **Then** the module is downloaded to `modules/@nf-core/fastqc/`, a `.checksum` file is created, and `nextflow_spec.json` is updated with the version +2. **Given** a workflow file with `include { FASTQC } from '@nf-core/fastqc'`, **When** user runs `nextflow run main.nf`, **Then** Nextflow resolves the module from local storage and executes the process +3. **Given** a module version declared in `nextflow.config`, **When** user includes the module, **Then** the declared version is used (not latest) + +--- + +### User Story 2 - Run Module Directly (Priority: P1) + +A user wants to run a module directly from the command line without writing a wrapper workflow. + +**Why this priority**: Enables immediate productivity - users can test and execute modules without boilerplate code, essential for AI agents and quick experimentation. + +**Independent Test**: Can be tested by running `nextflow module run nf-core/fastqc --input 'data/*.fq'` and verifying the process executes. + +**Acceptance Scenarios**: + +1. **Given** a module is available (locally or in registry), **When** user runs `nextflow module run nf-core/fastqc --input 'data/*.fastq'`, **Then** the module is executed with the provided inputs mapped to process parameters +2. **Given** a module with parameters defined in `meta.yaml`, **When** user runs `nextflow module run nf-core/bwa-align --batch_size 100000`, **Then** the parameter is validated and passed to the process +3. **Given** a module is not installed locally, **When** user runs `nextflow module run nf-core/salmon`, **Then** the module is automatically downloaded before execution + +--- + +### User Story 3 - Module Parameters (Priority: P1) + +A module author wants to define typed, documented parameters that provide a clear interface for module customization. + +**Why this priority**: Critical for module usability - provides type-safe, documented parameters that enable IDE autocompletion and validation, replacing the opaque `ext.args` pattern. + +**Independent Test**: Can be tested by configuring `params.batch_size = 100000` in config and verifying the parameter is applied in the script. + +**Acceptance Scenarios**: + +1. **Given** a module with `params` defined in `meta.yaml`, **When** user configures `params.batch_size = 100000` in config, **Then** the parameter is accessible in scripts via `params.batch_size` +2. **Given** a parameter with type validation, **When** user provides an invalid value type, **Then** a validation error is displayed +3. **Given** a module with documented parameters, **When** user runs `nextflow module run --help`, **Then** available parameters with descriptions are listed + +--- + +### User Story 4 - Module Version Management (Priority: P2) + +A pipeline developer wants to pin and manage module versions to ensure reproducible workflow executions. + +**Why this priority**: Reproducibility is important for scientific workflows - version pinning ensures consistent results. + +**Independent Test**: Can be tested by modifying `nextflow.config` module versions and verifying the correct version is used on workflow run. + +**Acceptance Scenarios**: + +1. **Given** a module is installed at version 1.0.0, **When** user changes `nextflow_spec.json` to specify version 1.1.0 and runs the workflow, **Then** version 1.1.0 is automatically downloaded and replaces the local copy +2. **Given** modules installed locally, **When** user runs `nextflow module list`, **Then** configured version, installed version, latest available version, and status are displayed for each module + +--- + +### User Story 5 - Module Integrity Protection (Priority: P2) + +A pipeline developer who has locally modified a module (for debugging or customization) wants to be protected from accidentally losing those changes. + +**Why this priority**: Protects user work - important for developer experience but not blocking core functionality. + +**Independent Test**: Can be tested by modifying a module's `main.nf` locally, then attempting to install a different version and verifying the warning appears. + +**Acceptance Scenarios**: + +1. **Given** a locally modified module (checksum mismatch with `.checksum`), **When** user tries to install a different version, **Then** Nextflow warns about local modifications and does NOT override +2. **Given** a locally modified module, **When** user runs `nextflow module install -force`, **Then** the local module is replaced with the registry version +3. **Given** a locally modified module, **When** user runs the workflow, **Then** a warning is displayed about checksum mismatch but execution continues + +--- + +### User Story 6 - Remove Module (Priority: P3) + +A pipeline developer wants to remove a module they no longer need. + +**Why this priority**: Housekeeping feature - useful but not blocking core workflows. + +**Independent Test**: Can be tested by running `nextflow module remove nf-core/fastqc` and verifying files are deleted and config is updated. + +**Acceptance Scenarios**: + +1. **Given** a module is installed, **When** user runs `nextflow module remove nf-core/fastqc`, **Then** the module directory is deleted and the entry is removed from `nextflow_spec.json` +2. **Given** a module is referenced in workflow files, **When** user runs `nextflow module remove`, **Then** a warning is displayed about the reference but removal proceeds + +--- + +### User Story 7 - Search and Discover Modules (Priority: P3) + +A pipeline developer wants to find available modules in the registry that match their analysis needs. + +**Why this priority**: Discovery feature - useful but users can find modules through documentation or registry web UI. + +**Independent Test**: Can be tested by running `nextflow module search bwa` and verifying results are displayed with name, version, and description. + +**Acceptance Scenarios**: + +1. **Given** modules exist in the registry, **When** user runs `nextflow module search alignment`, **Then** matching modules are displayed with name, latest version, description, and download count +2. **Given** user wants JSON output for scripting, **When** user runs `nextflow module search fastqc -json`, **Then** results are returned in parseable JSON format +3. **Given** many results exist, **When** user runs `nextflow module search quality -limit 5`, **Then** only 5 results are returned + +--- + +### User Story 8 - Publish Module to Registry (Priority: P3) + +A module author wants to publish their module to the Nextflow registry for others to use. + +**Why this priority**: Ecosystem contribution feature - important for growth but users can consume modules without publishing capability. + +**Independent Test**: Can be tested by creating a valid module structure and running `nextflow module publish -dry-run` to validate. + +**Acceptance Scenarios**: + +1. **Given** a valid module with `main.nf`, `meta.yaml`, and `README.md`, **When** user runs `nextflow module publish myorg/my-module`, **Then** the module is uploaded to the registry and becomes available for installation +2. **Given** an invalid module (missing required fields), **When** user runs `nextflow module publish`, **Then** validation errors are displayed listing the missing requirements +3. **Given** no authentication configured, **When** user runs `nextflow module publish`, **Then** a clear error message indicates authentication is required + +--- + +### Edge Cases + +- What happens when the registry is unreachable during module resolution? + - Nextflow uses locally cached modules if available, otherwise fails with a clear network error +- How does the system handle circular module dependencies? + - Dependency resolver detects cycles and fails with an error listing the cycle +- What happens when two modules require incompatible versions of the same dependency? + - System automatically selects the highest compatible version; if no compatible version exists, fails with error listing conflicting requirements +- How are modules resolved when multiple registries are configured? + - Registries are tried in order; first match wins +- What happens when `meta.yaml` is missing from a module? + - Module is treated as having no dependencies; basic functionality works +- What happens when local module directory is corrupted or incomplete? + - Checksum mismatch triggers warning; `-force` allows re-download + +## Requirements + +### Functional Requirements + +#### DSL Parser Extension + +- **FR-001**: System MUST recognize `@scope/name` syntax in `include` statements as registry module references +- **FR-002**: System MUST distinguish between local file paths (starting with `.` or `/`) and registry modules (starting with `@`) +- **FR-003**: System MUST resolve module versions from `nextflow_spec.json` before downloading +- **FR-004**: System MUST parse and validate `meta.yaml` files for module metadata and dependencies + +#### Module Resolution + +- **FR-005**: System MUST resolve modules at workflow parse time (after plugin resolution) +- **FR-006**: System MUST check local `modules/@scope/name/` directory before querying registry +- **FR-007**: System MUST verify module integrity using `.checksum` file on every run +- **FR-008**: System MUST download modules from registry when not present locally or when version differs +- **FR-009**: System MUST NOT override locally modified modules (checksum mismatch) unless `-force` is used +- **FR-010**: System MUST resolve version conflicts by selecting the highest compatible version; if no compatible version exists, MUST fail with error listing conflicting requirements + +#### Local Storage + +- **FR-011**: System MUST store modules in `modules/@scope/name/` directory structure (single version per module) +- **FR-012**: System MUST create `.checksum` file from registry's X-Checksum header on download +- **FR-013**: System MUST store module's `main.nf`, `meta.yaml`, and supporting files in the module directory + +#### CLI Commands + +- **FR-014**: System MUST provide `nextflow module install [scope/name]` command to download modules +- **FR-015**: System MUST provide `nextflow module search ` command to search the registry +- **FR-016**: System MUST provide `nextflow module list` command to show installed vs configured modules +- **FR-017**: System MUST provide `nextflow module remove scope/name` command to delete modules +- **FR-018**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry +- **FR-019**: System MUST provide `nextflow module run scope/name` command to execute modules directly +- **FR-019b**: System MUST provide `nextflow module info scope/name` command to display module metadata and a usage template + +#### Configuration + +- **FR-020**: System MUST persist module versions in `nextflow_spec.json`; MUST also read versions from `modules {}` block in `nextflow.config` as an alternative +- **FR-021**: System MUST support `registry {}` block with `url` and `apiKey` fields for configuring registry URL and authentication +- **FR-022**: System MUST support `NXF_REGISTRY_TOKEN` environment variable as fallback for `registry.apiKey` +- **FR-023**: System MUST support multiple registry URLs with fallback ordering + +#### Module Parameters + +- **FR-024**: System MUST parse module parameters from `params` section in `meta.yaml` +- **FR-025**: System MUST validate module parameters against `meta.yaml` schema (type) at workflow parse time +- **FR-026**: System MUST support boolean, integer, float, string, file, and path parameter types +- **FR-027**: System MUST make module parameters accessible via standard `params` variable in scripts + +#### Registry Communication + +- **FR-028**: System MUST communicate with registry via documented Module API endpoints +- **FR-029**: System MUST handle authentication using Bearer token in Authorization header +- **FR-030**: System MUST verify SHA-256 checksum on module download + +### Key Entities + +- **Module**: A reusable Nextflow process definition with `main.nf` entry point, optional `meta.yaml` manifest, and README documentation +- **Module Reference**: A scoped identifier (`@scope/name`) pointing to a registry module +- **Module Manifest (meta.yaml)**: YAML file containing module metadata, version, dependencies, and parameter definitions +- **Module Parameter**: A configurable parameter defined in `meta.yaml` with name, optional type, description, and example +- **Checksum File (.checksum)**: Local cache of registry checksum for integrity verification +- **Registry Configuration**: Settings for registry URL, authentication, and fallback ordering + +## Success Criteria + +### Measurable Outcomes + +- **SC-001**: Pipeline developers can install and use a registry module within 5 minutes of starting a new project +- **SC-002**: Module resolution adds less than 2 seconds to workflow startup time when modules are cached locally +- **SC-003**: Users can successfully search, install, and run any module from the registry without reading documentation +- **SC-004**: 100% of module version changes in `nextflow.config` result in automatic module updates without manual intervention +- **SC-005**: Users receive clear, actionable error messages for all failure scenarios (network, validation, authentication) +- **SC-006**: Module authors can publish a new module version within 3 minutes using the CLI +- **SC-007**: Locally modified modules are never accidentally overwritten during normal operations + +## Assumptions + +- Registry backend is fully implemented and available at `registry.nextflow.io` with the Module API as documented in the ADR +- Existing plugin authentication system can be reused for module registry authentication +- Module bundle size limit of 1MB (uncompressed) is enforced by the registry +- Network connectivity is available for initial module downloads; offline operation uses local cache only +- The `modules/` directory is intended to be committed to the pipeline's git repository +- Version constraints in `meta.yaml` follow the same syntax as existing Nextflow plugin version constraints +- SHA-256 is used for all checksum operations +- Module parameters use standard `--` CLI syntax + +## Dependencies + +- Registry backend API (Module API endpoints as specified in ADR) +- Existing Nextflow plugin system (for authentication reuse) +- Existing DSL parser infrastructure (for `include` statement extension) +- Existing config parser (for `modules {}` and `registry {}` blocks) + +## Clarifications + +### Session 2026-01-19 + +- Q: What should happen when incompatible dependency versions are detected? → A: Use highest compatible version automatically, warn if none exists +- Q: When should module parameter validation occur? → A: At workflow parse time (early, before any execution) \ No newline at end of file