Merge branch 'master' into 251117-module-system

Author: pditommaso

VERSION

@@ -1 +1 @@
-25.11.0-edge
+25.12.0-edge

adr/20251205-git-multi-revision-asset-management.md

@@ -0,0 +1,252 @@
+# Git Multi-Revision Asset Management with Strategy Pattern
+
+- Authors: Jorge Ejarque
+- Status: Approved
+- Deciders: Jorge Ejarque, Ben Sherman, Paolo Di Tommaso
+- Date: 2025-12-05
+- Tags: scm, asset-management, multi-revision
+
+## Summary
+
+Nextflow's asset management system has been refactored to support multiple revisions of the same pipeline concurrently through a bare repository approach with shared object storage, while maintaining backward compatibility with legacy direct-clone repositories using the Strategy design pattern.
+
+## Problem Statement
+
+The original asset management system (`AssetManager`) cloned each pipeline directly to `~/.nextflow/assets/<org>/<project>/.git`, creating several limitations:
+
+1. **No concurrent Git multi-revision support**: Only one revision of a pipeline could be checked out at a time, preventing concurrent execution of different versions
+2. **Update conflicts**: Pulling updates while a pipeline was running could cause conflicts or corruption
+3. **Testing limitations**: Users couldn't easily test different versions of a pipeline side-by-side
+
+The goal was to enable running multiple revisions of the same pipeline concurrently (e.g., production on v1.0, testing on v2.0-dev) while maintaining efficient disk usage through object sharing.
+
+## Goals or Decision Drivers
+
+- **Concurrent multi-revision execution**: Must support running different revisions of the same pipeline simultaneously
+- **Efficient disk usage**: Share Git objects between revisions to minimize storage overhead
+- **Backward compatibility**: Must not break existing pipelines using the legacy direct-clone approach
+- **API stability**: Maintain the existing `AssetManager` API for external consumers (K8s plugin, CLI commands, etc.)
+- **Minimal migration impact**: Existing repositories should continue to work without user intervention
+- **JGit compatibility**: Solution must work within JGit's capabilities to avoid relying on Git client installations
+- **Atomic updates**: Downloading new revisions should not interfere with running pipelines
+
+## Non-goals
+
+- **Migration of existing legacy repositories**: Legacy repos continue to work as-is; no forced migration
+- **Native Git worktree support**: Due to JGit limitations, not using Git's worktree feature
+- **Revision garbage collection**: No automatic cleanup of old revisions (users can manually drop)
+- **Multi-hub support**: Still tied to a single repository provider per pipeline
+
+## Considered Options
+
+### Option 1: Bare Repository with Git Worktrees
+
+Use Git's worktree feature to create multiple working directories from a single bare repository.
+
+**Implementation**:
+- One bare repository at `~/.nextflow/assets/<org>/<project>/.git`
+- Multiple worktrees at `~/.nextflow/assets/<org>/<project>/<revision>/`
+
+- Good, because it's the native Git solution for multiple checkouts
+- Good, because worktrees are space-efficient
+- Good, because Git handles all the complexity
+- **Bad, because JGit doesn't support worktrees** (deal-breaker)
+- Bad, because requires native Git installation
+
+**Decision**: Rejected due to JGit incompatibility 
+
+### Option 2: Bare Repository + Clones per Commit + Revision Map File
+
+Use a bare repository for storage and create clones for each commit, tracking them in a separate file.
+
+**Implementation**:
+- Bare repository at `~/.nextflow/assets/<org>/<project>/.nextflow/bare_repo/`
+- Clones at `~/.nextflow/assets/<org>/<project>/.nextflow/commits/<commit-sha>/`
+- Revision map file at `~/.nextflow/assets/<org>/<project>/.nextflow/revisions.json` mapping revision names to commit SHAs
+
+- Good, because it works with JGit
+- Good, because bare repo reduces remote repository interactions to checkout commits
+- Good, because explicit revision tracking
+- Bad, because disk space as git objects replicated in clones 
+- Bad, because revision map file can become stale
+- Bad, because requires file I/O for every revision lookup
+- Bad, because potential race conditions on map file updates
+- Bad, because adds complexity of maintaining external state
+
+**Decision**: Initially implemented but later refined
+
+### Option 3: Bare Repository + Shared Clones with Strategy Pattern
+
+Similar to Option 2 but eliminate the separate revision map file by using the bare repository itself as the source of truth. Additionally, use the Strategy pattern to maintain backward compatibility with existing legacy repositories without requiring migration.
+
+**Implementation**:
+- Bare repository at `~/.nextflow/assets/.repos/<org>/<project>/bare/`
+- Shared clones at `~/.nextflow/assets/.repos/<org>/<project>/clones/<commit-sha>/`
+- Use bare repository refs to resolve revisions to commit SHAs dynamically
+- JGit alternates mechanism for object sharing
+- `AssetManager` as facade with unchanged public API
+- `RepositoryStrategy` interface defining repository operations
+- `LegacyRepositoryStrategy` for existing direct-clone behavior
+- `MultiRevisionRepositoryStrategy` for new bare-repo approach
+- Strategy selection based on environment variable or repository state detection
+
+- Good, because no external state file to maintain
+- Good, because bare repository is always in sync (fetched on updates)
+- Good, because simpler and more reliable
+- Good, because atomic updates (Git operations are atomic)
+- Good, because works entirely within JGit
+- Good, because zero migration needed for existing repositories
+- Good, because maintains API compatibility
+- Good, because allows gradual adoption
+- Good, because isolates legacy code
+- Good, because makes future strategies easy to add
+- Neutral, because adds abstraction layer
+- Bad, because requires resolution on every access (minimal overhead)
+- Bad, because increases codebase size initially
+
+**Decision**: Selected
+
+## Solution or decision outcome
+
+Implemented **Option 3 (Bare Repository + Shared Clones with Strategy Pattern)** for multi-revision support with backward compatibility. Multi-revision is the default for new repositories, while legacy mode is available via `NXF_SCM_LEGACY` environment variable.
+
+## Rationale & discussion
+
+### Git Multi-Revision Implementation
+
+The bare repository approach provides efficient multi-revision support:
+
+```
+~/.nextflow/assets/.repos/nextflow-io/hello/
+├── bare/                       # Bare repository (shared objects)
+│   ├── objects/                # All Git objects stored here
+│   ├── refs/
+│   │   ├── heads/
+│   │   └── tags/
+│   └── config
+│
+└── clones/                     # Revisions-specific clones
+    ├── abc123.../              # Clone for commit abc123
+    │   └── .git/
+    │       ├── objects/        # (uses alternates → bare/objects)
+    │       └── info/
+    │           └── alternates  # Points to bare/objects
+    │
+    └── def456.../              # Clone for commit def456
+        └── .git/
+
+~/.nextflow/assets/nextflow-io/hello/
+└── .git/                       # Legacy repo location (HYBRID state)
+```
+
+**Key mechanisms:**
+
+1. **Bare repository as source of truth**: The bare repo is fetched/updated from the remote, keeping refs current
+2. **Dynamic resolution**: Revisions (branch/tag names) are resolved to commit SHAs using the bare repo's refs
+3. **Object sharing**: Clones use Git alternates to reference the bare repo's objects, avoiding duplication
+4. **Atomic operations**: Each clone is independent; downloading a new revision doesn't affect existing ones
+5. **Lazy creation**: Clones are created on-demand when a specific revision is requested
+
+**Advantages over revision map file:**
+- No external state to maintain or keep in sync
+- Bare repo fetch automatically updates all refs
+- Resolution is simple: `bareRepo.resolve(revision)` returns commit SHA
+- No race conditions on file updates
+- Simpler code with fewer failure modes
+
+### Strategy Pattern for Backward Compatibility
+
+The Strategy pattern provides clean separation and backward compatibility:
+
+```
+┌─────────────────────────┐
+│     AssetManager        │  ← Public API (unchanged)
+│     (Facade)            │
+└───────────┬─────────────┘
+            │
+            │ delegates to
+            ▼
+┌─────────────────────────┐
+│  RepositoryStrategy     │  ← Interface
+└───────────┬─────────────┘
+            △
+            │ implements
+    ┌───────┴────────┐
+    │                │
+┌───────────┐  ┌─────────────────┐
+│  Legacy   │  │ MultiRevision   │  ← Concrete strategies
+│ Strategy  │  │   Strategy      │
+└───────────┘  └─────────────────┘
+```
+
+**Strategy selection logic:**
+
+1. Check `NXF_SCM_LEGACY` environment variable → Use legacy if set
+2. Check if there is only the legacy asset of the repository (`isOnlyLegacy` method) → Use legacy (preserve existing)
+3. Otherwise -> Use multi-revision
+
+
+**Backward compatibility guarantees:**
+
+- Existing repositories continue to work without changes
+- `AssetManager` API remains identical
+- CLI commands work with both strategies transparently
+- Tests pass with minimal modifications
+- No forced migration; users opt-in naturally when creating new repos
+
+### Hybrid State Handling
+
+The system gracefully handles hybrid states where both legacy and multi-revision repositories coexist:
+
+- **Detection**: In hybrid states, a multi-revision strategy is selected by default.
+- **Fallback logic**: Multi-revision strategy can fall back to legacy repo for operations if needed
+- **No conflicts**: Strategies are designed to coexist; operations target different directories
+- **Explicit control**: Users can force a specific strategy via `setStrategyType()` or `NXF_SCM_LEGACY` environment variable
+
+### Migration Path
+
+Users naturally migrate as they pull new revisions:
+
+1. **Existing users**: Can continue with legacy repos (`NXF_SCM_LEGACY` state detected)
+2. **New users**: Get multi-revision by default
+3. **Opt-in migration**: Delete project directory to switch to multi-revision or pull with --migrate
+4. **Opt-out**: Set `NXF_SCM_LEGACY=true` to force legacy mode
+
+### Implementation Details
+
+**Key classes:**
+
+- `RepositoryStrategy`: Interface defining repository operations
+- `AbstractRepositoryStrategy`: Base class with shared helper methods
+- `LegacyRepositoryStrategy`: Direct clone implementation (original behavior)
+- `MultiRevisionRepositoryStrategy`: Bare repo + shared clones implementation
+
+**Critical methods:**
+
+- `download()`: Equivalent for both strategies (legacy pulls, multi-revision creates shared clone)
+- `getLocalPath()`: Returns appropriate working directory based on strategy
+- `getGit()`: Returns appropriate Git instance (legacy git, bare git, or commit git)
+
+### Performance Characteristics
+
+**Disk usage:**
+- Legacy: ~100% per repository (full clone with all git objects) + Worktree
+- Multi-revision: ~100% for bare + ~100K (.git with alternates) per revision + Worktree per revision
+
+**Operation speed:**
+- First download: Similar (both clone from remote)
+- Additional revisions: Multi-revision faster (only fetches new objects once, creates cheap clones)
+- Switching revisions: Multi-revision instant (different directories), legacy requires checkout
+
+### Known Limitations
+
+- No automatic migration of legacy repositories
+- Bare repository overhead even for users who only need one revision
+- JGit alternates slightly more complex than worktrees
+- Manual cleanup required for old revision clones
+
+## Links
+- [GitHub Issue #2870 - Multiple revisions of the same pipeline for concurrent execution](https://github.com/nextflow-io/nextflow/issues/2870)
+- [PR #6620 - Implementation of multiple revisions without revisions map](https://github.com/nextflow-io/nextflow/pull/6620)
+- Related PRs implementing the multi-revision approach (linked in #6620)
+

changelog.txt

@@ -1,5 +1,34 @@
 NEXTFLOW CHANGE-LOG
 ===================
+25.12.0-edge - 19 Dec 2025
+- Add `listDirectory()` to Path type and deprecate `listFiles()` (#6581) [56f0f007]
+- Add default maxSpotAttempts for fusion snapshots in Google Batch (#6652) [458ef97a]
+- Add onlyJobState option for SLURM executor (#6659) [3c3e9f52]
+- Add README files for all plugins (#6660) [bee8cff6]
+- Add runtimeClassName to the pod options (#6633) [ddcef4f4]
+- Add spot interruption tracking to trace records (#6606) [eecd8167]
+- Add URL encoding when revision name is used as HTTP query parameter (#6598) [7894e097]
+- Add warnings to JSON output in lint command (#6625) [bb066969]
+- Add wave.build.template config option (#6639) [d08a8952]
+- Check Nextflow version before loading plugins (#6591) [03da64eb]
+- Fix GitHub repository provider when providing token with auth property (#6662) [d01cbde1]
+- Fix optional param in params block (#6657) [bd8de5ca]
+- Fix String.format error when plugin URL contains percent chars (#6651) [59c4f4e1]
+- Fix validation of numeric types in params block (#6656) [664a26eb]
+- Fix WaveClient sending Bearer token to public S3 URLs (#6672) [ffaef0b6]
+- Fix: tolerate spaces in `$NXF_TASK_WORKDIR` (#6421) [7b386025]
+- Implementation of Git multiple revisions (#6620) [ce9d7b59]
+- Refactor Google Batch getExitCode to imperative style (#6649) [addd59e9]
+- Set local task exit status when time limit is exceeded (#6592) [d3f8e135]
+- Add Nextflow Development Constitution (#6578) [7047e6be]
+- docs: Add extra warnings as 25.10 is added to platform (#6655) [ae0e844f]
+- docs: Add longer NXF_SYNTAX_PARSER descriptions (#6637) [23c277ad]
+- docs: Document best practices for script and config params (#6631) [3421734d]
+- docs: Fix typos (#6641) [20f4631e]
+- docs: Improve preview feature warnings in documentation (#6663) [cdc7a586]
+- docs: Update note about AWS CLI (#6626) [bb7aecf8]
+- docs: Update NXF_SYNTAX_PARSER callouts (#6640) [1b284a19]
+
 25.11.0-edge - 28 Nov 2025
 - Add Google Batch LogsPolicy PATH option for logging to GCS (#6431) [5b61afe0]
 - Add default value to Apptainer pull timeout config paramter (#6534) [f4548bd1]

docs/_static/report-resources-min.png

@@ -25,21 +25,18 @@ The task hash is computed from the following metadata:
 - Task {ref}`Conda environment <process-conda>` (if applicable)
 - Task {ref}`Spack environment <process-spack>` and {ref}`CPU architecture <process-arch>` (if applicable)
 - Task {ref}`inputs <process-input>`
+- *New in 25.10:* Task {ref}`eval commands <process-out-eval>`
 - Task {ref}`script <process-script>`
-- Any global variables referenced in the task script
-- Any task {ref}`process-ext` properties referenced in the task script
+- Global variables referenced in the task script
+- *New in 23.10:* Task {ref}`process-ext` properties referenced in the task script
 - Any {ref}`bundled scripts <bundling-executables>` used in the task script
 - Whether the task is a {ref}`stub run <process-stub>`
 
 :::{note}
 Nextflow also includes an incrementing component in the hash generation process, which allows it to iterate through multiple hash values until it finds one that does not match an existing execution directory. This mechanism typically aligns with task retries (i.e., task attempts), however this is not guaranteed.
 :::
 
-:::{versionchanged} 23.09.2-edge
-The {ref}`process-ext` directive was added to the task hash.
-:::
-
-Nextflow computes this hash for every task when it is created but before it is executed. If resumability is enabled and there is an entry in the task cache with the same hash, Nextflow tries to recover the previous task execution. A cache hit does not guarantee that the task will be resumed, because it must also recover the task outputs from the [work directory](#work-directory).
+Nextflow computes this hash for every task before it is executed. If resumability is enabled, Nextflow checks whether the task cache contains a matching hash and whether the task outputs are still present in the [work directory](#work-directory). If both conditions are met, the task is resumed; otherwise, it is re-executed.
 
 Files are hashed differently depending on the caching mode. See the {ref}`process-cache` directive for more details.
 

docs/_static/report-summary-min.png

@@ -79,6 +79,13 @@ $ nextflow run nextflow-io/hello -r v1.1
 $ nextflow run nextflow-io/hello -r dev-branch
 $ nextflow run nextflow-io/hello -r a3f5c8e
 ```
+:::{versionadded} 25.12.0-edge
+:::
+Nextflow downloads and stores each explicitly requested Git branch, tag, or commit ID in a separate directory path, enabling you to run multiple revisions of the same pipeline simultaneously. Downloaded revisions are stored in a subdirectory of the local project: `$NXF_ASSETS/.repos/<org>/<repo>/clones/<commitId>`.
+
+:::{tip}
+Use tags or commit IDs instead of branches for reproducible pipeline runs. Branch references change as development progresses over time.
+:::
 
 (cli-params)=
 
@@ -171,12 +178,12 @@ Use this to understand a project's structure, see available versions, or verify
 $ nextflow info hello
 project name: nextflow-io/hello
 repository  : https://github.com/nextflow-io/hello
-local path  : $HOME/.nextflow/assets/nextflow-io/hello
+local path  : $HOME/.nextflow/assets/.repos/nextflow-io/hello
 main script : main.nf
 revisions   :
-* master (default)
+> master (default)
   mybranch
-  v1.1 [t]
+> v1.1 [t]
   v1.2 [t]
 ```
 
@@ -186,7 +193,7 @@ This shows:
 - Where it's cached locally
 - Which script runs by default
 - Available revisions (branches and tags marked with `[t]`)
-- Which revision is currently checked out (marked with `*`)
+- Which revisions are currently checked out (marked with `>`)
 
 ### Pulling or updating projects