|
| 1 | +# NIO Filesystem for Seqera Platform Datasets |
| 2 | + |
| 3 | +- Authors: Jorge Ejarque |
| 4 | +- Status: draft |
| 5 | +- Date: 2026-03-10 |
| 6 | +- Tags: nio, filesystem, seqera, datasets, nf-tower |
| 7 | + |
| 8 | +Technical Story: Enable Nextflow pipelines to read Seqera Platform datasets as ordinary file paths using `seqera://` URIs. |
| 9 | + |
| 10 | +## Summary |
| 11 | + |
| 12 | +Add a Java NIO `FileSystemProvider` to the `nf-tower` plugin that registers the `seqera://` scheme, allowing pipelines to reference Seqera Platform datasets (CSV/TSV) as standard file paths without manual download steps. The implementation reuses the existing `TowerClient` for all HTTP communication, inheriting authentication and retry behaviour. |
| 13 | + |
| 14 | +## Problem Statement |
| 15 | + |
| 16 | +Nextflow users managing datasets on the Seqera Platform must currently download dataset files manually or through custom scripts before referencing them in pipelines. There is no native integration between Nextflow's file abstraction and the Seqera Platform dataset API. This creates friction in workflows where datasets are the primary input and forces users to handle authentication, versioning, and file staging outside the pipeline definition. |
| 17 | + |
| 18 | +## Goals or Decision Drivers |
| 19 | + |
| 20 | +- Transparent access to Seqera Platform datasets using standard Nextflow file path syntax |
| 21 | +- Reuse of existing nf-tower plugin infrastructure (authentication, HTTP client, retry/backoff) |
| 22 | +- Hierarchical path browsing matching the platform's org/workspace/dataset structure |
| 23 | +- Extensible architecture that can support future Seqera-managed resource types (e.g. data-links) |
| 24 | +- No new plugin or module — feature lives within nf-tower |
| 25 | + |
| 26 | +## Non-goals |
| 27 | + |
| 28 | +- Streaming large datasets — the Platform API does not support streaming; content is fully buffered on download |
| 29 | +- Implementing resource types beyond `datasets` — only the extensible architecture is required |
| 30 | +- Local caching across pipeline runs — Nextflow's standard task staging handles caching |
| 31 | +- Dataset management operations (delete, rename) — the filesystem is read-only in the initial implementation |
| 32 | + |
| 33 | +## Considered Options |
| 34 | + |
| 35 | +### Option 1: Standalone plugin with dedicated HTTP client |
| 36 | + |
| 37 | +A new `nf-seqera-fs` plugin with its own HTTP client configuration and authentication setup. |
| 38 | + |
| 39 | +- Good, because it isolates the filesystem code from the nf-tower plugin |
| 40 | +- Bad, because it duplicates authentication configuration and HTTP client setup |
| 41 | +- Bad, because two separate HTTP clients sharing a refresh token would corrupt each other's auth state |
| 42 | + |
| 43 | +### Option 2: NIO filesystem within nf-tower using TowerClient delegation |
| 44 | + |
| 45 | +Add the filesystem to nf-tower, delegating all HTTP through the existing `TowerClient` singleton via a typed `SeqeraDatasetClient` wrapper. |
| 46 | + |
| 47 | +- Good, because it shares authentication and token refresh with TowerClient |
| 48 | +- Good, because it reuses existing retry/backoff configuration |
| 49 | +- Good, because no new dependencies are needed |
| 50 | + |
| 51 | +### Option 3: Direct HxClient usage within nf-tower |
| 52 | + |
| 53 | +Add the filesystem to nf-tower but use `HxClient` directly rather than going through TowerClient. |
| 54 | + |
| 55 | +- Good, because it gives full control over request construction |
| 56 | +- Bad, because exposing HxClient internals couples the filesystem to implementation details |
| 57 | +- Bad, because token refresh coordination with TowerClient becomes manual |
| 58 | + |
| 59 | +## Solution or decision outcome |
| 60 | + |
| 61 | +Option 2 — NIO filesystem within nf-tower using TowerClient delegation. All HTTP calls go through `TowerClient.sendApiRequest()`, ensuring a single point of authentication and retry logic. |
| 62 | + |
| 63 | +## Rationale & discussion |
| 64 | + |
| 65 | +### Path Hierarchy |
| 66 | + |
| 67 | +The `seqera://` path encodes the Platform's organizational structure directly: |
| 68 | + |
| 69 | +``` |
| 70 | +seqera:// → ROOT (directory, depth 0) |
| 71 | + └── <org>/ → ORGANIZATION (directory, depth 1) |
| 72 | + └── <workspace>/ → WORKSPACE (directory, depth 2) |
| 73 | + └── datasets/ → RESOURCE TYPE (directory, depth 3) |
| 74 | + └── <name>[@<version>] → DATASET (file, depth 4) |
| 75 | +``` |
| 76 | + |
| 77 | +Each level is a directory except the leaf dataset, which is a file. Version pinning uses an `@version` suffix on the dataset name segment (e.g. `seqera://acme/research/datasets/samples@2`). Without it, the latest non-disabled version is resolved. |
| 78 | + |
| 79 | +### Name-to-ID Resolution |
| 80 | + |
| 81 | +The path uses human-readable names but the Platform API requires numeric IDs. Resolution is built from two API calls at filesystem initialization: |
| 82 | + |
| 83 | +1. `GET /user-info` → obtain `userId` |
| 84 | +2. `GET /user/{userId}/workspaces` → returns all accessible org/workspace pairs |
| 85 | + |
| 86 | +This single source provides both directory listing content and name→ID mapping. Results are cached in `SeqeraFileSystem` with invalidation on write operations. `GET /orgs` is intentionally not used as it returns all platform orgs, not scoped to user membership. |
| 87 | + |
| 88 | +### Component Structure |
| 89 | + |
| 90 | +``` |
| 91 | +plugins/nf-tower/src/main/io/seqera/tower/plugin/ |
| 92 | +├── fs/ ← NIO layer |
| 93 | +│ ├── SeqeraFileSystemProvider ← FileSystemProvider (scheme: "seqera") |
| 94 | +│ ├── SeqeraFileSystem ← FileSystem with org/workspace/dataset caches |
| 95 | +│ ├── SeqeraPath ← Path implementation (depth 0–4) |
| 96 | +│ ├── SeqeraFileAttributes ← BasicFileAttributes |
| 97 | +│ ├── SeqeraPathFactory ← PF4J FileSystemPathFactory extension |
| 98 | +│ └── DatasetInputStream ← SeekableByteChannel over InputStream |
| 99 | +├── dataset/ ← API client layer |
| 100 | +│ ├── SeqeraDatasetClient ← Typed HTTP client wrapping TowerClient |
| 101 | +│ ├── DatasetDto ← Dataset API response model |
| 102 | +│ ├── DatasetVersionDto ← Version API response model |
| 103 | +│ ├── OrgAndWorkspaceDto ← Org/workspace list model |
| 104 | +│ └── WorkspaceOrgDto ← Workspace/org mapping model |
| 105 | +└── resources/META-INF/services/ |
| 106 | + └── java.nio.file.spi.FileSystemProvider |
| 107 | +``` |
| 108 | + |
| 109 | +### Key Design Decisions |
| 110 | + |
| 111 | +1. **TowerClient delegation**: `SeqeraDatasetClient` delegates all HTTP through `TowerFactory.client()` → `TowerClient.sendApiRequest()`. This ensures shared authentication state and avoids the token refresh corruption that would occur with separate HTTP client instances. |
| 112 | + |
| 113 | +2. **One filesystem per JVM**: `SeqeraFileSystemProvider` maintains a single `SeqeraFileSystem` keyed by scheme. This matches the `TowerClient` singleton-per-session pattern. |
| 114 | + |
| 115 | +3. **Read-only initial scope**: The filesystem reports `isReadOnly()=true`. Write support (dataset upload via multipart POST) is deferred to a future iteration. |
| 116 | + |
| 117 | +4. **Download filename constraint**: The Platform API's download endpoint (`GET /datasets/{id}/v/{version}/n/{fileName}`) requires the exact filename from upload time. The implementation always resolves `DatasetVersionDto.fileName` from `GET /datasets/{id}/versions` before constructing the download URL. |
| 118 | + |
| 119 | +5. **Extensible resource types**: The path hierarchy reserves depth 3 for a resource type segment (currently only `datasets`). Adding support for data-links or other resource types requires only a new handler at the directory listing and I/O layers, with no changes to path resolution or authentication. |
| 120 | + |
| 121 | +6. **Thread safety**: `SeqeraFileSystem` cache methods and `SeqeraFileSystemProvider` lifecycle methods are `synchronized`. The filesystem map uses `LinkedHashMap` with external synchronization rather than `ConcurrentHashMap`, matching the low-contention access pattern. |
| 122 | + |
| 123 | +### Limitations |
| 124 | + |
| 125 | +- **No size metadata**: `SeqeraFileAttributes.size()` returns 0 for all paths because the Platform API does not expose content length in dataset metadata. |
| 126 | +- **Single endpoint per JVM**: The filesystem key is scheme-only; concurrent access to different Platform endpoints in the same JVM is not supported. |
| 127 | + |
| 128 | +### Streaming Downloads |
| 129 | + |
| 130 | +Dataset downloads use `TowerClient.sendStreamingRequest()` which calls `HxClient.sendAsStream()` — the response body is returned as an `InputStream` streamed directly from the HTTP connection. This avoids the triple-buffering problem (`String` → `getBytes()` → `ByteArrayInputStream`) that would otherwise consume ~40 MB heap per 10 MB dataset. The `HxClient.sendAsStream()` method goes through the same `sendWithRetry()` path as `sendAsString()`, so retry logic and token refresh are preserved. |
| 131 | + |
| 132 | +## Links |
| 133 | + |
| 134 | +- [Spec](../specs/260310-seqera-dataset-fs/spec.md) |
| 135 | +- [Implementation plan](../specs/260310-seqera-dataset-fs/plan.md) |
| 136 | +- [Data model](../specs/260310-seqera-dataset-fs/data-model.md) |
0 commit comments