Port to iohub 0.3.2 ImageArray API

[alxndrkalinin]

Summary

Staging commit a10d4c4 bumped packages/viscy-data to iohub>=0.3.2, which rewrote ImageArray and requires Python ≥3.12. Fresh uv sync breaks ("No solution found"); once forced through, 81 tests fail / 1090 pass against this tip because callers still use the removed .name / .tensorstore(context=...) attributes. This PR ports all callers to the new API while preserving the tensorstore backend (async concurrent reads with data_copy_concurrency / cache_pool_bytes).

After this PR: fresh sync resolves cleanly, 8 fail / 1163 pass — the remaining 8 are the pre-existing out-of-scope failures listed below, not caused or touched by this change.

Commits

1. chore(deps) — bump requires-python >=3.12 in root + 9 sub-project pyprojects, drop the now-unsupported Python :: 3.11 classifier, regenerate uv.lock (iohub==0.3.2).
2. refactor(viscy-data) — ImageArray.name → ImageArray.path at 5 call sites (3 source files). Sliding-window sample-id tuple is emitted as f"/{img.path}" so PredictionWriter._create_image / MaskTestDataset continue to parse the 5-part /A/1/0/array layout via split("/").
3. refactor(data) — port .tensorstore(context=...) → .native and move tensorstore context config from per-dataset to per-plate (at open_ome_zarr time) via iohub.core.config.TensorStoreConfig.
4. refactor(data): use context managers for one-shot plate openers — wrap generate_normalization_metadata and run_qc_metrics plate opens in with blocks, per CLAUDE.md's context-manager rule.
5. refactor(data): drop unused cache_pool_bytes from Dataset classes — remove the no-op param from TripletDataset / MultiExperimentTripletDataset and the three call sites that still passed it. DataModule-level cache_pool_bytes stays (it's the one that actually feeds TensorStoreConfig at plate-open time), so YAML configs are unaffected.
6. docs(data): align docstrings and error text with .name -> .path port — update HCSStackIndex.image comment, _read_img_window return-docstring, and _get_windows error message to match the /A/1/0/array format the port emits.
7. fix(docs): address Copilot review typos — Z-sclice → Z-slice in prediction_writer.py; fix stale qc_metrics.run_qc_metrics reference in focus.py comment (real function is generate_qc_metadata).

How the tensorstore backend is preserved

iohub 0.3.2 removed per-array .tensorstore(context=...). The context is now set once at plate open:

from iohub.core.config import TensorStoreConfig
with open_ome_zarr(
    path,
    implementation="tensorstore",
    implementation_config=TensorStoreConfig(
        data_copy_concurrency=N,
        cache_pool_bytes=M,
    ),
) as plate:
    handle = plate[pos]["0"].native  # raw ts.TensorStore
    data = handle[sel].read().result()
    # or ts.stack([handle[sel].translate_to[0] for _ in patches]).read().result()

Four plate openers are upgraded to the tensorstore impl — the only ones in the repo whose positions flow into .native[sel].read().result() / ts.stack(...) reads:

Opener	Call site
TripletDataModule._align_tracks_tables_with_positions	packages/viscy-data/src/viscy_data/triplet.py
MultiExperimentIndex._store_cache opener	applications/dynaclr/src/dynaclr/data/index.py
generate_normalization_metadata	packages/viscy-utils/src/viscy_utils/meta_utils.py
run_qc_metrics → generate_qc_metadata	applications/qc/src/qc/qc_metrics.py

SLURM-aware CPU count (SLURM_CPUS_PER_TASK → os.cpu_count() fallback) is preserved. cache_pool_bytes is owned by the DataModule and threaded into TensorStoreConfig at plate-open time. The Dataset classes no longer carry the parameter.

All downstream .oindex[...], .translate_to[...], ts.stack(...).read().result() — including the batched-read pattern documented in applications/dynaclr/CLAUDE.md — work unchanged because .native on a tensorstore-backed plate returns exactly the same raw ts.TensorStore the old .tensorstore(context=ctx) factory returned. Other openers (hcs.py, mmap_cache.py, segmentation.py, cell_index.py, etc.) stay on the default zarr impl — they only use .oindex / img[...] / metadata, which work on both impls.

recheck_cached_data="open" is no longer plumbable through iohub and has been dropped; benign for the read-only training/predict paths that were the only consumers.

Smoke verification

handle type: TensorStore
slab dtype/shape: float32 (44, 10, 10)
ts.stack shape: (3, 10, 10)

An HCSDataModule smoke test on real SEC61B zarr builds and iterates through a batch end-to-end.

Not fixed in this PR (pre-existing, unrelated)

These 8 tests are still red on the base branch and this PR does not attempt them:

• applications/airtable/tests/test_database.py — 2 failures (mock call-count drift, env-vars fixture).
• applications/dynaclr/tests/test_multi_experiment_integration.py — 4 failures (missing configs/training/{fit,multi_experiment_fit}.yml after the dynaclr-dino / cleanup PRs).
• applications/dynaclr/tests/test_training_integration.py — 2 failures (one config-missing, one pytorch_metric_learning labels shape regression).

Test plan

• uv sync --all-packages --all-extras resolves cleanly on a fresh venv.
• Smoke: open_ome_zarr(..., implementation="tensorstore", implementation_config=TensorStoreConfig(data_copy_concurrency=8)) on SEC61B zarr returns a TensorStore handle with working .read().result() and ts.stack.
• uv run pytest across packages + applications: 8 failed / 1163 passed / 1 xfailed (vs. 81 failed / 1090 passed pre-port).
• HCSDataModule setup + batch iteration works end-to-end against real zarr.
• Full dynacell fit --trainer.fast_dev_run=true on a GPU (deferred — blocked on a /-partition space issue unrelated to this PR).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Ports the codebase to iohub>=0.3.2’s updated ImageArray API and preserves the tensorstore backend by configuring tensorstore at plate-open time (via TensorStoreConfig) and switching call sites from .tensorstore(context=...) to .native.

Changes:

• Bump requires-python to >=3.12 across the root project and multiple sub-projects, removing the Python 3.11 classifier.
• Update ImageArray.name usages to ImageArray.path, and adjust sliding-window sample IDs to keep the existing "/A/1/0/array" parsing behavior.
• Move tensorstore configuration to open_ome_zarr(..., implementation="tensorstore", implementation_config=TensorStoreConfig(...)) and update downstream reads to use .native.

Reviewed changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.
packages/viscy-utils/src/viscy_utils/meta_utils.py	Open plates with tensorstore + TensorStoreConfig; switch sampling to .native.
packages/viscy-utils/src/viscy_utils/callbacks/prediction_writer.py	Replace image.name with image.path in resize debug logging.
packages/viscy-utils/pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.
packages/viscy-transforms/pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.
packages/viscy-models/pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.
packages/viscy-data/src/viscy_data/triplet.py	Switch tensorstore access to .native; configure tensorstore at plate open with TensorStoreConfig.
packages/viscy-data/src/viscy_data/sliding_window.py	Replace img.name with img.path; emit sample IDs as f"/{img.path}" to preserve downstream parsing.
packages/viscy-data/src/viscy_data/segmentation.py	Replace target_img.name with target_img.path in debug logging.
packages/viscy-data/pyproject.toml	Raise minimum Python to 3.12; normalize iohub>=0.3.2 spec formatting.
applications/qc/tests/test_focus.py	Open test plate using tensorstore impl + TensorStoreConfig to support .native.read().result().
applications/qc/src/qc/qc_metrics.py	Open plate using tensorstore impl + TensorStoreConfig for metric execution.
applications/qc/src/qc/focus.py	Switch focus metric reads to .native[...] .read().result() and drop direct tensorstore context creation.
applications/qc/pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.
applications/dynaclr/src/dynaclr/data/index.py	Cache plates opened with tensorstore impl + configurable TensorStoreConfig.
applications/dynaclr/src/dynaclr/data/dataset.py	Switch tensorstore handle acquisition to .native; keep cache_pool_bytes as documented no-op.
applications/dynaclr/src/dynaclr/data/datamodule.py	Build and pass a shared TensorStoreConfig (SLURM-aware concurrency + cache pool) into the index.
applications/dynaclr/pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.
applications/dynacell/pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.
applications/cytoland/pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.
applications/airtable/pyproject.toml	Raise minimum Python to 3.12; drop 3.11 classifier.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[edyoshikun]

Here are some of my findings:

• @srivarra can we expose some TensorStoreConfig variables? 0.3.2 file_io_concurrency for NSF I/O parallelism ,recheck_cached_data for the heavy load NSF reads of metadata. These are critical for I/O boosts. These need to be exposed for us to evalute.
• Why do we have TensorStoreConfig instead of just the raw ts.Context (naive question). In VisCy we use the ts.Context. Currently, we do ts.stack([pos1["0"].native, pos2["0"].native, ...]).read().result() which works, but each .native call goes through iohub's Position wrapper and I think it re-parses .zattrs/.zarray on first access. An iohub helper like plate.stack_positions([path1, path2, ...], array="0") -> ts.TensorStore that batches the metadata reads would cut the per-FOV Python overhead. Even better: a plate-level "open all FOVs under this well" call that shares one ts.Context and one metadata read pass.
• @srivarra do we always need to through the ImageArray? Can we go Position.array_tensorstore(array="0") that returns ts.TensorStore directly without building a wrapper? Or is this beause the default backend is zarrpython?

[ieivanov]

@srivarra is there an opportunity to introduce better iohub defaults what will reduce some of the boilerplate code in this PR?

[edyoshikun]

czbiohub-sf/iohub#406 adding some changes to support the options we neeed for dynaclr

[srivarra]

@edyoshikun

can we expose some TensorStoreConfig variables

Yeah, we can do that.

a plate-level "open all FOVs under this well" call that shares one ts.Context and one metadata read pass.

Yeah we can try to set this up, the pydantic config was just limited, there isn't much of a reason for it other than it being a nice structured way to add settings together.

do we always need to through the ImageArray?

Yeah, unless you don't need the OME-Zarr group metadata.

We could set up a single ts.Context for a HCS plate as part of the config, will need to look into how to effectively do this.