Parallel writing to shards (#311)

* update signature type hints

* clarify the type of position keys

* expose sharding in plate creation

* isort

* fix default version

* fix example code block format

* fix type hints

* utility to split indices by shards

* separate apply and save

* simplify random number generation

* wip: batched writing in time

* match storage keys with values

* removed unused argument

* fix import

* fix string formatting

* add more version testing

* use tensorstore instead

* control tensorstore concurrency

* memory management

* isort

* format

* remove platform check

* warning if shards is specified for 0.4

* set shards to none for 0.4

* Update acquire-zarr OME v0.5 fixture / aqz test to reflect new config API and downsampling behavior

* add notes about upstream issues

* add example of sharded plate

* explicitly add layout in open_ome_zarr

* Fix tensorstore empty array handling (#326)

* Fix tensorstore empty array handling

- Add validation for empty arrays in _save_transformed before tensorstore write
- Skip write operations for empty arrays with warning messages
- Add comprehensive error handling with detailed diagnostics for tensorstore failures
- Improve error messages to include array shapes, sizes, and tensorstore details

This resolves the ValueError: Error aligning dimensions issue when empty arrays
are passed to tensorstore write operations.

* Add empty results check to prevent tensorstore alignment errors

Adds validation in apply_transform_to_tczyx_and_save() to check for empty
results dictionary before calling _save_transformed(). When no valid time
points are available, logs diagnostic message and skips write operation
instead of attempting to write empty arrays to tensorstore, which causes
alignment dimension mismatches.

* Revert "Fix tensorstore empty array handling"

This reverts commit 65c9ddb.

* better handling of output_time_indices

* style

* bugfix and better type hints

* better messaging

* style

* raise error if attempting sharding along channel dimension

---------

Co-authored-by: Alan Liddell <aliddell@chanzuckerberg.com>
Co-authored-by: Ivan Ivanov <ivan.ivanov@czbiohub.org>

Author: 3 people

iohub/ngff/nodes.py

@@ -383,7 +383,7 @@ def dask_array(self):
     def downscale(self):
         raise NotImplementedError
 
-    def tensorstore(self):
+    def tensorstore(self, concurrency: int | None = None):
         """Open the zarr array as a TensorStore object.
         Needs the optional dependency ``tensorstore``.
 
@@ -396,10 +396,20 @@ def tensorstore(self):
 
         ts_spec = {
             "driver": "zarr2" if self.metadata.zarr_format == 2 else "zarr3",
-            "kvstore": (Path(self.store.root) / self.path.strip("/")).as_uri(),
+            "kvstore": {
+                "driver": "file",
+                "path": str(Path(self.store.root) / self.path.strip("/")),
+            },
         }
         zarr_dataset = ts.open(
-            ts_spec, read=True, write=not self.read_only
+            ts_spec,
+            read=True,
+            write=not self.read_only,
+            context=(
+                ts.Context({"data_copy_concurrency": {"limit": concurrency}})
+                if concurrency
+                else None
+            ),
         ).result()
         return zarr_dataset