Improve TIFF I/O stack

[ziw-liu]

For now we use a combination of tifffile, custom code (multi-page OME-TIFF), and ndtiff (pycromanager NDTIFF) to read TIFF-based formats. Given that tifffile also supports NDTIFF, there may be room to simplify the reading stack.

Also, we should investigate if tifffile and AICSImageIO has improved their OME-TIFF reading performance.

@nclack also mentioned that there are additional strategies we can explore to accelerate TIFF I/O.

[nclack]

The ScanImageTiffReader scans through ifds to build an index when the file is first opened. I've found that to be reasonably fast in the past but (a) that was when read speeds from hard drives were more in the 500-600 MB/s range and (b) I was typically looking at stacks w < 1000 frames. There's a python library.

I've also wondered if there are readers, e.g tifffile, that could take advantage of SubIFD tags to accelerate stack access. SubIFDs are usually not interpreted that way by readers afaict, and the little investigation I've done down this path never yielded anything. But I'm still curious.

[ziw-liu]

Also, we should investigate if tifffile and AICSImageIO has improved their OME-TIFF reading performance.

Tifffile has seen major improvements (#66), and I think shallower reading stack is worth a try.

[ziw-liu]

Given #66, #94, and #99, I think it is time for us to have a wider discussion of our TIFF I/O stack. On the lower level, it now seems reasonable to use the upgraded tifffile as much as possible.

During the recent meeting, @talonchandler suggested that this can also be a opportunity to redesign the TIFF reader API as a step towards #40. I think it is a good idea to agree on some general API schema before we jump into implementation.

Tagging @AhmetCanSolak and @JoOkuma for visibility.

[mattersoflight]

I agree with this. It is much better to build on top of tifffile. In addition to aligning the API for TIFF formats with API for Zarr formats, we should make sure that we transfer metadata from TIFF to Zarr.

Once the metadata has been transferred from TIFF to zarr, we can delete all the raw TIFF stacks and save many TBs of space.

I suggest the following step-by-step process:

• iohub convert copies over TIFF metadata verbatim to a hidden folder in Zarr store. It may be named .mm_meta.
• We refine API to add metadata to .zattrs. This API can take an optional argument to parse metadata from the Micro-Manager metadata stored in the zarr store.

tagging @talonchandler, @ieivanov , @edyoshikun @rachel-banks to invite discussion.

[ziw-liu]

Thought I sent this earlier but apparently didn't:

Here's a POC of how the TIFF reading API can be more like a mapping of arrays as it is implemented for NGFF. Using Dask here for lazy transpose of raw axes ordering (necessary for utilizing tifffile's ZarrTiffStore implementation), and Xarray for labelling axes (not strictly necessary). Usage will look like:

In [1]: from iohub.multipagetiff import MMStack

In [2]: data_path = "pytest_temp/test_data/MM20_ome-tiffs/mm2.0-20201209_4p_3c_512k_1/"

In [3]: with MMStack(data_path) as dataset:
   ...:     print(dataset[0])
   ...:     print(dataset[1].isel(T=0, C=0, Z=0).max().compute())
   ...: 
<xarray.DataArray 'mm2.0-20201209_4p_3c_512k_1' (T: 1, C: 3, Z: 1, Y: 128,
                                                 X: 128)>
dask.array<getitem, shape=(1, 3, 1, 128, 128), dtype=uint16, chunksize=(1, 1, 1, 128, 128), chunktype=numpy.ndarray>
Dimensions without coordinates: T, C, Z, Y, X
<xarray.DataArray 'mm2.0-20201209_4p_3c_512k_1' ()>
array(213, dtype=uint16)

In [4]: dataset = MMStack(data_path)

In [5]: for position in dataset:
   ...:     print(position.mean())
   ...: 
<xarray.DataArray 'mm2.0-20201209_4p_3c_512k_1' ()>
dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=(), chunktype=numpy.ndarray>
<xarray.DataArray 'mm2.0-20201209_4p_3c_512k_1' ()>
dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=(), chunktype=numpy.ndarray>
<xarray.DataArray 'mm2.0-20201209_4p_3c_512k_1' ()>
dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=(), chunktype=numpy.ndarray>
<xarray.DataArray 'mm2.0-20201209_4p_3c_512k_1' ()>
dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=(), chunktype=numpy.ndarray>

In [6]: dataset.close()