Choose a faster default compressor

[ziw-liu]

@shinzlet found that zstd compression that we use as default in iohub makes I/O much slower for zarr arrays without significant (<5%) disk space gain. The best choice seems to be among the on-the-wire flavor of compressors, which is unsurprising considering that zarr-python's default is lz4.

[ziw-liu]

A grid search on numcodecs.Blosc parameters, with 16 zen 2 cores and DDR4 memory. Zarr-python's MemoryStore store is used to remove the filesystem from this measurement (the compressed buffer is written to RAM). No-compression (clevel=0) is used as a control for memory speed.

The current default is:

iohub/iohub/ngff/nodes.py

Lines 565 to 567 in 9c8db4c

	"compressor": Blosc(
	cname="zstd", clevel=1, shuffle=Blosc.BITSHUFFLE
	),

2-channel float32 image of phase (continuous) and fluorescence (discrete, originally uint16): CZYX=(2, 74, 1958, 1958), about 2.2 GB decompressed.

Only byte-shuffle (2):

Note that the 2 channels compress differently, both blosclz and lz4 saturate quickly for fluorescence.

Phase:

Fluorescence:

Code:

from timeit import Timer

import pandas as pd
import seaborn as sns
import zarr
from numcodecs import Blosc

image = zarr.open(
    "/hpc/projects/intracellular_dashboard/viral-sensor/2025_03_05_A549_pAL17_27h/1-preprocess/registered_truncated.zarr/0/1/000000/0",
    mode="r",
)[-1, :2]

print(image.shape, image.size * 4)


def bench(image, cname: str, clevel: int, memory_store=None):
    if memory_store is None:
        memory_store = zarr.storage.MemoryStore()
    zarr.save_array(
        memory_store,
        image,
        chunks=(1, 1, *image.shape[-2:]),
        compressor=Blosc(cname=cname, clevel=clevel, shuffle=Blosc.SHUFFLE),
        dtype=image.dtype,
    )
    size = int(zarr.storage.getsize(memory_store))
    return size


results = []
size_store = zarr.storage.MemoryStore()

for cname in ["blosclz", "lz4", "zstd"]:
    print("cname: ", cname)
    for clevel in [0, 1, 2, 3, 5]:
        print("clevel: ", clevel)
        for shuffle in (0, 1, 2):
            print("shuffle: ", shuffle)
            result = Timer(
                f"bench(image, cname='{cname}', clevel={clevel})",
                globals=globals(),
            ).repeat(3, number=1)
            size = bench(image, cname, clevel, memory_store=size_store)
            for replicate in result:
                results.append(
                    {
                        "cname": cname,
                        "clevel": clevel,
                        "shuffle": shuffle,
                        "time": replicate,
                        "compression ratio": (image.size * 4) / size,
                    }
                )

df = pd.DataFrame(results)
df["name-shuffle"] = df["cname"] + "-" + df["shuffle"].astype(str)

a = sns.scatterplot(
    data=df,
    x="time",
    y="compression ratio",
    style="clevel",
    hue="name-shuffle",
    s=10,
)
sns.move_legend(a, "lower right")
a.figure.savefig("compressor_benchmark.png", dpi=300)

Summary

On the tested hardware (AMD EPYC 7313P), numcodec's default (lz4, level 5, byte shuffle) is very good. While the current iohub default (zstd, level 1, bit shuffle) isn't objectively worse since it's just in a different segment of the trade-off frontier, it does come in 50%-80% slower for minor gain in compression ratio.

[ziw-liu]

A more complete study for (slightly different) QPI data: https://arxiv.org/abs/2503.18037

[shinzlet]

Interesting read! It seems pretty consistent with what I found in my testing too.

The biggest methodology difference between my work and the paper linked is probably chunk size - they use a 32x256x256 chunk for their compressor, I use chunks more like 100x2048x2048. That's a really massive difference and my testing shows that it definitely helps with bandwidth (and I imagine it must help with compression ratio).

When I get some free time, I would be really interested to do these benchmarks on windows / mac / linux / x64 / ARM platforms - when I started testing this months ago, I remember blosclz being a really strong performer compared to others, but all of our newer benchmarks seem to suggest that it's not very different than lz4 etc. I think the perf difference I saw might have been a result of doing those very first tests on my mac - perhaps the implementations perform differently on different builds / instruction sets.