add optional support for zstandard compression

[edwardhartnett]

I suggest we fully support the ZStandard library in the netCDF C, Fortran, and Java libraries. (This suggestion is part of a larger effort to improve netCDF compression #1545).

Compression is Essential, and Zlib is Slow

Compression has become a core component of netCDF. Large data producers, like NOAA, NASA, and ESA, all must compress their data in order to fit into their storage systems, which were sized with the assumption that data compression would be used operationally. That is, none of these data providers have the disk space to process and store these data uncompressed. Compression is essential.

The slowness of zlib with today's giant data files has become a real problem for ESA with the Copernicus satellite program, which has been generating large amounts of netCDF-4 data for many years. As with NASA Earth observing missions, the amazing increase in the capabilities of space instruments has been accompanied with a dramatic increase in the size of the data - the slow speed of zlib is becoming a serious problem.

As documented in #1545 NOAA is also running into the limits of zlib. For the recent GFS release, we dodged the bullet by adding the ability to use zlib with parallel I/O to netcdf-c-4.7.4. This allowed NOAA to regain its lost time budget and write the forecast data on time. But future versions of the GFS are planned with higher resolution. As we all understand, doubling the resolution in time and space results in a 16-fold multiplication of the data sizes, and, consequently, the time required to compress the data.

Why ZStandard?

The Zstandard compression library, currently available to netCDF users through the CCR project, provides significant performance improvements over zlib.

This chart from our recent AGU extended abstract shows typical performance improvement achieved with ZStandard:

When used with the new quantize feature, Zstandard performance is even more impressive:

Zstandard also offers a much wider range of tradeoff between compression and speed. For example in Klower's paper a setting of -10 was used to get very fast compression, which only compressed the data 50%. This allows operational requirements to be met. Zlib does not offer this range of speed/compression trade off. It is not possible to make zlib go much faster, whatever setting is used.

ZStandard is free and open software, readily available on all platforms: Unix, MacOS, and Windows. A reference C library is available which can be compiled anywhere, and pre-build Zstandard libraries are available on most package management systems. A pure-Java implementation is also available from the Zstandard website.

The Zstandard web site displays a table of Zstandard performance compared to other compression libraries, including zlib. They show better performance than I have measured but similar. They show a 5x speedup for writing, 4x for reading. (I have not measured read time - NOAA is more concerned with write times.)

Implementation

How would this look in implementation?

C and Fortran

The C and Fortran implementations are available in the CCR. They are very similar to the existing functions which currently support zlib:

    int nc_def_var_zstandard(int ncid, int varid, int level);
    int nc_inq_var_zstandard(int ncid, int varid, int *zstandardp, int *levelp);

In Fortran:

  function nf90_def_var_zstandard(ncid, varid, level) result(status)
  function nf90_inq_var_zstandard(ncid, varid, zstandardp, levelp) result(status)

The build systems would be updated to optionally include Zstandard, as Szip is handled now. Extra items in the build summary and netcdf_meta.h will help end users know if Zstandard is supported in any particular build. The Zstandard filter code would be included with netcdf-c, and installed in the HDF5 filter directory for the user.

(If released in the netCDF C and Fortran libraries, these functions would be removed from CCR.)

NetCDF-Java

There is a pure-Java Zstandard library. Code to use it as a filter would have to be written by the Unidata netCDF-Java team.

My understanding is that only Java read code would have to be modified. The pure Java part of netCDF-Java only reads NetCDF/HDF5 files. In this case the use of Zstandard would be similar to the way zlib is handled.

The NetCDF-Java library would be able to use the C library to write netCDF/HDF5 files with Zstandard compression, but support would have to be added to call the new functions.

Conclusion

The inclusion of Zstandard in all netCDF libraries will be a significant improvement for large data producers such as NOAA, NASA, and ESA. Zstandard provides both better performance and a much better range of performance, allowing data producers to tune the compression vs. speed of their data to match sometimes stringent operational requirements.

All readers of compressed data will also benefit from a performance speedup with Zstandard.

The greater ease of handling larger and larger data sets will also bring benefit to the science community, as more data can be written and distributed within the operational constraints of large data producers.

Although these benefits are currently available if all concerned install CCR, I believe there will be significant benefit to the netCDF community if Zstandard were supported out of the box by all netCDF libraries.

@DennisHeimbigner @WardF @dopplershift @lesserwhirls @czender and @gsjaardema your input is very much requested...

References

CCR web site: https://github.com/ccr/ccr

Zstandard web site: http://facebook.github.io/zstd/

Compressing atmospheric data into its real information content
Milan Klöwer, Miha Razinger, Juan J. Dominguez, Peter D. Düben & Tim N. Palmer, https://www.nature.com/articles/s43588-021-00156-2

Quantization and Next-Generation Zlib Compression for Fully Backward-Compatible, Faster, and More Effective Data Compression in NetCDF Files, Hartnett, Zender, Fisher, Heimbigner, Hang, Gerheiser, Curtis: https://www.researchgate.net/publication/357001251_Quantization_and_Next-Generation_Zlib_Compression_for_Fully_Backward-Compatible_Faster_and_More_Effective_Data_Compression_in_NetCDF_Files

[DennisHeimbigner]

Query: is zstandard the same as zlib-ng?
Also, is the zstandard API different from zlib (that is, is it a direct replacement).?

[czender]

@DennisHeimbigner Zstandard is distinct from zlib-ng, which is a drop-in replacement for zlib. The Zstandard API is different. You can see the Zstandard C API in action in this CCR codec code: https://github.com/ccr/ccr/blob/master/hdf5_plugins/ZSTANDARD/src/H5Zzstandard.c.
I would underscore @edwardhartnett's point above that Zstandard has wide support in a variety of native languages including Java.

[DennisHeimbigner]

Got it. What do you use as the zstandard implementation you use (i.e. code base)?

[edwardhartnett]

@DennisHeimbigner the zstandard code can be found here: http://facebook.github.io/zstd/

However I installed it on ubuntu with:

sudo apt-get install zstd

[edwardhartnett]

@DennisHeimbigner as noted by Charlie, Zstandard is a completely different library from zlib. It has it's own filter number in the HDF5 ecosystem.

zlib-ng is a great instant upgrade for everyone. It causes no compatibility issues and requires no code changes, it's fully backward compatible with old installs of netCDF, and it can be done today. But it only takes us a few steps back from the rising tide of data.

Zstandard offers features beyond zlib/zlib-ng, including better performance and compression. Fully supporting it in netCDF will take some time and effort. Once available it will not be backward-compatible with old installations of netCDF - readers will have to upgrade to the latest netCDF release. So it will take time to become widely useful.

But once implemented and digested by the community, Zstandard will allow our users to take a few more big steps back from that rising tide. ;-)

[DennisHeimbigner]

I see there is also a numcodecs std for this, so adding some tests for it should
be pretty easy.
But at some point, I would to consider leaving this in CCR and adding the necessary
codecs libraries to CCR.

[dopplershift]

To me the zstd algorithm definitely looks promising. If we were to do this, my only concerns would be:

1. Making sure the netCDF standard ("NUG") is properly updated first
2. Planning for this work in the netCDF-C, netCDF-FORTRAN, and especially netCDF-java roadmaps/schedules so that this is available in both libraries before we go shouting it from the rooftops.

[dopplershift]

cc @haileyajohnson

[edwardhartnett]

I am happy to update the NUG, if necessary.

The code for C/Fortran is already written, documented, and released in CCR. So it can be moved into the netcdf-c and netcdf-fortran libraries in a matter of hours, if the consensus is that zstandard should be fully supported in netCDF. I can do this at any time.

The Java library is the open question.

[edwardhartnett]

@DennisHeimbigner I don't understand your comments about codecs and testing.

A few other stray points that have not yet been mentioned:

• In NetCDF needs better and faster compression, especially for HPC applications #1545 I suggested adding a handful of new compression libraries. I think that if ZStandard is added, there will be relatively little appetite or need to add more. The others do not seem to offer any dramatic benefits over ZStandard. So I think adding ZStandard may be enough to satisfy compression needs for some years to come.
• I note that ZStandard is also used in Zarr. No doubt it is preferred there, for the same reasons we would like to use it in netCDF/HDF5 files.

[DennisHeimbigner]

The codec stuff is necessary for nczarr filter support.
Except for testing, I do not see what it means to add zstandard support
to netcdf-c. It is as simple/hard as adding any other filter.

[edwardhartnett]

To add zstandard support means:

• Look for zstandard library at build time in configure/cmake.
• Include additional functions nc_def_var_zstandard/nc_inq_var_zstandard.
• If zstandard found, build and install the zstandard filter as part of netcdf-c build.

This would make zstandard a privileged filter - one with greater support within the netCDF C, Fortran, and Java libraries.

As noted, zstandard is already fully supported in CCR. The point of this issue is to move that support into the netCDF C and Fortran libraries, and to get a plan for adding it to netCDF-Java.

[rsignell-usgs]

The default compressor in xarray when writing zarr is actually lz4:

 'compressor': Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0),

which worked even better than zstd in a genotype benchmark described in a blog post by @alimanfoo.

The conclusions of the blog post were:

Conclusions

For maximum all-round speed, Blosc with LZ4 and no shuffle is ludicrous, compressing at 11G/s and decompressing at over 7G/s.
For higher compression ratios, Blosc with Zstandard is excellent. Adding the bit-shuffle filter increases compression ratio even further to over 50X at a moderate cost to decompression speed.
Using Blosc with multiple threads accelerates both compression and decompression, but larger (16M) chunk sizes were needed to see the benefits (see also below).

Naturally there were benchmark result caveats also:

Caveats

There are multiple interacting factors at work here, including data, software and hardware. Your mileage may vary substantially. Running your own benchmark is highly recommended.

It would be great to have support for zstd for sure though.

[DennisHeimbigner]

I was afraid of this. Everytime we get a new and "better" compressor, it becomes
desirable to build it into netcdf-c. The only reason we support zlib and szip
is because they are in HDF5.
I really think a better idea is search at build time for ccr and let it do all the work.

[edwardhartnett]

The reason to put zstandard into netCDF is that it makes it easy for users to converge on something better than zlib, without splintering the community into many different compressions, each unreadable by other users until they figure out how to locate, build, and install the correct filter.

My understanding was that there was some concern at Unidata about this splintering of the community.

The current state of play is that users can use CCR and have Zstandard and soon BLOSC. NetCDF-Java can catch up as best it can, or be left behind. This is the path we are now on.

The goal with this issue is to guide the community into an acceptable, common choice. By providing full support for Zstandard, we make it the most attractive of the compression options. (BTW the LZ4 filter in HDF5 does not work, so that's why CCR does not support LZ4.)

What I would like to tell the GFS team is that netCDF will soon support a better compression method, which will help meet their operational requirements. If not that, then NOAA can use CCR and Zstandard. However, Unidata can then expect to get many support requests from users trying to read NOAA data, and needing to have filters and CCR explained to them. ;-)

And if ESA goes with BLOSC, NASA with LZ4, and NOAA with Zstandard, then there will be more work for the netCDF-Java team, or else they will simply have to give up on being able to read all netCDF files.

If there is concern for splintering of the community, we should select a common, acceptable path forward.

@DennisHeimbigner the amount of work to give zstandard preferential treatment is trivial. We spend a lot more time discussing it than it would take to do it...