NetCDF needs better and faster compression, especially for HPC applications

[edwardhartnett]

This issue is for general discussion about improving the compression capability of netCDF. I will address specific ideas in their own issues.

Compression is a key feature of netCDF-4. Whenever I ask why netCDF-4 was selected, the answer is always because it's netCDF, with built in compression. That is, the API is already understood and all the read program are working, and it's easy to turn on compression and suddenly save half your disk space. (The NOAA GFS is compressing a 33 GB file down to 6 GB. Without compression, storing and moving the file is much more expensive.)

However, currently we only offer zlib compression, which is not the best or most commonly desired kind. The work of @DennisHeimbigner with filters offers the chance to expose other HDF5 compression filters.

And what about the classic formats? Is there not a way to turn on compression? As a simple case, the entire file can be piped through gzip and automatically piped through unzip. It's pretty easy to do that, but there are much better ways.

When and if we add more compression options, we must of course continue to honor the existing API and code that has been written to it. Currently we do:

nc_def_var(...)
nc_def_var_deflate(...)

So we can imagine other forms of compression getting their own nc_def_var_XXXX function, which would be called after the var is defined and before enddef.

[gsjaardema]

Re: other forms of compression "getting their own nc_def_var_XXXX function". Would it not be OK from an API status to instead add an nc_def_var_compress(compression_type, compression_value) or something similar such that a new compression type could be added without adding a new API function... Maybe even a nc_def_var_property(). There has to be some way that this can be supported without adding a one or more new API functions per compression type. There would still be the need to add some defines, but please try to limit the number of API functions.

[gsjaardema]

Re: compression for classic formats. I remember some work from about a decade ago where there were some patches or something to the NetCDF library which provided compression. I think it wasn't just compressing the resulting file, but instead messed with NetCDF internals to compress on a dataset by dataset basis. Will try to see if I can find anything referencing that work...

[WardF]

@gsjaardema The API @DennisHeimbigner wrote to support filters (nc_def_var_filter, etc) is generalizable to different filters. I'm on my phone so can't check the reference at the moment. I don't think we need to define a function for each compression individually.

[edwardhartnett]

I don't want to get into the idea of just passing a void pointer and it means different things. That's a bit much. So I believe we will have separate functions for separate compression methods. I'm willing to reuse as much as possible, but different methods have different numbers and types of parameters.

[WardF]

This may need to be a broader discussion before we go too far with the implementation, it’s important we maintain consistent in our approach. Are you suggesting a handful of specialized API calls for a closed set of new compression algorithms? Or are you suggesting an open ended “every new compression method requires new functions?” The latter seems like it could expand quickly, and is not in line with the current approach.

What do you mean by passing a void pointer means different things?

[sluehrs]

From my side I think the available nc_def_var_filter is already pretty powerful for the netcdf4 format. E.g. I did some tests using ZFP (https://computing.llnl.gov/projects/floating-point-compression) within a parallel program (in a brute force way, by removing the parallel checks from the netcdf4 code, which normally blocks parallel filter calls). This worked pretty fine, e.g.:

H5Pset_zfp_reversible_cdata(cd_nelmts, cd_values);
nc_def_var_filter(nc_file_id,nc_variable,H5Z_FILTER_ZFP,cd_nelmts,cd_values);

Of course you have to include some H5 plugin code here, to configrue the individual plugin.
Adding additional compression mechanics directly into netcdf also means a lot of additional dependencies. This HDF5 plugin mechanism already solved this problem. So I think at least for the netcdf4 format the proper parallel access to the filter function can already offer a lot of new netcdf HPC approaches, without a lot of additional overhead.

[edwardhartnett]

I believe having a different nc_def_var_XXXX() for each compression method is perfectly acceptable and also in accordance with what we already have.

The post of @sluehrs actually proves my point. It does not sound easy! If I am to describe how to turn on compression to NOAA programmers, I need to just tell them a function, like nc_def_var_deflate().

I am perfectly willing to have a function cover multiple compression methods, but since each method has it's own type and number of parameters, how will that work?

I welcome concrete proposals...

[gsjaardema]

I do not have a concrete proposal at this time. But, I know of several compression algorithms, both lossy and non-lossy, and the thought of adding support for all of these to NetCDF through a new API function seems non-tenable. Some of these may be niche solutions -- e.g. they work only for an unstructured mesh representation -- and may not be worthwhile to have them fully supported in every installation of NetCDF.

I haven't looked closely at the filter implementation, but maybe something similar to that is needed. I know that HDF5 has a somewhat extensible filter API which supports compression; maybe there is something there that can be adopted.

While you are looking at the compression algorithms, please look at BLOSC (https://github.com/Blosc/hdf5-blosc and https://blosc.org/pages/blosc-in-depth/) as it looks like it gives good performance.

I guess I don't have a good API suggestion, but I would like it to be easy to add support for (or at least test) a new experimental lossy or non-lossy compression algorithm without having to modify the API. lossy, problem-specific, compression algorithms are an active area of research so many are being proposed and it would be nice to be able to evaluate them on my application area as easily as possible.

I'm also not sure how the API function per compression algorithm approach works for optional support. NetCDF is currently able to support multiple optiona formats (CDF1, CDF2, ..., CDF5) in parallel and serial without much affect on the API; hopefully the same can be done for compression and filter support.

I also see many older algorithms going away or not being used very often anymore and don't want to see the NetCDF API in 5 or 10 years cluttered with a bunch of functions for compression algorithms that are no longer supported.

[edwardhartnett]

@gsjaardema you make some excellent points, and all of these must be kept in mind as we move forward.

To address one issue, which is how do compression methods interact with formats: right now, none of the classic formats can use zlib. When one (or more) can, then yes, I would expect we would use the same nc_def_var_deflate() function; it would then work for both netcdf-4 and whatever classic formats can handle it. In other words, the APIs are independent of format, though not supported in all formats.

I am thinking of less than 5 new compression methods, so I don't believe a general purpose solution is needed. Also recall that whatever we do must be usable from Fortran, which restricts how clever we can be with parameters. What works best, and matches the rest of the api, is a specific type-safe function wherever we can provide one, even if they are redundant (for example we could use nc_put_vara() for all array puts - but we also have a function for each type, so the user gets type-checking. And this way, everything looks very obvious in fortran codes.

Let us discuss the algorithms on a case by case basis. I agree we only want ones that will stand the test of time. I have put a bunch out for discussion, but that doesn't mean all of them will be added. I just put out all the ones I can think of.

Thanks for pointing out BLOSC, I will add it to the list.

We should also learn in detail what is being done by the netcdf4 python library. I believe it handles unpacking, for example. @dopplershift do you know about that?