Provide a way to do bit grooming before compression

[edwardhartnett]

In the NOAA GFS data files they are doing something neat to get better compression. They are zeroing out a bunch of bits they don't care about in the float.

For this to work, all we need to know is the number of bits. So we could have:

nc_def_var_smoothing(ncid, varid, nbits);

to turn this on.

For some code from Jeff Whitaker see #1543. (They call this "quantizing" and maybe that's a better name. I'm not sure if "smoothing" is correct.)

This is part of #1545

[czender]

I support adding lossy compression to netCDF. However, bit smoothing is sub-optimal because it always rounds down. Bit Grooming alternately rounds up (to 1s) and down (to 0s) which reduces the statistical biases of rounding by orders of magnitude. See the comparisons in:

Zender, C. S. (2016), Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+), Geosci. Model Dev., 9, 3199-3211, doi:10.5194/gmd-9-3199-2016.

[edwardhartnett]

@czender excellent point! And great reference.

I propose to add this to netcdf-c after the agu. I am thinking of the following functions:

nc_def_var_quantize(ncid, varid, nbits)
nc_inq_var_quantize(ncid, varid, &nbits_in)

These would only apply to NC_DOUBLE and NC_FLOAT, calling with any other type will return an error.

nc_def_var_quantize() will add an attribute _netcdf_nbits=12 for example.

Some things to note:

• The savings in file size can be enormous.
• NOAA GFS is already doing this; other model teams also are or will be. This will be popular.
• This is fully backward compatible (as not all our new compression features will be).

[czender]

You are welcome to use the NCO code lines 576--860 (not all are necessary) of https://github.com/nco/nco/tree/master/src/nco/nco_ppc.c

[edwardhartnett]

@czender I was already planning to steal your code. ;-)

(With full attribution of course. We're classy that way on the netCDF project. We steal your code, but we do say "thank you"). ;-)

[WardF]

Tagging in @lesserwhirls and @DennisHeimbigner to this discussion, so that we can consider the potential impact on NetCDF java as well. Adding the FORTRAN and C++ interfaces shouldn’t be a problem.

[edwardhartnett]

This will have no impact on Java, because the files can be read in the normal way.

If netCDF-Java would also like to add this feature, that is of course possible. I would not hold back the rest of the netCDF community while waiting for Java.

I will add these calls to the Fortran APIs as well; just forgot to mention it, but of course we want both the F77 and F90 APIs to fully reflect the C API.

[edwardhartnett]

Also would welcome input from @jswhit and @gsjaardema on this issue...

[lesserwhirls]

I assume we’re talking about something like this?

https://www.unidata.ucar.edu/blogs/developer/en/entry/compression_by_bit_shaving

We don’t have a way of doing this (writing) via the netCDF-Java API currently, but would be easy to add (have code in our tests). Could also tie in to whatever the C library ends up doing since we use the C library for writing. Reading won’t be an issue.

One question though - what would a variable look like? How would nbits be encoded? Would it be exposed as an attribute (so a need to update the NUG)?

No technical concerns from a netCDF-Java point of view, but thank you @WardF for even asking the question. Surprises are never fun :-)

[edwardhartnett]

@lesserwhirls this is part of a larger discussion about netcdf-c and compression. See #1545

I was going to ask if netcdf-java is now using the C library. You say for writing only? Why is that? Why not read as well as write?

[WardF]

@lesserwhirls no problem, I don't want to (potentially) add to your roadmap/workload without at least waving in your direction first ;).

No particular hurry on this; when lossy compression has come up in meetings or at conferences, the scientists have expressed their... hesitancy to truncate any of their data, even when it objectively contains no information. The lack of people asking for it makes it a relatively low priority internally, but adding support through a PR contribution is effort I can justify ;). Anyways, back to prepping for AGU. Thanks all!

[lesserwhirls]

@lesserwhirls this is part of a larger discussion about netcdf-c and compression. See #1545

I was going to ask if netcdf-java is now using the C library. You say for writing only? Why is that? Why not read as well as write?

Note: this was all done before my time, but here is my take.

Writing because, I think, the HDF documentation was too unclear regarding how writing is to be done (for example, chunking, VLEN, I think?) to support all of the features needed to write netCDF-4. We'd need to rope in John Caron for the details.

In terms of reading, we do support reading through the C library, but it's not threads safe (we have to thread lock on the java side to ensure we're ok there). That's a pretty big bummer. We've enabled reading through C on the server side (TDS), and performance of the server was...well, we don't do that, and never recommend for users it outside of super rare debugging cases (once). John wrote native HDF5 read support, so he utilized that to support reading netCDF-4. We do run into some issues in making sure the Java library and the C library see things in a similar way (Enums, for example), but that's helpful for everyone as it exposes areas in need of clarification in the netCDF data model documentation (not always, but sometimes).

Also (my opinion), it's really hard to recommend an on disk data format for archive purposes that relies on a single implementation for reading. That's one of the reasons why I loved the netCDF-3 on disk formats before my time at Undiata.

[edwardhartnett]

@lesserwhirls thanks for the clarifications. Thread-safety is something Denis has been working on I believe.

The classic formats are all still alive and well, for those who don't want to depend on HDF5. That said, HDF5 is an official NASA format that is guaranteed support based on the large amount of very expensive Earth science data NASA owns in HDF5. ;-)

[lesserwhirls]

I don't doubt the support one bit, and my comments should not be taken as a ding (just in case it came across that way), but knowing that a separate read implementation can be done is certainly a plus :-)