Allow users to turn on szip compression for variables in netCDF-4 files

[edwardhartnett]

HDF5 offers native access to zlib and szlib compression. However, at the time I wrote netCDF-4, szlib had an unclear license. So Russ and I decided to make it a read-only capability in netCDF-4. That is, you can read HDF5 files written with szlib, but you could not create them.

(Probably we were being too cautious anyway. HDF5 uses it and no one cares.)

The szip compression library is beloved by NASA Goddard. It does better then zlib with arrays of floating point data.

Anyway, Elena has explained that they have changed their license and are now fully free software. So we can allow users to turn this form of compression on, pretty easily. (I actually used to have a function to do this, but I took it out when we got concerned over licensing.)

This is part of #1545

[ArchangeGabriel]

Interesting, didn’t now that they turned to free software licensing (can you give a link for this though, I still read “free for non-commercial use”?). We were relying on libaec to provide szip support on Arch, maybe we should re-evaluate that then.

[epourmal]

There is SZIP implementation from German Climate Computing Center that has BSD type license. It is fully compatible with SZIP supported by The HDF Group. You can get the source from here.

[ArchangeGabriel]

@epourmal Yeah, just as I implied above, this was the de-facto replacement when SZIP was non-free. But if they are now free, re-evaluating SZIP vs libaec might be interesting.

[edwardhartnett]

So here's the question: how do we handle it when people do not have libaec installed? Some alternatives include:
1 - Fail configure and demand they install it or build with --disable-netcdf-4.
2 - Build, but only include nc_def_var_szip() if libaec is installed?
3 - Include libaec in the builds (both autoconf and cmake), and build and install it if not already present.

3 seems safest - all users are guaranteed to be able to read files compressed with libaec.

[epourmal]

I agree, 3 is the safest. You can also include the source of szip with netCDF-4 source code.

[epourmal]

Forgot to mention that in HDF5 szip is a required filter, i.e., H5Dwrite will fail if the filter is not found for some reason. Do you build HDF5 libraries when you build netCDF-4 or some preinstalled binaries are used (probably both)? You will need to check that HDF5 was built with szip.

[edwardhartnett]

OK, so szip cannot be installed after HDF5 is built, however, other filters can?

To answer your question, we use an installed HDF5 library. We don't know if the user just built it, or installed it with apt-get, or what. We know nothing except what we can check. ;-)

[dopplershift]

Does libaec build on windows at all? Conda-forge has windows builds for libaec disabled. I have severe reservations of making this a major feature in netCDF if a major client platform won't be able to use it.

[dopplershift]

Actually, I'm going to take that a step further: I have HUGE problems with making it possible to use the netcdf-c library to write files that you can't open on Windows.

[edwardhartnett]

@dopplershift this horse has already left the barn. ;-)

With the recent addition of filters, netcdf-c can use any HDF5 filter, which means users can already write netcdf files with any filter. So it is quite possible to write netcdf-4 files which cannot be used on Windows, if there are filters that can't build on windows (and there may be).

I agree that full windows compatibility is a requirement for the compression features I am adding to netcdf-c. I will add that all compression features must be available in the F77 and F90 APIs as well. The Fortran APIs are how all modeling and climate groups use netcdf to write the giant data sets that are most concerned with compression.

My goal in the current effort is to add compression that can be assured on all platforms. That is, I want to not just use some filters, but actually ensure they are packaged with the netCDF distributions (automake and cmake) so that all users are known to have them. (If we find them already installed, we do nothing. If missing, we install them when netCDF is installed.)

This includes Windows. So we should select compression filters that work well, that can build under autotools and cmake, and that work on Windows. Any compression filters that don't meet those criteria will probably not make the cut for what I am trying to accomplish with this compression work.

But not netcdf-java. I rather doubt that all the filters exist in Java. So netcdf-java may have to fall back on using the C library for reading, as well as writing HDF5 some files. This is inevitable in any case. Eventually we will start using some new HDF5 features which John Caron did not know about, and so did not code into netcdf-java.

However, we must also not let the widespread penetration of netcdf hold us back from improving it.

As with the releases of netcdf-4.0 and netcdf-3.0, we must honor our commitments for full and complete backward compatibility in API and data formats. All existing netCDF code should continue to work, and all existing netCDF files continue to be fully readable by all future versions of the library.

[dopplershift]

With the recent addition of filters, netcdf-c can use any HDF5 filter, which means users can already write netcdf files with any filter.

I'm aware of this. I'm pushing back against adding more APIs to the C library that simplify the process of creating such files and exacerbates this problem. Weren't you just arguing how hard it is to use the filter API so that few people do it and that's why we need the simple API?

However, we must also not let the widespread penetration of netcdf hold us back from improving it.

That's an easy statement to make when you're not the one fielding support requests over on the netcdf-java repo. I'm all for making improvements to better serve our community. But such decisions need to be made in regards to what's good for the netCDF data model, file format(s), and above all our community. netcdf-c is but one implementation that Unidata maintains, and additions and changes to on-disk format need to be made considering the entire portfolio of netCDF implementations. Confusion when one version of the library can read a netCDF file and one can't is NOT good for the community.

So netcdf-java may have to fall back on using the C library for reading

Again, easy to say since you're not responsible for handling the support requests for clients of netcdf-java and having to ship Java packages that now would REQUIRE shipping multiple platforms' worth of compiled libraries. It's one thing to put some of that burden on people running the TDS (and even then just to write a certain format)--it's a whole other when that means we need to worry about bundling this with every copy of IDV, Panoply, or any other user of the netCDF-java library.

I love aspirational ideas and we should be considering every transformative change we can make to better serve the needs of the community. I think you have this covered really well right now @edwardhartnett; so I'm going to be here to represent reality. And the reality is, we support both netCDF-java and netCDF-c and we need to consider both, in terms of support risk and technical effort, when we're talking about new features that impact how data files are written and what clients will be able to read them. We're already doing that with the zarr work, and work on compression should be no different.

[DennisHeimbigner]

At least twice now, we (Ward and I) have rejected the idea
of netcdf-c including a bunch of additional filter implementations
because of security and maintenance issues. I for one
still stand by that position.

One partial approach would be to have a new include file,
netcdf_known_filters.h say. This file would contain wrappers
for nc_def_var_filter for specific filter s -- bzip, etc.
These wrappers would accept filter specific arguments and translate them
to proper form for nc_def_var_filter.
This would allow users to use a given filter with more semantically
specific parameters.
Notes:

1. this file would not be part of netcdf.h, so we do not pollute
   the core API.
2. Users could propose additions to this set of wrappers.
3. Presence of a wrapper in netcdf_known_filters.h would not
   mean that the implementation of that filter is available.
   The user is still responsible for finding and installing an
   implementation.
   None of this does addresses the problem of having files
   that cannot be completely read because it uses missing filters.
   We could build a program that takes a .nc file and prints
   out the filter ids of any filters used by that file (and which variables use them).
   Remember that until libaec, we had this problem already
   with the szip filter because it was supported, but no one had
   a non-proprietary implementation.

[edwardhartnett]

Well this has been a very interesting discussion!

@dopplershift do you object to Dennis' proposal?

@DennisHeimbigner would you be willing to allow me to extend this idea to the Fortran APIs?

I just participated in a NOAA telecom. Zlib is not meeting our operational constraints and we are now looking at other filters, starting with libaec/szlib. Speed of reading is a concern, because zlib is slow to uncompress data. Parallel I/O helps with that on HPC, but many programs run sequentially, or on non-HPC systems.