netcdf built with hdf5 1.10.8 error after 'nc_def_var_fletcher32'

[abhibaruah]

NetCDF version: v4.8.1
OS: Linux 10
Compiler gcc 8.3.0

I am a developer at The MathWorks Inc. For MATLAB, we were building netcdf with hdf5-1.8.12.
However, we are trying to change netcdf's dependency on hdf5 to v1.10.8.

I built netcdf successfully with hdf5 1.10.8. However, while qualifying I started seeing some of our tests fail, which I have narrowed down to an issue with 'nc_def_var_fletcher'.

I am attaching a C repro code below, which works fine when netcdf 4.8.1 is built with hdf5 1.8.12.

However, when netcdf 4.8.1 is built with hdf5 1.10.8, 'nc_close' on line 35 gives the error code -101 (NC_EHDFERR)
The error does not occur if the datatype (xtype) in 'nc_def_var' is anything other than NC_STRING.
Also, the error does not occur if the call to 'nc_def_var_fletcher32' is removed.

Any help in resolving this would be highly appreciated.

#include <iostream>
#include <vector>
#include "netcdf.h"

void checkErrorCode(int status, const char* message){
    if (status != NC_NOERR){
        std::cout << "Error code: " << status << " from " << message << std::endl;
    }
}

int main(int argc, const char * argv[]) {
    
    int ncid;
    int retval;
    
    retval = nc_create("myfile.nc", NC_NETCDF4, &ncid);
    checkErrorCode(retval, "nc_create");
    
    // Define dimensions
    int dimid_a;
	int dimlen = 100;
    retval = nc_def_dim(ncid, "a", dimlen, &dimid_a);
    checkErrorCode(retval, "nc_def_dim for 'a'");
    
    // Define variable
    int varid;
	int dimids[1] = {dimid_a};
    retval = nc_def_var(ncid, "var", NC_STRING, 1, dimids, &varid);
    checkErrorCode(retval, "nc_def_var");
        
    
    retval = nc_def_var_fletcher32(ncid, varid, true);
    checkErrorCode(retval, "nc_def_var_fletcher32");

    retval = nc_close(ncid);
    checkErrorCode(retval, "nc_close");
    
    
    return retval;
}

[WardF]

Thanks; taking a look at it now, I will also see about getting 1.10.8 into our test matrix.

[WardF]

What platform are you observing this failure on?

[WardF]

Nevermind my question, I see it in the header of the issue you opened, sorry, thanks :)

[WardF]

Observing this issue on OSX as well.

[WardF]

In linux with debug and logging turned on, observing the following.

TLDR: H5Dcreate2 is throwing an error for some reason.

wfisher@ubuntudev-arm:~/Desktop/github-test$ ./test 
				log_level changed to 5
		HDF5_def_dim: ncid 0x10000 name a len 100
				nc4_find_dim: dimid 0
		NC4_def_var: name var type 12 ndims 1
				dimid[0] 0
				nc4_get_typelen_mem xtype: 12
		nc_inq_atomic_type: typeid 12
				nc4_type_new: size 8 name string assignedid 12
				nc4_find_dim: dimid 0
				allocating array of 1 size_t to hold chunksizes for var var
				nc4_find_default_chunksizes2: name var dim 0 DEFAULT_CHUNK_SIZE 4194304 num_values 100.000000 type_size 8 chunksize 100
				total_chunk_size 800.000000
				nc4_get_typelen_mem xtype: 12
				new varid 0
		nc_def_var_extra: ncid 0x10000 varid 0
			extra: default: chunksizes for var varsizes=100
	NC4_close: ncid 0x10000
			nc4_close_hdf5_file: h5->path myfile.nc abort 0
			sync_netcdf4_file
		*** NetCDF-4 Internal Metadata: int_ncid 0x0 ext_ncid 0x10000
		FILE - path: myfile.nc cmode: 0x11009 parallel: 0 redef: 0 fill_mode: 0 no_write: 0 next_nc_grpid: 1
		 GROUP - / nc_grpid: 0 nvars: 1 natts: 0
		 DIMENSION - dimid: 0 name: a len: 100 unlimited: 0
		 VARIABLE - varid: 0 name: var ndims: 1 dimids: 0 storage: chunked
			nc4_rec_write_groups_types: grp->hdr.name /
			nc4_rec_write_metadata: grp->hdr.name /, bad_coord_order 0
				nc4_create_dim_wo_var: creating dim a
				nc4_create_dim_wo_var: about to H5Dcreate1 a dimscale dataset a
				write_netcdf4_dimid: writing secret dimid 0
				write_var: writing var var
			var_create_dataset:: name var
				var_create_dataset: about to H5Dcreate2 dataset var of type 0x13e
ERROR: file nc4hdf.c, line 988.
NetCDF: HDF error
Error -101: nc_close was received

[WardF]

For comparison, the following succeeds (netCDF 4.8.1, HDF5 1.8.21)

wfisher@ubuntudev-arm:~/Desktop/github-test$ ./test 
				log_level changed to 5
		HDF5_def_dim: ncid 0x10000 name a len 100
				nc4_find_dim: dimid 0
		NC4_def_var: name var type 12 ndims 1
				dimid[0] 0
				nc4_get_typelen_mem xtype: 12
		nc_inq_atomic_type: typeid 12
				nc4_type_new: size 8 name string assignedid 12
				nc4_find_dim: dimid 0
				allocating array of 1 size_t to hold chunksizes for var var
				nc4_find_default_chunksizes2: name var dim 0 DEFAULT_CHUNK_SIZE 4194304 num_values 100.000000 type_size 8 chunksize 100
				total_chunk_size 800.000000
				nc4_get_typelen_mem xtype: 12
				new varid 0
		nc_def_var_extra: ncid 0x10000 varid 0
			extra: default: chunksizes for var varsizes=100
	NC4_close: ncid 0x10000
			nc4_close_hdf5_file: h5->path myfile.nc abort 0
			sync_netcdf4_file
		*** NetCDF-4 Internal Metadata: int_ncid 0x0 ext_ncid 0x10000
		FILE - path: myfile.nc cmode: 0x11009 parallel: 0 redef: 0 fill_mode: 0 no_write: 0 next_nc_grpid: 1
		 GROUP - / nc_grpid: 0 nvars: 1 natts: 0
		 DIMENSION - dimid: 0 name: a len: 100 unlimited: 0
		 VARIABLE - varid: 0 name: var ndims: 1 dimids: 0 storage: chunked
			nc4_rec_write_groups_types: grp->hdr.name /
			nc4_rec_write_metadata: grp->hdr.name /, bad_coord_order 0
				nc4_create_dim_wo_var: creating dim a
				nc4_create_dim_wo_var: about to H5Dcreate1 a dimscale dataset a
				write_netcdf4_dimid: writing secret dimid 0
				write_var: writing var var
			var_create_dataset:: name var
				var_create_dataset: about to H5Dcreate2 dataset var of type 0x3000060
		attach_dimscales: attaching scale for dimid 0 to var var
					NC4_write_ncproperties
			nc4_rec_grp_HDF5_del: grp->name /
			closing HDF5 dataset 83886081
				nc4_rec_grp_HDF5_del: closing group /
			nc4_close_netcdf4_file: h5->path myfile.nc abort 0
					NC4_clear_provenance
			nc4_rec_grp_del: grp->name /
				var_free: deleting var var
				nc4_type_free: deleting type string
				dim_free: deleting dim a

[WardF]

Note to self: Potential resource to follow up on:

• https://forum.hdfgroup.org/t/fletcher32-filter-on-variable-length-string-datasets-not-suitable-for-filters/9038

[WardF]

Note to self, I've narrowed this down as a regression between HDF5 1.10.3 (works) and 1.10.7 (fails). It is unclear to me as of yet whether or not fletcher32 on a variable-length datatype variable is valid in the HDF5 API. I'll investigate this further tomorrow.

[WardF]

Confirmation that HDF5 1.10.6 appears to work, I'll have to dig into the release notes.

[WardF]

From the 1.10.7 release notes, I see the following note. It is confusing since it seems to indicate, at a glance, the reverse of what we are observing. It will merit further review.

   - Creation of dataset with optional filter

      When the combination of type, space, etc doesn't work for filter
      and the filter is optional, it was supposed to be skipped but it was
      not skipped and the creation failed.

      A fix is applied to allow the creation of a dataset in such
      situation, as specified in the user documentation.

      (BMR - 2020/8/13, HDFFV-10933)

[WardF]

I'm still taking a look into why this is happening, but the bottom line might be that it might be beyond our control in this circumstance. It should not, however, fail silently. We'll have to put some thought into how we will best handle this at the netcdf layer.

[DennisHeimbigner]

We could put in a check in nc_def_var_filter so that if one attempts to add a filter to
a string or vlen typed variable, the call to nc_def_var_filter will fail.

[WardF]

@DennisHeimbigner Agreed. I'm uncertain (and can't switch gears at the moment to think about it) whether we want to remove the filter definition when the combination doesn't work and silently proceed, or whether we want to fail with an informative message. The question is whether we want to break a workflow with a message that explains why, or do we want to simply retain previous behavior (one layer up).