HDF Error when attempting to create variable defined with compound under some conditions

[smndvgnc]

import netCDF4
import numpy as np


filt_shape = (8192)
n_filt = 1

with netCDF4.Dataset("my_netcdf.nc", "w") as ds:
    filter_defn = np.dtype(
        [
            ("filter", np.float64, filt_shape),
        ]
    )

    filter_type = ds.createCompoundType(filter_defn, "filters")

    index_dim = ds.createDimension("index", n_filt)

    v = ds.createVariable("filter_list", filter_type, index_dim)

The above code does not work and crashes with the following error:

Traceback (most recent call last):
  File "/Users/smndvgnc/Downloads/test_nc.py", line 8, in <module>
    with netCDF4.Dataset("my_netcdf.nc", "w") as ds:
  File "src/netCDF4/_netCDF4.pyx", line 2498, in netCDF4._netCDF4.Dataset.__exit__
  File "src/netCDF4/_netCDF4.pyx", line 2622, in netCDF4._netCDF4.Dataset.close
  File "src/netCDF4/_netCDF4.pyx", line 2585, in netCDF4._netCDF4.Dataset._close
  File "src/netCDF4/_netCDF4.pyx", line 2029, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

If you change filt_shape to (8191), the problem vanishes. This issue appeared with version 1.6.2. Version 1.6.1 does not produce error using the code snippet.

To spare you some calculations:
8191 elements in double precision: 65528 bytes
8192 elements in double precision: 65536 bytes (=2**16).

A size above 8192 also makes it crash.

[jswhit]

Nothing has changed in the python interface since 1.6.1 that would impact this, so I suspect it's a C lib issue. Can you check to see what versions of HDF5 and netcdf-c you are using with 1.6.1 and 1.6.2? You can do this by running

import netCDF4, numpy
print('netcdf4-python version: %s'%netCDF4.__version__)
print('HDF5 lib version:       %s'%netCDF4.__hdf5libversion__)
print('netcdf lib version:     %s'%netCDF4.__netcdf4libversion__)
print('numpy version           %s' % numpy.__version__)

Are you installing using the wheels from pypi?

[jswhit]

I have confirmed that this error occurs with both 1.6.1 and current master with hdf5 1.14.0 and netcdf-c 4.9.2.

[jswhit]

but the error does not occur with the current master using netcdf-c 4.8.1 and hdf5 1.12.1

[smndvgnc]

With mamba/conda.

So, with mamba install "netcdf4=1.6.1", I get:

>>> print('netcdf4-python version: %s'%netCDF4.__version__)
netcdf4-python version: 1.6.1
>>> print('HDF5 lib version:       %s'%netCDF4.__hdf5libversion__)
HDF5 lib version:       1.12.2
>>> print('netcdf lib version:     %s'%netCDF4.__netcdf4libversion__)
netcdf lib version:     4.8.1
>>> print('numpy version           %s' % numpy.__version__)
numpy version           1.25.2

And with mamba install "netcdf4=1.6.2", I get:

>>> print('netcdf4-python version: %s'%netCDF4.__version__)
netcdf4-python version: 1.6.2
>>> print('HDF5 lib version:       %s'%netCDF4.__hdf5libversion__)
HDF5 lib version:       1.12.2
>>> print('netcdf lib version:     %s'%netCDF4.__netcdf4libversion__)
netcdf lib version:     4.9.1
>>> print('numpy version           %s' % numpy.__version__)
numpy version           1.25.2

[jswhit]

a few more data points... (all with latest netcdf4-python master)

netcdf-c 4.9.2, hdf5 1.14.0 FAILS
netcdf-c 4.9.1, hdf5 1.14.0 FAILS
netcdf-c 4.9.0, hdf5 1.12.2 FAILS

but

netcdf-c 4.8.1, hdf5 1.12.2 WORKS

so it looks like the problem started with netcdf-c 4.9.0

[jswhit]

next step will be to create a c example program that triggers the crash and post it to netcdf-c

[jswhit]

@DennisHeimbigner do you know if there were any changes in netcdf-c 4.9.0 that would have affected creation of compound type elements with lengths greater than 2**16 -1 bytes?

[DennisHeimbigner]

Nothing comes to mind. I suppose it could have an unforeseen side-effect of some seemingly unrelated change.

[smndvgnc]

In C:

#include <stdio.h>
#include <netcdf.h>

#define FILE_NAME "output.nc"
#define FILT_SIZE 8192

typedef struct Filter{
    double values[FILT_SIZE];
} Filter;

int main() {
    int ncid; // NetCDF file ID
    int dimid; // Dimension ID
    int varid; // Variable ID
    Filter filters;

    // Create NetCDF file
    if (nc_create(FILE_NAME, NC_NETCDF4, &ncid) != NC_NOERR) {
        fprintf(stderr, "Error creating NetCDF file.\n");
        return 1;
    }

    // Define a dimension
    if (nc_def_dim(ncid, "dim", 1, &dimid) != NC_NOERR) {
        fprintf(stderr, "Error defining dimension.\n");
        return 1;
    }

    // Define a compound type
    nc_type compound_type;
    if (nc_def_compound(ncid, sizeof(Filter), "Filter", &compound_type) != NC_NOERR) {
        fprintf(stderr, "Error defining compound type.\n");
        return 1;
    }
    int nelts = FILT_SIZE;
    if (nc_insert_array_compound(ncid, compound_type, "values", NC_COMPOUND_OFFSET(Filter, values), NC_DOUBLE, 1, &nelts) != NC_NOERR) {
        fprintf(stderr, "Error inserting member into compound type.\n");
        return 1;
    }

    // Define the variable using the compound type
    if (nc_def_var(ncid, "variable", compound_type, 1, &dimid, &varid) != NC_NOERR) {
        fprintf(stderr, "Error defining variable.\n");
        return 1;
    }

    // End the definition phase
    if (nc_enddef(ncid) != NC_NOERR) {
        fprintf(stderr, "Error ending definition phase.\n");
        return 1;
    }

    // Initialize and populate the data array
    for (unsigned int i = 0; i < FILT_SIZE; ++i) {
        filters.values[i] = 0.;
    }

    // Write the data
    if (nc_put_var(ncid, varid, &filters) != NC_NOERR) {
        fprintf(stderr, "Error writing data.\n");
        return 1;
    }

    // Close the NetCDF file
    if (nc_close(ncid) != NC_NOERR) {
        fprintf(stderr, "Error closing NetCDF file.\n");
        return 1;
    }

    printf("NetCDF file created successfully.\n");
    return 0;
}

Works with libnetcdf in version 4.8.1 but fails with version 4.9.0. And works with version 4.9.0 with a size below or equal to 8191.

S.

P.S.: I'll post this to netcdf-c tomorrow.

[jswhit]

@smndvgnc thanks for doing this! Small point - you don't actually need the nc_endef in the C code above (that's only for the old classic format)

[smndvgnc]

Thanks, I removed that part. I posted the issue to netcdf-c: here.