Cannot create nested compound types when creating netCDF file in memory

[nmassey001]

With reference to the Python version of this issue: [https://github.com/Unidata/netcdf4-python/issues/970], I've now recreated the problem in C.

I've run into a problem when using the relatively new functionality of creating a netCDF4 file in memory, then writing the memory to a file on closure of the netCDF file.
It seems that it's not possible to use nested compound types - i.e. a compound type that contains another compound type.

The code below illustrates the problem.

When using the method below an invalid netCDF file is created. Using ncdump on the file results in the error:
ncdump: test_raw_nc.nc: test_raw_nc.nc: NetCDF: HDF error

#include <stdio.h>
#include <netcdf.h>
#include <netcdf_mem.h>

typedef struct Subarray {
    char* ncvar;
    char* file;
    char* format;
    int shape;
} Subarray;

typedef struct Partition {
    int index;
    int location;
    struct Subarray subarray;
} Partition;

void check_err(const int stat, const int line, const char *file)
{
    if (stat != NC_NOERR) {
        (void)fprintf(stderr,"line %d of %s: %s\n", line, file, nc_strerror(stat));
        fflush(stderr);
        exit(1);
    }
}

int main()
{
    const char* file_name = "./test_nc4_compound_types_fails.nc";
    int ncid, retval;
    int Subarray_typid, Partition_typid;
    size_t initialsize = 0;

    // create the netCDF4 file in memory
    retval = nc_create_mem("test", NC_NETCDF4, initialsize, &ncid);
    check_err(retval, __LINE__,__FILE__);

    // define the first compound type: Subarray
    retval = nc_def_compound(ncid, sizeof(Subarray),
                             "Subarray", &Subarray_typid);
    check_err(retval, __LINE__,__FILE__);

    // add the struct members to the Subarray compound type
    retval = nc_insert_compound(ncid, Subarray_typid, "ncvar",
                                NC_COMPOUND_OFFSET(Subarray, ncvar),
                                NC_STRING);
    check_err(retval, __LINE__,__FILE__);

    retval = nc_insert_compound(ncid, Subarray_typid, "file",
                                NC_COMPOUND_OFFSET(Subarray, file),
                                NC_STRING);
    check_err(retval, __LINE__,__FILE__);

    retval = nc_insert_compound(ncid, Subarray_typid, "format",
                                NC_COMPOUND_OFFSET(Subarray, format),
                                NC_STRING);
    check_err(retval, __LINE__,__FILE__);

    retval = nc_insert_compound(ncid, Subarray_typid, "shape",
                                NC_COMPOUND_OFFSET(Subarray, shape),
                                NC_INT);
    check_err(retval, __LINE__,__FILE__);

    // define the second compound type: Partition
    if ((retval = nc_def_compound(ncid, sizeof(Partition),
                                 "Partition", &Partition_typid)))
    check_err(retval, __LINE__,__FILE__);

    // add the struct members to the Partition compound type
    retval = nc_insert_compound(ncid, Partition_typid, "index",
                                NC_COMPOUND_OFFSET(Partition, index),
                                NC_INT);
    check_err(retval, __LINE__,__FILE__);

    retval = nc_insert_compound(ncid, Partition_typid, "location",
                                NC_COMPOUND_OFFSET(Partition, location),
                                NC_INT);
    check_err(retval, __LINE__,__FILE__);

    retval = nc_insert_compound(ncid, Partition_typid, "subarray",
                                NC_COMPOUND_OFFSET(Partition, subarray),
                                Subarray_typid);
    check_err(retval, __LINE__,__FILE__);

    // close the netCDF file in memory and get a pointer to the memory block used
    NC_memio file_memio;
    retval = nc_close_memio(ncid, &file_memio);
    check_err(retval, __LINE__,__FILE__);

    // open the file to write out
    FILE* fptr = fopen(file_name, "wb");
    fwrite(file_memio.memory, 1, file_memio.size, fptr);
    // close the file
    fclose(fptr);
}

when using the below code, which does not have nested compound types, a valid netCDF file is created.

#include <stdlib.h>
#include <stdio.h>
#include <netcdf.h>
#include <netcdf_mem.h>

typedef struct Subarray {
    char* ncvar;
    char* file;
    char* format;
    int shape;
} Subarray;

void check_err(const int stat, const int line, const char *file)
{
    if (stat != NC_NOERR) {
        (void)fprintf(stderr,"line %d of %s: %s\n", line, file, nc_strerror(stat));
        fflush(stderr);
        exit(1);
    }
}

int main()
{
    const char* file_name = "./test_nc4_compound_types_works.nc";
    int ncid, retval;
    int Subarray_typid;
    size_t initialsize = 0;

    // create the netCDF4 file in memory
    retval = nc_create_mem("test", NC_NETCDF4, initialsize, &ncid);
    check_err(retval, __LINE__,__FILE__);

    // define the first compound type: Subarray
    retval = nc_def_compound(ncid, sizeof(Subarray),
                             "Subarray", &Subarray_typid);
    check_err(retval, __LINE__,__FILE__);

    // add the struct members to the Subarray compound type
    retval = nc_insert_compound(ncid, Subarray_typid, "ncvar",
                                NC_COMPOUND_OFFSET(Subarray, ncvar),
                                NC_STRING);
    check_err(retval, __LINE__,__FILE__);

    retval = nc_insert_compound(ncid, Subarray_typid, "file",
                                NC_COMPOUND_OFFSET(Subarray, file),
                                NC_STRING);
    check_err(retval, __LINE__,__FILE__);

    retval = nc_insert_compound(ncid, Subarray_typid, "format",
                                NC_COMPOUND_OFFSET(Subarray, format),
                                NC_STRING);
    check_err(retval, __LINE__,__FILE__);

    retval = nc_insert_compound(ncid, Subarray_typid, "shape",
                                NC_COMPOUND_OFFSET(Subarray, shape),
                                NC_INT);
    check_err(retval, __LINE__,__FILE__);

    // close the netCDF file in memory and get a pointer to the memory block used
    NC_memio file_memio;
    retval = nc_close_memio(ncid, &file_memio);
    check_err(retval, __LINE__,__FILE__);

    // open the file to write out
    FILE* fptr = fopen(file_name, "wb");
    fwrite(file_memio.memory, 1, file_memio.size, fptr);
    // close the file
    fclose(fptr);
}

[WardF]

Tagging @DennisHeimbigner in case he has any immediate thoughts. We will take a look at this.

[DennisHeimbigner]

This is surprising. My immediate speculation is that it is related
to the layout of structs. and the relative field offsets.

[nmassey001]

Thanks @DennisHeimbigner @WardF.
I should have said that the first code example works fine if the file is written straight to disk, rather than going through the memory. However, my use case is streaming files from / to memory to / from object store over S3, without every touching the disk.

[DennisHeimbigner]

I did some experiments;

1. use NC_DISKLESS - works
2. change the char* field types to e.g. int - fails
3. use h5dump to view the file - works
   This is going to take a while.

[edwardhartnett]

A through refactor of all the nested type code has long been on my wish-list. I really just did not get deep nesting of types correctly. Oh well, even the great Homer nods.

However, it's not a trivial undertaking.

I am very curious as to the use-case here. What are these data? Why are we using nested compounds for them?

[nmassey001]

@edwardhartnett The example I gave is a trivialisation of the actual data structures used and, even with the current full data structures, I can flatten them so that I don't need to use nested compounds.
However, the bug still exists and could be a problem with my code in the future, when I expand the datatypes to have (for example) an array of a compound type in a compound type.

[DennisHeimbigner]

Ed is right about the refactoring of nested types. But I suspect that since
this works with files and DISKLESS, that the problem is in my memory code
in this case.

[DennisHeimbigner]

I am working now on this issue. I have a question.
Do you ever use the ability to over-allocate the incoming buffer memory
plus nc_open_mem
or do you always use the nc_create_mem function?

[nmassey001]

Hi Dennis. Thanks for looking at this.
I mainly interact with the netCDF library via the Python API, which uses both nc_create_mem and nc_open_mem.