Error/Crash on nc_close when writing a "nested" NC_VLEN attribute

[krisfed]

I am running into problems when I try to write a "nested" NC_VLEN attribute (meaning there are two NC_VLEN types, and base type for one of them is the other NC_VLEN type). I seem to be able to write the same data as a variable without the same issues.

Here are simplistic reproduction steps:

#include <iostream>
#include "netcdf.h"

void checkErrorCode(int status, const char* message){
    if (status != NC_NOERR){
        std::cout << "Error code: " << status << " from " << message << std::endl;
        std::cout << nc_strerror(status) << std::endl << std::endl;
    }
}

int main(int argc, const char * argv[]) {
    
    // Setup data
    const int DIM_LEN = 2;
    nc_vlen_t data[DIM_LEN];
    
    // first element in nested vlen
    const int first_nest_size = 2;
    nc_vlen_t first_nest[first_nest_size];

    const int first_size1 = 10;
    double first1[first_size1] = {2, 3, 5, 7, 11, 13, 17, 37, 79, 113};
    first_nest[0].p = first1;
    first_nest[0].len = first_size1;
    
    const int second_size1 = 5;
    double second1[second_size1] = {101, 121, 131, 141, 151};
    first_nest[1].p = second1;
    first_nest[1].len = second_size1;
    
    data[0].p = first_nest;
    data[0].len = first_nest_size;
    
    // second element in nested vlen
    const int second_nest_size = 1;
    nc_vlen_t second_nest[second_nest_size];

    const int first_size2 = 2;
    double first2[first_size2] = {55, 56};
    second_nest[0].p = first2;
    second_nest[0].len = first_size2;
    
    data[1].p = second_nest;
    data[1].len = second_nest_size;

    // Open file
    int ncid;
    int retval;
    
    retval = nc_create("nested_vlen_att.nc", NC_NETCDF4, &ncid);
    checkErrorCode(retval, "nc_create");
    
    // Define vlen type named MY_VLEN
    nc_type vlen_typeID;
    retval = nc_def_vlen(ncid, "MY_VLEN", NC_DOUBLE, &vlen_typeID);
    checkErrorCode(retval, "nc_def_vlen");
    
    // Define nested vlen type
    nc_type vlen_nested_typeID;
    retval = nc_def_vlen(ncid, "MY_NESTED_VLEN", vlen_typeID, &vlen_nested_typeID);
    checkErrorCode(retval, "nc_def_vlen (nested)");
    
    // Write vlen attribute
    retval = nc_put_att(ncid, NC_GLOBAL, "numbers_att", vlen_nested_typeID, DIM_LEN, data);
    checkErrorCode(retval, "nc_put_att");
    
//    // Write vlen variable
//    int dimid;
//    retval = nc_def_dim(ncid, "DIM", DIM_LEN, &dimid);
//    checkErrorCode(retval, "nc_def_dimid");
//    int varid;
//    retval = nc_def_var(ncid, "numbers_var", vlen_nested_typeID, 1, &dimid, &varid);
//    checkErrorCode(retval, "nc_def_var");
//    retval = nc_put_var(ncid, varid, data);
//    checkErrorCode(retval, "nc_put_var");
    
    retval = nc_close(ncid);
    checkErrorCode(retval, "nc_close");
    
    return retval;
}

Using netcdf-c version 4.7.4 on macOS 11.2.3 I sometimes get an error and sometimes a segfault:

$ ./a.out 
a.out(5539,0x101457e00) malloc: can't allocate region
:*** mach_vm_map(size=8893249870866587648, flags: 100) failed (error code=3)
a.out(5539,0x101457e00) malloc: *** set a breakpoint in malloc_error_break to debug
a.out(5539,0x101457e00) malloc: can't allocate region
:*** mach_vm_map(size=8893249870866587648, flags: 100) failed (error code=3)
a.out(5539,0x101457e00) malloc: *** set a breakpoint in malloc_error_break to debug
Error code: -107 from nc_close
NetCDF: Can't open HDF5 attribute

$ ./a.out 
Segmentation fault: 11

(I think segfault might be more likely to happen if I haven't deleted the nested_vlen_att.nc file, but it is not a clear pattern. It has happened after I have deleted the file too.)

On Debian 10, still with 4.7.4, I don't see an error (although I think I've seen the segfault sometimes). But ncdump shows that the attribute was not successfully written:

% ncdump nested_vlen_att.nc 
netcdf nested_vlen_att {
types:
  double(*) MY_VLEN ;
  MY_VLEN(*) MY_NESTED_VLEN ;

// global attributes:
		MY_NESTED_VLEN :numbers_att = {{2, 3, 5, 7, 11, 13, 17, 37, 79, 113}, {}}, {{}} ;
}

I couldn't find any documented limitations about this.. Is this a bug? Or is this a wrong way to created a "nested" NC_VLEN attribute?

[DennisHeimbigner]

The vlen handling code has a number of known errors. I will try to get to this
as soon as I can.

[krisfed]

Thank you, Dennis!

Are the other known vlen issues already on github (or documented elsewhere)? I might have run into some of them too, and was planning to report them if I get clear repro steps...

[DennisHeimbigner]

They may be noted in the code. But you should add an issue for those you find
as a reminder to us.