Segfault in H5FD__free_cls with netcdf 4.9.1

[opoplawski]

I'm testing updating Fedora to netcdf 4.9.1 and I'm seeing a new failure when running the tests for the octave-netcdf package. This might be tricky to track down, but I'm not seeing the failure with the current netcdf 4.9.0 package. HDF5 is remaining constant at 1.12.1.

The segfault occurs when exiting:

(gdb) bt
#0  0x00007fffcee93eb0 in ?? ()
#1  0x00007ffff5522b63 in H5FD__free_cls (cls=0x555555f09810) at ../../src/H5FD.c:188
#2  0x00007ffff557e57a in H5I__mark_node (key=0x0, _udata=<synthetic pointer>, _info=0x5555555a6c60) at ../../src/H5Iint.c:393
#3  H5I_clear_type (type=<optimized out>, force=false, app_ref=<optimized out>) at ../../src/H5Iint.c:339
#4  0x00007ffff5522af8 in H5FD_term_package () at ../../src/H5FD.c:147
#5  0x00007ffff5466e54 in H5_term_library () at ../../src/H5.c:377
#6  0x00007ffff546771d in H5_term_library () at ../../src/H5.c:460
#7  0x00007ffff546772d in H5close () at ../../src/H5.c:989
#8  0x00007ffff79077dd in octave::load_save_system::~load_save_system (this=<optimized out>, this=<optimized out>) at libinterp/corefcn/load-save.cc:274
#9  0x00007ffff78f75b5 in octave::interpreter::~interpreter (this=<optimized out>, this=<optimized out>) at libinterp/corefcn/interpreter.cc:661
#10 0x00007ffff703ae47 in std::default_delete<octave::interpreter>::operator() (this=<optimized out>, __ptr=0x5555555c62f0) at /usr/include/c++/12/bits/unique_ptr.h:95
#11 std::default_delete<octave::interpreter>::operator() (__ptr=0x5555555c62f0, this=<optimized out>) at /usr/include/c++/12/bits/unique_ptr.h:89
#12 std::unique_ptr<octave::interpreter, std::default_delete<octave::interpreter> >::~unique_ptr (this=<optimized out>, this=<optimized out>) at /usr/include/c++/12/bits/unique_ptr.h:396
#13 octave::application::~application (this=<optimized out>, this=<optimized out>) at libinterp/octave.cc:296
#14 0x0000555555556590 in octave::cli_application::~cli_application (this=<optimized out>, this=<optimized out>) at libinterp/octave.h:377
#15 main (argc=<optimized out>, argv=<optimized out>) at src/main-cli.cc:122
(gdb) up
#1  0x00007ffff5522b63 in H5FD__free_cls (cls=0x555555f09810) at ../../src/H5FD.c:188
188         if (cls->terminate && cls->terminate() < 0)
(gdb) list
183
184         /* If the file driver has a terminate callback, call it to give the file
185          * driver a chance to free singletons or other resources which will become
186          * invalid once the class structure is freed.
187          */
188         if (cls->terminate && cls->terminate() < 0)
189             HGOTO_ERROR(H5E_VFL, H5E_CANTCLOSEOBJ, FAIL, "virtual file driver '%s' did not terminate cleanly",
190                         cls->name)
191
192         H5MM_xfree(cls);
(gdb) print cls
$1 = (H5FD_class_t *) 0x555555f09810
(gdb) print *cls
$2 = {name = 0x7fffcef4c0d0 <error: Cannot access memory at address 0x7fffcef4c0d0>, maxaddr = 9223372036854775807, fc_degree = H5F_CLOSE_WEAK, terminate = 0x7fffcee93eb0, sb_size = 0x0, sb_encode = 0x0, sb_decode = 0x0, 
  fapl_size = 0, fapl_get = 0x0, fapl_copy = 0x0, fapl_free = 0x0, dxpl_size = 0, dxpl_copy = 0x0, dxpl_free = 0x0, open = 0x7fffcee97280, close = 0x7fffcee97230, cmp = 0x7fffcee94170, query = 0x7fffcee93ed0, get_type_map = 0x0, 
  alloc = 0x7fffcee93f40, free = 0x0, get_eoa = 0x7fffcee93f70, set_eoa = 0x7fffcee93fa0, get_eof = 0x7fffcee93fd0, get_handle = 0x7fffcee94060, read = 0x7fffcee97610, write = 0x7fffcee940f0, flush = 0x7fffcee94000, truncate = 0x0, 
  lock = 0x7fffcee94020, unlock = 0x7fffcee94040, fl_map = {H5FD_MEM_SUPER, H5FD_MEM_SUPER, H5FD_MEM_SUPER, H5FD_MEM_DRAW, H5FD_MEM_DRAW, H5FD_MEM_SUPER, H5FD_MEM_SUPER}}
(gdb) print *cls->terminate
Cannot access memory at address 0x7fffcee93eb0
(gdb) print cls->terminate
$3 = (herr_t (*)(void)) 0x7fffcee93eb0

(gdb) up
#3  H5I_clear_type (type=<optimized out>, force=false, app_ref=<optimized out>) at ../../src/H5Iint.c:339
339                 if (H5I__mark_node((void *)item, NULL, (void *)&udata) < 0)
(gdb) print *item
$6 = {id = 576460752303423489, count = 1, app_count = 1, object = 0x555555e33f50, marked = false, hh = {tbl = 0x55555558ebf0, prev = 0x55555558eb90, next = 0x0, hh_prev = 0x0, hh_next = 0x0, key = 0x5555555a6c60, keylen = 8, 
    hashv = 3217010591}}

valgrind:

==166== Jump to the invalid address stated on the next line
==166==    at 0x32D48EB0: ???
==166==    by 0x6D5F579: UnknownInlinedFun (H5Iint.c:393)
==166==    by 0x6D5F579: H5I_clear_type (H5Iint.c:339)
==166==    by 0x6D03AF7: H5FD_term_package (H5FD.c:147)
==166==    by 0x6C47E53: H5_term_library.part.0 (H5.c:377)
==166==    by 0x6C4872C: H5close (H5.c:989)
==166==    by 0x55597DC: octave::load_save_system::~load_save_system() (load-save.cc:274)
==166==    by 0x55495B4: octave::interpreter::~interpreter() (interpreter.cc:661)
==166==    by 0x4C8CE46: UnknownInlinedFun (unique_ptr.h:95)
==166==    by 0x4C8CE46: UnknownInlinedFun (unique_ptr.h:89)
==166==    by 0x4C8CE46: UnknownInlinedFun (unique_ptr.h:396)
==166==    by 0x4C8CE46: octave::application::~application() (octave.cc:296)
==166==    by 0x10A58F: UnknownInlinedFun (octave.h:377)
==166==    by 0x10A58F: main (main-cli.cc:122)
==166==  Address 0x32d48eb0 is not stack'd, malloc'd or (recently) free'd
==166== 
fatal: caught signal Segmentation fault -- stopping myself...
==166== 
==166== Process terminating with default action of signal 11 (SIGSEGV)
==166==    at 0x6A8EB94: __pthread_kill_implementation (in /usr/lib64/libc.so.6)
==166==    by 0x6A3DAED: raise (in /usr/lib64/libc.so.6)
==166==    by 0x6A3DB9F: ??? (in /usr/lib64/libc.so.6)
==166==    by 0x32D48EAF: ???
==166==    by 0x6D5F579: UnknownInlinedFun (H5Iint.c:393)
==166==    by 0x6D5F579: H5I_clear_type (H5Iint.c:339)
==166==    by 0x6D03AF7: H5FD_term_package (H5FD.c:147)
==166==    by 0x6C47E53: H5_term_library.part.0 (H5.c:377)
==166==    by 0x6C4872C: H5close (H5.c:989)
==166==    by 0x55597DC: octave::load_save_system::~load_save_system() (load-save.cc:274)
==166==    by 0x55495B4: octave::interpreter::~interpreter() (interpreter.cc:661)
==166==    by 0x4C8CE46: UnknownInlinedFun (unique_ptr.h:95)
==166==    by 0x4C8CE46: UnknownInlinedFun (unique_ptr.h:89)
==166==    by 0x4C8CE46: UnknownInlinedFun (unique_ptr.h:396)
==166==    by 0x4C8CE46: octave::application::~application() (octave.cc:296)
==166==    by 0x10A58F: UnknownInlinedFun (octave.h:377)
==166==    by 0x10A58F: main (main-cli.cc:122)

There are quite a lot of iterations through the loop in H5I_clear_type before the segfault. I have no idea what other information would be useful for tracking this down.

[edwardhartnett]

You should try with the newly released hdf5-1.14.0. If fixes a lot of bugs, perhaps this is one of them.

[WardF]

Interesting; is this happening on particular hardware? I'm trying to get our big-endian machine back online to try to track down the other issue you've reported, @opoplawski, (#1338) and if this is also on specific hardware, let me know. Thanks!

[opoplawski]

Seems to happen on all - and definitely on x86_64.

[WardF]

Thanks! That will make the issue easier to track down. Looking at the traces you provided, I'm trying to pin down at what point a function in libnetcdf is being called? I'm trying to grab the latest octave release, I assume this test is failing as part of the regular suite of tests, I will see if I can replicate it locally.

[opoplawski]

Presumably netcdf calls are made during the tests, but at the point of the crash I think we're just closing down the HDF5 library.

I'm afraid this is multiple levels deep - it's not the tests from octave, but from the octave-netcdf package https://gnu-octave.github.io/packages/netcdf/

later I'll try to see if it's sensitive to updating hdf5.

[edwardhartnett]

Could it be that both netcdf-c and octave are shutting down the HDF5 library? The netcdf-c one succeeds, and then the octave attempt fails?

[opoplawski]

I set a breakpoint on H5close(), but I only see it called the once leading to the segfault.

[opoplawski]

I've at least stripped it down to a simple reproducer in octave with the netcdf package installed:

octave -H -q --no-window-system --no-site-file --eval "pkg('load','netcdf');import_netcdf;ncid = netcdf.create('test-netcdf.nc','NC_CLOBBER'); netcdf.close(ncid);"

So it really does seem to be something to do with library tear down in the octave environment as we aren't doing much else. It also crashed without the netcdf.close(ncid) call, but then you don't get a valid netcdf file. Since octave is built with HDF5 support, it calls H5close() on exit. But it still seems like the HDF5 state must be getting corrupted somehow for it to segfault.

I've also just discovered that I've still been building netcdf with -DH5_USE_110_API though I suspect that probably hasn't been needed for ages. I'm doing some rebuilds without that and with hdf5 1.12.2 and 1.14.0 and will report those results when done. octave isn't built with and H5_*_API defines so it would be good to make that consistent.

[DennisHeimbigner]

Any chance of running your program with valgrind?

[opoplawski]

The valgrind output is the same as before - see the initial comment.

[opoplawski]

Dropping -DH5_USE_110_API didn't have any effect.

[edwardhartnett]

Some time ago I changed the netcdf-c code so that it does not depend on H5_USE_110_API or any other macro redefinition schemes that they use at HDF5. Simpler to just call the desired functions directly, and not worry about the redefines of their APIs (a misguided approach, IMO).

Is there a way of telling if the HDF5 library has been closed down?