libnetcdf 4.9.3 fails to write from multi-threaded application

[djhoese]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

I recently updated an environment and started getting failures related to writing a NetCDF file. Unfortunately my use case is not a simple one. I have a Python application that takes the following steps:

1. Read NetCDF data using xarray's open_dataset lazily using dask.
2. Do stuff with the data
3. Split the data into chunks and save each chunk to their own NetCDF file. Each array chunk is passed to a function running on a separate thread using dask's blockwise and is written using xarray's to_netcdf.

When going from and to the builds mentioned below:

The following packages will be UPDATED:

  libnetcdf                        4.9.2-nompi_h0134ee8_117 --> 4.9.3-nompi_h81b047f_102 
  netcdf4                     1.7.2-nompi_py313ha67fc0a_103 --> 1.7.2-nompi_py313hfae5b86_104 

I get:

HDF5-DIAG: Error detected in HDF5 (1.14.6) thread 1:
  #000: H5F.c line 496 in H5Fis_accessible(): unable to determine if file is accessible as HDF5
    major: File accessibility
    minor: Not an HDF5 file
  #001: H5VLcallback.c line 3913 in H5VL_file_specific(): file specific failed
    major: Virtual Object Layer
    minor: Can't operate on object
  #002: H5VLcallback.c line 3848 in H5VL__file_specific(): file specific failed
    major: Virtual Object Layer
    minor: Can't operate on object
  #003: H5VLnative_file.c line 344 in H5VL__native_file_specific(): error in HDF5 file check
    major: File accessibility
    minor: Can't get value
  #004: H5Fint.c line 1055 in H5F__is_hdf5(): unable to open file
    major: File accessibility
    minor: Unable to initialize object
  #005: H5FD.c line 787 in H5FD_open(): can't open file
    major: Virtual File Layer
    minor: Unable to open file
  #006: H5FDsec2.c line 323 in H5FD__sec2_open(): unable to open file: name = '/local/path/to/output/my.nc', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
    major: File accessibility
    minor: Unable to open file

When I run my application. With the previous versions mentioned above, it works fine. This even happens if I tell dask to use only one threaded worker. If I tell dask to use the special DASK_SCHEDULER="single-threaded" then it succeeds.

I realize this isn't much to go on and I will try to get a minimal example in the next 24 hours.

Installed packages

# packages in environment at /home/davidh/miniforge3/envs/axi_tools_py313:
#
# Name                              Version                   Build                    Channel
_libgcc_mutex                       0.1                       conda_forge              conda-forge
_openmp_mutex                       4.5                       2_gnu                    conda-forge
affine                              2.4.0                     pyhd8ed1ab_1             conda-forge
attr                                2.5.2                     h39aace5_0               conda-forge
attrs                               25.3.0                    pyh71513ae_0             conda-forge
aws-c-auth                          0.9.1                     h48c9088_3               conda-forge
aws-c-cal                           0.9.2                     he7b75e1_1               conda-forge
aws-c-common                        0.12.4                    hb03c661_0               conda-forge
aws-c-compression                   0.3.1                     h92c474e_6               conda-forge
aws-c-event-stream                  0.5.6                     h82d11aa_3               conda-forge
aws-c-http                          0.10.4                    h94feff3_3               conda-forge
aws-c-io                            0.22.0                    h57f3b0d_1               conda-forge
aws-c-mqtt                          0.13.3                    h2b1cf8c_6               conda-forge
aws-c-s3                            0.8.6                     h4e5ac4b_5               conda-forge
aws-c-sdkutils                      0.2.4                     h92c474e_1               conda-forge
aws-checksums                       0.2.7                     h92c474e_2               conda-forge
aws-crt-cpp                         0.34.4                    h60c762c_0               conda-forge
aws-sdk-cpp                         1.11.606                  h32384e2_4               conda-forge
axi-tools                           1.0.0                     pypi_0                   pypi
azure-core-cpp                      1.16.0                    h3a458e0_1               conda-forge
azure-identity-cpp                  1.12.0                    ha729027_0               conda-forge
azure-storage-blobs-cpp             12.14.0                   hb1c9500_1               conda-forge
azure-storage-common-cpp            12.10.0                   hebae86a_2               conda-forge
azure-storage-files-datalake-cpp    12.12.0                   h8b27e44_3               conda-forge
blosc                               1.21.6                    he440d0b_1               conda-forge
bokeh                               3.8.0                     pyhd8ed1ab_0             conda-forge
brotli-python                       1.1.0                     py313h7033f15_4          conda-forge
bzip2                               1.0.8                     hda65f42_8               conda-forge
c-ares                              1.34.5                    hb9d3cd8_0               conda-forge
ca-certificates                     2025.8.3                  hbd8a1cb_0               conda-forge
cached-property                     1.5.2                     hd8ed1ab_1               conda-forge
cached_property                     1.5.2                     pyha770c72_1             conda-forge
certifi                             2025.8.3                  pyhd8ed1ab_0             conda-forge
cffi                                2.0.0                     py313hf01b4d8_0          conda-forge
cftime                              1.6.4                     py313h29aa505_2          conda-forge
charset-normalizer                  3.4.3                     pyhd8ed1ab_0             conda-forge
click                               8.3.0                     pyh707e725_0             conda-forge
click-plugins                       1.1.1.2                   pyhd8ed1ab_0             conda-forge
cligj                               0.7.2                     pyhd8ed1ab_2             conda-forge
cloudpickle                         3.1.1                     pyhd8ed1ab_0             conda-forge
colorama                            0.4.6                     pyhd8ed1ab_1             conda-forge
configobj                           5.0.9                     pyhd8ed1ab_1             conda-forge
contourpy                           1.3.3                     py313h7037e92_2          conda-forge
crc32c                              2.7.1                     py313h54dd161_2          conda-forge
cytoolz                             1.0.1                     py313h536fd9c_0          conda-forge
dask                                2025.9.1                  pyhcf101f3_0             conda-forge
dask-core                           2025.9.1                  pyhcf101f3_0             conda-forge
defusedxml                          0.7.1                     pyhd8ed1ab_0             conda-forge
deprecated                          1.2.18                    pyhd8ed1ab_0             conda-forge
distributed                         2025.9.1                  pyhcf101f3_0             conda-forge
donfig                              0.8.1.post1               pyhd8ed1ab_1             conda-forge
freexl                              2.0.0                     h9dce30a_2               conda-forge
fsspec                              2025.9.0                  pyhd8ed1ab_0             conda-forge
geos                                3.14.0                    h480dda7_0               conda-forge
geotiff                             1.7.4                     h239500f_2               conda-forge
gflags                              2.2.2                     h5888daf_1005            conda-forge
giflib                              5.2.2                     hd590300_0               conda-forge
glog                                0.7.1                     hbabe93e_0               conda-forge
h2                                  4.3.0                     pyhcf101f3_0             conda-forge
h5py                                3.14.0                    nompi_py313h253c126_101  conda-forge
hdf4                                4.2.15                    h2a13503_7               conda-forge
hdf5                                1.14.6                    nompi_h6e4c0c1_103       conda-forge
hpack                               4.1.0                     pyhd8ed1ab_0             conda-forge
hyperframe                          6.1.0                     pyhd8ed1ab_0             conda-forge
icu                                 75.1                      he02047a_0               conda-forge
idna                                3.10                      pyhd8ed1ab_1             conda-forge
importlib-metadata                  8.7.0                     pyhe01879c_1             conda-forge
jinja2                              3.1.6                     pyhd8ed1ab_0             conda-forge
json-c                              0.18                      h6688a6e_0               conda-forge
keyutils                            1.6.3                     hb9d3cd8_0               conda-forge
krb5                                1.21.3                    h659f571_0               conda-forge
lcms2                               2.17                      h717163a_0               conda-forge
ld_impl_linux-64                    2.44                      ha97dd6f_2               conda-forge
lerc                                4.0.0                     h0aef613_1               conda-forge
libabseil                           20250512.1                cxx17_hba17884_0         conda-forge
libaec                              1.1.4                     h3f801dc_0               conda-forge
libarchive                          3.8.1                     gpl_h98cc613_100         conda-forge
libarrow                            21.0.0                    h73424eb_6_cpu           conda-forge
libarrow-acero                      21.0.0                    h635bf11_6_cpu           conda-forge
libarrow-compute                    21.0.0                    h8c2c5c3_6_cpu           conda-forge
libarrow-dataset                    21.0.0                    h635bf11_6_cpu           conda-forge
libarrow-substrait                  21.0.0                    h3f74fd7_6_cpu           conda-forge
libblas                             3.9.0                     36_h4a7cf45_openblas     conda-forge
libbrotlicommon                     1.1.0                     hb03c661_4               conda-forge
libbrotlidec                        1.1.0                     hb03c661_4               conda-forge
libbrotlienc                        1.1.0                     hb03c661_4               conda-forge
libcblas                            3.9.0                     36_h0358290_openblas     conda-forge
libcrc32c                           1.1.2                     h9c3ff4c_0               conda-forge
libcurl                             8.14.1                    h332b0f4_0               conda-forge
libdeflate                          1.24                      h86f0d12_0               conda-forge
libedit                             3.1.20250104              pl5321h7949ede_0         conda-forge
libev                               4.33                      hd590300_2               conda-forge
libevent                            2.1.12                    hf998b51_1               conda-forge
libexpat                            2.7.1                     hecca717_0               conda-forge
libffi                              3.4.6                     h2dba641_1               conda-forge
libfreetype                         2.14.1                    ha770c72_0               conda-forge
libfreetype6                        2.14.1                    h73754d4_0               conda-forge
libgcc                              15.1.0                    h767d61c_5               conda-forge
libgcc-ng                           15.1.0                    h69a702a_5               conda-forge
libgfortran                         15.1.0                    h69a702a_5               conda-forge
libgfortran5                        15.1.0                    hcea5267_5               conda-forge
libgomp                             15.1.0                    h767d61c_5               conda-forge
libgoogle-cloud                     2.39.0                    hdb79228_0               conda-forge
libgoogle-cloud-storage             2.39.0                    hdbdcf42_0               conda-forge
libgrpc                             1.73.1                    h1e535eb_0               conda-forge
libiconv                            1.18                      h3b78370_2               conda-forge
libjpeg-turbo                       3.1.0                     hb9d3cd8_0               conda-forge
libkml                              1.3.0                     hf539b9f_1021            conda-forge
liblapack                           3.9.0                     36_h47877c9_openblas     conda-forge
liblzma                             5.8.1                     hb9d3cd8_2               conda-forge
libmpdec                            4.0.0                     hb9d3cd8_0               conda-forge
libnetcdf                           4.9.3                     nompi_h81b047f_102       conda-forge
libnghttp2                          1.67.0                    had1ee68_0               conda-forge
libnsl                              2.0.1                     hb9d3cd8_1               conda-forge
libopenblas                         0.3.30                    pthreads_h94d23a6_2      conda-forge
libopentelemetry-cpp                1.21.0                    hb9b0907_1               conda-forge
libopentelemetry-cpp-headers        1.21.0                    ha770c72_1               conda-forge
libparquet                          21.0.0                    h790f06f_6_cpu           conda-forge
libpng                              1.6.50                    h421ea60_1               conda-forge
libprotobuf                         6.31.1                    h9ef548d_1               conda-forge
libre2-11                           2025.08.12                h7b12aa8_1               conda-forge
librttopo                           1.1.0                     h96cd706_19              conda-forge
libspatialite                       5.1.0                     h7250436_15              conda-forge
libsqlite                           3.50.4                    h0c1763c_0               conda-forge
libssh2                             1.11.1                    hcf80075_0               conda-forge
libstdcxx                           15.1.0                    h8f9b012_5               conda-forge
libstdcxx-ng                        15.1.0                    h4852527_5               conda-forge
libthrift                           0.22.0                    h454ac66_1               conda-forge
libtiff                             4.7.1                     h8261f1e_0               conda-forge
libutf8proc                         2.11.0                    hb04c3b8_0               conda-forge
libuuid                             2.41.2                    he9a06e4_0               conda-forge
libwebp-base                        1.6.0                     hd42ef1d_0               conda-forge
libxcb                              1.17.0                    h8a09558_0               conda-forge
libxml2                             2.13.8                    h04c0eec_1               conda-forge
libzip                              1.11.2                    h6991a6a_0               conda-forge
libzlib                             1.3.1                     hb9d3cd8_2               conda-forge
locket                              1.0.0                     pyhd8ed1ab_0             conda-forge
lz4                                 4.4.4                     py313hdd09ace_1          conda-forge
lz4-c                               1.10.0                    h5888daf_1               conda-forge
lzo                                 2.10                      h280c20c_1002            conda-forge
markupsafe                          3.0.2                     py313h8060acc_1          conda-forge
minizip                             4.0.10                    h05a5f5f_0               conda-forge
msgpack-python                      1.1.1                     py313h7037e92_1          conda-forge
narwhals                            2.5.0                     pyhcf101f3_0             conda-forge
ncurses                             6.5                       h2d0b736_3               conda-forge
netcdf4                             1.7.2                     nompi_py313hfae5b86_104  conda-forge
nlohmann_json                       3.12.0                    h54a6638_1               conda-forge
numcodecs                           0.16.1                    py313h08cd8bf_1          conda-forge
numpy                               2.3.3                     py313hf6604e3_0          conda-forge
openjpeg                            2.5.4                     h55fea9a_0               conda-forge
openssl                             3.5.3                     h26f9b46_1               conda-forge
orc                                 2.2.0                     h1bc01a4_0               conda-forge
packaging                           25.0                      pyh29332c3_1             conda-forge
pandas                              2.3.2                     py313h08cd8bf_0          conda-forge
partd                               1.4.2                     pyhd8ed1ab_0             conda-forge
pcre2                               10.46                     h1321c63_0               conda-forge
pillow                              11.3.0                    py313ha492abd_3          conda-forge
pip                                 25.2                      pyh145f28c_0             conda-forge
platformdirs                        4.4.0                     pyhcf101f3_0             conda-forge
pooch                               1.8.2                     pyhd8ed1ab_3             conda-forge
proj                                9.6.2                     h18fbb6c_2               conda-forge
prometheus-cpp                      1.3.0                     ha5d0236_0               conda-forge
psutil                              7.1.0                     py313h07c4f96_0          conda-forge
pthread-stubs                       0.4                       hb9d3cd8_1002            conda-forge
pyarrow                             21.0.0                    py313h78bf25f_0          conda-forge
pyarrow-core                        21.0.0                    py313he109ebe_0_cpu      conda-forge
pycparser                           2.22                      pyh29332c3_1             conda-forge
pykdtree                            1.4.3                     py313h29aa505_1          conda-forge
pyorbital                           1.10.2                    pyhd8ed1ab_0             conda-forge
pyparsing                           3.2.5                     pyhcf101f3_0             conda-forge
pyproj                              3.7.2                     py313hcfca4fd_1          conda-forge
pyresample                          1.34.2                    py313h46c70d0_0          conda-forge
pysocks                             1.7.1                     pyha55dd90_7             conda-forge
pyspectral                          0.13.6                    pyhd8ed1ab_0             conda-forge
python                              3.13.7                    h2b335a9_100_cp313       conda-forge
python-dateutil                     2.9.0.post0               pyhe01879c_2             conda-forge
python-geotiepoints                 1.8.0                     py313h29aa505_0          conda-forge
python-tzdata                       2025.2                    pyhd8ed1ab_0             conda-forge
python_abi                          3.13                      8_cp313                  conda-forge
pytz                                2025.2                    pyhd8ed1ab_0             conda-forge
pyyaml                              6.0.2                     py313h8060acc_2          conda-forge
re2                                 2025.08.12                h5301d42_1               conda-forge
readline                            8.2                       h8c095d6_2               conda-forge
requests                            2.32.5                    pyhd8ed1ab_0             conda-forge
s2n                                 1.5.26                    h5ac9029_0               conda-forge
satpy                               0.58.1.dev121+ga4e58f082  pypi_0                   pypi
scipy                               1.16.2                    py313h11c21cd_0          conda-forge
setuptools                          80.9.0                    pyhff2d567_0             conda-forge
shapely                             2.1.2                     py313hfc84eb1_0          conda-forge
six                                 1.17.0                    pyhe01879c_1             conda-forge
snappy                              1.2.2                     h03e3b7b_0               conda-forge
snuggs                              1.4.7                     pyhd8ed1ab_2             conda-forge
sortedcontainers                    2.4.0                     pyhd8ed1ab_1             conda-forge
sqlite                              3.50.4                    hbc0de68_0               conda-forge
tblib                               3.1.0                     pyhd8ed1ab_0             conda-forge
tk                                  8.6.13                    noxft_hd72426e_102       conda-forge
toolz                               1.0.0                     pyhd8ed1ab_1             conda-forge
tornado                             6.5.2                     py313h07c4f96_1          conda-forge
tqdm                                4.67.1                    pyhd8ed1ab_1             conda-forge
trollsift                           0.6.0                     pyhd8ed1ab_0             conda-forge
typing_extensions                   4.15.0                    pyhcf101f3_0             conda-forge
tzdata                              2025b                     h78e105d_0               conda-forge
uriparser                           0.9.8                     hac33072_0               conda-forge
urllib3                             2.5.0                     pyhd8ed1ab_0             conda-forge
wrapt                               1.17.3                    py313h07c4f96_1          conda-forge
xarray                              2025.9.0                  pyhd8ed1ab_0             conda-forge
xerces-c                            3.2.5                     h988505b_2               conda-forge
xorg-libxau                         1.0.12                    hb9d3cd8_0               conda-forge
xorg-libxdmcp                       1.1.5                     hb9d3cd8_0               conda-forge
xyzservices                         2025.4.0                  pyhd8ed1ab_0             conda-forge
yaml                                0.2.5                     h280c20c_3               conda-forge
zarr                                3.1.3                     pyhcf101f3_0             conda-forge
zict                                3.0.0                     pyhd8ed1ab_1             conda-forge
zipp                                3.23.0                    pyhd8ed1ab_0             conda-forge
zlib                                1.3.1                     hb9d3cd8_2               conda-forge
zstandard                           0.25.0                    py313h54dd161_0          conda-forge
zstd                                1.5.7                     hb8e6e7a_2               conda-forge

Environment info

active environment : axi_tools_py313
    active env location : /home/davidh/miniforge3/envs/axi_tools_py313
            shell level : 2
       user config file : /home/davidh/.condarc
 populated config files : /home/davidh/miniforge3/.condarc
                          /home/davidh/.condarc
          conda version : 25.7.0
    conda-build version : not installed
         python version : 3.10.14.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=25.7.0=0
                          __glibc=2.35=0
                          __linux=6.12.10=0
                          __unix=0=0
       base environment : /home/davidh/miniforge3  (writable)
      conda av data dir : /home/davidh/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/davidh/miniforge3/pkgs
                          /home/davidh/.conda/pkgs
       envs directories : /home/davidh/miniforge3/envs
                          /home/davidh/.conda/envs
               platform : linux-64
             user-agent : conda/25.7.0 requests/2.31.0 CPython/3.10.14 Linux/6.12.10-76061203-generic pop/22.04 glibc/2.35 solver/libmamba conda-libmamba-solver/25.3.0 libmambapy/2.0.8
                UID:GID : 53807:1000
             netrc file : None
           offline mode : False

[dopplershift]

libnetcdf (netcdf-c) isn't threadsafe, so it working previously feels like coincidence.

[djhoese]

🤦‍♂️ Ya know, you'd think with how long I've been working with NetCDF files that I would have thought of this. Thanks @dopplershift. I'll see if throwing a lock around the whole thing works out.

[djhoese]

Actually, this still fails with a threading Lock controlling access to xarray's to_netcdf. Additionally, I'm only writing one file. So I suppose if this is an expected threading issue then it must be conflicting with the reading of the input data.

Edit: Ok so it is related to also reading data from NetCDF first. If I replace the arrays with equivalent "zeros" from-scratch then it works. Additionally, if the file exists already and I open it in "a" mode then it also works fine. Also note when the HDF5 files are received it still seems to write everything fine.

[djhoese]

I'm still not convinced this isn't something wrong with the build or with the C library. I've narrowed this down alot and hope to narrow it down further. For my and others' reference here is what I've got so far:

import dask.array as da
import numpy as np
import xarray as xr


def delayed_save(arr):
    data_arr = xr.DataArray(arr, dims=("y", "x"))
    data_arr.to_netcdf("test.nc")
    return arr


if __name__ == "__main__":
    nc = xr.open_dataset("/tmp/OR_ABI-L1b-RadM2-M6C07_G19_s20252672155535_e20252672156004_c20252672156046.nc",
                         chunks="auto")
    data = da.zeros((500, 500), chunks=500, dtype=np.float32)

    res = da.map_blocks(
        delayed_save,
        data,
        meta=np.ndarray((), dtype=data.dtype),
        dtype=data.dtype,
    )
    res.compute()

Basically:

1. Open a NetCDF file with xarray.open_dataset
2. Don't even use the file
3. Save some data to a new NetCDF file via dask with a single worker thread. There is only one chunk in the data being saved so this function is only called once.

This produces:

HDF5-DIAG: Error detected in HDF5 (1.14.6) thread 1:
  #000: H5F.c line 496 in H5Fis_accessible(): unable to determine if file is accessible as HDF5
    major: File accessibility
    minor: Not an HDF5 file
  #001: H5VLcallback.c line 3913 in H5VL_file_specific(): file specific failed
    major: Virtual Object Layer
    minor: Can't operate on object
  #002: H5VLcallback.c line 3848 in H5VL__file_specific(): file specific failed
    major: Virtual Object Layer
    minor: Can't operate on object
  #003: H5VLnative_file.c line 344 in H5VL__native_file_specific(): error in HDF5 file check
    major: File accessibility
    minor: Can't get value
  #004: H5Fint.c line 1055 in H5F__is_hdf5(): unable to open file
    major: File accessibility
    minor: Unable to initialize object
  #005: H5FD.c line 787 in H5FD_open(): can't open file
    major: Virtual File Layer
    minor: Unable to open file
  #006: H5FDsec2.c line 323 in H5FD__sec2_open(): unable to open file: name = '/home/davidh/repos/git/axi_tools/davidh_tests/test.nc', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
    major: File accessibility
    minor: Unable to open file

Edit: Here's the script with xarray removed and still produces the error:

import dask.array as da
import numpy as np
from netCDF4 import Dataset


def delayed_save(arr):
    Dataset("test.nc", "w")
    return arr


if __name__ == "__main__":
    nc = Dataset("/tmp/OR_ABI-L1b-RadM2-M6C07_G19_s20252672155535_e20252672156004_c20252672156046.nc", "r")
    data = da.zeros((500, 500), chunks=500, dtype=np.float32)

    res = da.map_blocks(
        delayed_save,
        data,
        meta=np.ndarray((), dtype=data.dtype),
        dtype=data.dtype,
    )
    res.compute()

I'll work on removing dask tomorrow and seeing if I can run this from a normal python thread.

[djhoese]

Ok something weird is going on. This fails for me:

from netCDF4 import Dataset
import threading


def delayed_save():
    Dataset("test.nc", "w")


if __name__ == "__main__":
    t = threading.Thread(target=delayed_save)
    t.start()
    t.join()

Just make sure you remove test.nc before running the script multiple times.

Note: If test.nc already exists, then it doesn't fail.

[djhoese]

So some more updates on this:

1. I asked ChatGPT to generate a C version equivalent to this and I made modifications to use the same functions that I think netcdf4-python are using and it does not fail.
2. I tried installed netcdf4-python from PyPI and using it's bundled version of libnetcdf. In the newest release it uses 4.9.4-development and in an earlier release it used 4.9.3-development. In both cases my above python script doesn't fail.

@ocefpaf Seeing as you create the netcdf4 python manylinux build images (https://github.com/ocefpaf/netcdf-manylinux) and merged the 4.9.3 PR here (#207), do you have any idea what might be going on here? This really seems to be something wrong with the conda-forge build only.

If anyone has ideas for how to get the below C example to be more like the python example then let me know and maybe we can get a failure that is pure C.

Non-failing C example

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <netcdf.h>

#define FILE_PATH "created_file.nc"  // Will be created (overwritten if exists)

// Macro to handle NetCDF errors
#define NC_CHECK(call)                                 \
    do {                                                \
        int retval = call;                              \
        if (retval != NC_NOERR) {                       \
            fprintf(stderr, "NetCDF error: %s\n", nc_strerror(retval)); \
            exit(EXIT_FAILURE);                         \
        }                                               \
    } while (0)

// Thread function to create a NetCDF-4 file
void* create_netcdf4_file(void* arg) {
    const char* path = (const char*)arg;
    int ncid;

    printf("Thread: Creating NetCDF-4 file: %s\n", path);

    NC_CHECK(nc_set_default_format(NC_FORMAT_NETCDF4, NULL));

    // Create a new NetCDF-4 file, overwriting if it exists
    //NC_CHECK(nc_create(path, NC_NETCDF4 | NC_CLOBBER, &ncid));
    NC_CHECK(nc_create(path, NC_CLOBBER, &ncid));
    printf("Thread: File created (ID: %d)\n", ncid);

    // Close the file
    NC_CHECK(nc_close(ncid));
    printf("Thread: NetCDF-4 file created and closed successfully.\n");

    return NULL;
}

int main() {
    pthread_t thread;

    printf("Main: Starting thread to create NetCDF-4 file...\n");

    if (pthread_create(&thread, NULL, create_netcdf4_file, (void*)FILE_PATH) != 0) {
        perror("pthread_create");
        return EXIT_FAILURE;
    }

    // Wait for the thread to finish
    pthread_join(thread, NULL);

    printf("Main: Thread finished.\n");
    return EXIT_SUCCESS;
}

[djhoese]

Ok more info on this after building netcdf-c and netcdf-python from source, adding debug flags, logging, etc and adding my own printf messages:

1. The problem comes down to netcdf-c's NC_infermodel in libdispatch/dinfermodel.c with code that was added as part Check if HDF5 "file" is a DAOS object Unidata/netcdf-c#2021. Specifically the isdaoscontainer function in this file is called.
2. The isdaoscontainer function is supposed to check if the provided NetCDF files is a "DAOS container", but since we're creating the file and nothing exists on disk this ends up failed with the referenced errors about the file not existing.
3. Functionally this is FINE because this check for it being a DAOS container fails in the non-error message case, but for some reason when run in a separate thread it causes HDF5 to print out the full stack where the file is discovered to not exist.

I haven't been able to dive further than this as I'm already way out of my element as far as NetCDF-C and HDF5 source code. The above makes me think that either:

1. There is some error state in HDF5 that is either:
   i. Not being initialized by NetCDF4 or HDF5 in the new thread
   ii. Not being checked by HDF5 properly after the failure so it thinks it needs to display the error message.
2. For some reason NetCDF4 or HDF5 is not checking plain existence of the file before calling the lower level H5Fis_accessible function.

[ocefpaf]

@ocefpaf Seeing as you create the netcdf4 python manylinux build images (https://github.com/ocefpaf/netcdf-manylinux) and merged the 4.9.3 PR here (#207), do you have any idea what might be going on here? This really seems to be something wrong with the conda-forge build only.

It seems you are already in the right direction there, but I don't think the builds here are that different. I used most of the same flags. I never investigated how different they may be though.

[djhoese]

Sorry @ocefpaf. This indeed has nothing to do with conda-forge's specific way of building or at least it doesn't seem that way right now. I've tracked things further and I think this is a "bug" in netcdf-c in that it doesn't set error handling functions per-thread when HDF5's error handlers are per-thread. OR this is documented somewhere and I or netcdf-python should be doing this.

It comes down to the low-level macros in HDF5 where it builds up a stack of calls in an error log and when the top-level function is done the macro at the end of the function will check if there is an error and if so check if there is a callback to do something with that error. If NULL then don't print out anything (this is what netcdf-c does in their libhdf5/hdf5internal.c in the nc4_hdf5_initialize function -> set_auto(NULL, NULL) call. Otherwise, the default in HDF5 is to print out the stack as seen in my error above. In the HDF5 error handling functions a call to H5E__get_stack is made which creates this error stack information per-thread.

So the questions/problems I'm currently tackling for anyone following this journey:

1. Why didn't this happen in libnetcdf 4.9.2?
2. My biggest problem right now is why can I only get this failing in the python (netcdf4-python) case and not the C-based (libnetcdf only) case. Kind of hard to file a bug report when I can't reproduce it in the language the library is written in.

[dopplershift]

I'm still not seeing behavior here that can't be chalked up to "netcdf-c doesn't support multi-threaded access". Am I missing something? (cc @WardF)

[djhoese]

I thought it was the difference between thread-safe and can be used in a separate thread. In the above example(s), I'm not sharing anything user-wise between threads.

[dopplershift]

I'll defer to @WardF here.

[djhoese]

I should have said, I'm also not using NetCDF from multiple threads except for the initial python import. So I hoped that this example where no race conditions are possible/expected would be do-able. It was also my understanding that if locks were used carefully then netcdf could be used in multiple threads, but I'll admit this could have come from a Google AI overview.

I think my main obsession with this and as an excuse for the amount of time I've spent, is that this used to show no errors in the previous release (again, the error/failure is expected, just not the printed stack). Of course this might be expected, or maybe not unexpected is more accurate, but I've only just tracked it to this thread-local error stack today so I wasn't sure what to think before that.