Error when dimension shares name with variable in netcdf-4 classic file

[ellenjohnson]

Hello everyone!

Short question: In a netcdf4-classic file, can a fixed-size dimension share the same name with a previously-defined variable?

A customer reported a MATLAB crash with following steps using our nccreate function. To me, this wouldn’t make sense to name the second variable’s dimension the same name as the first variable’s name. Wondering if this is related to issue #1528, but in our case they are fixed-size dimensions. We're using netcdf 4.7.3 and this is not OS-specific.

nccreate('test.nc','var1','Dimensions',{'dim1',3}); % one-dimension, vector of length 3
nccreate('test.nc','var2','Dimensions',{'var1',2}); % one-dimension, vector of length 2

This errors during nc_close() with error -101 NC_EHDFERR, and then we crash (oops). Note: nccreate defaults to netcdf4-classic. Calling it twice in succession like that means we first create new file test.nc and write var1 with dim1 to it, then close. Then reopen for append and create var2 with dimension name var1, then close.

I wrote a netcdf-c program (attached, netcdfRepro_VarDimError) to reproduce it (using netcdf-c v4.7.3), and still get the same NC_EHDFERR error, but no segfault or similar crash. (My repro is .c, but looks like github won't let me attach a .c file, so renamed it .txt).
netcdfRepro_VarDimError.txt

[DennisHeimbigner]

It looks like this is driven by that bane of our existence: HDF5 dim scales.
My speculation is that this particular combination ends up trying to
create two datasets (one a hidden dimscale) with the same name.
But Ed Hartnett might know for sure.

[DennisHeimbigner]

Short answer: you probably can't do this using the current HDF5+netcdf-c.
It would require us to avoid using dimension scales and that is not likely
any time soon.

[ellenjohnson]

Thanks Dennis. So to clarify: the NC_EHDFERR from nc_close() is expected netcdf behavior (i.e. limitation)? Or unexpected (i.e., bug) but there won't be a fix anytime soon? We'll fix our MATLAB code so we don't crash upon this error, but it would be great to know if this is a netcdf limitation (then we can document the lijmitation to our customers), or a netcdf bug.

In our case, the customer has an easy workaround -- don't name the dimension the same name as the previously defined variable. I'm not sure why the were doing that in the first place, unless it has something to do with netcdf coordinate variables. I know a variable should only be given the same name as a dimension if it's to be used as a coordinate variable -- but does that apply in reverse? Meaning are there restrictions (other than the error) on naming a dimension the same name as a predefined variable? And does that have anything to do with coordinate variables?

[DennisHeimbigner]

Technically, it is an error since the spec says nothing AFAIK about this being illegal.
But calling it a limitation of the netcdf-4 over HDF5 implementation seems
proper. I will see if I can add a description of this somewhere in our
documentation.
And no, it is unlikely to be fixed any time soon.
You can get an insite into this by setting this env variable:
export NETCDF_LOG_LEVEL=0
and running your test program. In the HDF5 error log output
you will see a complaint about a duplicate name.

As for this question:

are there restrictions (other than the error) on naming a dimension the same name as a predefined variable?
I do not know for sure.

[edwardhartnett]

The best way to get a fix would be to submit a C test which demonstrates the problem.

There are netCDF-4 limitations with respect to the naming of dimensions and variables. This is due to the lack of support for shared dimensions in HDF5. So creating a netCDF dimension actually creates a HDF5 dataset (i.e. variable) with some special metadata.

If you then create a var with the same name, it needs to delete the dataset created for the dim, and create a new one, which is a coordinate variable, representing both the variable and the dimension in one HDF5 dataset.

However, apparently the code does not handle the case where you want a var of the same name, but not a coordinate var, when you do things in a certain order. This will be order dependent, so try doing things in a different order and that could well work. (For example, I think if you create the var first, and then a dim with the same name, that will work correctly.)

[DennisHeimbigner]

I stand corrected. Ed's answer is the one you should accept.

[ellenjohnson]

Thanks Ed and Dennis.
@edhartnett: I did create a C repro code that gives the error -- I attached it in my initial question (see netcdfRepro_VarDimError). It's a C file but i had to rename it to .txt because github wouldn't let me attach a .c file. The NC_EHDFERR happens at call to nc_close(). I linked with netCDF 4.7.3 and HDF5 1.8.12.

I mimicked what our nccreate function is doing by calling it twice in a row like the customer did -- you'll see it in the repro code. Can you please take a look at the C repro and let me know if this helps explain the problem?

Let me know if you'd rather have me copy and paste the C repro directly i the comment.

[nvishnoiMW]

@edwardhartnett and @DennisHeimbigner - I was working on the crash that Ellen mentioned happened in MATLAB (after we got NC_EHDFERR error). I found this crash got triggered when nc_close(ncid) was called again after the first nc_close reported the error. I have attached the modified repro code in this post. Here is the sample output of running the code with netcdf 4.7.3:

status after close = 0
error code after close = -33
reopening file to write a second variable...
status = 0
error code = 0
error code after close = -101
main.o: nc4hdf.c:1438: attach_dimscales: Assertion `dsid > 0' failed.
Abort (core dumped)

Interestingly, I could only reproduce the crash using netcdf 4.6.1 and later versions. Calling nc_close(ncid) twice did not crash but gave the following output with 4.3.3.1:

status after close = 0
error code after close = -33
reopening file to write a second variable...
status = 0
error code = 0
error code after close = -101
error code after close = -33
End of test.

which is expected as as you can see in the case of "no error" calling nc_close twice returned -33. I would expect the same in the case of any error as well. Can you please confirm if this is a bug in later versions of netcdf? Probably something changed between 4.3.3.1 and 4.6.1 which caused this crash to happen.

Thanks,
Nalini

netcdfRepro_VarDimErrorAndCrash.txt

.

[nvishnoiMW]

Hi @edwardhartnett and @DennisHeimbigner,

Any comments on the crash I saw?

Thanks,
Nalini

[edwardhartnett]

Firstly, thanks for all the hard work tracking this down!

I'm sorry that this issue, like so many, does not get the attention it deserves. ;-) I wish that there was a giant team of programmers working on netCDF, but unfortunately there is only a small team of (in one case) giant programmers.

The question of what happens when you call nc_close() again, after calling it once and getting an error, is undefined. Basically, don't do that.

I agree it should not crash however. So this is a second bug.

In terms of the first bug, the dimension scales issue is a difficult one. I will take a look when I get a chance.

[MT2017090]

Hi @edwardhartnett and @DennisHeimbigner ,

I work closely with Nalini and Ellen for providing NetCDF functionality in MATLAB. I wanted to check if there is any update regarding the crash issue reported by Nalini. Please let me know if you have any questions for me to clarify our workflow.

Thanks,
Rajat

[edwardhartnett]

Sorry, no update nor likely to be any progress for some time.

[pp-mo]

Just noting that this is still an issue.
Our experience mediated via Python, though.

Minimal:

import netCDF4 as nc
ds = nc.Dataset('tmp.nc', mode='w', format='NETCDF4')
ds.createVariable(varname='same_name', datatype=int, dimensions=())
ds.createDimension(dimname='same_name', size=4)
ds.close()

Resulting..

Traceback (most recent call last):
  File "minimal_hfd_error_case.py", line 5, in <module>
    ds.close()
  File "src/netCDF4/_netCDF4.pyx", line 2465, in netCDF4._netCDF4.Dataset.close
  File "src/netCDF4/_netCDF4.pyx", line 2429, in netCDF4._netCDF4.Dataset._close
  File "src/netCDF4/_netCDF4.pyx", line 1927, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

Alternatives:

• creating the dimension before the variable does not hit the error.
• creating the file with format "NETCDF3-CLASSIC " has no problem.

We here are writing a generic library which just uses this as a standard storage format.
I must say it seems to cause a nasty lack of generality (!!).
I guess we will need to worry about the names being different in these cases.

[edwardhartnett]

Sorry this is a long-known issue, deeply embedded in the core netCDF-4 code and HDF5's way of dealing with dimensions.

The work-around is to define the dimension first.