new ncdump failure to print NCZarr variable

[czender]

Hi, After building today's netcdf-c snapshot that includes PR #2492, I noticed that ncdump and ncks now fail to print some NCZarr variables that they previously worked well with. Here is the symptom:

zender@spectral:~/nco/data$ ncgen -k netCDF-4 -b -o in_zarr4.nc in_zarr4.cdl
zender@spectral:~/nco/data$ ncgen -k netCDF-4 -lb -o "file://in_zarr4#mode=nczarr,file" in_zarr4.cdl 
zender@spectral:~/nco/data$ ncdump "file://in_zarr4#mode=nczarr,file"
....
normal output
....
	float ppc_bgr(time) ;
		ppc_bgr:long_name = "Precision-Preserving Compression, bigger numbers" ;
		ppc_bgr:purpose = "test --ppc switches" ;
		ppc_bgr:original_values = "1234567890e20,1234567890e19,1234567890e18,1234567890e17,1234567890e16,1234567890e15,1234567890e14,1234567890e13,1234567890e12,1234567890e11" ;
NetCDF: Invalid argument
Location: file ncdump.c; line 1762
zender@spectral:~/nco/data$ 

Note that ncdump and ncks continue to print the in_zarr4.nc file correctly.
The problem can be demonstrated by trying to dump the NCZarr version of the attached in_zarr4.cdl.txt

FYI, ncks fails to dump the Zarr file with this error:
zender@spectral:~/nco/data$ ncks "file://in_zarr4#mode=nczarr,file"

ERROR: nco_inq_var() fails for variable "ppc_bgr"
ERROR NC_EINVAL Invalid argument
HINT: NC_EINVAL errors can occur for many reasons. Three common ones are described here. 1. When invalid chunking directives are given, e.g., to unchunk a record variable or to chunk a variable with an invalid size (such as zero or larger than a fixed dimension). This is the most frequent/likely cause in our experience. 2. When HDF4-enabled NCO attempts to directly write to an HDF4 (not HDF5) file. Not all HDF5 calls are available with HDF4, and the operators ncrename and ncatted are liklier to trigger this problem. We are working to solve this (fxm TODO nco1104). Please let us know if it affects you. For now the workaround is to convert the HDF4 file to netCDF4 first (e.g., with ncks -4 in.hdf out.nc), then use ncrename or ncatted. 3. When NCO operators attempt to open netCDF4 files using the diskless option, usually invoked with --diskless_all, --ram_all, or --open_ram. Is your input file netCDF4 format?  (http://nco.sf.net/nco.html#fmt_inq shows how to tell.) If so then omitting the diskless option may solve this problem. 4. When floating-point quantization algorithms (e.g., BitGroom) are applied to integer-containing variables.
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_inq_var()
nco_err_exit(): ERROR Error code is -36. Translation into English with nc_strerror(-36) is "NetCDF: Invalid argument"
nco_err_exit(): ERROR NCO will now exit with system call abort()
Abort trap: 6
zender@spectral:~/nco/data$ 

[WardF]

Thanks @czender, I'll take a look. @DennisHeimbigner, does anything leap out at you immediately?

[WardF]

@czender I'm having problems finding a version of netCDF-C that worked with the process outlined above. Using v4.9.0 gives me errors when trying to generate the zarr data, or when trying to read the zarr data created by the latest snapshot. Can you point me to the version that works from you, and I can start from there? I'll also keep poking through to try to figure out what's going on.

[czender]

@WardF thanks for taking a look. Everything reported above was done with the latest (today's) snapshot of netcdf-c 4.9.1-development. I just repeated the procedure, and obtain the same results as above. 4.9.0 will fail to generate the Zarr file because strings were not supported until PR #2492 was merged a few days ago. Please retry with the latest main trunk.

[WardF]

@czender Thanks! I'm digging through this and am narrowing down the 'where', and then can start to figure out the why.

I've run ncdump using a build with logging turned on, and passing -L10. I'm seeing the following:

		ppc_bgr:original_values = "1234567890e20,1234567890e19,1234567890e18,1234567890e17,1234567890e16,1234567890e15,1234567890e14,1234567890e13,1234567890e12,1234567890e11" ;
		NCZ_inq_var_all: ncid 0x10000 varid 94
				ncz_find_grp_var_att: ncid 65536 varid 94 attnum 0 use_name 0
		nc_inq_atomic_type: typeid 2
				nc4_get_typelen_mem xtype: 2
		nc_inq_atomic_type: typeid 2
			nc4_att_list_add: name long_name 
		nc_inq_atomic_type: typeid 2
				nc4_get_typelen_mem xtype: 2
		nc_inq_atomic_type: typeid 2
			nc4_att_list_add: name purpose 
		nc_inq_atomic_type: typeid 2
NetCDF: Invalid argument
Location: file /home/wfisher/Desktop/gitprojects/netcdf-c/ncdump/ncdump.c; line 1762
					NC4_clear_provenance
					NCZ_clear_provenance

Still working on this; we are hoping to have a 4.9.1 maintenance release out ASAP, and this is the sort of Zarr-related issue we want to get fixed for it. Thanks!

[DennisHeimbigner]

I need to remember to indicate that I am working on an Issue.
In any case, I am testing a fix for this now.

[WardF]

Thanks @DennisHeimbigner I'll pivot back towards other issues/esupport.

[DennisHeimbigner]

Hopefully fixed by #2497