nc_open(<file schema>) returns NC_NOERR on non-NCZarr stores

[czender]

Hi Gurus,

I invoke nc_open() with a file:// schema to determine whether a given
directory contains a valid NCZarr store.
It appears that nc_open(ncz_fl,NC_NOWRITE,&id) always succeeds if ncz_fl
is a valid path that already exists, regardless of whether
the existing path leads to an NCZarr store, a non-netCDF file, an empty directory, or a
directory full of non-NCZarr stuff. I am using netCDF 4.9.2, current
development snapshot on MacOS.

For example, the nc_open(ncz_fl,NC_NOWRITE,&id) returns NC_NOERR if

ncz_fl == file:///Users/zender/ncz_dnd/foo#mode=nczarr,file
and the directory (or file) /Users/zender/ncz_dnd/foo exists. Full stop.
nc_open(ncz_fl) succeeds when 1. foo contains an NCZarr store (yay!),
2. when foo exists but is a completely empty directory (boo!),
and 3. when foo is a directory filled with completely non-NCZarr contents (boo!),
and 4. when foo is any file (including non-netCDF files).
In the latter three cases I expect nc_open() to fail with an error such as

#define NC_ENOTNC (-51) /**< Not a netcdf file */

or

#define NC_ENOOBJECT (-141) /**< Some object not found */

Is this the observed behavior of nc_open() intentional? If so, why?
I thought nc_open() would return NC_NOERR only for existing netCDF files,
as implied by the nc_open() documentation.
Or am I missing something NCZarr-y?

Many thanks,
Charlie

[DennisHeimbigner]

Off hand, this looks like an error. I will investigate.

[czender]

Thanks for looking into this, Dennis. I should perhaps also mention that nc_close(id) successfully closes (i.e., returns NC_NOERR) the handles returned by nc_open(ncz_fl,NC_NOWRITE,&id) on non-NCZarr stores. Not sure whether this is the same underlying issue, or if I'm just losing it.

[DennisHeimbigner]

I can sort-of fix this, but there is a cost.
The Zarr format is highly decentralized.
There is no easily testable magic number or superblock to look for.
In effect the only way to see if a directory is Zarr is to successfully read it.
So, the only other way to see if some directory looks like a Zarr file is
to search the file tree looking for occurrences of e.g. .zarray objects.
This could, in principle, be costly.

[czender]

That makes sense. What do you have in mind for a "sort-of" fix?

I do not fully understand the Zarr format, yet It seems to me that the path component of the scheme://host:port/path?query#fragment specification must lead to either or both a .zgroup and a .zattrs file for nc_open() to return NC_NOERR.
Is that always a valid assumption? If so, that could constitute a minimal condition for the library to check.
Minimal in that it involves only one or two stat() system calls to check whether that .zattrs and possible .zarray files exist.
That seems like a small cost to pay for a more accurate return code than nc_open() (and nc_close()?) currently provide.

Figuring out whether an NCZarr path lead to a valid NCZarr store is a big deal for NCO. NCO needs to know this before it decides to overwrite or move output files around. The wrong return value from nc_open() could means the difference between deleting or overwriting files and entire directories. I can implement the above minimal procedure in NCO as a safeguard, though I'd prefer the netCDF library to do the check. Thoughts?

[DennisHeimbigner]

I do not fully understand the Zarr format, yet It seems to me that the path component of the scheme://host:port/path?query#fragment specification must lead to either or both a .zgroup and a .zattrs file for nc_open() to return NC_NOERR.

Actually no.

A path of the form

/a/b/c/.../.zarray

constitutes a legal Zarr dataset. Neither .zgroup nor .zattrs need exist and the depth of the path (/a/b/...)
can be arbitrarily long. So, the only way to test is to do a breadth or depth first search of the whole
file tree looking for an occurrence of a .zarray object. On average, I would hope that the depth is not
great, so the cost of walking the tree is not overwhelming, but it could be.