Allow lazy-read of netCDF-4/HDF5 files

[edhartnett]

At this point, this represents more an aspiration that a plan, but there has been some discussion (see PR #849) of how to enable lazy reads of netCDF-4 file metadata.

Files with a very large amount of metadata take a long time to load because netCDF reads all metadata at file open. For classic files, this doesn't seem to bother people much. But for netCDF-4/HDF5 files, it does. Perhaps this can be explained by the use of netCDF-4/HDF5 for some really complex and large datasets, which end up with tens of thousands of attributes, variables, dimensions, and/or groups. Or perhaps the classic formats, having all their metadata in a block at the beginning of the file, just load faster.

This has already cost us satellite users - the NPP uses netCDF-4, but the follow-on JPSS spacecraft switched to HDF5 without netCDF, due to the slow load times. I was told a similar story about a ESA satellite system by a very active netCDF user in the Netherlands. (Satellite L2 data files generally contain a very large number of attributes, some of which may be reasonably large arrays.)

One idea I suggested is to read each group only as needed. This would be pretty easy to implement I think. It would help where there's lot of groups. @DennisHeimbigner points out that this will not help with files that contain lots of vars. He indicates a known use case with a very large number of vars, all in the root group.

Well, that's another good idea all shot to hell. ;-)

In order to do lazy reads as Dennis suggests I think much of the libsrc4 code would have to be rewritten. (The good news is that with #849 soon to merge, and #856 to follow, the libsrc4 code will be a fair bit smaller than it is now.)

For example, if we open a file and read nothing, and then the user does an nc_inq(), we need to find out how many variables there are. In the current code, we count our list, because we have already read them. In the lazy-read code, we would rsee if there's a way we can get the numbers we need without reading every variable's metadata. That is probably possible in HDF5, but not how the code is currently written.

Handling dimensions in a lazy-read is going to be particularly tricky. They may be in different groups from a variable. So if the user opens a file and does a nc_inq_var() on a var deep in the group structure, we will have to have code smart enough to find all the dimensions in whatever group they are in. All this information is in the HDF5 file, but the code to read it and use it properly remains to be written.

[DennisHeimbigner]

I should note that the metadata read speedup that I already have
still reads all the metadata at one time (i.e., not lazily). It gets
its speedup from pervasive use of hashing. I just put up a PR
for the initial integration of this speed up. Waiting for the netcdf-4
rename stuff to be completed before I do the final step.

[DennisHeimbigner]

Another issue I discovered was that if we do lazy eval, then the various integer
ids (dimid, varid, etc) will probably change depending on the order in which
the meta-data elements are read. Specifically, any user code that assumes anything
about the ids is potentially going to fail. Issue #851 is relevant here.

[edhartnett]

Maybe an easy way to do this would be a special attribute that contains a table of metadata. So instead of reading each object in a lazy way, we read the table. Then all our data structure code continues to work.

Files that do not have the table of metadata can not be read quickly. But they could add the table and then quick reads would work on the file.

We could store the metadata in various ways, like a file level attribute that lists all the dims in the file, a group-level table that lists all vars in a group, etc. Or we can package it up as one big metadata element (perhaps just store the output of ncdump -h as a char attribute).

[DennisHeimbigner]

Not a bad idea. The problem is that if we have large amounts of metadata, then
that extra attribute itself becomes very large. We would need some very
good encoding method.

[edhartnett]

Even a very large ncgen -h output, stored as a char array, would read hugely faster than opening all the objects in HDF5 and querying them each. The whole thing would be 1 disk access, instead of tens or hundreds of thousands. Essentially it would be O(1) instead of O(N), right?

The more I think about it, the more I like it. Instead of re-writing all the libsrc4 code, it all will still work just fine.

[edhartnett]

Would it be at all possible/practical to put the existing ncgen/ncdump parsing code into the library, so that we could literally use the output of ncdump -h? We could zip it to make it smaller.

Or is that just nutty?

[DennisHeimbigner]

That is just nutty :-)
It also occurs to me that we need to allow for non-lazy (bulk) read of the
metadata when the caller is e.g. something like ncdump.

[edhartnett]

Let us say we have an optional text attribute in the root group with a protected name which contains some compressed, easy to parse representation of the metadata in the file.

Each time any metadata is added or altered, the attribute is updated so it is always correct.

We add a function nc_put_metadata() which does the bulk read of metadata and (re-)creates this attribute.

Then why wouldn't ncdump be able to use the metadata too? Seems like that would be useful and a lot faster.

[DennisHeimbigner]

But this does not solve anything; innstead of reading all the metadata at once
as we do now, we are reading the equivalent in that single attribute. Plus
we have the overhead of parsing it, which is potentially a big cost. This
reminds me of the HDF-EOS solution.

[edhartnett]

It does remind me of the HDF-EOS thing too, and that is certainly not a compliment to the idea. So perhaps not the way to go.

[DennisHeimbigner]

Before this goes much further, we need to have
a software architecture and design document to
codfy the API and the critical implementation
details. Something in markdown like a cross between
this https://github.com/Unidata/netcdf-c/blob/master/docs/filters.md
and this https://support.hdfgroup.org/HDF5/doc/Advanced/DynamicallyLoadedFilters/HDF5DynamicallyLoadedFilters.pdf

[edhartnett]

Having worked with this code a bunch, I do see an optimization which would be pretty easy, and could result in significant speed-up of file opens.

In NC_VAR_INFO_T we we add a field atts_read.

When opening the file we don't read any variable atts.

When the user asks for a variable att, we check atts_read. If zero, we read all atts for that variable and set it to 1.

Anyone have any thoughts or opinions on this approach?

[Dave-Allured]

Developers, have you considered no attribute cacheing as an alternative strategy to cacheing and lazy reads, in the case of netCDF4? HDF5 offers efficient attribute access by name and by index number. It seems to me that a thin wrapper approach for attributes would result in less memory demand and simpler library code, with little impact on performance in most real cases.

This approach may also be valid for other alternative storage formats such as cloud.