Use the built-in HDF5 byte-range reader, if available.#1849
Use the built-in HDF5 byte-range reader, if available.#1849WardF merged 1 commit intoUnidata:masterfrom
Conversation
re: Issue Unidata#1848 The existing Virtual File Driver built to support byte-range read-only file access is quite old. It turns out to be extremely slow (reason unknown at the moment). Starting with HDF5 1.10.6, the HDF5 library has its own version of such a file driver. The HDF5 developers have better knowledge about building such a driver and what incantations are needed to get good performance. This PR modifies the byte-range code in hdf5open.c so that if the HDF5 file driver is available, then it is used in preference to the one written by the Netcdf group. Misc. Other Changes: 1. Moved all of nc4print code to ncdump to keep appveyor quiet.
WardF
left a comment
WardF
left a comment
There was a problem hiding this comment.
Looks good, thank you Dennis! Running the edge case tests and will get merged when they pass. Thanks!
|
So to be clear...that driver from HDF5 only works on S3, right? So we’ll still see the pathological behavior on e.g. THREDDS servers? Also, I’m not sure you really want to rely on that HADCrut data that’s sitting in a file on S3. @rsignell-usgs can correct me if I’m wrong, but the URL even says testing so I’m not sure how long it will live there. |
|
Note also that the HDF5 driver will work against any server that supports |
|
re: HADCrut data that’s sitting in a file on S3. |
|
Well, at this one points to a server we control: https://thredds-test.unidata.ucar.edu/thredds/fileServer/casestudies/harvey/goes16/CONUS/Channel03/20170827/GOES16_CONUS_20170827_001717_0.87_1km_33.3N_91.4W.nc4 |
3 participants
re: Issue #1848
The existing Virtual File Driver built to support byte-range
read-only file access is quite old. It turns out to be extremely
slow (reason unknown at the moment).
Starting with HDF5 1.10.6, the HDF5 library has its own version
of such a file driver. The HDF5 developers have better knowledge
about building such a driver and what incantations are needed to
get good performance.
This PR modifies the byte-range code in hdf5open.c so
that if the HDF5 file driver is available, then it is used
in preference to the one written by the Netcdf group.
Note also that the HDF5 driver will work against any server that supports
byte-range. See the test case nc_test/test_byterange.sh.
Misc. Other Changes: