Add bitgroom support to NCZarr#2140
Closed
DennisHeimbigner wants to merge 14 commits intoUnidata:mainfrom
Closed
Conversation
re: PR Unidata#2088 Primary changes: * Add NCZarr-specific quantize function to the dispatch table. * Copy quantize code from libhdf5 * Add quantize invocation to zvar.c * Add support for _QuantizeBitgroomNumberOfSignificantDigits to ncgen. * Copy quantize test from nc_test4 to nczarr_tests. Remove some parts that are not relevant to NCZarr. Other Changes: * Break zsync.c into zsync.c (writing) and zload.c (reading). * Clean up the fill value handling (many changes) * Disable atexit() under Windows * Move ncjson to libdispatch * Add documentation of differences between netcdf-4 and NCZarr, especially WRT fill value. * Some mingw fixes * Remove some cruft * Cleanup the handling of scalars
|
This pull request introduces 1 alert when merging 2543a8e into 0e205f9 - view on LGTM.com new alerts:
|
re: PR Unidata#2140 Fix some outstanding problems with the above PR. * Minor consistency change to --enable-libxml2 * Simplify the makelib construction rule in libncxml/Makefile.am by eliminating the need for unistd and the file apis.
|
|
||
| /* Create a netcdf-4 file with two vars. Attempt | ||
| * quantization. It will work, eventually... */ | ||
| if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; |
Contributor
There was a problem hiding this comment.
@DennisHeimbigner I don't understand how this tests the nczarr quantize. Seems like you would need to change the mode flags of the nc_create() calls, right? Otherwise these tests are just creating a netCDF/HDF5 file and testing quantize works there...
Collaborator
Author
There was a problem hiding this comment.
The file_name is a URL in this case: e.g. s3://bucket/zarrfile#mode=nczarr,s3
Look at the file run_quantize.sh to see how this works.
Contributor
There was a problem hiding this comment.
So you don't need to use NC_ZARR if the filename is a URL?
Does NC_NETCDF4 have any effect in this case?
Collaborator
Author
There was a problem hiding this comment.
If the URL is properly structured, then even NC_NETCDF4 is not needed.
Collaborator
Author
|
Close to split into multiple PRs. |
DennisHeimbigner
added a commit
to DennisHeimbigner/netcdf-c
that referenced
this pull request
Jan 24, 2022
re: PR Unidata#2088 re: PR Unidata#2130 replaces: Unidata#2140 Changes: * Add NCZarr-specific quantize functions to the dispatch table. * Copy (modified) quantize code from libhdf5 to NCZarr * Add quantize invocation to zvar.c * Add support for _QuantizeBitgroomNumberOfSignificantDigits and _QuantizeGranularBitgroomNumberOfSignificantDigits to ncgen. * Modify nc_test4/tst_quantize.c to allow it to be used both for hdf5 and for nczarr. * Make dap4 properly handle quantize functions in dispatch table. * Add quantize attribute support to ncgen. Other changes: * Caught and fixed some S3 problems * Fixed some nczarr fillvalue problems. * Fixed some nczarr cache problems. * Cleanup some flaws in libdispatch/dinfermodel.c * Allow byterange requests to S3 be readable by dinfermodel.c/check_file_type * Remove the libnczarr ztracedispatch code (big change).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
re: PR #2088
Primary changes:
Other Changes: