Allow user to turn on zlib, shuffle, and/or fletcher32 filters with parallel I/O for HDF5-1.10.2+#1582
Merged
WardF merged 29 commits intoUnidata:masterfrom Jan 13, 2020
NOAA-GSD:ejh_parallel_zlib
Merged
Allow user to turn on zlib, shuffle, and/or fletcher32 filters with parallel I/O for HDF5-1.10.2+#1582WardF merged 29 commits intoUnidata:masterfrom NOAA-GSD:ejh_parallel_zlib
WardF merged 29 commits intoUnidata:masterfrom
NOAA-GSD:ejh_parallel_zlib
Conversation
…ffle, or fletcher32 filters are in use
gsjaardema
reviewed
Dec 20, 2019
Contributor
Author
|
@WardF normally I would like you to merge my PRs in the order I write them, but in this case it would be great to skip ahead and merge this one. It will make a great holiday gift to NOAA and other data producers around the world. ;-) Happy holidays to all at Unidata and throughout the netCDF community! ;-) |
Contributor
|
@WardF can we get this merged soon please? The NOAA GFS people will build it and test to ensure this problem is resolved. Then they will know that, starting with the next release of netcdf-c, they will be able to do parallel writes of compressed data... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1580
This is the complete PR, with documentation changes, and Cmake and autotools support for determining if the user has access to HDF5 1.10.2.
If the user does not have 1.10.2, they will continue to see NC_EINVAL if they try to set shuffle, zlib, or fletch32 filters. If they have HDF5-1.10.2 or better, then instead of returning an error, these filters will work.
I have change documentation everywhere to reflect and explain the changes.
NOAA is particularly eager for this feature, but many other large data producers will welcome it.