Skip to content

Clean up the handling of chunking parameters (the -c flag) in nccopy.#1087

Merged
WardF merged 3 commits intomasterfrom
ncchunking.dmh
Jul 31, 2018
Merged

Clean up the handling of chunking parameters (the -c flag) in nccopy.#1087
WardF merged 3 commits intomasterfrom
ncchunking.dmh

Conversation

@DennisHeimbigner
Copy link
Copy Markdown
Collaborator

@DennisHeimbigner DennisHeimbigner commented Jul 27, 2018

This PR primarily addresses Issue #725.
After a long discussion, I implemented the rules at the end of that issue.
They are documented in nccopy.1.

Additionally, I added a new, per-variable, -c flag that allows
for the direct setting of the chunking parameters for a variable.
The form is
-c var:c1,c2,...ck
where var is the name of the variable (possibly a fully qualified name)
and the ci are the chunksizes for that variable. It must be the case
that the rank of the variable is k. If the new form is used as well
as the old form, then the new form overrides the old form for the
specified variable. Note that multiple occurrences of the new form
-c flag may be specified.

Misc. Other fixes

  1. Added -M option to nccopy to specify the minimum
    allowable chunksize.
  2. Removed the unused variables from bigmeta.c
    (Issue Seeing compile warnings in bigmeta.c #1079)
  3. Fixed failure of nc_test4/tst_filter.sh by using the new -M
    flag (Netcdf cmake #1) to allow filter test on a small chunk size.

After a long discussion, I implemented the rules at the end of that issue.
They are documented in nccopy.1.

Additionally, I added a new, per-variable, -c flag that allows
for the direct setting of the chunking parameters for a variable.
The form is
    -c var:c1,c2,...ck
where var is the name of the variable (possibly a fully qualified name)
and the ci are the chunksizes for that variable. It must be the case
that the rank of the variable is k. If the new form is used as well
as the old form, then the new form overrides the old form for the
specified variable. Note that multiple occurrences of the new form
-c flag may be specified.

Misc. Other fixes
1. Added -M <size> option to nccopy to specify the minimum
   allowable chunksize.
2. Removed the unused variables from bigmeta.c
   (Issue #1079)
3. Fixed failure of nc_test4/tst_filter.sh by using the new -M
   flag (#1) to allow filter test on a small chunk size.
@DennisHeimbigner DennisHeimbigner requested a review from WardF as a code owner July 27, 2018 02:17
@WardF WardF self-assigned this Jul 30, 2018
@WardF WardF modified the milestones: future, 4.7.0 Jul 30, 2018
@WardF
Copy link
Copy Markdown
Member

WardF commented Jul 30, 2018

Reviewing now then will merge. Thanks @DennisHeimbigner !

@WardF
Copy link
Copy Markdown
Member

WardF commented Jul 30, 2018

Seeing a failure on Windows in ncdump/tst_nccopy4.sh. Specifically, in the 'test that nccopy -c can chunk and unchunk files' I'm gettin ga 'Variable not found' error. This is different from the error sometimes seen on master currently.

@WardF
Copy link
Copy Markdown
Member

WardF commented Jul 30, 2018

Specifically, it is failing on nccopy -c / tmp-chunked.nc tmp-unchunked.nc

@WardF
Copy link
Copy Markdown
Member

WardF commented Jul 30, 2018

Simple fix, changing -c / to -c //, appears to fix the problem. @DennisHeimbigner any issues with this approach?

@DennisHeimbigner
Copy link
Copy Markdown
Collaborator Author

DennisHeimbigner commented Jul 30, 2018

nccopy -c / tmp-chunked.nc tmp-unchunked.nc
Interesting. It seemed to be ok with my version of visual studio.
If the string that nccopy gets internally is '/' for '//' then ok. If it is getting
'//', then not ok.
Will '/' work? (that is use quotes on the command line).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Seeing compile warnings in bigmeta.c Nccopy ignores chunk spec when input is netcdf-4 contiguous

2 participants