nccopy to compressed NetCDF-4 file is magnitudes slower on version 4.6.0+

[edhartnett]

Mike Chen mike.scchen@gmail.com via unidata.ucar.edu
7:51 AM (1 hour ago)

to netcdfgroup

Hi,
We have a storage shortage and we'd like to leverage the compression to save space.
The idea works fine for some time.
However, I was testing different NetCDF versions and happens to find that the nccopy to compressed NetCDF-4 file is magnitudes slower on version 4.6.0+.

The test method follows:
OS: CentOS 6.10
HDF5: 1.10.5
NetCDF: 4.4.1.1, 4.5.0, 4.6.2, 4.7.0, HPC-NetCDF 4.6.2.4 from NOAA
All building uses system-provided GCC suite 4.4.1.

The test is simple: nccopy a ~400MB classic NetCDF file to NetCDF-4 with:
nccopy -4 -d 1 -s input.nc output.nc4
With version v4.5.0 and lower, the process takes ~7 seconds.
HPC-NetCDF also gives similar speed.
(gzip-ing the same file takes similar time.)
But with version v4.6.2+, the copy can take ~25 minutes!

Our production system still uses 4.4.0, so the speed should be okay.
But I'd like to know more so I can plan the updates accordingly.

Mike Chen
Research Assistant / System Engineer
Dept. of Atmospheric Science
National Taiwan University

[neumannd]

Probably related to issue #1409 and closed issue #1398? Or the same?

[DennisHeimbigner]

Is the fix for #1398
in version 4.7.0 or is it still only in master?

[DennisHeimbigner]

In any case, you might try forcing the chunk sizes to see if that makes
the speed problem go away. That will tell us if #1398 is the same issue.

[DennisHeimbigner]

I am pretty sure the problem is the default chunking.

[mike-scchen]

I'm the original mail poster.
It looks like the default chunking is the blame.
I'll turn away for now and see if the NCO (our colleagues uses ncks rather than the nccopy) with newer NetCDF has this issue too.

[DennisHeimbigner]

Nccopy is actually overriding the default chunking incorrectly.
The problem appears to be with UNLIMITED dimensions.
As a temporary work-around, add the option "-c U/1"
to the nccopy command where U is the name of any unlimited dimension.

[DennisHeimbigner]

If you need a quick fix, then use that attached nccopy.c and tell me
if it solves your problem.
nccopy.zip