Nccopy ignores chunk spec when input is netcdf-4 contiguous

[Dave-Allured]

Environment

Linux and Mac, 64-bit
Version tested: nccopy with netcdf-C 4.4.1.1, hdf5-1.10.1

Summary

When using nccopy to convert a netcdf-4 file from contiguous to chunked, the chunk spec on the command line is ignored, and the output file contains invented chunk sizes. IMO, nccopy is not working as advertised in this case.

Remarkably, when the input file is chunked rather than contiguous, the command line chunk spec is respected. (Example not shown.)

Steps to reproduce

Test input file, 5.2 Mbytes: test31.contig.nc.gz

Run this command:

nccopy -d1 -c time/1,lat/180,lon/180 test31.contig.nc test33.chunked.nc

Input file header:

netcdf test31.contig {
dimensions:
	time = 40 ;
	lat = 180 ;
	lon = 180 ;
variables:
	float x(time, lat, lon) ;
		x:_Storage = "contiguous" ;
		x:_Endianness = "little" ;

// global attributes:
		:_NCProperties = "version=1|netcdflibversion=4.4.1.1|hdf5libversion=1.10.1" ;
		:_SuperblockVersion = 0 ;
		:_IsNetcdf4 = 1 ;
		:_Format = "netCDF-4" ;
}

Expected output chunk sizes:

		x:_ChunkSizes = 1, 180, 180 ;

Actual result with unexpected chunk sizes:

dimensions:
	time = 40 ;
	lat = 180 ;
	lon = 180 ;
variables:
	float x(time, lat, lon) ;
		x:_Storage = "chunked" ;
		x:_ChunkSizes = 20, 90, 90 ;
		x:_DeflateLevel = 1 ;
		x:_Endianness = "little" ;

// global attributes:
		:_NCProperties = "version=1|netcdflibversion=4.4.1.1|hdf5libversion=1.10.1" ;
		:_SuperblockVersion = 0 ;
		:_IsNetcdf4 = 1 ;
		:_Format = "netCDF-4" ;
}

[WardF]

Thanks the report, taking a look now.

[WardF]

Duplicated and am observing the same behavior. Trying to narrow down where this is happening. Also, seeing this wil hdf5libversion=1.8.19 and the master branch.

[Dave-Allured]

@WardF, thanks for trying an alternate hdf5 version. This seems to me like a relatively simple bug in nccopy code, rather than a deep support library problem.

This might be related to unresolved #391, "Using nccopy, setting deflate level to 0 ignores chunking specification".

You can mark this as low priority as far as I am concerned.

[WardF]

The alternate HDF5 version was incidental; it's what was on hand in my dev environment, and I made a note of it so that I wouldn't forget. Thanks! Hoping it is as simple as it seems, going to try to knock it out in short order.

[adrfantini]

I'm trying to re-chunk some files that have 1, 46, 113 chunking into 721,1,1 and I also found that -c is not working as intended, with the output chunks being 121, 47, 113, our roughly 1/6 of the variable size in each dimension (which is the usual default).
I'm using version 4.6.3.

[DennisHeimbigner]

The primary problem is that the code in nccopy.c is incorrect.
At about line 673, it is testing for contig==1 to decide if the output
should be contigous or chunked. As near as I can tell, the value of contiq
is the one for the input file, so if the input file is contiquous, it forces the output file
to also be contiguous.
The mystery is the the output file actually is shown to be chunked, so
somewhere in the netcdf-c library, the contiguous flag is being changed
to be chunking. It is probably the case that with this change, default chunking
is being used.
Ok, the reason that chunking is being forced is that the output file has
deflation set.
So, the fix would be to change the condition at line 673 to properly
use contig only when no chunking was specified and the input file is
contiguous and no output compression is set.

[DennisHeimbigner]

Temporary workaround:

1. copy the contiguous file with -d1 to force the output file to be chunked. e.g.

nccopy -d1 test31.contig.nc tmp.nc

2. since the tmp file is chunked, specifying new chunking will work, so
   copy again to rechunk as desired. e.g.

nccopy -d1 -c time/1,lat/180,lon/180 tmp.nc test33.chunked.nc