nc_put_var_double execution time increases in subsequent runs when a variable is written with chunking and compression

[abhibaruah]

NetCDF version: 4.9.1
HDF5 version: 1.10.10
Platform: Windows 11

I have a repro code where I am creating a file, creating an NC_DOUBLE variable and writing a 2000 x 512 x 512 variable to it (with chunk sizes of 20 x 10 x 10 and deflate compression of level 5). I run this code in a for loop of 10 iterations and at the end of every loop, delete the created nc file.

I time the call to nc_put_var_double and I observe that its execution time increases for every subsequent iteration. This happens only on Windows and only when I write using both chunking and compression.
I see similar behavior with 'nc_get_var_int' as well.

Here is the output from the program:

In main
Index: 0
In execution
### Execution time is 26.4505
Index: 1
In execution
### Execution time is 30.2897
Index: 2
In execution
### Execution time is 40.5599
Index: 3
In execution
### Execution time is 48.5356
Index: 4
In execution
### Execution time is 51.8821
Index: 5
In execution
### Execution time is 55.844
Index: 6
In execution
### Execution time is 60.3397
Index: 7
In execution
### Execution time is 62.5748
Index: 8
In execution
### Execution time is 67.3835
Index: 9
In execution
### Execution time is 72.4015

Here is my reproduction code:

#include <stdio.h>
#include <string>
#include <netcdf.h>
#include <chrono>
#include <iostream>

/* NetCDF file names */
#define FILE_NAME "sample_xyz.nc"


/* Test with 3D data */
#define NDIMS 3
#define NX 2000
#define NY 512
#define NZ 512

#define CHUNKX 20
#define CHUNKY 10
#define CHUNKZ 10

#define ERRCODE 2
#define ERR(e) {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

using namespace std;
using namespace std::chrono;

void execution(double *arr)
{
    std::cout << "In execution" << std::endl;
    int status;
    int ncid;
    int varid;
    int dimids[NDIMS];
    int x_dimid, y_dimid, z_dimid;
    int retval;
    const size_t chunksize[NDIMS] = { CHUNKX, CHUNKY, CHUNKZ };

    

    /* Create a NetCDF4 file. */
    if ((retval = nc_create(FILE_NAME, NC_NETCDF4, &ncid)))
        ERR(retval);
    if (retval != NC_NOERR) {
        printf("Error creating .NC file.\n");
    }

    /* Define dimensions */
    if ((retval = nc_def_dim(ncid, "dim_512", NY, &y_dimid)))
        ERR(retval);
    if ((retval = nc_def_dim(ncid, "dim_2000", NX, &x_dimid)))
        ERR(retval);

    dimids[0] = x_dimid;
    dimids[1] = y_dimid;
    dimids[2] = y_dimid;

    /* Define the variable */
    if ((retval = nc_def_var(ncid, "data", NC_DOUBLE, NDIMS,
        dimids, &varid)))
        ERR(retval);
    
    if ((retval = nc_def_var_chunking(ncid, varid, NC_CHUNKED, chunksize)))
        ERR(retval);
        
    
    if ((retval = nc_def_var_deflate(ncid, varid, 0, 1, 5)))
        ERR(retval);
    

    std::chrono::time_point<std::chrono::high_resolution_clock> timestart, timeend;
    timestart = std::chrono::high_resolution_clock::now();
    retval = nc_put_var_double(ncid, varid, arr);
    timeend = std::chrono::high_resolution_clock::now();
    std::chrono::duration<double> duration = timeend - timestart;
    double durationInSeconds = duration.count();
    std::cout << "### Execution time is " << durationInSeconds << std::endl;

    /* close the file */
    status = nc_close(ncid);

    remove(FILE_NAME);

}

int main() {

    // Dynamically allocate memory for the 3D array
    double* arr = new double[NX * NY * NZ];
    int index = 0;
    // Traverse the 3D array
    for (int i = 0; i < NX; i++) {
        for (int j = 0; j < NY; j++) {
            for (int k = 0; k < NZ; k++) {
                *(arr + index) = 5;
                index++;

            }
        }
    }

    std::cout << "In main" << std::endl;

    for (int i = 0; i < 10; i++)
    {
        std::cout << "Index: " << i << std::endl;
        execution(arr);
    }

delete[] arr;
    
}

Let me know if I am doing anything wrong or if any additional information is needed from my side.

[abhibaruah]

Wanted to mention here that I could reproduce something similar to this in HDF5 as well.
This might well be an HDF5 issue, instead of netCDF.

[WardF]

Thanks for your patience; I suspect this is an HDF5 issue (similar to other recent netCDF4 speed issues only observed on Windows), but will see if I can replicate!

[captainkirk99]

We have added a regression/diagnostic test to track this issue: nc_test4/tst_chunked_deflate_perf.c.

The test reproduces the exact scenario from the original report — repeated cycles of create/write/close/delete on a 3D chunked+deflate-compressed NC_DOUBLE variable — and prints the CPU time for each nc_put_var_double call across 10 iterations. It uses clock()/CLOCKS_PER_SEC for portable timing on all platforms including Windows MSVC (which lacks sys/time.h).

The test is wired into the standard nc_test4 test suite (both CMake and Autotools), so it will run automatically in CI on Windows. It never fails on timing alone — it only fails if a netCDF API call returns an error — but the printed output in CI logs will make it immediately visible whether write times are stable or still growing across iterations.

This will let us confirm whether the issue has been resolved by changes made since 4.9.1, or whether it is still present and needs further investigation.