Performance change with NC_SHARE

[barronh]

As documented in nc_open, the NC_SHARE mode often enhances read speeds for NetCDF3 files with many (e.g. 100+) variables with large dimension sizes (t=UNLIMITED, z=50, y=300, x=500). This has been true for me on many systems with different versions of the netcdf library. My understanding is that this was related to buffering of variables along the unlimited dimension that was inefficient.

In a recent test, I found that I could not reproduce the speed enhancements with v4.4.1. Has this speed enhancement changed or can someone describe a test case known to reproduce the enhancements?

• environmental information (i.e. Operating System, compiler info, java version, python version, etc.)

  Operating System: Red Hat Enterprise Linux
  CPE OS Name: cpe:/o:redhat:enterprise_linux:7.8:GA:server
  Kernel: Linux 3.10.0-1062.12.1.el7.x86_64
  Architecture: x86-64

  Tested in python and nco via 4.9.4-alpha02..

• a description of the issue with the steps needed to reproduce it

The commands below will reproduce the similar performance (not faster) using the netdf4-python bindings on a linux system using bash.

cat << EOF > test.py
import sys
import netCDF4 as ncf
import numpy as np
import os

writemode = sys.argv[1]
readmode = sys.argv[2]
keys = ['test%03d' % i for i in range(100)]
replications = 1

for i in range(replications):
    os.remove('test.nc')

    testf = ncf.Dataset('test.nc', format='NETCDF3_CLASSIC', mode=writemode)
    testf.createDimension('t', None)
    testf.createDimension('z', 10)
    testf.createDimension('y', 100)
    testf.createDimension('x', 100)
    getlen = lambda k: len(testf.dimensions[k])
    outshape = (24, getlen('z'), getlen('y'), getlen('x'))
    for key in keys:
        var = testf.createVariable(key, 'f', ('t', 'z', 'y', 'x'))
        var.units = 'none'
        var.long_name = key

    for key in keys:
        print(key, flush=True, end='.')
        vals = np.random.random(size=np.prod(outshape))
        var = testf.variables[key]
        var[:outshape[0]] = vals.reshape(outshape)

    print()
    testf.close()

testf = ncf.Dataset('test.nc', mode=readmode)
for key in keys:
    a = testf.variables[key][:]
    b = a + 2
    del a, b
EOF
time python test.py ws rs
time python test.py w r

Related discussion https://sourceforge.net/p/nco/discussion/9829/thread/85b29e80c7/?limit=25

[DennisHeimbigner]

My suspicion is that the issue is not NC_SHARE directly; as near as I can tell
that is the same between 4.4.1 and 4.7.4.
Rather,I think we (me?) mistakenly removed the LOCKNUMREC code
and that is the cause of the slowdown.

[DennisHeimbigner]

THe LOCKNUMREC code causes the use of the shmem package.
Is there any way for you to tell if your faster version is using shmem?

[barronh]

I’m on vacation, so can’t check that until next week (VPN etc). I’m not familiar with shmem . What should I do to check when I get back?

[DennisHeimbigner]

It appears that shmem is CRAY specific, which I assume you are not using.
So I am not sure what is going on. Do use use nc_set_base_pe() or nc_inq_base_pe( )?