Allow users to specify data alignment

[hmaarrfk]

In the mailing list I raised the question about data being aligned.

I provided a small code example that showed how data could be misaligned (code in python with a recent version of netcdf4-python)

Code Example

import xarray as xr
import numpy as np
import netCDF4
from pathlib import Path

basic_filename = "basic_file_netcdf4.nc"
if Path(basic_filename).exists():
    Path(basic_filename).unlink()

dataset = xr.DataArray(
    np.zeros((3072, 3072), dtype='uint8'),
    dims=("y", "x"),
    coords={
        "y": np.arange(3072, dtype=int),
        "x": np.arange(3072, dtype=int),
    },
    name='images').to_dataset()

dataset.to_netcdf(basic_filename, format="NETCDF4", engine="netcdf4")

import h5py
h5file = h5py.File(basic_filename)
h5dataset = h5file.get("images")
offset = h5dataset.id.get_offset()
print(offset % 4096)
print(offset % 2048)
print(offset % 1024)
print(offset % 512)
print(offset % 128)
print(offset % 64)

"""
3206
1158
134
134
6
6
"""

We came to the point where we determined that he user should set the File Access Property at Creation time, near:

netcdf-c/libhdf5/hdf5open.c

Line 772 in 988e771

if ((fapl_id = H5Pcreate(H5P_FILE_ACCESS)) < 0)

I'm opening this as an issue to keep track of the conversation on github.

cc: @edhartnett

Mailing list link: https://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2022/msg00000.html

[hmaarrfk]

Ed asked:

Is this change always on? Or does the user turn it on and off?

I believe this change is "always on" from my understanding, at file opening time???

Data alignment
    Sometimes file access is faster if certain data elements are aligned in a specific manner. 
    This can be controlled by setting alignment properties via the H5Pset_alignment function. 
    There are two values involved:

        A threshhold value
        An alignment interval

    Any allocation request at least as large as the threshold will be aligned on an address that 
    is a multiple of the alignment interval.

I presume you could change the alignment by closing and opening the file. It seems silly that it is a property for the file as a whole at opening time, but I understand that memory management is tricky and they might be trying to avoid a corner case.

My code for patching h5netcdf is the following, and should be quite similar

from h5py._hl.files import make_fapl as _make_fapl
import h5py
from functools import wraps

@wraps(_make_fapl)
def make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0, locking,
              page_buf_size, min_meta_keep, min_raw_keep, **kwds):
    fapl = _make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0, locking,
                      page_buf_size, min_meta_keep, min_raw_keep, **kwds)
    fapl.set_alignment(1024 * 1024 - 4096, 4096)
    return fapl

h5py._hl.files.make_fapl = make_fapl

fapl is called at creation / opening time:
https://github.com/h5py/h5py/blob/9c9033849e647d4254923bf9b50fcdedc1a320b3/h5py/_hl/files.py#L502

Cross referencing the post on h5py: h5py/h5py#2034

[hmaarrfk]

Just writing down other notes: it seems that there is some interplay between the file space strategy too:
https://docs.hdfgroup.org/hdf5/v1_12/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a

[hmaarrfk]

cc @edwardhartnett i realize you have two accounts.

[DennisHeimbigner]

I am going to try to set the alignment on an open file. Can it be detected by h5dump?

[hmaarrfk]

I don't think the data alignment is persistent.

It is part of the File Access Property List, not File Creation Property List.

I've been:
Using the following h5py code for chunked data

import h5py
h5file = h5py.File(file_saved)
h5dataset = h5file.get("data")

for i in range(h5dataset.id.get_num_chunks()):
    print(h5dataset.id.get_chunk_info(i).byte_offset % 4096)

and for non chunked data

import h5py
h5file = h5py.File(file_saved)
h5dataset = h5file.get("data")
offset = h5dataset.id.get_offset() % 4096
print(offset)

[hmaarrfk]

The datasets are generated by:

1. Using h5netcdf with a similar hack.
2. Data that is larger than the threshold is created.
3. Confirmed that it is aligned.

[hmaarrfk]

I think you can use:

h5ls --verbose --address YOUR_NC_FILE.nc

[hmaarrfk]

In light of this whole discussion (especially around the backward compatible API), there may be a slew of other options that would help those of us that are performance minded. Things like the cache size, and other access property list items.

It might be a good idea to think of how to allow the users to have the most flexibility for future tweaks.

[Dave-Allured]

How about this? Add a new function to the netCDF API, to be optionally called BEFORE opening a file. A new function does not harm backward API compatibility. This is for the C API, similar for Python. Tweak the name as you see fit, I don't care.

nc_set_alignment (threshold, alignment)

This would simply change the default behavior of nc_create and nc_open, and invoke H5Pset_alignment under the hood, at the appropriate moment before H5Fcreate or H5Fopen.

Because there is no existing file handle or FAPL at the user level before the netCDF open calls, the simplest implementation would be for the alignment settings to be global in some sense, and persist for multiple file opens, unless changed though another call.

My understanding is that changing object alignment like this, should have no impact at all on backward format compatibility.

[DennisHeimbigner]

That is a really good solution, thanks. And nothing is lost because the user can set and reset
the global flag before opening a file. so it would be possible to effectively make it per-file
if necessary.

Looking ahead, and as has been noted, there are other properties that may need to be set
via either an access fapl or a create fcpl. So we can either make this a one-off function,
or can generalize it to set other kinds of properties. I suppose it is ironic that we
are in-effect re-inventing the HDF5 properties mechanism :-)

[DennisHeimbigner]

In order to implement Greg's solution. I propose adding this function
to the netcdf API:

nc_set_hdf5_alignment(int threshold, int alignment)

[hmaarrfk]

I see alot of trouble with "global settings".

The main challenge is that you want this to happen at the "application" level, but the definition of an "application" can change quite a bit.

It starts to somewhat start a conflict between different libraries, each optimizing for a different "default" usecase. In python at least, where "loading" a library is not very different from "executing" the script, it may start to conflict with the import order of things. Things might get lazy loaded in different order causing conflict between settings.

Furthermore, global settings also start to require "an access control mechanism", locks, coordination, or other.

If you want like to have global settings to start, I think that is a good way to begin things. However, would it be possible to request have an explicit API too? I'm happy to count it a "future" addition, but so long as it is understood that the "global settings" are not always desirable.

[edwardhartnett]

@hmaarrfk I do not understand your point about global settings. Actually we already have some global settings:

/* Set the default nc_create format to NC_FORMAT_CLASSIC, NC_FORMAT_64BIT,
 * NC_FORMAT_CDF5, NC_FORMAT_NETCDF4, or NC_FORMAT_NETCDF4_CLASSIC */
EXTERNL int
nc_set_default_format(int format, int *old_formatp);

/* Set the cache size, nelems, and preemption policy. */
EXTERNL int
nc_set_chunk_cache(size_t size, size_t nelems, float preemption);

/* Get the cache size, nelems, and preemption policy. */
EXTERNL int
nc_get_chunk_cache(size_t *sizep, size_t *nelemsp, float *preemptionp);

So this function would act the same way and should be named similarly. I would suggest not mentioning "hdf5" in the function name, since it's certainly conceivable that some other backend might one day allow control of alignment. (Does Zarr?)

I would suggest simply:
nc_set_alignment()/nc_get_alignment().

Just as with, for example nc_set_chunk_cache(), calling nc_set_alignment() will apply to all subsequent file opens.

[hmaarrfk]

It is very conceivable to me, that one application may be using two netcdf files with vastly different optimization settings.

One might be on a network drive, the other on a local drive.

As such, global settings like this typically cause a clash. It would be nice to be able to override them, even if the initial API is "global-first".