Allow users to specify the use of the DIRECT driver

[hmaarrfk]

This is somewhat a followup to: #2177

In order to achieve maximum performance on writing large data blocks, I had to do a few things:

1. Ensure the datasets were aligned in the user's ram (out of scope for netcdf-c)
2. Ensure the destination in the file was aligned to a block : Allow users to specify data alignment #2177
3. Use the direct driver to bypass many caching mechanism in the operating system.

With #2177 approaching being fixed #2206

I'm hoping that we can provide users to use different virtual file drivers.

The DIRECT driver fapl documention is:
https://support.hdfgroup.org/HDF5/doc/RM/H5P/H5Pset_fapl_direct.htm

Without the DIRECT driver, on linux, I'm limited to about 1GB/s.
With the DIRECT driver, on linux, I can reach read/write speeds that are limited by the PCIe Interface and the SSD reaching up to 3GB/s.

A secondary effect is that it bypasses the operating system file cache, so the operating system doesn't need to spend time evicting it (or other files) during large disk reads/writes.

Thank you for considering this feature.

I presume that the changes would have to provide a path similar to

        if (H5Pset_fapl_mpio(fapl_id, comm, info) < 0)
            BAIL(NC_EPARINIT);

In the file: libhdf5/hdf5create.c and libhdf5/hdf5open.c.

Let me know how I can help get this through.

For reference, here is my PR to h5py that exposes the same feature: h5py/h5py#2041

[DennisHeimbigner]

Since we use a couple of file drivers already, this is certainly possible.
I will have to investigate a reasonable way to specify the use of some
file driver. Is the DIRECT driver already part of HDF5 or is it your
own custom driver?

[DennisHeimbigner]

Is PR 2066 a possible solution when it gets integrated?

[hmaarrfk]

Do you mean to ask:
"Is a solution similar to PR2206 an acceptable solution?"

The answer is yes.

[DennisHeimbigner]

A couple of questions:
1, Is the DIRECT driver already part of HDF5 or is it your own custom driver?
2. Does it take any parameters?
Is there a pointer to the source code for this driver?

[DennisHeimbigner]

https://docs.hdfgroup.org/hdf5/v1_8/group___f_a_p_l.html#title14

Is this the driver to which you refer?

[hmaarrfk]

https://docs.hdfgroup.org/hdf5/v1_8/group___f_a_p_l.html#title14

Is this the driver to which you refer?

yes.

I guess my original reference to the official documentation was missed.

[hmaarrfk]

The difference is subtle, but eventually you get down deep in thesource code.

H5FDSec2 on the left, H5FDDirect On the right

• SEC2: https://github.com/HDFGroup/hdf5/blob/develop/src/H5FDsec2.c#L329
• Direct: https://github.com/HDFGroup/hdf5/blob/develop/src/H5FDdirect.c#L495

I think there are other differences too, but the O_DIRECT flag is what gives you more direct access to the hardware.

https://github.com/HDFGroup/hdf5/blob/develop/src/H5FDsec2.c#L329

[DennisHeimbigner]

Sorry, do you want Direct or do you want SEC2?

[hmaarrfk]

SEC2 is the default. Direct is the one I'm requesting.

[hmaarrfk]

(I've definitely been in the HDF5 weeds for too long, sorry for the jargon)

[DennisHeimbigner]

I see. Anyway, This will require some thinking. Solution will not be soon.

[hmaarrfk]

Cross referencing other benchmarks I ran:
#2206 (comment)

[hmaarrfk]

I must say that something I found rather annoying when building support for this in h5py was that the direct driver symbols are not always visible.

This means that you have to selectively enable (and disable!) them at compile time.

[DennisHeimbigner]

I must say that something I found rather annoying when building support for this in h5py was that the direct driver symbols are not always visible.
This means that you have to selectively enable (and disable!) them at compile time.

Not sure I follow; can you elaborate?