set HDF5_PLUGIN_PATH programmatically

[edwardhartnett]

For filters (and for Zarr?) we are using the HDF5_PLUGIN_PATH environment variable to specify where plugins should be found.

But as I just learned today, we can actually set the plugin path programmatically now.

https://docs.hdfgroup.org/hdf5/develop/group___h5_p_l.html#title3

Turns out they have had this capability for years, so we should have known this when we did the compression work. But we did not. I guess there is a good reason to go to workshops in person.

How should we use this?

We could:

1. Allow the builder to specify a default plugin path that will always be used whenever netCDF is run. We could do this by calling the appropriate functions on library startup. Then we don't have to care about the environment variable HDF5_PLUGIN_PATH (but would interoperate with it). This is easy and we should do this immediately.
2. We could put new functions in the netCDF API to allow the user to directly append to, delete, and inquire the plugin path. This would be maximally flexible. However I don't think this is a good enough reason to extend the API.

So I suggest 1 but not 2. However I defer to @DennisHeimbigner the expert on filters.

[DennisHeimbigner]

I note that there are also prepend (put at front) and insert (put anywhere in list) functions.

[DennisHeimbigner]

Allow the builder to specify a default plugin path that will always be used whenever netCDF is run.

It occurs to me that I do not quite understand this proposal. I guess that you are proposing an option.
say --with-default-plugin-path=XXX.
Is my guess correct?

[edwardhartnett]

Yes, that's what I had in mind.

I think we already allow them to specify a plugin directory, right? And then they have to set the environment varialble HDF5_PLUGIN_PATH to that location before running any programs that use the filter.

Instead, netcdf-c could set that programmatically, so there would be no need for the environment var to be set. This would help a lot, since many users are sure to get confused and forget to set the environment var.

[DennisHeimbigner]

Also, we already have --with-plugin-dir to specify where to install plugins. How, if at all, should
this proposal interact with --with-plugin-dir?

[edwardhartnett]

I think it should just take that directory, and set it programmatically whenever netcdf starts. So the user does not have to set HDF5_PLUGIN_PATH environment variable for filters to work.

[DennisHeimbigner]

I don't follow. the --with-plugin-dir options says where to install the plugins, while --with-default-plugin-path
(my name for the option you propose) would specify from where to load plugins.
We could do, say, the following:
1.If --with-plugin-dir is set, and --with-default-plugin-path is not set, we use the former as the value of the latter.
2. If --with-plugin-dir is not set, and --with-default-plugin-path is set, we use the latter as the value of the former.
3. We leave them as completely independent.

[edwardhartnett]

What good can possibly come of allowing the user to install filters in one place, and then look for them in another?

Let there be zero or one directory specified at netCDF configure time. If zero, the HDF5 default plugin dir is used.
If one, then filters are installed there, and every time netcdf-c starts up, it prepends that directory to the plugin path programmatically.

I assume the HDF5 code would give the environment variable HDF5_PLUGIN_PATH the final say in the matter, so that is still available to advanced users.

[DennisHeimbigner]

What good can possibly come of allowing the user to install filters in one place, and then look for them in another?

It is quite conceivable that an organization might have a directory of globally available plugins and a user might have his own directory of plugins. In this case, the local directory is for installation and the path "global;local" is the path for dynamic loading. This raises the question if the default path should be a single
directory or should be allowed to a true path (i.e. a sequence of directories).

[edwardhartnett]

I would say that the netcdf installation has the global path and the user should set the HDF5_PLUGIN_PATH environment variable to include his local plugin directory.

Yes, a true path should be allowed.

[DennisHeimbigner]

Proposal for Default Plugin Path

The plugin path list manipulation functions in HDF5 operate basically as follows.

1. On startup, the internal plugin path list is one path that is either the built-in default (i.e. /usr/local/hdf5/lib/plugin) or the value of HDF5_PLUGIN_PATH (if defined).
2. If HDF5_PLUGIN_PATH has multiple directories (e.g. dirpath1:dirpath2) then it is parsed as a list so the internal plugin path list is as follows.

element [0]: dirpath1
element [1]: dirpath2

3. The append and prepend operations work as expected to insert a single directory at the front or end of the internal plugin path.

So, the proposal is as follows.

Add configure option

--with-default-plugin-path=*;\<path\>;\<path\>;...;*

The * indicates where the existing default plugin path goes.
So, the sequence of <path>s is parsed to produce a list
and the list is inserted (in order) at either the front or the end of the
existing internal plugin list as controlled by the placement of the star.
The leading star (or no star) means insert at the end of the existing path.
Trailing start means insert at the beginning of the existing path.

Note that semicolon separator is always allowed even on linux machines.
The colon is also allowed on non-windows platforms, but not on
platforms that allow the "X:" windows drive letter notation in paths.

The --with-plugin-dir option is renamed --with-plugin-install-dir
but otherwise its semantics remains unchanged.
For back compatibility, the --with-plugin-dir option
is deprecated but aliased to --with-plugin-install-dir for some
period of time before removing it.

As noted above, if HDF5_PLUGIN_PATH is defined at startup, then it serves to initialize the internal path list.

[Alexander-Barth]

Being able to set the plugin path programmatically would help us a lot in in the julia community where all packages are installed in the users home directory (typically) and the path contains a hashsum of the package (e.g. /home/abarth/.julia/artifacts/980adb65c877c384f59bf7b2b800699e22a38815/lib/libhdf5.so). Multiple installation of HDF5 can coexists. But for a user, these path are difficult to know. I assume that the HDF5_PLUGIN_PATH is used when the HDF5 library is loaded (and later changes will be ignored, this is correct?). If a NetCDF package or the user can tell HDF5/NetCDF where to find the plugin within a julia session after the HDF5 library is loaded, this would help us a lot.

[DennisHeimbigner]

This just slipped down the stack. I will put together a proposal for this and run it by
you and Unidata. May take a little while since it is not my highest priority.

[DennisHeimbigner]

I assume that the HDF5_PLUGIN_PATH is used when the HDF5 library is loaded (and later changes will be ignored, this is correct?).

Sort of. AFAIK the value of HDF5_PLUGIN_PATH is parsed when the HDF5 library is initialized. You are correct
that any changes to HDF5_PLUGIN_PATH are ignored after that point.

If a NetCDF package or the user can tell HDF5/NetCDF where to find the plugin within a julia session after the HDF5 library is loaded, this would help us a lot.

If I understand, your requirements are somewhat more extensive than Ed's original proposal above.
It sounds like you want to be able to do the equivalent of having later changes to HDF5_PLUGIN_PATH
to be propagated to the HDF5 library plugin search immediately. As opposed to Ed's at build time.

[Alexander-Barth]

Yes, indeed I would like a way to set the plugin path after the HDF5 and netCDF libraries are loaded via an environment variable or via an API call. A somewhat similar problem was the discovering of the SSL certificates which is possible via nc_rc_set API that you implemented (thanks again:-)). Maybe an API call would be preferable, so that we do not leave a modified HDF5_PLUGIN_PATH variable once the julia program finishes.

As I understand, HDF5 expects all plugins inside its own plugin directory per default and the NetCDF installation script copies them there automatically so that the user does not have to worry about setting the HDF5_PLUGIN_PATH variable.

In julia we currently do not ship NetCDF/HDF5 filters because we cannot modify the HDF5 package after building the HDF5 source code. Including the NetCDF filters in the HDF5 package would result that HDF5 becomes dependency on netCDF (and vice versa). So one possibility for us would be to ship the NetCDF/HDF5 plugin with the NetCDF package. But then we need to tell HDF5 where to look for the plugins.

Maybe an alternative (simpler?) approach for our problem, could be that when libnetcdf.so is initialized the path plugins or ../plugins (relative to libnetcdf.so) is appended using H5PLappend (as in julia all libraries are installed with different path prefix unlike Linux systems when all libraries are install in a common prefix /usr for example)