netCDF 4.9.0 tests fail when RPATH support is enabled

[bedroge]

When using --rpath, all tests for netCDF-4.9.0-gompi-2022a.eb fail due to:

error while loading shared libraries: libnetcdf.so.19: cannot open shared object file: No such file or directory

The binaries do actually have an RPATH section, but they point to the installation dir, while the tests run before the installation step. It also includes $ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64, but the library is located in ../liblib/ in the build dir...

[casparvl]

I think this is not so much because you enabled rpath support, but because you filter LD_LIBRARY_PATH. Could you check that @bedroge ? If so, maybe adapt the title of the issue to make clear it's related to filtering LD_LIBRARY_PATH, not setting rpath.

[boegel]

This rings a bell, I could've sworn I ran into this (and maybe even fixed it somehow), but I'm not sure it was with netCDF...
Will take another look, and come back with more info (or ping me if I don't)

[bedroge]

I think this is not so much because you enabled rpath support, but because you filter LD_LIBRARY_PATH. Could you check that @bedroge ? If so, maybe adapt the title of the issue to make clear it's related to filtering LD_LIBRARY_PATH, not setting rpath.

Unless it somehow does this automatically (but I don't think that's the case?), I'm not filtering LD_LIBRARY_PATH. Also reproduced this on another machine with just passing --rpath, and having more or less default settings for everything else.

[bedroge]

I'm now seeing similar issues for other packages that use the CMakeMake easyblock and have runtest = 'test', e.g. Xerces and json-c. I think it's due to:

CMake by default sets the RPATH property on executables that link to shared libraries in the same project. This makes it possible to run executables (like tests) from the build directory without having to set LD_LIBRARY_PATH.
https://gitlab.kitware.com/cmake/cmake/-/issues/18413

I checked the build dir of a Xerces build (with EB RPATH support disabled), and can indeed confirm that the executables that were called for the tests had an RPATH pointing to, in this case, $builddir/easybuild_obj/src. This gets removed during the install step.

I guess that, with EB RPATH support enabled, this RPATH entry gets overwritten / ignored by the RPATH wrappers of EB, resulting in executables that cannot find the shared libraries in the build directory anymore. I'm not sure if it's easy to detect this, but it would be nice if these directories that CMake wants to add to the RPATH, can somebow be added to (and, later on, removed again from) the ones from EB. Alternatively, we would have to set $LD_LIBRARY_PATH for the tests, but I'm not sure if there's an easy way to figure out which directories would have to be added in that case: sometimes these shared libraries are in the root of the build directory, sometimes they're in a subdirectory (for Xerces they're in src, for netCDF in liblib).

[bedroge]

@boegel
It seems related to this: easybuilders/easybuild-easyblocks#1031.

This settings instructs CMake to not do any RPATH stuff, and hence the tests (performed in the build directory) fail due to binaries not being able to locate shared libraries in the build directory.

I just did some tests with -DCMAKE_SKIP_RPATH=OFF instead of ON, and for me it seems to append paths only. So maybe something has changed in CMake, and it no longer strips all RPATHs?

[bedroge]

Some more details: I did a json-c installation with --stop=build --rpath to make EB stop after the build, right before the test step. With the original settings/easyconfig, I see the following RPATH in the test binaries in the build directory:

  RPATH                /home/bob/easybuildinstall/software/json-c/0.16-GCCcore-12.2.0/lib:/home/bob/easybuildinstall/software/json-c/0.16-GCCcore-12.2.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/data/apps/software/binutils/2.39-GCCcore-12.2.0/lib64:/data/apps/software/GCCcore/12.2.0/lib64:/data/apps/software/GCCcore/12.2.0/lib:/data/apps/software/libarchive/3.6.1-GCCcore-12.2.0/lib/../lib64:/data/apps/software/XZ/5.2.7-GCCcore-12.2.0/lib/../lib64:/data/apps/software/cURL/7.86.0-GCCcore-12.2.0/lib/../lib64:/home/bob/.local/easybuild/software/OpenSSL/1.1/lib/../lib64:/data/apps/software/bzip2/1.0.8-GCCcore-12.2.0/lib/../lib64:/data/apps/software/ncurses/6.3-GCCcore-12.2.0/lib/../lib64:/data/apps/software/zlib/1.2.12-GCCcore-12.2.0/lib/../lib64:/data/apps/software/GCCcore/12.2.0/lib/gcc/x86_64-pc-linux-gnu/12.2.0:/data/apps/software/libarchive/3.6.1-GCCcore-12.2.0/lib:/data/apps/software/XZ/5.2.7-GCCcore-12.2.0/lib:/data/apps/software/cURL/7.86.0-GCCcore-12.2.0/lib:/home/bob/.local/easybuild/software/OpenSSL/1.1/lib:/data/apps/software/bzip2/1.0.8-GCCcore-12.2.0/lib:/data/apps/software/ncurses/6.3-GCCcore-12.2.0/lib:/data/apps/software/zlib/1.2.12-GCCcore-12.2.0/lib

Then I added configopts = '-DCMAKE_SKIP_RPATH=OFF ' to the easyconfig, and got the following RPATH:

  RPATH                /home/bob/easybuildinstall/software/json-c/0.16-GCCcore-12.2.0/lib:/home/bob/easybuildinstall/software/json-c/0.16-GCCcore-12.2.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/data/apps/software/binutils/2.39-GCCcore-12.2.0/lib64:/data/apps/software/GCCcore/12.2.0/lib64:/data/apps/software/GCCcore/12.2.0/lib:/data/apps/software/libarchive/3.6.1-GCCcore-12.2.0/lib/../lib64:/data/apps/software/XZ/5.2.7-GCCcore-12.2.0/lib/../lib64:/data/apps/software/cURL/7.86.0-GCCcore-12.2.0/lib/../lib64:/home/bob/.local/easybuild/software/OpenSSL/1.1/lib/../lib64:/data/apps/software/bzip2/1.0.8-GCCcore-12.2.0/lib/../lib64:/data/apps/software/ncurses/6.3-GCCcore-12.2.0/lib/../lib64:/data/apps/software/zlib/1.2.12-GCCcore-12.2.0/lib/../lib64:/data/apps/software/GCCcore/12.2.0/lib/gcc/x86_64-pc-linux-gnu/12.2.0:/data/apps/software/libarchive/3.6.1-GCCcore-12.2.0/lib:/data/apps/software/XZ/5.2.7-GCCcore-12.2.0/lib:/data/apps/software/cURL/7.86.0-GCCcore-12.2.0/lib:/home/bob/.local/easybuild/software/OpenSSL/1.1/lib:/data/apps/software/bzip2/1.0.8-GCCcore-12.2.0/lib:/data/apps/software/ncurses/6.3-GCCcore-12.2.0/lib:/data/apps/software/zlib/1.2.12-GCCcore-12.2.0/lib:/data/eb/build/jsonc/0.16/GCCcore-12.2.0/easybuild_obj

So, that seems to contain both the paths added by EB (just like with the original easyconfig) and the one (the last one) added by CMake.

Finally, when doing a full installation and checking a shared library in the installation directory:

  RPATH                /home/bob/easybuildinstall/software/json-c/0.16-GCCcore-12.2.0/lib:/home/bob/easybuildinstall/software/json-c/0.16-GCCcore-12.2.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/data/apps/software/binutils/2.39-GCCcore-12.2.0/lib64:/data/apps/software/GCCcore/12.2.0/lib64:/data/apps/software/GCCcore/12.2.0/lib:/data/apps/software/libarchive/3.6.1-GCCcore-12.2.0/lib/../lib64:/data/apps/software/XZ/5.2.7-GCCcore-12.2.0/lib/../lib64:/data/apps/software/cURL/7.86.0-GCCcore-12.2.0/lib/../lib64:/home/bob/.local/easybuild/software/OpenSSL/1.1/lib/../lib64:/data/apps/software/bzip2/1.0.8-GCCcore-12.2.0/lib/../lib64:/data/apps/software/ncurses/6.3-GCCcore-12.2.0/lib/../lib64:/data/apps/software/zlib/1.2.12-GCCcore-12.2.0/lib/../lib64:/data/apps/software/GCCcore/12.2.0/lib/gcc/x86_64-pc-linux-gnu/12.2.0:/data/apps/software/libarchive/3.6.1-GCCcore-12.2.0/lib:/data/apps/software/XZ/5.2.7-GCCcore-12.2.0/lib:/data/apps/software/cURL/7.86.0-GCCcore-12.2.0/lib:/home/bob/.local/easybuild/software/OpenSSL/1.1/lib:/data/apps/software/bzip2/1.0.8-GCCcore-12.2.0/lib:/data/apps/software/ncurses/6.3-GCCcore-12.2.0/lib:/data/apps/software/zlib/1.2.12-GCCcore-12.2.0/lib

So I don't see any signs of CMake stripping the EB-added paths in RPATH here.

[boegel]

We saw a similar problem with GMP, see #11188, but there CMake is not involved.
The workaround used there may still be useful here, but it does seems like there's a broader issue we're not dealing with properly...