make OpenSSL aware of host CA certs

[jpecar]

While developing a PythonBundle eb I came across a situation where one of the entries in exts_lists was trying to fetch additional dependencies from pypi.org. This was failing on [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate. Looking at strace of what was going on I got the impression that eb installed OpenSSL is looking into its ssl/certs folder for CA certificates, which is empty. It appears as if it is looking for cert hashes and not even considering ca-bundle.crt and similar.

I'm not familiar enough with inner workings of OpenSSL to come up with a solution, but it looks like it will differ slightly between distributions.

[jpecar]

Additional info: we found out that this only happens when openssl easyblock doesn't find openssl headers on host and decides to build openssl from source. In this case variable OPENSSLDIR points to installdir/ssl instead of proper host location in /etc.

[boegel]

@lexming I think you were involved in this discussion, anything to fix here in the OpenSSL wrapper easyblock?

[akesandgren]

1 - If the wrapper easyblock finds libraries and engines of the required version but not the include files, the first option should be to error out and ask the user to get the corresponding development package installed. Of course we need an option to go ahead and install from source anyway in case sysadmins refuse.
2 - if the wrapper easyblocks finds ANY installed openssl version it should ask it for the path to the system installed ca-certs and pass it on to the build-from-source step. Easily done through a env var if there is no other easy way.

If there is NO openssl version installed then the system is almost guaranteed to be broken and we should inform the users they need to specify the path to the systems ca-certs. I.e. the ca-certs package must be installed on the OS level, always.

[lexming]

This is indeed only an issue whenever OpenSSL is built from source by EB. Regardless if it is built by the wrapper or a regular OpenSSL easyconfig. Certificates are not part of OpenSSL sources.

If OpenSSL in the host is successfully wrapped by EB, the path to the certificates in the system will be known by OpenSSL. The path in OPENSSLDIR is set at build time, so it is unaffected by the wrapper and our symlinks.

Making the build of OpenSSL aware of any certificates in the host system is not trivial. The certificates may or may not be installed and the path to the certificates is distro dependent (eg /etc/pki/tls in RHEL, /usr/lib/ssl in Debian). The only reliable way to locate the certificates is through a working OpenSSL installation in the host to be able to query for OPENSSLDIR with openssl version -d.

Cases where we the wrapper will build OpenSSL from source but we could still have a working openssl version -d:

1. OpenSSL in host is just missing the headers
   In this case I agree with @akesandgren, if only the headers are missing the wrapper should fail the installation and notify the user. That's usually easy to correct. (maybe add an os_dependency? but that is complicated with the wrapper as it can also build from source)
2. OpenSSL in host is outdated
   In this case I don't see why we should trust the certificates in the system, as those will probably be outdated as well. But it would be relatively easy to get the path and use it in our builds.
3. User is manually forcing the build of OpenSSL from source
   In this case we can try to catch OPENSSLDIR. If that fails, the alternative is to add an option so that the path to certificates can be set by the user.

[akesandgren]

Actually /etc/ssl/certs work on both RHEL and Debian :-) the magic of symlinks
But others may be different.

[akesandgren]

2. OpenSSL is outdated, that may just be that whatever you're building requires a newer libssl, it does NOT imply that the ca-cert info is outdated. And the system installed ca-certs should still always be used. (even if you don't really trust them for other reasons)

[FokkeDijkstra]

I ran into the same issue, as I reported for the easyconfig of OpenSSL at: easybuilders/easybuild-easyconfigs#14901
This was for an CentOS 7 system where the standard (supported) system OpenSSL is version 1.02k, whereas EasyBuild requires OpenSSL 1.1 for more recent packages. The wrapper then installs its own version of OpenSSL 1.1.

OpenSSL does indeed not come with certificates and depends on the certificates on the host installed with the OS. The documentation on the location of the certificates in the openssl build instructions is very sketchy, as the path of the certificates obtained from --openssldir is or was related to the --prefix setting. Details can be found at:
https://wiki.openssl.org/index.php/Compilation_and_Installation#PREFIX_and_OPENSSLDIR

Setting --openssldir (as is done in the spec file for rpms) may not be possible, as the install step seems to try to install stuff into that system path, which easybuild should not be allowed to do. In my current EasyBuild based OpenSSL/1.1 installation the path contains additional files and folders installed by the EasyBuild OpenSSL installation, so I assume it will try to do so.
Creating a symbolic link to the system certificate path works fine however, but I am not sure how robust that solution is.

And there is still the issue of how to obtain the system certificate path. This could be indeed by done using an existing openssl, which a system should have anyway. This requirement could be put in the osdependencies.