Unable to compile a statically linked wheel

[mrossinek]

Please apologize the length of this bug report in advance, but I am trying to provide as much information as possible.

Informations

• Qiskit Aer version: master
• Python version: 3.6.9
• Operating system: uname -a provides:

Linux qt00 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

To provide some context: I am trying to compile Aer from source on a compute server to allow multiple users to access the same master-branch installation from a centrally managed virtualenv. While doing so I ran into the problem of not being able to produce a correctly statically linked wheel.
Specifically this means that I log in onto a login node, create a virtualenv there and expect all computations on the respective compute nodes to work normally.

I am using the following Python script for testing:

import sys
print(sys.version)
from qiskit.providers.aer import Aer
print(Aer)

and the following bash script for testing the execution on the compute nodes:

#!/bin/bash
locate libopenblas.so.0
source ~/path/to/venv/bin/activate
which python
which pip
pip list | grep qiskit
python test.py  # this is the above script

What is the current behavior?

In order to explain the problem at hand I will step through a number of different scenarios and explain the results which I have achieved with each:

A stable installation using pip install qiskit-terra qiskit-aer

Starting from a clean virtualenv and running the above pip installation command everything works fine.
On the login node I obtain:

3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0]
/home/a/oss/software/qiskit-stable/venv/lib/python3.6/site-packages/qiskit/__init__.py:75: RuntimeWarning: Could not import the IBMQ provider from the qiskit-ibmq-provider package. Install qiskit-ibmq-provider or check your installation.
  RuntimeWarning)
AerProvider

And on the compute nodes:

/path/to/stable/venv/bin/python
/path/to/stable/venv/bin/pip
qiskit-aer         0.7.0
qiskit-terra       0.16.0
/home/a/oss/software/qiskit-stable/venv/lib/python3.6/site-packages/qiskit/__init__.py:75: RuntimeWarning: Could not import the IBMQ provider from the qiskit-ibmq-provider package. Install qiskit-ibmq-provider or check your installation.
  RuntimeWarning)
3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0]
AerProvider

A master installation using python setup.py bdist_wheel

When trying to compile Aer manually from master everything works as expected on the login node (i.e. same as above) but on the compute node I obtain the following error:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    from qiskit.providers.aer import Aer
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/__init__.py", line 64, in <module>
    from .aerprovider import AerProvider
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/aerprovider.py", line 20, in <module>
    from .backends.qasm_simulator import QasmSimulator
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/__init__.py", line 17, in <module>
    from .qasm_simulator import QasmSimulator
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/qasm_simulator.py", line 25, in <module>
    from .controller_wrappers import qasm_controller_execute
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

This was to be expected as the first line of the bash script (locate libopenblas.so.0) does not return anything which shows that OpenBLAS cannot be found on the compute nodes (while it is available normally on the login node).
Having talked to the system administrator, we agreed that it would be best if, thus, we can compile Aer with static linking (as this appears to work with the stable installation using pip install which ships a Python wheel bundled with libopenblas).

Static linking: python setup.py bdist_wheel -- -DSTATIC_LINKING=True

This attempt causes even the installation on the login node to break with the following error:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    from qiskit.providers.aer import Aer
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/__init__.py", line 64, in <module>
    from .aerprovider import AerProvider
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/aerprovider.py", line 20, in <module>
    from .backends.qasm_simulator import QasmSimulator
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/__init__.py", line 17, in <module>
    from .qasm_simulator import QasmSimulator
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/qasm_simulator.py", line 25, in <module>
    from .controller_wrappers import qasm_controller_execute
ImportError: /home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/controller_wrappers.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _gfortran_concat_string

Static linking with FindBLAS.cmake.fix-static-linking

I then found out about this patch: https://github.com/Qiskit/qiskit-aer/blob/master/cmake/FindBLAS.cmake.fix-static-linking
But after applying that and trying the same -DSTATIC_LINKING=True compilation as above, I still obtain the same error.

Steps to reproduce the problem

The following summarizes the steps necessary to reproduce this issue. However, your mileage may vary as I have tried to reproduce this on my Fedora laptop where I am unable to do so. Both of the last two steps do not result in the problem but I am not sure how to verify that this is because the static linking was successful or whether it falls back to dynamic linking.
The Fedora laptop specs are:

$ uname -a
Linux pulsano 5.8.15-201.fc32.x86_64 #1 SMP Thu Oct 15 15:56:44 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ python --version
Python 3.6.12  # the same works with Python 3.8.6
$ gcc --version
gcc (GCC) 9.3.0
$ cmake --version
cmake version 3.18.2  # same as on the server

mkdir tmp
cd tmp
python3 -m virtualenv venv
source venv/bin/activate
git clone https://github.com/Qiskit/qiskit-terra.git
cd qiskit-terra
pip install -c constraints.txt -r requirements.txt -r requirements-dev.txt
pip install -e .
cd ..
git clone https://github.com/Qiskit/qiskit-aer.git
cd qiskit-aer
pip install -c constraints.txt -r requirements-dev.txt
python setup.py bdist_wheel -- -DSTATIC_LINKING=True
cd ..
python
% from qiskit.providers.aer import Aer

What is the expected behavior?

I would expect to be able to compile a statically linked Python wheel of Aer

Suggested solutions

I am at a loss to be honest, which is why I came here 🙂

[mtreinish]

We build statically linked wheels for release on pypi. We use the cibuildwheel tool to automate this in CI. But the basic workflow it uses is:

pip wheel . --no-deps
auditwheel repair qiskit_aer-*.whl

The auditwheel repair command there vendors the shared libs into the binary. Although it may only work if the system has a version of system libs (like glibc) that is compatible with a manylinux python packaging spec (which is the other thing cibuildwheel does for us, it runs everything in the manylinux docker images).

[mrossinek]

Thanks a lot for the information! I went ahead and tried it out and as slightly anticipated by your comment about auditwheel it complains about the versions of the system libs:

auditwheel: error: cannot repair "qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl" to "manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

From reading a little bit through auditwheels Readme it doesn't seem like there is anything I can do about this unless I have access to older versions of these libraries, correct?

[vvilpas]

Under the hood we use manylinux2010 to build the linux wheels. Not sure if you can in your system, but maybe you can follow the workflow @mtreinish described, using the manylinux2010 docker image.

[mrossinek]

On this compute server where I would like to compile Aer I do not have access to docker. And I would like to avoid having to compile this on a different system just to push the wheel onto the server every time I want to update it.
I checked the PEPs for manylinux1, manylinux2010 and manylinux2014 for the maximally allowed system library versions. But it appears as though none of them will support what is available on the server. The output of auditwheel --verbose repair *.whl confirms this as it shows incompatibility of all three manylinux* variants.

Is there no way of statically linking the libraries into this specific wheel? It doesn't need to support any platform but just the compute nodes which should be identical to the login node (but apparantly missing the libraries because otherwise the dynamic linking should work...)

[vvilpas]

Can you compile with static_linking enabled (but forget about FindBLAS.cmake.fix-static-linking) and then check for dependency on shared libraries (ldd qiskit/providers/aer/backends/controller_wrappers*.so under _skbuild/**YOUR_OS**/cmake-install/)?

[mrossinek]

Here is the ouptut of what you described:

(venv) oss@qt00:~/software/qiskit-master/src/qiskit-aer/_skbuild/linux-x86_64-3.6/cmake-install$ ldd qiskit/providers/aer/backends/controller_wrappers.cpython-36m-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffd47d76000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5905a27000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5905689000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f590545a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5905069000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5907912000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5904e65000)

[mrossinek]

So I took a closer look at auditwheel as it appeared odd to me that it shouldn't be possible to restrict the platform to something else than manylinux*. Turns out I must have overlooked the --plat argument to the repair command earlier.
Anyways, I believe that this should work. However, now I need to get patchelf to work on the server. I will update once that is done and hopefully be able to close this issue then 👍

[vvilpas]

ok, let us know. In any case, not sure if the problem comes from openblas as it seems to be correctly linked statically. However, it seems there is a missing gfortran dependency. Maybe it works adding -static-libgfortran.

[mrossinek]

Okay there is still a problem..

1. Adding -static-libgfortran unfortunately does not resolve it
2. I also got patchelf installed now but auditwheel is unable to repair the wheel. The scenario is identical to A master installation using python setup.py bdist_wheel from my original post

Here are the steps and their output:

(venv) oss@qt00:~/software/qiskit-master/src/qiskit-aer$ pip wheel . --no-deps
Processing /home/a/oss/software/qiskit-master/src/qiskit-aer
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: qiskit-aer
  Building wheel for qiskit-aer (PEP 517) ... done
  Created wheel for qiskit-aer: filename=qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl size=8028382 sha256=98b142254a3b7c0f2104cc86dae770c0a823d0ddb1e3d505849e4d6303949719
  Stored in directory: /home/a/oss/.cache/pip/wheels/ff/60/9c/5cd68d848ea8e1317a62926e8c6f2c8d8230ea305a58763e51
Successfully built qiskit-aer

(venv) oss@qt00:~/software/qiskit-master/src/qiskit-aer$ auditwheel show qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl

qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_4.3.0',
'GCC_3.4', 'GCC_4.2.0', 'GCC_3.3', 'GCC_4.0.0', 'GCC_3.0'},
libgomp.so.1 with versions {'OMP_1.0', 'GOMP_4.0', 'GOMP_1.0'},
libc.so.6 with versions {'GLIBC_2.4', 'GLIBC_2.3.4', 'GLIBC_2.3.2',
'GLIBC_2.14', 'GLIBC_2.3', 'GLIBC_2.10', 'GLIBC_2.6', 'GLIBC_2.17',
'GLIBC_2.7', 'GLIBC_2.2.5'}, libm.so.6 with versions {'GLIBC_2.23',
'GLIBC_2.15', 'GLIBC_2.27', 'GLIBC_2.2.5'}, libpthread.so.0 with
versions {'GLIBC_2.3.2', 'GLIBC_2.3.4', 'GLIBC_2.2.5'}, libstdc++.so.6
with versions {'GLIBCXX_3.4.9', 'CXXABI_1.3.5', 'CXXABI_1.3.9',
'GLIBCXX_3.4.18', 'GLIBCXX_3.4', 'CXXABI_1.3', 'GLIBCXX_3.4.21',
'CXXABI_1.3.7', 'CXXABI_1.3.3', 'GLIBCXX_3.4.20', 'CXXABI_1.3.8',
'CXXABI_1.3.2', 'GLIBCXX_3.4.11', 'GLIBCXX_3.4.19'}, libdl.so.2 with
versions {'GLIBC_2.2.5'}, libquadmath.so.0 with versions
{'QUADMATH_1.0'}, libgfortran.so.4 with versions {'GFORTRAN_7'}

This constrains the platform tag to "linux_x86_64". In order to
achieve a more compatible tag, you would need to recompile a new wheel
from source on a system with earlier versions of these libraries, such
as a recent manylinux image.

(venv) oss@qt00:~/software/qiskit-master/src/qiskit-aer$ auditwheel repair --plat linux_x86_64 qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl
INFO:auditwheel.main_repair:Repairing qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl
warning: working around a Linux kernel bug by creating a hole of 2076672 bytes in ‘qiskit_aer.libs/libpthread-2-3dba1c1f.27.so’
warning: working around a Linux kernel bug by creating a hole of 2101248 bytes in ‘qiskit_aer.libs/libc-2-f0ad9639.27.so’
INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
INFO:auditwheel.wheeltools:New filename tags: linux_x86_64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp36-cp36m-linux_x86_64
INFO:auditwheel.wheeltools:No WHEEL info change needed.
INFO:auditwheel.main_repair:
Fixed-up wheel written to /home/a/oss/software/qiskit-master/src/qiskit-aer/wheelhouse/qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl

and for good measure an inspection of the wheel after auditwheel repair:

(venv) oss@qt00:~/software/qiskit-master/src/qiskit-aer$ auditwheel show qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl

qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_4.3.0',
'GCC_4.0.0', 'GCC_3.0', 'GCC_3.4', 'GCC_3.3', 'GCC_4.2.0'},
libgomp.so.1 with versions {'GOMP_4.0', 'GOMP_1.0', 'OMP_1.0'},
libc.so.6 with versions {'GLIBC_2.7', 'GLIBC_2.14', 'GLIBC_2.10',
'GLIBC_2.4', 'GLIBC_2.17', 'GLIBC_2.3.2', 'GLIBC_2.6', 'GLIBC_2.3.4',
'GLIBC_2.3', 'GLIBC_2.2.5'}, libm.so.6 with versions {'GLIBC_2.23',
'GLIBC_2.15', 'GLIBC_2.2.5', 'GLIBC_2.27'}, libpthread.so.0 with
versions {'GLIBC_2.3.2', 'GLIBC_2.2.5', 'GLIBC_2.3.4'}, libstdc++.so.6
with versions {'GLIBCXX_3.4.21', 'GLIBCXX_3.4.19', 'GLIBCXX_3.4.20',
'GLIBCXX_3.4.11', 'CXXABI_1.3.2', 'CXXABI_1.3.8', 'GLIBCXX_3.4.18',
'CXXABI_1.3.3', 'CXXABI_1.3.7', 'CXXABI_1.3.5', 'CXXABI_1.3',
'GLIBCXX_3.4.9', 'CXXABI_1.3.9', 'GLIBCXX_3.4'}, libdl.so.2 with
versions {'GLIBC_2.2.5'}, libgfortran.so.4 with versions
{'GFORTRAN_7'}, libquadmath.so.0 with versions {'QUADMATH_1.0'}

This constrains the platform tag to "linux_x86_64". In order to
achieve a more compatible tag, you would need to recompile a new wheel
from source on a system with earlier versions of these libraries, such
as a recent manylinux image.

Inspecting the wheel as a zip file also does not show a .libs subfolder as the PyPi version does.

[mrossinek]

Actually scratch the above. I did not notice that auditwheel stored the new wheel in a different path. Actually installing the wheel under wheelhouse/... still results in a problem however:

Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

[mrossinek]

When I compile with python setup.py bdist_wheel -- -DSTATIC_LINKING=True -DCMAKE_CXX_FLAGS="-static-libgfortran" -- -j16 I still obtain the same inconsistency as above but I can finally verify that some of the libraries are correctly included in the wheel:

qiskit_aer.libs/libc-2-f0ad9639.27.so
qiskit_aer.libs/libdl-2-a017a02c.27.so
qiskit_aer.libs/libpthread-2-3dba1c1f.27.so
qiskit_aer.libs/libgomp-52f2fd74.so.1.0.0
qiskit_aer.libs/libm-2-b631b1f5.27.so

However, I don't understand why libopenblas does not show up there...

[mrossinek]

And now for a final test with python setup.py bdist_wheel -- -- -j16 and subsequent auditwheel repair --plat linux_x86_64 dist/qiskit_aer*.whl I again obtain the inconsistency error but this time all libraries appear to show up correctly inside the wheel:

qiskit_aer.libs/libc-2-f0ad9639.27.so
qiskit_aer.libs/libgfortran-d87b8024.so.4.0.0
qiskit_aer.libs/libdl-2-a017a02c.27.so
qiskit_aer.libs/libstdc++-a7ce5e3a.so.6.0.25
qiskit_aer.libs/libquadmath-9dd08b42.so.0.0.0
qiskit_aer.libs/libpthread-2-3dba1c1f.27.so
qiskit_aer.libs/libgomp-52f2fd74.so.1.0.0
qiskit_aer.libs/libm-2-b631b1f5.27.so
qiskit_aer.libs/libopenblasp-r0-164709ac.2.20.so
qiskit_aer.libs/libgcc_s-2f0e4866.so.1

Come to think of it, now that I see this, I probably understand that in the previous case those libraries were not included since the static linking directly incorporated these libraries into the wrapper .so...

[mrossinek]

Okay I think I found the solution. The same happened to another user of auditwheel here: pypa/auditwheel#103
As he outlines in the first comment this patch from patchelf (NixOS/patchelf#85) fixes this but the Ubuntu version is still fixed at 0.9 (prior to that PR).

I will compile patchelf from source and try again. Should this finally work I will close this issue without further comment.
Please excuse the excessive commenting/spamming in the past hour...