Python packages are being installed to local/lib/python*/dist-packages due to tweaks to sysconfig.py in OS (Ubuntu 22.04)

[anselmicz]

Problem description

Hi. Recently, one of my machines started experiencing problems with installing EasyBuild via the bootstrap method.

== 2022-11-26 22:18:10,235 easyblock.py:3774 INFO Running method sanity_check_step part of step sanitycheck
== 2022-11-26 22:18:10,281 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/eb_tmp/local/lib/python3.10/dist-packages/easybuild/base/exceptions.py:124 in __init__): Failed to isolate installed egg dir for easybuild-framework (at easybuild/eb_tmp/local/lib/python3.10/dist-packages/easybuild/easyblocks/e/easybuildmeta.py:235 in sanity_check_step)

I use a custom build script I wrote based on the official instructions in the Lua, Lmod and EasyBuild documentations:
https://github.com/anselmicz/easybuild-installer/blob/main/easybuild.sh

My current builds fail during the sanity check phase, I'm enclosing the build log, environment at the time of installation, and trace output as a gist:
https://gist.github.com/anselmicz/72d1a0239cf67b685ed878aef2f61359

The error itself comes from https://github.com/boegel/easybuild-easyblocks/blob/develop/easybuild/easyblocks/e/easybuildmeta.py#L235

In a Slack conversation, @ocaisa mentioned that the error may be somehow connected to
#2805

System

OS
Debian bookworm/sid

Custom dependencies

Lua/5.1.4.9
Lmod/8.5

System packages

python3/3.10.6
python3-pip/22.3
wget/1.21.3
gcc/12.2.0
make/4.3
rsync/3.2.6
tcl-dev/8.6.11
libreadline-dev/8.2
libibverbs-dev/43.0
xdot/1.2

Python packages

setuptools/65.6.3
python-graph-core/1.8.2
python-graph-dot/1.8.2
archspec/0.2.0
autopep8/2.0.0
GitPython/3.1.29
pep8/1.7.1
pycodestyle/2.10.0
Rich/12.6.0

Do you have any thoughts on this? I tried lowering both the temporary and to-be-installed versions of EasyBuild, but the issue persists both on 4.6.1 and 4.6.0, and I can't try 4.5.4 because then the setuptools and setup.py problems arise.

[boegel]

@anselmicz Can you share the contents of /apps/all/EasyBuild/4.6.2, in particular the contents of the lib*/python*/site-packages subdirectory that should be there?

[anselmicz]

I have also tried lowering the setuptools and pip versions to no avail.

@boegel I have updated the gist with a directory tree of the /apps/all/EasyBuild/4.6.2

lib/python3.10/site-packages is empty, however, everything seems to install under /apps/all/EasyBuild/4.6.2/local, including lib... this is probably the distribution difference, but here the path is .../lib/python3.10/dist-packages.

Same goes for the temporary EasyBuild installation, instead of installing under the prefix, /tmp/$USER/eb_tmp, the actual location of bin, lib, etc. is under /tmp/$USER/eb_tmp/local

I remember dealing with this problem before, but the issue has always been exclusively with the temporary installation of EasyBuild that could have always been fixed by accommodating the PYTHONPATH, i. e. setting export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/local/lib*/python*/dist-packages | tail -1):$PYTHONPATH.

[crubb]

Here is a Dockerfile to reproduce the error using Ubuntu 22.04 LTS:

FROM ubuntu:jammy-20221130
#FROM ubuntu:focal-20221130

ARG DEBIAN_FRONTEND="noninteractive"

RUN set -eux \
  && apt-get update -qq \
  && apt-get -y upgrade \
  && apt-get install -y -q --no-install-recommends \
      lmod \
      python3-pip \
  && apt-get clean \
  && python3 -m pip install --ignore-installed --prefix /tmp/eb_tmp easybuild \
  && rm -rf /tmp/hsperfdata* /var/*/apt/*/partial /var/lib/apt/lists/* /var/log/apt/term*

RUN set -x \
  && export LOGNAME=root \
  && export PATH=$(find /tmp/eb_tmp -type d -name bin):$PATH \
  && export PYTHONPATH=$(find /tmp/eb_tmp/ -type d -name *-packages | tail -1) \
  && export EB_PYTHON=python3 \
  && . /etc/profile.d/lmod.sh \
  && eb --allow-use-as-root-and-accept-consequences --install-latest-eb-release -d --prefix /tmp/easybuild \
  ;  cat /tmp/eb-*/easybuild-EasyBuild-4.6.2-*.log

ENTRYPOINT ["/bin/bash"]

When you docker build the same thing using FROM ubuntu:focal-20221130 it works. So there has to be something different with Ubuntu 22.04 LTS, which I haven't figured out, yet. One of the obvious things, as mentioned: jammy installs everything in /tmp/eb_tmp/local/, while focal installs directly into /tmp/eb_tmp.

@anselmicz, is it the latest Ubuntu LTS for you, too?

Any ideas? Is there a fix or config change needed in Ubuntu, or with EasyBuild?

[anselmicz]

@crubb my report deals with Debian Unstable

I confirm I can reproduce the error in jammy. However, this is not at all that surprising, because, based on my understanding of Ubuntu versioning, jammy is based on Debian bookworm (current sid / unstable). It also makes sense that it might work correctly in focal, given it's based on bullseye where it works too (at least for me).

[crubb]

root cause of the problems seems to be the following: https://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240/14

Essentially, some distros patched Python's sysconfig to change the default installation scheme from posix_prefix to posix_local.

I found a bug report for Ubuntu, which essentially sits dormant after more or less stating that it's a design choice: https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1967920

[boegel]

@crubb I can see how that affect the installation of EasyBuild with EasyBuild (since that's done with the system Python), but how does that affect installing Python packages with a Python that was installed with EasyBuild?!

[anselmicz]

@crubb Good catch! This (temporary) workaround seems to work for me:

--- /lib/python3.10/sysconfig.py.orig	2022-12-13 17:38:06.221735514 +0100
+++ /lib/python3.10/sysconfig.py	2022-12-13 17:40:28.549111500 +0100
@@ -271,7 +271,7 @@
         if deb_build in ('deb', 'deb_system'):
             prefix_scheme = 'deb_system'
         else:
-            prefix_scheme = 'posix_local'
+            prefix_scheme = 'posix_prefix'

     return {
         'prefix': prefix_scheme,

@boegel, looking at the links @crubb provided, it seems like in Debian, sysconfig.py got some extreme mods between the versions:
https://lists.debian.org/debian-python/2022/03/msg00039.html.
So far this seems to be distro specific, so the EasyBuild Python should be unaffected.

[crubb]

I actually borrowed from Debian and stole their DEB_PYTHON_INSTALL_LAYOUT environment variable. Set it to eb, and it uses posix_prefix as the prefix_scheme:

--- sysconfig.py-orig   2022-12-13 21:37:25.489016711 +0000
+++ sysconfig.py        2022-12-13 21:37:52.088975055 +0000
@@ -270,6 +270,8 @@
         deb_build = os.environ.get('DEB_PYTHON_INSTALL_LAYOUT', 'posix_local')
         if deb_build in ('deb', 'deb_system'):
             prefix_scheme = 'deb_system'
+        elif deb_build in ('eb'):
+            prefix_scheme = 'posix_prefix'
         else:
             prefix_scheme = 'posix_local'

[SebastianAchilles]

I just run into the same problem while generating a test-report on a ubuntu-22.04 container (recipe, image @ ghcr.io ) for EasyBuild-4.7.0.eb: easybuilders/easybuild-easyconfigs#17065 (comment)

With the described workaround I was able to build it successfully.