zlib issue EasyBuild 4.3.2

[MH2410]

Hi, I do have an issue with installing software via eb.

OS: OpenSuse 15.2 (incl. zlib, gcc, lmod, etc.)

eb --version
This is EasyBuild 4.3.2 (framework: 4.3.2, easyblocks: 4.3.2) on host easybuild.

python --version
Python 3.6.12

ml avail
------------------------------------------ /home/UID/.local/easybuild/modules/all --------------------------
Bison/3.3.2 Bison/3.7.1(D) M4/1.4.18 flex/2.6.4 zlib/1.2.8
Bison/3.5.3 EasyBuild/4.3.2 cowsay/3.04 help2man/1.47.4 zlib/1.2.11(D)

ISSUE:

If I try to install binutils for example R I am always receiving an zlib issue (zlib is not statically linked in).

eb binutils-2.35.eb
== temporary log file in case of crash /tmp/eb-3m_m1xd2/easybuild-7ry5scw7.log
== found valid index for /home/UID/.local/easybuild/software/EasyBuild/4.3.2/easybuild/easyconfigs, so using it...
== processing EasyBuild easyconfig
/home/UID/.local/easybuild/software/EasyBuild/4.3.2/easybuild/easyconfigs/b/binutils/binutils-2.35.eb
== building and installing binutils/2.35...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== restore after iterating...
== postprocessing...
== sanity checking...
== FAILED: Installation ended unsuccessfully (build directory: /home/UID/.local/easybuild/build/binutils/2.35/system-system): build failed (first 300 chars): zlib is not statically linked in /home/UID/.local/easybuild/software/binutils/2.35/bin/ld: linux-vdso.so.1 (0x00007ffc1fbc4000)
libbfd-2.35.so => /home/UID/.local/easybuild/software/binutils/2.35/lib64/libbfd-2.35.so (0x00007fb716208000)
libctf.so.0 => /usr/lib64/libctf.so.0 (0x00007fb715fec0 (took 7 min 20 sec)

I hope the problem can be solved with your help.

[boegel]

@MH2410 Can you share the full log for inspection?

[MH2410]

Thanks for the reply.

The eb install does not generate logfiles, I have checked the R folder (see below),

I have tried to run eb –robot but it does show the same error

eb --show-config

buildpath      (D) = /home/UID/.local/easybuild/build
containerpath  (D) = /home/UID/.local/easybuild/containers
installpath    (D) = /home/UID/.local/easybuild
repositorypath (D) = /home/UID/.local/easybuild/ebfiles_repo
robot-paths    (D) = /usr/easybuild/easyconfigs
sourcepath     (D) = /home/UID/.local/easybuild/sources

~/.local/easybuild/software/R/4.0.0-foss-2020a> ls -la
total 8
drwxr-xr-x 2 UID users 4096 Jan  8 17:59 .
drwxr-xr-x 3 UID users 4096 Jan  8 17:59 ..

~/.local/easybuild/build/R/4.0.0/foss-2020a/R-4.0.0> ls
ChangeLog    configure     COPYING  etc      m4           Makefile.fw  po      share  SVN-REVISION  tools    VERSION-NICK
config.site  configure.ac  doc      INSTALL  Makeconf.in  Makefile.in  README  src    tests         VERSION

~/.local/easybuild/build/R/4.0.0/foss-2020a/R-4.0.0> eb R-4.0.0-foss-2020a.eb --robot
== temporary log file in case of crash /tmp/eb-2g9l_9u8/easybuild-dhv3hm9y.log
== found valid index for /usr/easybuild/easyconfigs, so using it...
== found valid index for /usr/easybuild/easyconfigs, so using it...
== resolving dependencies ...
== processing EasyBuild easyconfig /usr/easybuild/easyconfigs/b/binutils/binutils-2.34.eb
== building and installing binutils/2.34...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== restore after iterating...
== postprocessing...
== sanity checking...
== FAILED: Installation ended unsuccessfully (build directory: /home/UID/.local/easybuild/build/binutils/2.34/system-system): build failed (first 300 chars): zlib is not statically linked in /home/UID/.local/easybuild/software/binutils/2.34/bin/ld:      linux-vdso.so.1 (0x00007ffece763000)
        libbfd-2.34.so => /home/UID/.local/easybuild/software/binutils/2.34/lib64/libbfd-2.34.so (0x00007fc5c7b90000)
        libctf.so.0 => /usr/lib64/libctf.so.0 (0x00007fc5c79740 (took 7 min 31 sec)
== Results of the build can be found in the log file(s) /tmp/eb-2g9l_9u8/easybuild-binutils-2.34-20210110.165306.hGslx.log
ERROR: Build of /usr/easybuild/easyconfigs/b/binutils/binutils-2.34.eb failed (err: 'build failed (first 300 chars): zlib is not statically linked in /home/UID/.local/easybuild/software/binutils/2.34/bin/ld: \tlinux-vdso.so.1 (0x00007ffece763000)\n\tlibbfd-2.34.so => /home/UID/.local/easybuild/software/binutils/2.34/lib64/libbfd-2.34.so (0x00007fc5c7b90000)\n\tlibctf.so.0 => /usr/lib64/libctf.so.0 (0x00007fc5c79740')

[boegel]

@MH2410 The log file I'm interested in is /tmp/eb-2g9l_9u8/easybuild-binutils-2.34-20210110.165306.hGslx.log, I'd like to see more details on how zlib is being installed, and what the full output is of the commands that are running during the sanity check...

[MH2410]

Sorry my mistake I am new in easybuild.

zlib has been installed via zypper (OS):

easybuild:~ # zypper install libz1
Loading repository data...
Reading installed packages...
'libz1' is already installed.
No update candidate for 'libz1-1.2.11-lp152.8.6.1.x86_64'. The highest available version is already installed.
Resolving package dependencies...
'libz1' providing 'zlib' is already installed.
Nothing to do.

Does it have to be installed via be installed via toolchain installation?

Logfile:
easybuild-binutils-2.34-20210110.165306.hGslx.log

This is the command I am using:
eb R-4.0.0-foss-2020a.eb --robot

[boegel]

The problem is that zlib is not getting statically linked into the binutils installation for some reason, but it's supposed to:

== 2021-01-10 17:00:38,381 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:124 in __init__): zlib is not statically linked in /home/UID/.local/easybuild/software/binutils/2.34/bin/ld: 	linux-vdso.so.1 (0x00007ffece763000)
	libbfd-2.34.so => /home/UID/.local/easybuild/software/binutils/2.34/lib64/libbfd-2.34.so (0x00007fc5c7b90000)
	libctf.so.0 => /usr/lib64/libctf.so.0 (0x00007fc5c7974000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fc5c7770000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fc5c73b5000)
	libz.so.1 => /home/UID/.local/easybuild/software/zlib/1.2.11/lib/libz.so.1 (0x00007fc5c7197000)
	libbfd-2.35.1.20201123-lp152.4.6.so => /usr/lib64/libbfd-2.35.1.20201123-lp152.4.6.so (0x00007fc5c6b8c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc5c7ecf000)

That libz.so.1 => entry should not be there (since that means libz.so.1 is dynamically linked.

I don't really understand why that's the case though, because the patching we do to try and sure that the static libz.a gets linked is taking place, and I don't see -lz being linked...

[lexming]

After discussion in Slack, this seems very related to the issue reported by @zao: #10056
The RPATH of some system library or binary (probably libctf.so in this case) are pulling system libs dynamically linking to zlib.
However, this was already fixed in EB by prepending in RPATH the libs in EB. So our fix might not be fully propagated to all link steps or something is prepending itself later on.

[zao]

I set up a Leap 15.2 VM and gave it a go:
The freshly built /eb/software/binutils/2.34/bin/ld has this RUNPATH:

 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib:/usr/lib:/usr/lib64:/eb/software/binutils/2.34/lib64]

The $ORIGIN/../lib part looks correct at first glance, but on this distribution the files are actually in $ORIGIN/../lib64:

/eb/software/binutils/2.34/lib:
libiberty.a
/eb/software/binutils/2.34/lib64:
libbfd-2.34.so  libbfd.so  libctf-nobfd.a   libctf-nobfd.so.0      libctf.so.0         libopcodes.a
libbfd.a        libctf.a   libctf-nobfd.la  libctf-nobfd.so.0.0.0  libctf.so.0.0.0     libopcodes.la
libbfd.la       libctf.la  libctf-nobfd.so  libctf.so              libopcodes-2.34.so  libopcodes.so

Due to this, it finds system libraries like libctf.so.0 in /usr/lib64 where they end up depending transitively down to libz.so:

easybuild@eb-suse152:~> tldd /eb/software/binutils/2.34/bin/ld
/eb/software/binutils/2.34/bin/ld
├─libbfd-2.34.so => /eb/software/binutils/2.34/lib64/libbfd-2.34.so (0x00007fd4b7c6f000)
│ └─libdl.so.2 => /lib64/libdl.so.2 (0x00007fd4b784f000)
│   └─libc.so.6 => /lib64/libc.so.6 (0x00007fd4b7494000)
│     └─ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fd4b7fae000)
└─libctf.so.0 => /usr/lib64/libctf.so.0 (0x00007fd4b7a53000)
  ├─libz.so.1 => /lib64/libz.so.1 (0x00007fd4b727d000)
  └─libbfd-2.35.1.20201123-lp152.4.6.so => /usr/lib64/libbfd-2.35.1.20201123-lp152.4.6.so (0x00007fd4b6c72000)

[zao]

If I nudge the paths in the correct direction, it looks much better:

easybuild@eb-suse152:~> LD_LIBRARY_PATH=/eb/software/binutils/2.34/lib64 tldd /eb/software/binutils/2.34/bin/ld
/eb/software/binutils/2.34/bin/ld
├─libbfd-2.34.so => /eb/software/binutils/2.34/lib64/libbfd-2.34.so (0x00007fc1492b5000)
│ └─libdl.so.2 => /lib64/libdl.so.2 (0x00007fc148e86000)
│   └─libc.so.6 => /lib64/libc.so.6 (0x00007fc148acb000)
│     └─ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc1495f4000)
└─libctf.so.0 => /eb/software/binutils/2.34/lib64/libctf.so.0 (0x00007fc14908a000)

[ocaisa]

I kind of see that as a problem with the binutils easyblock, it just assumes it should install libiberty.a in lib rather than check where everything else is first, see https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/b/binutils.py#L168
This causes the lib dir to be non-empty and that probably has knock-on effects that kick in further down the line.

[ocaisa]

I took a stab at fixing the binutils easyblock in easybuilders/easybuild-easyblocks#2308

[ocaisa]

@MH2410 Can you try using the updated easyblock and see if that fixes your problem:

eb --include-easyblocks-from-pr=2308 binutils-2.35.eb

[MH2410]

I took a stab at fixing the binutils easyblock in easybuilders/easybuild-easyblocks#2308

Thanks, it's working now, however installing R is still failing.