Rust 1.52.1 (2021a) attempts to fetch files from static.rust-lang.org during build phase

[LHurst-JM]

Will not build within a HPC environment without internet access as a result, even after pre-fetching sources with 'eb --fetch':

== 2021-08-02 18:26:51,149 build_log.py:265 INFO building...
== 2021-08-02 18:26:51,149 easyblock.py:3410 INFO Starting build step
== 2021-08-02 18:26:51,149 easyconfig.py:1654 INFO Generating template values...
== 2021-08-02 18:26:51,150 easyconfig.py:1673 INFO Template values: arch='x86_64', bitbucket_account='rust', builddir='/dev/shm/eb-build/Rust/1.52.1/GCCcore-10.3.0', github_account='rust', installdir='/mnt/apps/easybuild/2021a/EL-7-sky/software/Rust/1.52.1-GCCcore-10.3.0', module_name='Rust/1.52.1-GCCcore-10.3.0', name='Rust', nameletter='R', nameletterlower='r', namelower='rust', parallel='1', pymajver='3', pyminver='9', pyshortver='3.9', pyver='3.9.5', toolchain_name='GCCcore', toolchain_version='10.3.0', version='1.52.1', version_major='1', version_major_minor='1.52', version_minor='52', versionprefix='', versionsuffix=''
== 2021-08-02 18:26:51,150 easyblock.py:3418 INFO Running method build_step part of step build
== 2021-08-02 18:26:51,150 configuremake.py:338 INFO Building target ''
== 2021-08-02 18:26:51,150 run.py:233 INFO running cmd: export CARGO_HOME=/dev/shm/eb-build/Rust/1.52.1/GCCcore-10.3.0/cargo &&  make  -j 1  
== 2021-08-02 18:26:52,871 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/sources/easybuild-framework/easybuild/base/exceptions.py:124 in __init__): cmd "export CARGO_HOME=/dev/shm/eb-build/Rust/1.52.1/GCCcore-10.3.0/cargo &&  make  -j 1 " exited with exit code 2 and output:
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static.rust-lang.org:443 

spurious failure, trying again
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static.rust-lang.org:443 

spurious failure, trying again
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static.rust-lang.org:443 

spurious failure, trying again
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static.rust-lang.org:443 

spurious failure, trying again
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static.rust-lang.org:443 
failed to run: curl -s -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/eb-uym7_pkg/tmplu3bai4_.sha256 https://static.rust-lang.org/dist/2021-03-25/rust-std-1.51.0-x86_64-unknown-linux-gnu.tar.xz.sha256
Build completed unsuccessfully in 0:00:01
make: *** [all] Error 1
 (at easybuild/sources/easybuild-framework/easybuild/tools/run.py:577 in parse_cmd_output)
== 2021-08-02 18:26:52,871 build_log.py:265 INFO ... (took 1 secs)
== 2021-08-02 18:26:52,871 filetools.py:1883 INFO Removing lock /mnt/apps/easybuild/2021a/EL-7-sky/software/.locks/_mnt_apps_easybuild_2021a_EL-7-sky_software_Rust_1.52.1-GCCcore-10.3.0.lock...
== 2021-08-02 18:26:52,877 filetools.py:358 INFO Path /mnt/apps/easybuild/2021a/EL-7-sky/software/.locks/_mnt_apps_easybuild_2021a_EL-7-sky_software_Rust_1.52.1-GCCcore-10.3.0.lock successfully removed.
== 2021-08-02 18:26:52,877 filetools.py:1887 INFO Lock removed: /mnt/apps/easybuild/2021a/EL-7-sky/software/.locks/_mnt_apps_easybuild_2021a_EL-7-sky_software_Rust_1.52.1-GCCcore-10.3.0.lock
== 2021-08-02 18:26:52,877 easyblock.py:3704 WARNING build failed (first 300 chars): cmd "export CARGO_HOME=/dev/shm/eb-build/Rust/1.52.1/GCCcore-10.3.0/cargo &&  make  -j 1 " exited with exit code 2 and output:
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static.rust-lang.org:443 

spurious failure, trying again
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCAL
== 2021-08-02 18:26:52,877 easyblock.py:300 INFO Closing log for application name Rust version 1.52.1

[LHurst-JM]

Early indications are that this is not going to be an easy fix:

As with many other programming languages, rustc (the rust compiler) needs a binary from which to bootstrap. It will download a stage0 binary and many cargo crates (these are actually .tar.gz source archives) at the start of the build, so you cannot compile it without an internet connection.

(from: https://www.linuxfromscratch.org/blfs/view/svn/general/rust.html)

[LHurst-JM]

Looking at src/bootstrap/bootstrap.py, the path to the bits it needs (cargo, rustc, rustfmt) can be individually set in a configuration file, passed by RUST_BOOTSTRAP_CONFIG or --config. I think it could potentially be configured to point to a pre-extracted standalone installer (https://forge.rust-lang.org/infra/other-installation-methods.html#standalone-installers) for boostrapping (which is all this download and extract stage of the build does in boostrap.py).

It can also be told to use a path instead of fetching from the internet via RUSTUP_DIST_SERVER. It would probably be far neater and less obscure to someone familiar with EB to simply add the required sources to the easyconfig's sources and set this environment variable.

[jerowe]

@LHurst-JM I'm having the same problem. Do you have an easybuild config workaround?

[LHurst-JM]

Hi @jerowe I'm afraid not. I gave up on testing 2021a and stuck with our existing 2020b toolchain for now - the Intel compilers now depend on rust somehow (I've not dug throught the whole dependency tree to find out what it pulling it in) in 2021a so we can't do anything meaninful with 2021a in our environment, at the moment.

[jerowe]

Oh wow, thanks for responding so quickly!

I'll just stick with the 2020b toolchain too then. ;-)

Thanks again!

[ocaisa]

It looks like there are ways to control this, see https://github.com/dtolnay/bootstrap

[LHurst-JM]

It looks like there are ways to control this, see https://github.com/dtolnay/bootstrap

Downloading an existing Rust binary shouldn't be a huge issue, if it can be done via sources in the easyconfig (although doing it in a platform-agnostic way....?).

Rust do provide offline installers (https://forge.rust-lang.org/infra/other-installation-methods.html#standalone-installers), I guess the key question is should the easyconfig just use one of them or fetch one and then do a local build from a (temporary) install? (i.e. is there an optimisation advantage to building locally c.f. just using their provided "offline" binaries?)

[casparvl]

For what it's worth, I ran into the same issue, and tried two approaches. Both approaches below are manual. My idea was: let's first find what works in a manual way, and then (maybe) we can automate this with EasyBuild. It's a long story, but it might help get someone on track to find a proper solution and implement it, since I already spent more time on this than I have...

Approach 1: trying to get the current EasyConfig to run offline by putting sources in a cachedir

If you run the EasyConfig for Rust-1.52.1-GCCcore-10.3.0.eb right now, one of the first things you'll see is

downloading https://static.rust-lang.org/dist/2021-03-25/rust-std-1.51.0-x86_64-unknown-linux-gnu.tar.xz
extracting /tmp/jenkins/build/Rust/1.52.1/GCCcore-10.3.0/rustc-1.52.1-src/build/cache/2021-03-25/rust-std-1.51.0-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-03-25/rustc-1.51.0-x86_64-unknown-linux-gnu.tar.xz
extracting /tmp/jenkins/build/Rust/1.52.1/GCCcore-10.3.0/rustc-1.52.1-src/build/cache/2021-03-25/rustc-1.51.0-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-03-25/cargo-1.51.0-x86_64-unknown-linux-gnu.tar.xz
extracting /tmp/jenkins/build/Rust/1.52.1/GCCcore-10.3.0/rustc-1.52.1-src/build/cache/2021-03-25/cargo-1.51.0-x86_64-unknown-linux-gnu.tar.xz

Clearly, you want to avoid those files being downloaded, and you can: by putting those sources in the cache directory before the build. This would require two things:
a. Download all of these sources, ideally as part of the fetch step. That way, one could get them in the conventional way by running eb --fetch
b. Make sure the sources get copied to the cache directory somewhere in between that directory being created, and the installation actually starting. From my manual test, I think you can manually create that cache directory, and then start the install - starting the install should not clear the cache dir. But not: the cachedir is something like <buildpath>/build/cache/<some_date>. Note that <some_date> is not the current date, for version 1.52.1 it's always '2021-03-25' (maybe the date of the release?).

This works quite fine in a manual setting. But then, even if the sources for rustc, rust-std and cargo are present, the next step will start downloading stuff again...

    Updating crates.io index
 Downloading crates ...
  Downloaded crossbeam-utils v0.7.2
  Downloaded version_check v0.9.1
  Downloaded walkdir v2.3.1
  Downloaded toml v0.5.7
  Downloaded getopts v0.2.21
  Downloaded lazy_static v1.4.0
  Downloaded unicode-xid v0.2.1
... etc ...

I'm completely unfamiliar with Rust & Cargo, but as I understand it, a 'crate' is being installed here, and a 'crate' defines its dependencies and downloads them at installation time, based on this crates.io index. That makes things difficult. As I understand it, the vendor argument to cargo can be used to include all dependcies in the project you want to ship (see https://stackoverflow.com/questions/32267233/how-to-build-a-project-using-cargo-in-an-offline-environment). As they say there:

First, run cargo vendor. This will setup a new directory named vendor in the root of your crate. It will then download dependencies from crates.io and git, and store them in this new directory.

My first thought: ok, we could potentially do that on an online system, in which case this would have to be part of the fetch step in EasyBuild. One challenge is that you need to have the cargo command present in order to run this. That command is one of the things installed by the Rust EasyConfig. So it would be a bit of a chicken and egg problem: you wouldn't be able to use this in the fetch step, because the cargo command is not there yet.

But: good news. Looking in the src directory of the rustc-1.52.1-src tarball, there is already a vendor directory!

[casparl@tcn1 rustc-1.52.1-src]$ ls -l src/vendor/
total 254
drwxr-xr-x 6 casparl casparl 4096 May  3 10:27 aho-corasick
drwxr-xr-x 5 casparl casparl 4096 May  3 10:26 ammonia
drwxr-xr-x 4 casparl casparl 4096 May  3 10:26 ansi_term
drwxr-xr-x 5 casparl casparl 4096 May  3 10:27 arrayvec
drwxr-xr-x 5 casparl casparl 4096 May  3 10:26 assert_cli
drwxr-xr-x 4 casparl casparl 4096 May  3 10:26 atty
drwxr-xr-x 6 casparl casparl 4096 May  3 10:27 backtrace
drwxr-xr-x 3 casparl casparl 4096 May  3 10:27 backtrace-sys
.. etc ..

So, why isn't it used? Turns out you need to pass a configure flag:

./configure --enable-vendor

Alternatively, you can modify the config.toml (instead of running ./configure):

# Indicate whether the vendored sources are used for Rust dependencies or not
vendor = true

One thing to note (since e.g. #14320 mentioned lack of Power support when using prebuilt cryptography wheels): this approach uses the 'traditional' installation method of Rust, which means an older Rust compiler is used to build a newer one. That means and older, binary rustc has to be downloaded that is compatible with one's architecture. I think Rust provides binaries for a lot of architectures, but you're still relying on some binary and thus will not be able to just build for 'any' architecture. Also, it will mean that in EasyBuild's fetch step, we'll need to manually determine which tarbal we need, but it should be possible to figure that out.

Approach 2: trying to use mrustc to build a rust compiler

This is based on @ocaisa 's linked https://github.com/dtolnay/bootstrap repo. Note that https://github.com/thepowersgang/mrustc is actually the more up-to-date version of this, as this contains an mrustc version 0.10 (and properly tagged versions and everything), but the bootstrap repo has some scripts that string the compilations together in a bootstrapped way.

mrustc is a compiler for Rust source that itself is written in C. In theory, this means there is no need for bootstrapping a build, i.e. building the next Rust compiler with a previous Rust compiler. In practice however, they support only a few versions to build directly with mrustc and still rely on the bootstrapping for the rest.

Supports (and can bootstrap) rustc 1.19.0, 1.29.0, 1.39.0, and 1.54.0

I succeeded in building 1.29.0 (the 'default') with mrustc-0.10 by running

• make RUSTCSRC (to download the sources)
• make -f minicargo.mk (according to docs: builds mrustc and minicargo, then libstd, libtest, rustcandcargo`)
• make -C run_rustc (according to docs: builds libstd and a "Hello, world" using the above rustc`)
  As per the instructions for Linux GNU/MacOX on https://github.com/thepowersgang/mrustc

To be honest, I think the third just goes through the same, full compilation I went through in the 2nd step, before compiling a 'hello, world', since I now had rustc compilers in two output directories, output and run_rustc (actually, the latter contained two builds for some reason):

[casparl@tcn1 mrustc-0.10]$ find -name rustc
./rustc-1.29.0-src/src/doc/rustc
./rustc-1.29.0-src/src/rustc
./output/rustc
./run_rustc/output/prefix-2/bin/rustc
./run_rustc/output/prefix-s/bin/rustc

Also, only the 3nd option seems to seperate into bin and lib directories, but spread over different prefixes for some reason. E.g.

[casparl@tcn1 mrustc-0.10]$ find -name libstd.so
./run_rustc/output/build-std2/x86_64-unknown-linux-gnu/release/libstd.so
./run_rustc/output/build-std2/x86_64-unknown-linux-gnu/release/deps/libstd.so
./run_rustc/output/prefix-2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.so
[casparl@tcn1 mrustc-0.10]$ find -name libcore.rlib
./output/libcore.rlib
./run_rustc/output/prefix-s/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore.rlib

To me, it's rather unclear why these two 'make' commands behave so differently, why the 3rd one creates multiple prefixes, and which are the versions we would need to install (or actually: use to build the next rustc version).

There are some other inconveniences there. The first is that make -f minicargo.mk resulted in a CXXABI error since it picked up libstdc++ from the host instead of the one from my GCCcore-10.3.0 EasyBuild installation. I had to edit minicargo.mk because it overwrites LD_LIBRARY_PATH instead of prepending/appending. This line fixes it:

sed -i 's/LD_LIBRARY_PATH=$(abspath output)/LD_LIBRARY_PATH=$(abspath output):${LD_LIBRARY_PATH}/g' mrustc-0.10/minicargo.mk

Also, the make -C run_rustc ended with

    Finished release [optimized] target(s) in 15.77s
[CARGO] ../rustc-1.29.0-src/src/rustc/Cargo.toml > output/build-rustc/
output/prefix/bin/cargo: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory
make: *** [Makefile:195: output/prefix/bin/rustc] Error 127
make: Leaving directory '/gpfs/home4/casparl/TMP/Rust_from_source3/mrustc-0.10/run_rustc'

It seems to have completed the build, but then still print that error probably because cargo is invoked. This might be because the makefile in run_rustc/Makefile also overwrites instead of appends the LD_LIBRARY_PATH, but I'm not 100% sure. If I ran

[casparl@tcn1 mrustc-0.10]$ ldd run_rustc/output/prefix/bin/cargo
        linux-vdso.so.1 (0x00007fffc81fd000)
        libc.so.6 => /lib64/libc.so.6 (0x000014aa6a3db000)
        libm.so.6 => /lib64/libm.so.6 (0x000014aa6a059000)
        librt.so.1 => /lib64/librt.so.1 (0x000014aa69e51000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x000014aa69c31000)
        libutil.so.1 => /lib64/libutil.so.1 (0x000014aa69a2d000)
        libgcc_s.so.1 => /home/casparl/.local/easybuild/Centos8/2021/software/GCCcore/10.3.0/lib64/libgcc_s.so.1 (0x000014aa6a99f000)
        libdl.so.2 => /lib64/libdl.so.2 (0x000014aa69829000)
        libz.so.1 => /sw/arch/Centos8/EB_production/2021/software/zlib/1.2.11-GCCcore-10.3.0/lib/libz.so.1 (0x000014aa6a97b000)
        libssl.so.1.1 => /sw/arch/Centos8/EB_production/2021/software/OpenSSL/1.1/lib/libssl.so.1.1 (0x000014aa69595000)
        libcrypto.so.1.1 => /sw/arch/Centos8/EB_production/2021/software/OpenSSL/1.1/lib/libcrypto.so.1.1 (0x000014aa690af000)
        libcurl.so.4 => /sw/arch/Centos8/EB_production/2021/software/cURL/7.76.0-GCCcore-10.3.0/lib/libcurl.so.4 (0x000014aa6a8dc000)
        libatomic.so.1 => /home/casparl/.local/easybuild/Centos8/2021/software/GCCcore/10.3.0/lib64/libatomic.so.1 (0x000014aa6a8d3000)
        /lib64/ld-linux-x86-64.so.2 (0x000014aa6a7a0000)
        libidn2.so.0 => /lib64/libidn2.so.0 (0x000014aa68e91000)
        libunistring.so.2 => /lib64/libunistring.so.2 (0x000014aa68b10000)

Everything got resolved correctly, so I don't think it's a real issue, but it's hard to be 100% sure that it based all steps of the make before exiting due to this error.

From the various rustc / cargo exectuables I had available, I added the following to the PATH:

[casparl@tcn1 rustc-1.30.0-src]$ which cargo
~/TMP/Rust_from_source3/mrustc-0.10/run_rustc/output/prefix/bin/cargo
[casparl@tcn1 rustc-1.30.0-src]$ which rustc
~/TMP/Rust_from_source3/mrustc-0.10/run_rustc/output/prefix-s/bin/rustc

And then tried:

./configure --enable-local-rust

Looking at the config.toml created by this, I saw that it contained

# Instead of downloading the src/stage0.txt version of Cargo specified, use
# this Cargo binary instead to build all Rust code
cargo = '/home/casparl/TMP/Rust_from_source3/mrustc-0.10/./run_rustc/output/prefix/bin/cargo'

# Instead of downloading the src/stage0.txt version of the compiler
# specified, use this rustc binary instead as the stage0 snapshot compiler.
rustc = '/home/casparl/TMP/Rust_from_source3/mrustc-0.10/./run_rustc/output/prefix-s/bin/rustc'

so that looks as expected. Then, I ran

./x.py

It no longer tries to download the an older version of the compilers, i.e. I no longer get things like:

downloading https://static.rust-lang.org/dist/2018-10-12/rust-std-1.29.2-x86_64-unknown-linux-gnu.tar.gz

Now, it seems to no longer be downloading from crates.io, but it indeed seems to use the vendored sources from src/vendor. However, I run into:

error[E0463]: can't find crate for `proc_macro`
  --> vendor/proc-macro2/src/lib.rs:53:1
   |
53 | extern crate proc_macro;
   | ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error: Could not compile `proc-macro2`.

There is a libproc_macro.rlib in the mrustc-0.10/output directory, but if I use the rustc and cargo from that directory, a python x.py errors out with:

[casparl@tcn1 rustc-1.30.0-src]$ python x.py
info: Downloading and building bootstrap before processing --help
      command. See src/bootstrap/README.md for help with common
      commands.
   Compiling proc-macro2 v0.4.13
error[E0463]: can't find crate for `std`

error: Could not compile `proc-macro2`.

To learn more, run the command again with --verbose.
failed to run: /home/casparl/TMP/Rust_from_source3/mrustc-0.10/output//cargo build --manifest-path /gpfs/home4/casparl/TMP/Rust_from_source3/rustc-1.30.0-src/src/bootstrap/Cargo.toml --frozen

This is where I gave up on this approach...

Conclusion
Approach 1 just works (in a manual setting) on an offline system. You'll need to download and ship the cargo rustc and rust-std tarballs for you architecture and put them in the build directory (<buildpath>/build/cache/<some_date>) on your offline system. Then, running the configure command as it is in the current EasyConfig, but with the addition of --enable-vendor (i.e. ./configure --prefix=<installpath> --enable-vendor --enable-extended --sysconfdir=<installpath>/etc --set=llvm.ninja=false), then ./x.py build then ./x.py install should do the job.

I think approach 2 is getting pretty close, but my lack of understanding of mrustc is probably why the proc-macro and/or std crates are either not build, or not found in the right place. The build procedure with mrustc feels not very mature (e.g. with the makefile that overwrites LD_LIBRARY_PATH instead of append/prepend, and the messy output structure). Approach 1 is definitely easier and less error prone. Approach 2 would only be needed if you fully wanted to build from source, and not boostrap with a downloaded version of a previous compiler. The list of supported/downloadable compiler binaries for bootstrapping is likely very similar to the standalone installers listed here. That includes pretty much all the major architectures, you'd have to have something pretty exotic for it not to be supported.

TODO
What I did not get to, is create a custom EasyBlock for Rust that follows approach 1. It should be pretty trivial though

• Adaptation of the fetch step, so that the cargo rustc and rust-std tarballs of the previous release are downloaded for your architecture.
• A pre-build step that creates the cachedir, and puts the pre-fetched tarballs for the bootstrap compilers there.
• Configure step should include --enable-vendor.
• Build step should just be ./x.py build --stage 2. It could also use make but the docs say this is not the 'recommended build system' (neither is the configure script, but since the recommended way to configure is modifying the config.toml by hand, I think we'll stick with the configure script :) ). Note that we might want to set CARGO_BUILD_JOBS for build parallelism.
• Install step should just be ./x.py install (or make install, if we prefer)

In theory, that should do it...

[casparvl]

Oh and as a woraround for now: until we have a proper offline installation method for Rust via a dedicated EasyBlock, the dirty workaround can just be to install cryptography as a wheel if you only needed Rust as a build dependency for that. For x86, I modified the Python-3.9.5-GCCcore-10.3.0.eb EasyConfig to:

...
# For easier offline install, use cryptography from a wheel
#    ('Rust', '1.52.1'),  # required for setuptools-rust, which is needed for cryptography
...
    ('cryptography', '3.4.7', {
        'source_tmpl': '%(namelower)s-%(version)s-cp36-abi3-manylinux2014_x86_64.whl',
        'checksums': ['1e056c28420c072c5e3cb36e2b23ee55e260cb04eee08f702e0edfec3fb51959'],
    }),
...

You'll need to run the same customized EasyConfig with --fetch on an online system to grab the wheel of course, and then you can easily run it on an offline system. Worked for me :)

[boegel]

@casparvl Thanks a lot for that very detailed breakdown!

That indeed looks like promising progress towards a fully offline Rust installation...

I would be happy if we could get option 1 working. Although that implies downloading pre-built binaries, they're only build dependencies, so I don't consider that a huge problem. And if no pre-built binaries are available, we're likely in for a world of pain anyway (since that probably means that CPU architecture is not well supported by Rust).

[casparvl]

Although that implies downloading pre-built binaries, they're only build dependencies, so I don't consider that a huge problem. And if no pre-built binaries are available, we're likely in for a world of pain anyway (since that probably means that CPU architecture is not well supported by Rust).

I fully agree with this. The pre-build binaries are no threat to performance, as they are build dependencies, and they support a very wide range of binaries. If you're on an architecture that isn't on that list, it's indeed probably best to assume that Rust is simply not supported there at all.