set `CMAKE_CUDA_HOST_COMPILER`, `CMAKE_CUDA_COMPILER`, and `CMAKE_CUDA_ARCHITECTURES` when using CUDA in CMakeMake easyblock by bedroge · Pull Request #3523 · easybuilders/easybuild-easyblocks

bedroge · 2024-11-30T12:47:48Z

When compiling a CUDA version of GROMACS in EESSI we noticed that nvcc would pick up the g++ from the compat layer sysroot as host compiler (instead of the GCC from the EB stack), leading to weird issues/errors:

-- Check for working CUDA compiler: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software/CUDA/12.4.0/bin/nvcc
-- Check for working CUDA compiler: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software/CUDA/12.4.0/bin/nvcc - broken
CMake Error at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/CMake/3.27.6-GCCcore-13.2.0/share/cmake-3.27/Modules/CMakeTestCUDACompiler.cmake:100 (message):
  The CUDA compiler

    "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software/CUDA/12.4.0/bin/nvcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/tmp/bot/easybuild/build/GROMACS/2024.4/foss-2023b-CUDA-12.4.0/easybuild_obj/CMakeFiles/CMakeScratch/TryCompile-JFpCnw'
    
    Run Build Command(s): /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/CMake/3.27.6-GCCcore-13.2.0/bin/cmake -E env VERBOSE=1 /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/gmake -f Makefile cmTC_b4aca/fast
    /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/gmake  -f CMakeFiles/cmTC_b4aca.dir/build.make CMakeFiles/cmTC_b4aca.dir/build
    gmake[1]: Entering directory '/tmp/bot/easybuild/build/GROMACS/2024.4/foss-2023b-CUDA-12.4.0/easybuild_obj/CMakeFiles/CMakeScratch/TryCompile-JFpCnw'
    Building CUDA object CMakeFiles/cmTC_b4aca.dir/main.cu.o
    /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software/CUDA/12.4.0/bin/nvcc -forward-unknown-to-host-compiler   -std=c++17 "--generate-code=arch=compute_52,code=[compute_52,sm_52]" -MD -MT CMakeFiles/cmTC_b4aca.dir/main.cu.o -MF CMakeFiles/cmTC_b4aca.dir/main.cu.o.d -x cu -c /tmp/bot/easybuild/build/GROMACS/2024.4/foss-2023b-CUDA-12.4.0/easybuild_obj/CMakeFiles/CMakeScratch/TryCompile-JFpCnw/main.cu -o CMakeFiles/cmTC_b4aca.dir/main.cu.o
    Linking CUDA executable cmTC_b4aca
    /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/CMake/3.27.6-GCCcore-13.2.0/bin/cmake -E cmake_link_script CMakeFiles/cmTC_b4aca.dir/link.txt --verbose=1
    /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/g++ -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/mpi4py/3.1.5-gompi-2023b/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/mpi4py/3.1.5-gompi-2023b/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/networkx/3.2.1-gfbf-2023b/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/networkx/3.2.1-gfbf-2023b/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/SciPy-bundle/2023.11-gfbf-2023b/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/SciPy-bundle/2023.11-gfbf-2023b/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software/CUDA/12.4.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software/CUDA/12.4.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/scikit-build-core/0.9.3-GCCcore-13.2.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/scikit-build-core/0.9.3-GCCcore-13.2.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/FFTW.MPI/3.3.10-gompi-2023b/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/FFTW.MPI/3.3.10-gompi-2023b/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/ScaLAPACK/2.2.0-gompi-2023b-fb/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/ScaLAPACK/2.2.0-gompi-2023b-fb/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.3.1-GCC-13.2.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/FlexiBLAS/3.3.1-GCC-13.2.0/lib -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/GCCcore/13.2.0/lib64 -L/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/GCCcore/13.2.0/lib  @CMakeFiles/cmTC_b4aca.dir/objects1.rsp -o cmTC_b4aca @CMakeFiles/cmTC_b4aca.dir/linkLibs.rsp -L"/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software/CUDA/12.4.0/targets/x86_64-linux/lib/stubs" -L"/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software/CUDA/12.4.0/targets/x86_64-linux/lib"
    /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/10/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/cmTC_b4aca.dir/main.cu.o: relocation R_X86_64_32 against `.bss' can not be used when making a PIE object; recompile with -fPIE
    /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/10/../../../../x86_64-pc-linux-gnu/bin/ld: failed to set dynamic section sizes: bad value
    collect2: error: ld returned 1 exit status
    gmake[1]: *** [CMakeFiles/cmTC_b4aca.dir/build.make:102: cmTC_b4aca] Error 1
    gmake[1]: Leaving directory '/tmp/bot/easybuild/build/GROMACS/2024.4/foss-2023b-CUDA-12.4.0/easybuild_obj/CMakeFiles/CMakeScratch/TryCompile-JFpCnw'
    gmake: *** [Makefile:127: cmTC_b4aca/fast] Error 2

The right host compiler can be set using the $CUDAHOSTCXX environment variable, which is preferred over CMAKE_CUDA_HOST_COMPILER according to https://cmake.org/cmake/help/latest/envvar/CUDAHOSTCXX.html (though that may change in the future, see https://gitlab.kitware.com/cmake/cmake/-/issues/23160).

edit: for consistency reasons, I did switch to the CMAKE_CUDA_HOST_COMPILER, see #3523 (comment).

bedroge · 2024-11-30T18:34:30Z

@boegelbot please test @ jsc-zen3-a100
EB_ARGS="AMGX-2.4.0-foss-2023a-CUDA-12.1.1.eb waLBerla-6.1-foss-2023a-CUDA-12.1.1.eb bitsandbytes-0.43.3-foss-2023a-CUDA-12.1.1.eb dorado-0.8.0-foss-2023a-CUDA-12.1.1.eb GROMACS-2024.4-foss-2023b-CUDA-12.4.0-PLUMED-2.9.2.eb GROMACS-2024.4-foss-2023b-CUDA-12.4.0.eb --installpath /tmp/$USER/pr3523"
CORE_CNT=16

boegelbot · 2024-11-30T18:53:09Z

@bedroge: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3523 EB_ARGS="AMGX-2.4.0-foss-2023a-CUDA-12.1.1.eb waLBerla-6.1-foss-2023a-CUDA-12.1.1.eb bitsandbytes-0.43.3-foss-2023a-CUDA-12.1.1.eb dorado-0.8.0-foss-2023a-CUDA-12.1.1.eb GROMACS-2024.4-foss-2023b-CUDA-12.4.0-PLUMED-2.9.2.eb GROMACS-2024.4-foss-2023b-CUDA-12.4.0.eb --installpath /tmp/$USER/pr3523" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3523 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 5360

Test results coming soon (I hope)...

Details

- notification for comment with ID 2509135920 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-11-30T20:03:24Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS waLBerla-6.1-foss-2023a-CUDA-12.1.1.eb
SUCCESS bitsandbytes-0.43.3-foss-2023a-CUDA-12.1.1.eb
SUCCESS dorado-0.8.0-foss-2023a-CUDA-12.1.1.eb
SUCCESS GROMACS-2024.4-foss-2023b-CUDA-12.4.0-PLUMED-2.9.2.eb
SUCCESS GROMACS-2024.4-foss-2023b-CUDA-12.4.0.eb
SUCCESS CCCL-2.3.0-GCCcore-12.3.0-CUDA-12.1.1.eb
FAIL (build issue) AMGX-2.4.0-foss-2023a-CUDA-12.1.1.eb (partial log available at https://gist.github.com/boegelbot/34d86b74b7066587909cbe883a3d8b3c)

Build succeeded for 6 out of 7 (6 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.19
See https://gist.github.com/boegelbot/f9fa4c623c01f9f2f5a3549caae7b683 for a full test report.

edit: not sure why AMGX failed, but it looks like a runtime issue with the test suite, so I guess it's not related to this PR (unfortunately the original easyconfig PR for AMGX was not tested on jsc-zen3-a100).

Co-authored-by: ocaisa <alan.ocais@cecam.org>

bedroge · 2024-12-02T14:20:09Z

@boegelbot please test @ jsc-zen3-a100
EB_ARGS="waLBerla-6.1-foss-2023a-CUDA-12.1.1.eb bitsandbytes-0.43.3-foss-2023a-CUDA-12.1.1.eb dorado-0.8.0-foss-2023a-CUDA-12.1.1.eb GROMACS-2024.4-foss-2023b-CUDA-12.4.0-PLUMED-2.9.2.eb GROMACS-2024.4-foss-2023b-CUDA-12.4.0.eb --installpath /tmp/$USER/pr3523"
CORE_CNT=16

boegelbot · 2024-12-02T14:33:09Z

@bedroge: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3523 EB_ARGS="waLBerla-6.1-foss-2023a-CUDA-12.1.1.eb bitsandbytes-0.43.3-foss-2023a-CUDA-12.1.1.eb dorado-0.8.0-foss-2023a-CUDA-12.1.1.eb GROMACS-2024.4-foss-2023b-CUDA-12.4.0-PLUMED-2.9.2.eb GROMACS-2024.4-foss-2023b-CUDA-12.4.0.eb --installpath /tmp/$USER/pr3523" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3523 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 5364

Test results coming soon (I hope)...

Details

- notification for comment with ID 2511671250 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-12-02T15:31:40Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS waLBerla-6.1-foss-2023a-CUDA-12.1.1.eb
SUCCESS bitsandbytes-0.43.3-foss-2023a-CUDA-12.1.1.eb
SUCCESS dorado-0.8.0-foss-2023a-CUDA-12.1.1.eb
SUCCESS GROMACS-2024.4-foss-2023b-CUDA-12.4.0-PLUMED-2.9.2.eb
SUCCESS GROMACS-2024.4-foss-2023b-CUDA-12.4.0.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.19
See https://gist.github.com/boegelbot/0513ce6b96fc957dcf860c6aa9dc8537 for a full test report.

ocaisa

LGTM

pass the preferred host compiler to the CUDA compiler

ee21659

bedroge added bug fix EESSI Related to EESSI project labels Nov 30, 2024

ocaisa reviewed Dec 2, 2024

View reviewed changes

Comment thread easybuild/easyblocks/generic/cmakemake.py Outdated

also set CMAKE_CUDA_COMPILER and CMAKE_CUDA_ARCHITECTURES

0857b9f

ocaisa reviewed Dec 2, 2024

View reviewed changes

Comment thread easybuild/easyblocks/generic/cmakemake.py Outdated

use absolute path for nvcc

90495ed

Co-authored-by: ocaisa <alan.ocais@cecam.org>

ocaisa approved these changes Dec 2, 2024

View reviewed changes

ocaisa merged commit 9a8da09 into easybuilders:develop Dec 2, 2024

bedroge deleted the cmakemake_set_cudahostcxx branch December 2, 2024 15:44

bedroge changed the title ~~pass the preferred host compiler to the CUDA compiler in the CMakeMake easyblock~~ set CMAKE_CUDA_HOST_COMPILER, CMAKE_CUDA_COMPILER, and CMAKE_CUDA_ARCHITECTURES when using CUDA in CMakeMake easyblock Dec 2, 2024

boegel added this to the release after 4.9.4 milestone Dec 3, 2024

boegel modified the milestones: release after 4.9.4, 5.0.0 Mar 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

set `CMAKE_CUDA_HOST_COMPILER`, `CMAKE_CUDA_COMPILER`, and `CMAKE_CUDA_ARCHITECTURES` when using CUDA in CMakeMake easyblock#3523

set `CMAKE_CUDA_HOST_COMPILER`, `CMAKE_CUDA_COMPILER`, and `CMAKE_CUDA_ARCHITECTURES` when using CUDA in CMakeMake easyblock#3523
ocaisa merged 3 commits intoeasybuilders:developfrom
bedroge:cmakemake_set_cudahostcxx

bedroge commented Nov 30, 2024 •

edited

Loading

Uh oh!

bedroge commented Nov 30, 2024

Uh oh!

boegelbot commented Nov 30, 2024

Uh oh!

boegelbot commented Nov 30, 2024 •

edited by bedroge

Loading

Uh oh!

Uh oh!

Uh oh!

bedroge commented Dec 2, 2024

Uh oh!

boegelbot commented Dec 2, 2024

Uh oh!

boegelbot commented Dec 2, 2024

Uh oh!

ocaisa left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

bedroge commented Nov 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedroge commented Nov 30, 2024

Uh oh!

boegelbot commented Nov 30, 2024

Uh oh!

boegelbot commented Nov 30, 2024 • edited by bedroge Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of tested easyconfigs (in order)

Uh oh!

Uh oh!

Uh oh!

bedroge commented Dec 2, 2024

Uh oh!

boegelbot commented Dec 2, 2024

Uh oh!

boegelbot commented Dec 2, 2024

Overview of tested easyconfigs (in order)

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bedroge commented Nov 30, 2024 •

edited

Loading

boegelbot commented Nov 30, 2024 •

edited by bedroge

Loading