convert all generic easyblocks to `run_shell_cmd` by bartoldeman · Pull Request #3046 · easybuilders/easybuild-easyblocks

bartoldeman · 2023-12-06T13:02:40Z

For ConfigureMake the default for the verbose= parameter for build_step is reversed in line with the default for run_shell_cmd.

The default for the verbose= parameter for build_step is reversed in line with the default for run_shell_cmd.

boegelbot · 2023-12-06T13:43:37Z

@bartoldeman: Tests failed in GitHub Actions, see https://github.com/easybuilders/easybuild-easyblocks/actions/runs/7115210289
Output from first failing test suite run:

ERROR: test_det_cmake_version (test.easyblocks.easyblock_specific.EasyBlockSpecificTest)
Tests for det_cmake_version function provided along with CMakeMake generic easyblock.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/easybuild-easyblocks/easybuild-easyblocks/test/easyblocks/easyblock_specific.py", line 231, in test_det_cmake_version
    self.assertErrorRegex(EasyBuildError, "Failed to determine CMake version", det_cmake_version)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/easybuild/base/testing.py", line 164, in assertErrorRegex
    call(*args, **kwargs)
  File "/home/runner/work/easybuild-easyblocks/easybuild-easyblocks/easybuild/easyblocks/generic/cmakemake.py", line 66, in det_cmake_version
    cmd_res = run_shell_cmd(cmd, hidden=True)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/easybuild/tools/run.py", line 166, in cache_aware_func
    res = func(cmd, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/easybuild/tools/run.py", line 323, in run_shell_cmd
    raise_run_shell_cmd_error(res)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/easybuild/tools/run.py", line 149, in raise_run_shell_cmd_error
    raise RunShellCmdError(cmd_res, caller_info)
easybuild.tools.run.RunShellCmdError: Shell command 'cmake' failed!

----------------------------------------------------------------------
Ran 597 tests in 52.171s

FAILED (errors=1)
ERROR: Not all tests were successful.

bleep, bloop, I'm just a bot (boegelbot v20200716.01)
Please talk to my owner @boegel if you notice me acting stupid),
or submit a pull request to https://github.com/boegel/boegelbot fix the problem.

It handles errors itself already by just looking at the output, no need also do it in run_shell_cmd here. This fixes the test case for det_cmake_version().

bartoldeman · 2023-12-12T16:17:13Z

All generic easyblocks have been converted. Time for testing...

boegel · 2024-01-03T13:57:08Z

@SebastianAchilles Can you look into testing this a bit, as soon as easybuilders/easybuild-framework#4423 is merged?

SebastianAchilles · 2024-01-03T14:02:37Z

Yes, I will test this PR, as soon as easybuilders/easybuild-framework#4423 is merged.

SebastianAchilles · 2024-01-04T10:52:38Z

Test report by @SebastianAchilles

Overview of tested easyconfigs (in order)

SUCCESS GCC-system.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
jscclxl1.int.jsc-clx.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, Intel Xeon Processor (Cascadelake) (cascadelake), Python 3.9.18
See https://gist.github.com/SebastianAchilles/4d8f88f630344f13ce8be3286e6a10fe for a full test report.

SebastianAchilles · 2024-01-05T15:07:54Z

I think it makes sense to test, if possible, one EasyConfig per modified EasyBlock. To find recent EasyConfigs for an EasyBlock <EasyBlock> I used:

find . -type f | xargs grep -l "easyblock = '<EasyBlock>'" | xargs ls -rt | tail -20

I intend to test the following EasyConfigs:

EasyBlock	Corresponding EasyConfig to test
`Binary`	`Nextflow-23.10.0.eb`
`Cargo`	`cryptography-41.0.1-GCCcore-12.3.0.eb`
`Cargo`	`Evcxr-REPL-0.14.2-GCCcore-12.2.0-Rust-1.65.0.eb`
`CMakeMake`	`RapidJSON-1.1.0-20230928-GCCcore-12.3.0.eb`
`CmdCp`	`Ninja-1.11.1-GCCcore-13.2.0.eb`
`Conda`	`TRUST4-1.0.5.1.eb`
`ConfigureMake`	`strace-6.6-GCCcore-12.2.0.eb`
`ConfigureMakePythonPackage`	`MUSCLE3-0.7.0-foss-2022b.eb`
`FortranPythonPackage`	using derived EasyBlock numpy `SciPy-bundle-2023.11-gfbf-2023b.eb`
`GoPackage`	`SeqKit-2.3.1.eb`
`IntelBase`	using derived EasyBlock Advisor `Advisor-2023.2.0.eb`
`JuliaPackage`	`ResistanceGA-4.2-5-foss-2022a-R-4.2.1-Julia-1.9.2.eb`
`MesonNinja`	`libGLU-9.0.3-GCCcore-13.2.0.eb`
`OCamlPackage`	no easyconfig found
`OctavePackage`	`Octave-7.1.0-foss-2021b.eb`
`PerlModule`	`Bio-SearchIO-hmmer-1.7.3-GCC-11.2.0.eb`
`PythonPackage`	`Pillow-10.0.0-GCCcore-12.3.0.eb`
`RPackage`	`castor-1.7.11-foss-2022a.eb`
`RPM`	no easyconfig found
`RubyGem`	`Ruby-3.2.2-GCCcore-12.2.0.eb`
`SCons`	`Cantera-3.0.0-foss-2023a.eb`
`SystemCompiler`	`GCC-system.eb`
`SystemMPI`	dependencies not installed
`Tarball`	`VSCode-1.85.0.eb`
`VersionIndependentPythonPackage`	`JUBE-2.4.1.eb`
`Waf`	no easyconfig found

SebastianAchilles · 2024-01-05T18:10:22Z

Test report by @SebastianAchilles

Overview of tested easyconfigs (in order)

SUCCESS Nextflow-23.10.0.eb
SUCCESS cryptography-41.0.1-GCCcore-12.3.0.eb
SUCCESS Evcxr-REPL-0.14.2-GCCcore-12.2.0-Rust-1.65.0.eb
SUCCESS RapidJSON-1.1.0-20230928-GCCcore-12.3.0.eb
SUCCESS Ninja-1.11.1-GCCcore-13.2.0.eb
SUCCESS TRUST4-1.0.5.1.eb
SUCCESS strace-6.6-GCCcore-12.2.0.eb
SUCCESS MUSCLE3-0.7.0-foss-2022b.eb
SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
SUCCESS SeqKit-2.3.1.eb
SUCCESS Advisor-2023.2.0.eb
SUCCESS ResistanceGA-4.2-5-foss-2022a-R-4.2.1-Julia-1.9.2.eb
SUCCESS libGLU-9.0.3-GCCcore-13.2.0.eb
SUCCESS Bio-SearchIO-hmmer-1.7.3-GCC-11.2.0.eb
SUCCESS Pillow-10.0.0-GCCcore-12.3.0.eb
SUCCESS castor-1.7.11-foss-2022a.eb
SUCCESS Ruby-3.2.2-GCCcore-12.2.0.eb
SUCCESS Cantera-3.0.0-foss-2023a.eb
SUCCESS GCC-system.eb
SUCCESS VSCode-1.85.0.eb
SUCCESS JUBE-2.4.1.eb

Build succeeded for 21 out of 21 (21 easyconfigs in total)
jscclxc1.int.jsc-clx.fz-juelich.de - Linux Rocky Linux 9.3, x86_64, Intel Xeon Processor (Cascadelake) (cascadelake), Python 3.9.18
See https://gist.github.com/SebastianAchilles/d676bcc8bde9132013cb2c0d11476d3e for a full test report.

SebastianAchilles

lgtm

Requested changes have been implemented.

SebastianAchilles · 2024-01-06T09:25:21Z

Going in, thanks @bartoldeman!

boegel · 2024-01-07T16:05:06Z


                cmd = self.compose_install_command(test_installdir, extrapath=extrapath)
-                run_cmd(cmd, log_all=True, simple=True, verbose=False)
+                run_shell_cmd(cmd, hidden=True)


@bartoldeman Did you intentionally hide the installation command before running the tests in PythonPackage.test_step?

Just wondering, not opposing it, sort of makes sense, but it is different from what we had before (verbose=False in run_cmd is not equivalent with hidden=True in run_shell_cmd)

This looks like an oversight, since trace=False (now hidden=True) was used in many other places in this easyblock. I'll file a new PR for consistency.

To address easybuilders#3046 (comment) as a followup to easybuilders#3046

Convert ConfigureMake easyblock to run_shell_cmd.

714bd2c

The default for the verbose= parameter for build_step is reversed in line with the default for run_shell_cmd.

bartoldeman marked this pull request as draft December 6, 2023 13:02

bartoldeman added the EasyBuild-5.0 EasyBuild 5.0 label Dec 6, 2023

bartoldeman added this to the 5.0 milestone Dec 6, 2023

bartoldeman added 4 commits December 6, 2023 13:20

Convert CMakeMake easyblock to run_shell_cmd.

1cabe3c

Convert MesonNinja easyblock to run_shell_cmd.

93de2a9

Convert Binary easyblock to run_shell_cmd.

a4485b4

Convert CmdCp easyblock to run_shell_cmd.

4a3442f

boegel previously requested changes Dec 6, 2023

View reviewed changes

Comment thread easybuild/easyblocks/generic/configuremake.py Outdated

Fix use of out in CMakeMake easyblock

d71b71e

boegel changed the title ~~Convert generic easyblocks to run_shell_cmd.~~ Convert generic easyblocks to run_shell_cmd Dec 6, 2023

bartoldeman added 18 commits December 7, 2023 14:26

Deprecate 'verbose' parameter for build_step in ConfigureMake

c5b7f46

Use fail_on_error=False in det_cmake_version().

d5e54a9

It handles errors itself already by just looking at the output, no need also do it in run_shell_cmd here. This fixes the test case for det_cmake_version().

Convert PythonPackage easyblock to run_shell_cmd.

a9be397

Convert ConfigureMakePythonPackage easyblock to run_shell_cmd

d7b1d46

Convert FortranPythonPackage easyblock to run_shell_cmd

5d80cd8

Convert VersionIndependentPythonPackage easyblock to run_shell_cmd

53ef067

Convert Tarball easyblock to run_shell_cmd

3b31e92

Convert PerlModule easyblock to run_shell_cmd

d7c6474

Convert RPackage easyblock to run_shell_cmd

52fa3c3

Convert Cargo easyblock to run_shell_cmd

30a907e

Convert Conda easyblock to run_shell_cmd

c039ab4

Convert GoPackage easyblock to run_shell_cmd

d89524c

Convert JuliaPackage easyblock to run_shell_cmd

816af4d

Convert OCamlPackage easyblock to run_shell_cmd

853a9f7

Convert OctavePackage easyblock to run_shell_cmd

8b4d0c3

Convert RubyGem easyblock to run_shell_cmd

a87daf2

Convert IntelBase easyblock to run_shell_cmd

bce0644

Convert Rpm easyblock to run_shell_cmd

73364a6

bartoldeman added 7 commits December 12, 2023 15:47

Convert SCons easyblock to run_shell_cmd

ff8e9ec

Convert SystemCompiler easyblock to run_shell_cmd

e43be23

Convert SystemMPI easyblock to run_shell_cmd

1a7fe10

Convert Waf easyblock to run_shell_cmd

40d10fa

pythonpackage: log_ok=False -> fail_on_error=False to use ec

f402ac9

rpackage: log_ok=False -> fail_on_error=False

5e71a44

Fix use of out in systemcompiler.py

6f69813

bartoldeman marked this pull request as ready for review December 12, 2023 16:17

SebastianAchilles self-assigned this Jan 5, 2024

SebastianAchilles approved these changes Jan 6, 2024

View reviewed changes

SebastianAchilles merged commit 437a74c into easybuilders:5.0.x Jan 6, 2024

easybuilders deleted a comment from boegelbot Jan 7, 2024

boegel reviewed Jan 7, 2024

View reviewed changes

bartoldeman added a commit to bartoldeman/easybuild-easyblocks that referenced this pull request Jan 15, 2024

pythonpackage: Remove hidden=True from run_shell_cmd for consistency

0aafc8c

To address easybuilders#3046 (comment) as a followup to easybuilders#3046

bartoldeman mentioned this pull request Jan 15, 2024

remove hidden=True from run_shell_cmd in PythonPackage easyblock for consistency #3074

Merged

boegel changed the title ~~Convert generic easyblocks to run_shell_cmd~~ convert all generic easyblocks to run_shell_cmd Jan 17, 2024

bartoldeman added a commit to ComputeCanada/easybuild-easyblocks that referenced this pull request Apr 3, 2025

pythonpackage: Remove hidden=True from run_shell_cmd for consistency

3dae7a2

To address easybuilders#3046 (comment) as a followup to easybuilders#3046

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

convert all generic easyblocks to `run_shell_cmd`#3046

convert all generic easyblocks to `run_shell_cmd`#3046
SebastianAchilles merged 31 commits intoeasybuilders:5.0.xfrom
bartoldeman:generic-run-cmd

bartoldeman commented Dec 6, 2023

Uh oh!

Uh oh!

boegelbot commented Dec 6, 2023

Uh oh!

bartoldeman commented Dec 12, 2023

Uh oh!

boegel commented Jan 3, 2024

Uh oh!

SebastianAchilles commented Jan 3, 2024

Uh oh!

SebastianAchilles commented Jan 4, 2024

Uh oh!

SebastianAchilles commented Jan 5, 2024

Uh oh!

SebastianAchilles commented Jan 5, 2024

Uh oh!

SebastianAchilles left a comment

Uh oh!

SebastianAchilles commented Jan 6, 2024

Uh oh!

boegel Jan 7, 2024

Uh oh!

bartoldeman Jan 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

bartoldeman commented Dec 6, 2023

Uh oh!

Uh oh!

boegelbot commented Dec 6, 2023

Uh oh!

bartoldeman commented Dec 12, 2023

Uh oh!

boegel commented Jan 3, 2024

Uh oh!

SebastianAchilles commented Jan 3, 2024

Uh oh!

SebastianAchilles commented Jan 4, 2024

Overview of tested easyconfigs (in order)

Uh oh!

SebastianAchilles commented Jan 5, 2024

Uh oh!

SebastianAchilles commented Jan 5, 2024

Overview of tested easyconfigs (in order)

Uh oh!

SebastianAchilles left a comment

Choose a reason for hiding this comment

Uh oh!

SebastianAchilles commented Jan 6, 2024

Uh oh!

boegel Jan 7, 2024

Choose a reason for hiding this comment

Uh oh!

bartoldeman Jan 8, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants