Skip to content

PoC: update PythonPackage easyblock to build wheel first when installing Python package with pip#3068

Closed
Flamefire wants to merge 2 commits intoeasybuilders:developfrom
Flamefire:20240105123332_new_pr_pythonpackage
Closed

PoC: update PythonPackage easyblock to build wheel first when installing Python package with pip#3068
Flamefire wants to merge 2 commits intoeasybuilders:developfrom
Flamefire:20240105123332_new_pr_pythonpackage

Conversation

@Flamefire
Copy link
Copy Markdown
Contributor

@Flamefire Flamefire commented Jan 5, 2024

(created using eb --new-pr)

This is a Proof of Concept only.

It adds the option to set use_pip = 'wheel' which matches more our build-test-install cycle.

Currently when using pip we have this:

  • build step does nothing
  • test step installs into a temporary directory with pip install --prefix /tmp/foo .
  • install step installs to the final location with pip install --prefix /final/foo .

Confusingly all build options need to be passed to (pre)installopts and as Python usually creates a "wheel file" (basically a compiled, installable zip) this duplicates work.

This option when set does the following:

  • build step: pip wheel --wheel-dir /new-tmp-dir
  • test step installs into a temporary directory with pip install --prefix /tmp/foo /new-tmp-dir/*.whl
  • install step installs to the final location with pip install --prefix /final/foo /new-tmp-dir/*.whl

This avoids duplicate work and ensures that what is tested is what gets installed (currently we build twice which could yield different results). It is also nice to have separate timing and output for the actual build

However it turned out a bit more involved than anticipated. E.g. not all options for pip install apply to pip wheel and vice versa. Also the time savings are not that great. I tested PyTorch 2.1.2 on a (admittedly high-cpu) machine. And after an initial build/install the times are:

  • setup.py install: ~35s
  • pip install .: ~63s
  • pip install *.whl: ~17s

So the time saving is around 11s which IMO isn't worth the added complexity.
However it brings back a dedicated build step, which shows up as such in EB status/progress and not confusingly in the test or install step.

@Flamefire Flamefire marked this pull request as draft January 5, 2024 11:33
@boegel boegel added this to the 5.0 milestone Jan 17, 2024
@boegel boegel changed the title PoC: Build wheel first when installing Python package with pip PoC: update PythonPackage eaysyblock to build wheel first when installing Python package with pip Jan 17, 2024
@boegel boegel changed the title PoC: update PythonPackage eaysyblock to build wheel first when installing Python package with pip PoC: update PythonPackage easyblock to build wheel first when installing Python package with pip Jan 17, 2024
@Flamefire Flamefire force-pushed the 20240105123332_new_pr_pythonpackage branch from 3b1123c to 30fb1bc Compare January 19, 2024 07:47
@boegel boegel modified the milestones: 5.0.0, release after 5.0.0 Mar 18, 2025
@Flamefire Flamefire force-pushed the 20240105123332_new_pr_pythonpackage branch from 15680d1 to d487e9f Compare March 20, 2025 17:06
@boegel boegel modified the milestones: next release (5.1.0), 5.x May 22, 2025
@Flamefire Flamefire force-pushed the 20240105123332_new_pr_pythonpackage branch from d487e9f to 36349b5 Compare May 23, 2025 08:56
Check that replacement actually happens and wheel files are created.
Also change config values for following install step(s).
@Flamefire Flamefire force-pushed the 20240105123332_new_pr_pythonpackage branch from 36349b5 to 144af11 Compare May 23, 2025 09:31
@Flamefire Flamefire closed this Oct 16, 2025
@Flamefire
Copy link
Copy Markdown
Contributor Author

Closing as the added complexity is not worth the gain

@Flamefire Flamefire deleted the 20240105123332_new_pr_pythonpackage branch October 16, 2025 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants