[SPARK-55367][PYTHON] Use venv for run-pip-tests#54154
[SPARK-55367][PYTHON] Use venv for run-pip-tests#54154gaogaotiantian wants to merge 7 commits intoapache:masterfrom
Conversation
JIRA Issue Information=== Bug SPARK-55367 === This comment was automatically generated by GitHub Actions |
|
It is a nice change if we can drop the usage of conda. |
|
Yeah it's an unnecessary layer of complexity to setup conda just for pip test. We should be able to do it with pure python. |
|
Can this fix resolve the following issues? 
|
|
@LuciferYang yes and the PR that made the error above is already reverted. |
| RUN add-apt-repository ppa:deadsnakes/ppa | ||
| RUN apt-get update && apt-get install -y \ | ||
| python3.14-nogil \ | ||
| python3.14-venv \ |
There was a problem hiding this comment.
it seems we cannot install packages in this way, deadsnakes doesn't support such package.
And for venv, do we need to additionally install it? It should be available after install python
There was a problem hiding this comment.
Huh? Could you post the failure? It's only not supported in 3.14 or 3.14t? That's how I install venv.
So venv, like many other "standard libraries", is actually optional to Python package. How to package it is determined by the distribution. On Linux I believe venv is not always packaged with python.
We can use uv, but this should in theory work. deadsnake should have it.
There was a problem hiding this comment.
https://github.com/zhengruifeng/spark/actions/runs/21749252022/job/62742834544
it acutually failed with python3.14-pip
When upgrading the os to ubuntu 24, it seems the venv is already available after intsalling python3.14
There was a problem hiding this comment.
Hmm, okay. In that case I'll remove this line.
|
@zhengruifeng the notification failed. Build result in https://github.com/gaogaotiantian/spark/actions/runs/21883385818 I removed |
|
merged to master |
|
@gaogaotiantian it seems python 3.14t workflow is broken https://github.com/apache/spark/actions/runs/21880964892/job/63163219324 I guess it needs a non-gil version venv? |
|
@zhengruifeng this run was still using the old workflow (before this PR). 3.14t runs every 3 days. Let's wait for the latest run. |
### What changes were proposed in this pull request? Use `venv` instead of `conda` or `virtualenv` for `run-pip-tests`. Remove the `conda` dependency in our CI. ### Why are the changes needed? `run-pip-tests` require a virtual environment which we used to achieve with `conda` or `virtualenv`. However, `venv`(https://docs.python.org/3/library/venv.html) is the recommended way to create a virtual environment since python 3.5. It's a standard library so we don't need any new dependency. It just require python to work. In this way we can just remove the conda part which is messing with our CI when it installs the same version of python as our docker image. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I tried it locally and it worked. Let's wait for CI results. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#54154 from gaogaotiantian/redo-pip-test. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
3 participants
What changes were proposed in this pull request?
Use
venvinstead ofcondaorvirtualenvforrun-pip-tests. Remove thecondadependency in our CI.Why are the changes needed?
run-pip-testsrequire a virtual environment which we used to achieve withcondaorvirtualenv. However,venv(https://docs.python.org/3/library/venv.html) is the recommended way to create a virtual environment since python 3.5. It's a standard library so we don't need any new dependency. It just require python to work.In this way we can just remove the conda part which is messing with our CI when it installs the same version of python as our docker image.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
I tried it locally and it worked. Let's wait for CI results.
Was this patch authored or co-authored using generative AI tooling?
No.