Skip to content

Commit b6a9813

Browse files
committed
[SPARK-53835][INFRA] Install pyarrow/torch/torchvision packages to Python 3.14 Dockefile
### What changes were proposed in this pull request? This PR aims to install `pyarrow/torch/torchvision` packages to Python 3.14 Dockefile. After this PR, the only missing dependency will be `MLFlow`. ### Why are the changes needed? Finally, they supports `Python 3.14` officially. - https://pypi.org/project/pyarrow/22.0.0/ (2025-10-24) - https://pypi.org/project/torch/2.9.0/ ### Does this PR introduce _any_ user-facing change? No, this is an infra change. ### How was this patch tested? Manual review. After merging, `Python 3.14` CI will provide a test coverage for this. https://github.com/apache/spark/actions/workflows/build_python_3.14.yml ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52751 from dongjoon-hyun/SPARK-53835. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 89167c0 commit b6a9813

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

dev/spark-test-image/python-314/Dockerfile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,13 +67,14 @@ RUN apt-get update && apt-get install -y \
6767
&& rm -rf /var/lib/apt/lists/*
6868

6969

70-
ARG BASIC_PIP_PKGS="numpy six==1.16.0 pandas==2.3.3 scipy plotly<6.0.0 coverage matplotlib openpyxl memory-profiler>=0.61.0 scikit-learn>=1.3.2"
70+
ARG BASIC_PIP_PKGS="numpy pyarrow>=22.0.0 six==1.16.0 pandas==2.3.3 scipy plotly<6.0.0 coverage matplotlib openpyxl memory-profiler>=0.61.0 scikit-learn>=1.3.2"
7171
# Python deps for Spark Connect
7272
ARG CONNECT_PIP_PKGS="grpcio==1.75.1 grpcio-status==1.71.2 protobuf==5.29.5 googleapis-common-protos==1.65.0 graphviz==0.20.3"
7373

7474
# Install Python 3.14 packages
7575
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.14
7676
RUN python3.14 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs this
7777
RUN python3.14 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting $CONNECT_PIP_PKGS lxml && \
78+
python3.14 -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu && \
7879
python3.14 -m pip install torcheval && \
7980
python3.14 -m pip cache purge

0 commit comments

Comments
 (0)