Skip to content

Commit 40f3ea7

Browse files
committed
[SPARK-52927][PYTHON][INFRA] Upgrade pyarrow to 15.0 in image python-minimum
### What changes were proposed in this pull request? Upgrade pyarrow to 15.0 in image `python-minimum` I will test pyarrow version for `python-ps-minimum`, and finally update the docs, in separate PRs. ### Why are the changes needed? recent changes used features in newer versions, e.g. 1. `pa.Table.from_struct_array` used in 537f813 is only available since 15.0 2. `pa.array(list of pa.Scalar)` used in arrow UDF, is only available since 13.0 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? PR builder with ``` default: '{"PYSPARK_IMAGE_TO_TEST": "python-minimum", "PYTHON_TO_TEST": "python3.10"}' ``` https://github.com/zhengruifeng/spark/runs/46547147080 ### Was this patch authored or co-authored using generative AI tooling? No Closes #51632 from zhengruifeng/fix_arrow_old_version. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>
1 parent 79ba12a commit 40f3ea7

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

dev/spark-test-image/python-minimum/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ RUN apt-get update && apt-get install -y \
6161
wget \
6262
zlib1g-dev
6363

64-
ARG BASIC_PIP_PKGS="numpy==1.22.4 pyarrow==11.0.0 pandas==2.2.0 six==1.16.0 scipy scikit-learn coverage unittest-xml-reporting"
64+
ARG BASIC_PIP_PKGS="numpy==1.22.4 pyarrow==15.0.0 pandas==2.2.0 six==1.16.0 scipy scikit-learn coverage unittest-xml-reporting"
6565
# Python deps for Spark Connect
6666
ARG CONNECT_PIP_PKGS="grpcio==1.67.0 grpcio-status==1.67.0 googleapis-common-protos==1.65.0 graphviz==0.20 protobuf"
6767

0 commit comments

Comments
 (0)