Skip to content

Commit 3b2ab27

Browse files
zhengruifengzhengruifeng
committed
[SPARK-55234][PYTHON][INFRA] Remove unused packages from remaining pyspark test images
### What changes were proposed in this pull request? Remove unused packages from remaining pyspark test images ### Why are the changes needed? for more disk space for testing ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? We had checked other similar images with python 3.10 ~ 3.14, for pypy images, I will check the scheduled jobs ### Was this patch authored or co-authored using generative AI tooling? No Closes #53998 from zhengruifeng/infra_others. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>
1 parent 63ea260 commit 3b2ab27

File tree

7 files changed

+15
-100
lines changed

7 files changed

+15
-100
lines changed

‎dev/spark-test-image/pypy-310/Dockerfile‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark wi
2424
# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
2525
LABEL org.opencontainers.image.version=""
2626

27-
ENV FULL_REFRESH_DATE=20251224
27+
ENV FULL_REFRESH_DATE=20260127
2828

2929
ENV DEBIAN_FRONTEND=noninteractive
3030
ENV DEBCONF_NONINTERACTIVE_SEEN=true
@@ -36,27 +36,14 @@ RUN apt-get update && apt-get install -y \
3636
gfortran \
3737
git \
3838
gnupg \
39-
libcurl4-openssl-dev \
40-
libfontconfig1-dev \
41-
libfreetype6-dev \
42-
libfribidi-dev \
4339
libgit2-dev \
44-
libharfbuzz-dev \
45-
libjpeg-dev \
4640
liblapack-dev \
4741
libopenblas-dev \
48-
libpng-dev \
49-
libpython3-dev \
5042
libssl-dev \
51-
libtiff5-dev \
52-
libwebp-dev \
53-
libxml2-dev \
5443
openjdk-17-jdk-headless \
5544
pkg-config \
56-
qpdf \
5745
tzdata \
5846
software-properties-common \
59-
wget \
6047
zlib1g-dev \
6148
&& apt-get autoremove --purge -y \
6249
&& apt-get clean \

‎dev/spark-test-image/pypy-311/Dockerfile‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark wi
2424
# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
2525
LABEL org.opencontainers.image.version=""
2626

27-
ENV FULL_REFRESH_DATE=20251224
27+
ENV FULL_REFRESH_DATE=20260127
2828

2929
ENV DEBIAN_FRONTEND=noninteractive
3030
ENV DEBCONF_NONINTERACTIVE_SEEN=true
@@ -36,27 +36,14 @@ RUN apt-get update && apt-get install -y \
3636
gfortran \
3737
git \
3838
gnupg \
39-
libcurl4-openssl-dev \
40-
libfontconfig1-dev \
41-
libfreetype6-dev \
42-
libfribidi-dev \
4339
libgit2-dev \
44-
libharfbuzz-dev \
45-
libjpeg-dev \
4640
liblapack-dev \
4741
libopenblas-dev \
48-
libpng-dev \
49-
libpython3-dev \
5042
libssl-dev \
51-
libtiff5-dev \
52-
libwebp-dev \
53-
libxml2-dev \
5443
openjdk-17-jdk-headless \
5544
pkg-config \
56-
qpdf \
5745
tzdata \
5846
software-properties-common \
59-
wget \
6047
zlib1g-dev \
6148
&& apt-get autoremove --purge -y \
6249
&& apt-get clean \

‎dev/spark-test-image/python-312-classic-only/Dockerfile‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark Cl
2424
# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
2525
LABEL org.opencontainers.image.version=""
2626

27-
ENV FULL_REFRESH_DATE=20251224
27+
ENV FULL_REFRESH_DATE=20260127
2828

2929
ENV DEBIAN_FRONTEND=noninteractive
3030
ENV DEBCONF_NONINTERACTIVE_SEEN=true
@@ -36,27 +36,14 @@ RUN apt-get update && apt-get install -y \
3636
gfortran \
3737
git \
3838
gnupg \
39-
libcurl4-openssl-dev \
40-
libfontconfig1-dev \
41-
libfreetype6-dev \
42-
libfribidi-dev \
4339
libgit2-dev \
44-
libharfbuzz-dev \
45-
libjpeg-dev \
4640
liblapack-dev \
4741
libopenblas-dev \
48-
libpng-dev \
49-
libpython3-dev \
5042
libssl-dev \
51-
libtiff5-dev \
52-
libwebp-dev \
53-
libxml2-dev \
5443
openjdk-17-jdk-headless \
5544
pkg-config \
56-
qpdf \
5745
tzdata \
5846
software-properties-common \
59-
wget \
6047
zlib1g-dev
6148

6249
# Install Python 3.12

‎dev/spark-test-image/python-312-pandas-3/Dockerfile‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark wi
2727
# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
2828
LABEL org.opencontainers.image.version=""
2929

30-
ENV FULL_REFRESH_DATE=20260110
30+
ENV FULL_REFRESH_DATE=20260127
3131

3232
ENV DEBIAN_FRONTEND=noninteractive
3333
ENV DEBCONF_NONINTERACTIVE_SEEN=true
@@ -39,27 +39,14 @@ RUN apt-get update && apt-get install -y \
3939
gfortran \
4040
git \
4141
gnupg \
42-
libcurl4-openssl-dev \
43-
libfontconfig1-dev \
44-
libfreetype6-dev \
45-
libfribidi-dev \
4642
libgit2-dev \
47-
libharfbuzz-dev \
48-
libjpeg-dev \
4943
liblapack-dev \
5044
libopenblas-dev \
51-
libpng-dev \
52-
libpython3-dev \
5345
libssl-dev \
54-
libtiff5-dev \
55-
libwebp-dev \
56-
libxml2-dev \
5746
openjdk-17-jdk-headless \
5847
pkg-config \
59-
qpdf \
6048
tzdata \
6149
software-properties-common \
62-
wget \
6350
zlib1g-dev
6451

6552
# Install Python 3.12

‎dev/spark-test-image/python-314-nogil/Dockerfile‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark wi
2424
# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
2525
LABEL org.opencontainers.image.version=""
2626

27-
ENV FULL_REFRESH_DATE=20251224
27+
ENV FULL_REFRESH_DATE=20260127
2828

2929
ENV DEBIAN_FRONTEND=noninteractive
3030
ENV DEBCONF_NONINTERACTIVE_SEEN=true
@@ -36,27 +36,14 @@ RUN apt-get update && apt-get install -y \
3636
gfortran \
3737
git \
3838
gnupg \
39-
libcurl4-openssl-dev \
40-
libfontconfig1-dev \
41-
libfreetype6-dev \
42-
libfribidi-dev \
4339
libgit2-dev \
44-
libharfbuzz-dev \
45-
libjpeg-dev \
4640
liblapack-dev \
4741
libopenblas-dev \
48-
libpng-dev \
49-
libpython3-dev \
5042
libssl-dev \
51-
libtiff5-dev \
52-
libwebp-dev \
53-
libxml2-dev \
5443
openjdk-17-jdk-headless \
5544
pkg-config \
56-
qpdf \
5745
tzdata \
5846
software-properties-common \
59-
wget \
6047
zlib1g-dev
6148

6249
# Install Python 3.14 (no GIL)

‎dev/spark-test-image/python-minimum/Dockerfile‎

Lines changed: 5 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark wi
2424
# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
2525
LABEL org.opencontainers.image.version=""
2626

27-
ENV FULL_REFRESH_DATE=20251225
27+
ENV FULL_REFRESH_DATE=20260127
2828

2929
ENV DEBIAN_FRONTEND=noninteractive
3030
ENV DEBCONF_NONINTERACTIVE_SEEN=true
@@ -37,30 +37,20 @@ RUN apt-get update && apt-get install -y \
3737
gfortran \
3838
git \
3939
gnupg \
40-
libcurl4-openssl-dev \
41-
libfontconfig1-dev \
42-
libfreetype6-dev \
43-
libfribidi-dev \
4440
libgit2-dev \
45-
libharfbuzz-dev \
46-
libjpeg-dev \
4741
liblapack-dev \
4842
libopenblas-dev \
49-
libpng-dev \
50-
libpython3-dev \
5143
libssl-dev \
52-
libtiff5-dev \
53-
libwebp-dev \
54-
libxml2-dev \
5544
openjdk-17-jdk-headless \
5645
pkg-config \
5746
python3.10 \
5847
python3-psutil \
59-
qpdf \
6048
tzdata \
6149
software-properties-common \
62-
wget \
63-
zlib1g-dev
50+
zlib1g-dev \
51+
&& apt-get autoremove --purge -y \
52+
&& apt-get clean \
53+
&& rm -rf /var/lib/apt/lists/*
6454

6555
ARG BASIC_PIP_PKGS="numpy==1.22.4 pyarrow==18.0.0 pandas==2.2.0 six==1.16.0 scipy scikit-learn coverage unittest-xml-reporting"
6656
# Python deps for Spark Connect

‎dev/spark-test-image/python-ps-minimum/Dockerfile‎

Lines changed: 5 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For Pandas API
2424
# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
2525
LABEL org.opencontainers.image.version=""
2626

27-
ENV FULL_REFRESH_DATE=20251225
27+
ENV FULL_REFRESH_DATE=20260127
2828

2929
ENV DEBIAN_FRONTEND=noninteractive
3030
ENV DEBCONF_NONINTERACTIVE_SEEN=true
@@ -37,30 +37,20 @@ RUN apt-get update && apt-get install -y \
3737
gfortran \
3838
git \
3939
gnupg \
40-
libcurl4-openssl-dev \
41-
libfontconfig1-dev \
42-
libfreetype6-dev \
43-
libfribidi-dev \
4440
libgit2-dev \
45-
libharfbuzz-dev \
46-
libjpeg-dev \
4741
liblapack-dev \
4842
libopenblas-dev \
49-
libpng-dev \
50-
libpython3-dev \
5143
libssl-dev \
52-
libtiff5-dev \
53-
libwebp-dev \
54-
libxml2-dev \
5544
openjdk-17-jdk-headless \
5645
pkg-config \
5746
python3.10 \
5847
python3-psutil \
59-
qpdf \
6048
tzdata \
6149
software-properties-common \
62-
wget \
63-
zlib1g-dev
50+
zlib1g-dev \
51+
&& apt-get autoremove --purge -y \
52+
&& apt-get clean \
53+
&& rm -rf /var/lib/apt/lists/*
6454

6555

6656
ARG BASIC_PIP_PKGS="pyarrow==18.0.0 pandas==2.2.0 six==1.16.0 numpy scipy coverage unittest-xml-reporting"

0 commit comments

Comments
 (0)
⚡