Open
Conversation
Spark 4 officially supports both Java 17 and 21. This adds a JDK 21 profile next to the existing JDK 17 profile in the Spark 4 matrix across pr_build_linux.yml and spark_sql_test.yml, extends the conditional JAVA_TOOL_OPTIONS to apply the existing --add-opens / --add-exports flags on JDK 21, mirrors the known sql_hive-1 exclude (issue apache#2946) for the new JDK 21 row, and updates the Spark 4 entry in the installation compatibility table. Closes apache#4059
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #4059
Rationale for this change
Spark 4 officially supports both Java 17 and 21 (ref). The current Comet CI and compatibility docs only cover Java 17 for Spark 4.0, which leaves Java 21 — an LTS release, widely adopted across the Spark / Arrow / Netty / Iceberg / Delta stack — without official validation.
This PR adds Java 21 as a supported runtime for the experimental Spark 4.0 tier so downstream consumers can adopt it with an upstream signal.
What changes are included in this PR?
.github/workflows/pr_build_linux.yml: add aSpark 4.0, JDK 21profile next to the existingSpark 4.0, JDK 17profile in both matrix sections. Extend the conditionalJAVA_TOOL_OPTIONSso the existing--add-opens/--add-exportsflags also apply under JDK 21 (they remain valid — same as Spark 4's own launcher scripts set for both JDK versions)..github/workflows/spark_sql_test.yml: add{spark-short: '4.0', spark-full: '4.0.1', java: 21, scan-impl: 'auto'}to theconfig:matrix. Mirror the existingsql_hive-1exclude (tracked under Spark 4.0.x hive-1 jdk17 tests failing consistently in multiple PRs #2946) for the new JDK 21 row.docs/source/user-guide/latest/installation.md: update the Spark 4.0.1 row of the experimental compatibility table from17to17/21.Skipped because they have no Spark 4 rows to mirror:
spark_sql_test_native_iceberg_compat.yml(Spark 3.4/3.5 × Java 11 only)iceberg_spark_test.yml(Spark 3.4/3.5 × Java 11/17 only)How are these changes tested?
By the CI itself — this PR's purpose is to add the test profile. The new matrix rows exercise the existing full test suite under a JDK 21 runtime; any genuine incompatibility surfaces as a row-specific failure. If the JDK 21 rows go green, Comet can legitimately claim JDK 21 support for Spark 4 in the compatibility matrix.
The
--add-opens/--add-exportsflag set is the same one Spark 4 ships in its launcher scripts for both JDK 17 and 21, so no flag divergence is expected.Additionally, I've run comet a bit in JDK 21 successfully, but not extensively.