[doc] Add Spark 4.0 compatibility warning for JVM packages#12136
Open
wbo4958 wants to merge 1 commit intodmlc:masterfrom
Open
[doc] Add Spark 4.0 compatibility warning for JVM packages#12136wbo4958 wants to merge 1 commit intodmlc:masterfrom
wbo4958 wants to merge 1 commit intodmlc:masterfrom
Conversation
Spark 4.0.0 has a binary incompatibility in `org.apache.spark.ml.param.Param` (SPARK-52259) that breaks all third-party ML libraries including XGBoost4J-Spark. The issue is fixed in Spark 4.0.1 and 4.1.0. Add warnings to the Spark tutorial and migration guide so users know to avoid Spark 4.0.0 and upgrade to at least Spark 4.0.1. Made-with: Cursor
Member
|
I think a simple troubleshooting note should be sufficient |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
doc/jvm/xgboost4j_spark_tutorial.rst) about Spark 4.0.0 binary incompatibilitydoc/jvm/xgboost_spark_migration.rst)Context
Apache Spark 4.0.0 introduced a breaking change to
org.apache.spark.ml.param.Paramconstructor (SPARK-52259), which causesNoSuchMethodErrorwhen instantiating any XGBoost estimator (e.g.,XGBoostClassifier,XGBoostRegressor). This affects all third-party ML libraries that useParam.Test results:
NoSuchMethodErroronParamconstructorThe upstream fix is included in Spark 4.0.1 (released 2025-09-02) and Spark 4.1.0. XGBoost4J-Spark JARs built against Spark 3.5 work on Spark 4.0.1+ without any code changes.
This PR adds documentation warnings so users know to avoid Spark 4.0.0 and use Spark 4.0.1 or later.
Related
Made with Cursor