Skip to content

[doc] Add Spark 4.0 compatibility warning for JVM packages #12136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wbo4958 wants to merge 1 commit into
base: master
Choose a base branch
from
+30 −0
Open

[doc] Add Spark 4.0 compatibility warning for JVM packages #12136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions doc/jvm/xgboost4j_spark_tutorial.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,21 @@ in order to add XGBoost4J-Spark as a dependency for your project. We provide bot
Also, make sure to install Spark directly from `Apache website <https://spark.apache.org/>`_. **Upstream XGBoost is not guaranteed to
work with third-party distributions of Spark, such as Cloudera Spark.** Consult appropriate third parties to obtain their distribution of XGBoost.

.. warning:: Spark 4.0.0 is not compatible with XGBoost4J-Spark

Apache Spark 4.0.0 introduced a breaking change to the ``org.apache.spark.ml.param.Param`` class
constructor (`SPARK-52259 <https://issues.apache.org/jira/browse/SPARK-52259>`_), which causes a
``NoSuchMethodError`` when instantiating any XGBoost estimator:

.. code-block:: text

java.lang.NoSuchMethodError: 'void org.apache.spark.ml.param.Param.<init>(
org.apache.spark.ml.util.Identifiable, java.lang.String, java.lang.String, scala.Function1)'

This issue affects all third-party ML libraries that use ``Param``. It was fixed in
**Spark 4.0.1** and **Spark 4.1.0**. If you are using Spark 4.x, please upgrade to
**Spark 4.0.1 or later**.

Data Preparation
================

Expand Down
15 changes: 15 additions & 0 deletions doc/jvm/xgboost_spark_migration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,3 +113,18 @@ Starting from xgboost4j-spark 3.0, below parameters are removed.
- singlePrecisionHistogram
- lambdaBias
- objectiveType

***************************
Spark 4.0 Compatibility
***************************

XGBoost4J-Spark JARs built against Spark 3.5 are binary compatible with **Spark 4.0.1+** and
**Spark 4.1.0+**. No code changes are required.

.. warning::

**Spark 4.0.0** is **not** compatible due to an upstream bug
(`SPARK-52259 <https://issues.apache.org/jira/browse/SPARK-52259>`_) that changed the
constructor signature of ``org.apache.spark.ml.param.Param``. This causes a
``NoSuchMethodError`` at runtime when instantiating any XGBoost estimator. The fix is included
in Spark 4.0.1 and later. Please avoid Spark 4.0.0 and upgrade to at least **Spark 4.0.1**.
Loading
⚡