[pyspark] Add support for Spark Connect ML

[medb]

Fixes #9780 #11510

ci todos:

• Revert change to CI tag.

[medb]

@WeichenXu123 @wbo4958 may you take a look at this draft PR? It's in very early stages, but I would like to get feedback whether this is a right direction.

[trivialfis]

Thank you for the PR! I will leave the review to @WeichenXu123 and @wbo4958

Out of curiosity, last time @wbo4958 work on this topic, it was suggested to call the XGBoost JVM package from the Python client. Is this still relevant? (I was against the idea, just asking to understand the current status)

[wbo4958]

Could you be able to add some unit tests for it?

[medb]

@wbo4958 I have modified existing unit tests to run for Spark Classic and Spark Connect, PTAL

[medb]

@trivialfis @RAMitchell @wbo4958 @WeichenXu123 I have rebased PR on the main branch HEAD and it's ready for review now. PTAL

[trivialfis]

E pyspark.errors.exceptions.base.PySparkImportError: [PACKAGE_NOT_INSTALLED] zstandard >= 0.25.0 must be installed; however, it was not found.

err.

I will run some tests locally first.

[trivialfis]

ok, I managed to sort out the dependencies. Once the review from @wbo4958 is resolved, I will merge the devop PR and update the tag here.

[wbo4958]

LGTM, Thx.