Release next version of OpenSearch Hadoop

[harshavamsi]

We've had substantial changes in the hadoop client which now supports OpenSearch serverless, We should release a new minor version with an upcoming major version with changes supporting OpenSearch 3.x

[lawofcycles]

Thanks for creating this issue. Happy to help if there is anything I can do for the release.

Regarding the upcoming major version with changes supporting OpenSearch 3.x, are there specific OS 3.x API breaking changes that would require client updates?

[harshavamsi]

Thanks! Unfortunately we don't have a client generator for the hadoop client like we do for Java, Python etc. It uses the API definitions from https://github.com/opensearch-project/opensearch-api-specification/pulls to generate new APIs. We could do a draft PR updating the required version in the build files to be 3.x and update the github workflows to consume from 3.x to see if any existing API is breaking

[lawofcycles]

I did a local build test with OpenSearch 3.5.0 to see what breaks. Here's what I found.

Changes needed to build with build-tools 3.x

1. buildSrc/opensearch-hadoop-version.properties updated opensearch/lucene/build-tools to 3.5.0/10.3.2/3.5.0
2. buildSrc/build.gradle bumped JDK check from 11 to 21 (build-tools 3.x requires JDK 21)
3. build.gradle removed the explicit Shadow plugin declaration (com.gradleup.shadow 9.2.2), since build-tools 3.x pulls in shadow-gradle-plugin 8.3.9 as a transitive dependency and the two conflict

Integration test results against OpenSearch 3.5.0

Module	Result
MR	All passed
Hive	All passed
Spark 3.x (sql-30)	All passed
Spark 4.0 (sql-40)	All passed

No code changes were needed. The OpenSearch 3.5.0 test cluster started successfully and all REST API interactions (bulk indexing, search, scroll, etc.) worked without issues.

I can put together a PR with these changes. Some items to consider going forward.

1. Should we keep 2.x tests alongside 3.x, or move entirely to 3.x?
2. The runtimeJavaHome in BaseBuildPlugin.groovy currently requires JDK 8 and is used for both the OpenSearch test cluster and the Hadoop test cluster (qa/kerberos). Since OpenSearch 3.x requires JDK 21, we'll need to separate these.

[andrross]

Catch All Triage - 1

@lawofcycles Is JDK-8 support a requirement here? I'd recommend dropping support for it as its been end of life for quite a while now.

[lawofcycles]

@andrross I agree with dropping JDK 8 support. Currently JDK 8 is used in two ways in this project.

1. Build dependency: BaseBuildPlugin requires JDK 8 as runtimeJavaHome, which is used for running integration tests, generating Javadoc, and providing rt.jar for Scala bootclasspath. These can all be migrated to a higher JDK.

2. Runtime target: sourceCompatibility/targetCompatibility are currently set to 1.8. Spark 3.4 and 3.5 still support JDK 8, but they also support JDK 11 and 17 (Spark 3.4, Spark 3.5). Spark 4.0+ requires JDK 17+ (Spark 4.0).

Given these factors, I think we can drop JDK 8 for both build and runtime.

[lawofcycles]

#730 has been merged. OpenSearch 3.x build-tools support, JDK 8 drop, and Hadoop 3.4.3 upgrade are now on main.

Note that Hive integration tests are currently skipped because Hive 3.1.2's HiveServer2 requires JDK 8. Upgrading to Hive 4.x will be addressed in a separate issue.

I'd like to start working on the remaining tasks for the next release. I'll put together a task list and share it here.

[lawofcycles]

As a separate topic for future discussion: it may be worth considering whether to continue maintaining the Hive and MapReduce modules. Looking at the issues and the users I've interacted with, the demand seems to be concentrated on the Spark integration. Not suggesting any immediate action, but it could simplify maintenance significantly if we eventually focus on Spark only.

[lawofcycles]

Here's the task list I have in mind for the next release.

Version number

The next release includes breaking changes (JDK 8 dropped, OpenSearch 3.x required, Spark 2.x removed), so a major version bump to 2.0.0 seems appropriate. Open to discussion on this.

Tasks

1. Bump version from 1.3.0 to 2.0.0 in buildSrc/opensearch-hadoop-version.properties
2. Update release and snapshot workflows to include Spark 3.5 and 4.0 artifacts. publish_snapshot.yml and release_drafter.yml currently only publish opensearch-spark-30_2.12 and opensearch-spark-30_2.13. Need to add opensearch-spark-35_2.12, opensearch-spark-35_2.13, and opensearch-spark-40_2.13.
3. Update dist/build.gradle to include Spark 3.5 and 4.0 jars in the distribution zip
4. Finalize CHANGELOG: replace [Unreleased] with [2.0.0]
5. Tag and run release workflow
6. Maven Central publication. I believe this is handled via the opensearch-build release process once the draft release is created, based on [release]: OpenSearch Hadoop Client opensearch-build#3385. @harshavamsi could you confirm?
7. Draft release notes
8. Release blog post