[RELEASE] Release version 2.4.0

[github-actions]

Release OpenSearch and OpenSearch Dashboards 2.4.0

I noticed that a manifest was automatically created in manifests/2.4.0. Please follow the following checklist to make a release.

How to use this issue

This Release Issue

This issue captures the state of the OpenSearch release, its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. More details are included in the Maintainers Release owner section.

Release Steps

There are several steps to the release process, these steps are completed as the whole release and components that are behind present risk to the release. The release owner completes the tasks in this ticket, whereas component owners resolve tasks on their ticket in their repositories.

Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release.

Component List

To aid in understanding the state of the release there is a table with status indicating each component state. This is updated based on the status of the component issues.

Preparation

• Assign this issue to a release owner.
• Declare a pencils down date for new features to be merged.
• 2022/11/04 is pencils down date for code freeze.
• Update the Campaigns section to include monitoring campaigns during this release.
• Update this issue so all __REPLACE_RELEASE-__ placeholders have actual dates.
• Document any new quality requirements or changes.
• Finalize scope and feature set and update the Public Roadmap.
• Create a release issue in every component repo that links back to this issue, update Components section with these links.
• Ensure the label is created in each component repo for this new version, and the next minor release. Create a version label
• Ensure that all release issues created above are assigned to an owner in the component team.
• Increase the build frequency for the this release from once a day (H 1 * * *) to once every hour (H/60 * * * *) in jenkinsFile

Campaigns

• [META] Add Windows Support opensearch-plugins#95
• [Meta] Add automated license header check to all plugins opensearch-plugins#138
• [PROPOSAL] Add flaky test retry  opensearch-plugins#139
• [Meta] Deprecate non-inclusive naming  OpenSearch#2589
  • Deprecate the "Master" nomenclature OpenSearch-Dashboards#1318
  • Change the "Master" nomenclature OpenSearch#472
• [IntegTest] Onboard all plugins into automated integration tests for the distribution #58
• [Campaign] Ensure 1.x and 2.x branches (main should be 3.0) opensearch-plugins#142
• [Campaign] opensearch.opensearchplugin add groupID field to org.opensearch.plugin #2521
• [CAMPAIGN] Bump jackson and snakeyaml dependencies opensearch-plugins#168

Release Readiness

Repo	Untriaged issues	Issues tagged to 2.4 version
OpenSearch
OpenSearch-Dashboards
alerting
anomaly-detection
asynchronous-search
common-utils
cross-cluster-replication
dashboards-reports
geospatial
index-management
job-scheduler
k-NN
ml-commons
notifications
observability
performance-analyzer
performance-analyzer-rca
security
sql
alerting-dashboards-plugin
anomaly-detection-dashboards-plugin
dashboards-maps
index-management-dashboards-plugin
dashboards-visualizations
opensearch-dashboards-functional-test
security-dashboards-plugin
security-analytics
security-analytics-dashboards-plugin
neural-search
dashboards-search-relevance

Release branch readiness - Ends 2022/10/31

• Component versions are auto-incremented to 2.4.0 version
• Plugins team to ensure working CI in component repositories and merge the version increment PR's.
• OpenSearch and OpenSearch-Dashboards core create 2.4 release branch
• All components create 2.4 branch for the release.

CI/CD Freeze - Ends 2022/11/02

• OpenSearch Engineering Effectiveness team will update the input manifests to point to 2.4 branch and configure the CI to run against 2.4 branch.
• Create Jenkins workflows that run daily snapshot builds for OpenSearch and OpenSearch Dashboards.

Code Complete - Ends 2022/11/03 2022/11/04

• Code Complete: Make sure that the code for this specific version of the release is ready and the branch corresponding to this release has been added to this release version manifest.
• OpenSearch / OpenSearch-Dashboards core and components teams finalize their features
• Verify pull requests to add each component to manifests/2.4.0/opensearch-2.4.0.yml and manifests/2.4.0/opensearch-dashboards-2.4.0.yml have been merged.
• Gather, review and combine the release notes from components repositories.
  • Proposal: Hyperlink directly to version download in release notes #2707
  • Add the consolidated release notes for 2.4.0 release #2887

Release testing - 2022/11/07 - 2022/11/08

• Declare a release candidate build, and provide the instructions with the release candidates for teams on testing (2022/11/08).
  • [RELEASE] Release version 2.4.0 #2649 (comment)
• Stop builds for this version of OpenSearch and/or OpenSearch Dashboards in order to avoid accidental commits going in unknowingly. Restart only if necessary else manually run the build workflow and declare new release candidate.
  • Disable auto build for 2.4.0 #2852
• After the initial release candidate creation, send PR to lock input manifest refs of both OS and OSD builds to use HEAD commit ids of the components on branch 2.4.
  • Make input manifest ci checks accept commit id as refs #2856
  • Update the ref with 2.4 branch for OSD. #2855
• Sanity Testing (2022/11/07 - 2022/11/08): Sanity testing and fixing of critical issues found by teams. Teams test their components within the distribution, ensuring integration, backwards compatibility, and perf tests pass.
• Publish all test results in the comments of this issue.

Performance testing validation - 2022/11/10 - 2022/11/11

• Performance tests do not show a regression
• Longevity tests do not show any issues

Release - ** 2022/11/10 2022/11/15**

• Verify all issues labeled v2.4.0 in all projects have been resolved.
• Complete documentation for this release.
  • @kolchfa-aws Adds windows installation documentation documentation-website#1825
• Author blog post for this release.
  • @krisfreedain Download pages for 2.4.0 release (target 11/10/2022) project-website#1133
• 2022/11/09 - Publish this release on opensearch.org.
• 2022/11/10 - Publish a blog post - release is launched!

Post Release

• Create release tags for each component (Jenkins job name: release-tag-creation).
• Replace refs in manifests/2.4.0 with tags and remove checks.
  • Update ref for 2.4.0 with tags #2906
• If this is a major or minor version release, stop building previous patch version.
• Generate distribution release notes reviewed by PM team for opensearch-build repository.
  • Add the consolidated release notes for 2.4.0 release #2887
• Increment version for Helm Charts (sample PR) for the 2.4.0 release.
  • Update appVersion for 2.4.0 release helm-charts#350
• Increment version for Ansible Charts (sample PR) for the 2.4.0 release.
  • Update versions for 2.4.0 release ansible-playbook#103
• Prepare for next patch release by incrementing patch versions for each component.
  • [AUTO] Incremented version to 2.4.1. OpenSearch#5268
  • Bump version on 2.x branch to 2.5.0 OpenSearch-Dashboards#2884
  • Bump version to 2.4.1 OpenSearch-Dashboards#2883
• Update this template with any new or missed steps.
• Create an issue for a retrospective, solicit feedback, and publish a summary.
  • [Retrospective] Release 2.4.0 #2905

Components

Component	On track	Notes
opensearch-project/OpenSearch#4755	🟢	Owner Assigned Now
opensearch-project/OpenSearch-Dashboards#2556	🟢	Owner Assigned Now
opensearch-project/alerting#600	🟢	Owner Assigned Now
opensearch-project/anomaly-detection#689	🟢	Owner Assigned Now
opensearch-project/asynchronous-search#191	🟢	Owner Assigned Now
opensearch-project/reporting#505	🟢	Owner Assigned Now
opensearch-project/dashboards-visualizations#118	🟢	Owner Assigned Now
opensearch-project/index-management#570	🟢	Owner Assigned Now
opensearch-project/job-scheduler#252	🟢	Owner Assigned Now
opensearch-project/k-NN#576	🟢	Owner Assigned Now
opensearch-project/performance-analyzer#299	🟢	Owner Assigned Now
opensearch-project/security-dashboards-plugin#1140	🟢	Owner Assigned Now
opensearch-project/security#2160	🟢	Owner Assigned Now
opensearch-project/sql#905	🟢	Owner Assigned Now
opensearch-project/observability#1114	🟢	Owner Assigned Now
opensearch-project/alerting-dashboards-plugin#347	🟢	Owner Assigned Now
opensearch-project/notifications#548	🟢	Owner Assigned Now
opensearch-project/index-management-dashboards-plugin#281	🟢	Owner Assigned Now
opensearch-project/anomaly-detection-dashboards-plugin#327	🟢	Owner Assigned Now
opensearch-project/cross-cluster-replication#588	🟢	Owner Assigned Now
opensearch-project/common-utils#280	🟢	Owner Assigned Now
opensearch-project/ml-commons#466	🟢	Owner Assigned Now
opensearch-project/geospatial#166	🟢	Owner Assigned Now
opensearch-project/dashboards-maps#60	🟢	Owner Assigned Now
opensearch-project/security-analytics#63	🟢	Owner Assigned Now
opensearch-project/security-analytics-dashboards-plugin#16	🟢	Owner Assigned Now
opensearch-project/neural-search#21	🟢	Owner Assigned Now
opensearch-project/dashboards-search-relevance#7	🟢	Owner Assigned Now

Legend

Symbol	Meaning
🟢	On track with overall release
🟡	Missed last milestone
🔴	Missed multiple milestones

PRs:

• Slow the 240 build freq as it is breaking groupId due to #2521 #2731
• Move JS to the correct position in 2.4.0 manifest. #2738
• Add null checker to distribution platform parameter #2767
• Remove additional quotes and bump build freq of 2.4.0 OS #2783
• Reduce 3.0.0 to only linux build for now #2798
• Add codeowner dashboards-search-relevance#43
• Opensearch 2.4.0 switch from 2.x to 2.4 branches and add test manifest #2838
• Add data source config to opensearch-dashboards-docker dashboards-docker-images#2
• Add data source config to opensearch-dashboards-docker #2758
• Resolve test topology check if there is no integ-test #2843
• Added support to run integTests, in either single-cluster or multi-cluster mode. #2663
• Add support for vis_builder and data_source config in build scripts #2853
• [VisBuilder] Change VisBuilder flag for docker config OpenSearch-Dashboards#2804
• Enables visbuilder tests by default opensearch-dashboards-functional-test#376
• Update 2.4.0 changes dashboards-docker-images#3
• Add Windows workflow on cypress bundle build opensearch-dashboards-functional-test#367
• Make input manifest ci checks accept commit id as refs #2856
• Add support for linux e2e template to run no-sec tests opensearch-dashboards-functional-test#383
• Add default copy for opensearch-dashboards yml and docker entrypoint #2861
• [Backport 2.4] Bump nebula-publishing-plugin from v4.4.0 to v4.6.0 OpenSearch#5131
• [2.4] Bump jackson to 2.14.0 sql#1058
• Add fix to fromXContent and toXContent in ModelGraveyard k-NN#618
• [Backport 2.4]Revert "Cluster manager task throttling feature [Final PR] (#5071)  OpenSearch#5203
• [RELEASE] Release version 2.4.0 #2649
• Freeze the ref with commit ID for RC OS 6452 & OSD 4304 #2882
• Resolve Window IntegTest copy and startup issues #2892
• Update opensearch-2.4.0.yml with OS core 5250 changes to resolve performance issues #2896
  • Revert changes in AbstractPointGeometryFieldMapper OpenSearch#5250