Merge branch 'main' into release-automation-notes

Author: ZilongX

.github/workflows/backport.yml

@@ -12,6 +12,17 @@ jobs:
       contents: write
       pull-requests: write
     name: Backport
+    # Only react to merged PRs for security reasons.
+    # See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target.
+    if: >
+      github.event.pull_request.merged
+      && (
+        github.event.action == 'closed'
+        || (
+          github.event.action == 'labeled'
+          && contains(github.event.label.name, 'backport')
+        )
+      )
     steps:
       - name: GitHub App token
         id: github_app_token
@@ -22,10 +33,9 @@ jobs:
           # opensearch-trigger-bot installation ID
           installation_id: 22958780
 
-      # Using fork of https://github.com/tibdex/backport
-      # https://github.com/tibdex/backport/pull/81
       - name: Backport
-        uses: VachaShah/backport@v1.1.4
+        uses: VachaShah/backport@v2.1.0
         with:
           github_token: ${{ steps.github_app_token.outputs.token }}
-          branch_name: backport/backport-${{ github.event.number }}
+          head_template: backport/backport-<%= number %>-to-<%= base %>
+          files_to_skip: "CHANGELOG.md"

.github/workflows/cypress_workflow.yml

@@ -11,7 +11,8 @@ env:
   FTR_PATH: 'ftr'
   START_CMD: 'node ../scripts/opensearch_dashboards --dev --no-base-path --no-watch'
   OPENSEARCH_SNAPSHOT_CMD: 'node ../scripts/opensearch snapshot'
-  SPEC: 'cypress/integration/core-opensearch-dashboards/opensearch-dashboards/*.js,'
+  SPEC: 'cypress/integration/core-opensearch-dashboards/opensearch-dashboards/**/*.js,'
+  CYPRESS_ENV: 'env CYPRESS_VISBUILDER_ENABLED=true '
 
 jobs:
   cypress-tests:
@@ -75,7 +76,7 @@ jobs:
           working-directory: ${{ env.FTR_PATH }}
           start: ${{ env.OPENSEARCH_SNAPSHOT_CMD }}, ${{ env.START_CMD }}
           wait-on: 'http://localhost:9200, http://localhost:5601'
-          command: yarn cypress:run-without-security --browser chromium --spec ${{ env.SPEC }}
+          command: ${{ env.CYPRESS_ENV }} yarn cypress:run-without-security --browser chromium --spec ${{ env.SPEC }}
 
       # Screenshots are only captured on failure, will change this once we do visual regression tests
       - uses: actions/upload-artifact@v3

CHANGELOG.md

@@ -10,6 +10,10 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ### 🛡 Security
 
+- [Legacy Maps Plugin] Prevent reverse-tabnabbing ([#2540](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2540))
+- Eliminate dependency on `got` versions older than 11.8.5 ([#2801](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2801))
+- [Multi DataSource] Add explicit no spellcheck on password fields ([#2818](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2818))
+
 ### 📈 Features/Enhancements
 
 - [MD] Support legacy client for data source ([#2204](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2204))
@@ -32,41 +36,60 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - [Vis Builder] Rename wizard to visBuilder in i18n id and formatted message id ([#2635](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2635))
 - [Vis Builder] Rename wizard to visBuilder in class name, type name and function name ([#2639](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2639))
 - [Vis Builder] Rename wizard on save modal and visualization table ([#2645](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2645))
+- [Vis Builder] Adds functional tests to CI ([#2728](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2728))
 - [Vis Builder] Enable VisBuilder by default ([#2725](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2725))
 - Change save object type, wizard id and name to visBuilder #2673 ([#2673](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2673))
 - [Multi DataSource] Update MD data source documentation link ([#2693](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2693))
 - [Save Object Aggregation View] Add extension point in saved object management to register namespaces and show filter ([#2656](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2656))
 - [Save Object Aggregation View] Fix for export all after scroll count response changed in PR#2656 ([#2696](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2696))
+- [Vis Builder] Add an experimental table visualization in vis builder ([#2705](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2705))
+- [Vis Builder] Add field summary popovers ([#2682](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2682))
+- Add yarn opensearch arg to setup plugin dependencies ([#2544](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2544))
 
 ### 🐛 Bug Fixes
 
 - [Vis Builder] Fixes auto bounds for timeseries bar chart visualization ([2401](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2401))
 - [Vis Builder] Fixes visualization shift when editing agg ([2401](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2401))
 - [Vis Builder] Renames "Histogram" to "Bar" in vis type picker ([2401](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2401))
 - [Vis Builder] Update vislib params and misc fixes ([2610](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2610))
-* [Vis Builder] Bug fixes for datasource picker and auto time interval ([2632](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2632))
+- [Vis Builder] Bug fixes for datasource picker and auto time interval ([2632](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2632))
 - [MD] Add data source param to low-level search call in Discover ([#2431](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2431))
 - [Multi DataSource] Skip data source view in index pattern step when pick default ([#2574](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2574))
 - [Multi DataSource] Address UX comments on Edit Data source page ([#2629](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2629))
 - [BUG] Fix suggestion list cutoff issue ([#2607](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2607))
 - [Multi DataSource] Address UX comments on index pattern management stack ([#2611](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2611))
 - [Multi DataSource] Apply get indices error handling in step index pattern ([#2652](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2652))
 - [Vis Builder] Last Updated Timestamp for visbuilder savedobject is getting Generated ([#2628](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2628))
-- Removed Leftover X Pack references ([#2638](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2638)) 
+- Removed Leftover X Pack references ([#2638](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2638))
+- Removes Add Integration button ([#2723](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2723))
+- Change geckodriver version to make consistency ([#2772](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2772))
+- [Multi DataSource] Update default audit log path ([#2793](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2793))
+- [Table Visualization] Fix first column sort issue ([#2828](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2828))
+- Temporary workaround for task-kill exceptions on Windows when it is passed a pid for a process that is already dead ([#2842](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2842))
+- [Vis Builder] Fix empty workspace animation does not work in firefox ([#2853](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2853))
+- Bumped `del` version to fix MacOS race condition ([#2847](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2873))
 
 ### 🚞 Infrastructure
 
 - Add CHANGELOG.md and related workflows ([#2414](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2414))
+- Update backport custom branch name to utilize head template ([#2766](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2766))
 
 ### 📝 Documentation
 
-* [MD] Add design documents of multiple data source feature [#2538](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2538)
+- Add the release runbook to RELEASING.md ([#2533](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2533))
+- [MD] Add design documents of multiple data source feature [#2538](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2538)
+- [MD] Tweak multiple data source design doc [#2724](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2724)
+- Add `current-usage.md` and more details to `README.md` of `charts` plugin ([#2695](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2695))
+
 ### 🛠 Maintenance
 
 - Adding @zhongnansu as maintainer. ([#2590](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2590))
 
 ### 🪛 Refactoring
-* [MD] Refactor data source error handling ([#2661](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2661))
+
+- [MD] Refactor data source error handling ([#2661](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2661))
+- Refactor and improve Discover field summaries ([#2391](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2391))
+- [Vis Builder] Removed Hard Coded Strings and Used i18n to transalte([#2867](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2867))
 
 ### 🔩 Tests
 
@@ -87,6 +110,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Resolve sub-dependent d3-color version and potential security issue ([#2454](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2454))
 - [CVE-2022-3517] Bumps minimatch from 3.0.4 to 3.0.5 and [IBM X-Force ID: 220063] unset-value from 1.0.1 to 2.0.1 ([#2640](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2640))
 - [CVE-2022-37601] Bump loader-utils to 2.0.3 ([#2689](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2689))
+- [WS-2021-0638][Security] bump mocha to 10.1.0 ([#2711](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2711))
 
 ### 📈 Features/Enhancements
 

DEVELOPER_GUIDE.md

@@ -59,6 +59,30 @@ Dashboards. In a separate terminal you can run the latest snapshot built using:
 $ yarn opensearch snapshot 
 ```
 
+If you would like to download a specific OpenSearch plugin on the cluster snapshot, pass the `--P` flag after `yarn opensearch snapshot`. We can use the flag multiple times to install multiple plugins on the cluster snapshot. The argument value can be URL to the plugin's zip file, maven coordinates of the plugin or for local zip files, use `file:` followed by the absolute or relative path to the plugin zip file. Below is the example help command:
+
+```
+$ yarn opensearch snapshot --P https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-test-plugin/2.4.0.0/opensearch-test-plugin-2.4.0.0.zip
+```
+
+Following are the list of options that can be passed after `yarn opensearch snapshot` to configure the cluster snapshot.
+Options:
+
+      --license         Run with a 'oss', 'basic', or 'trial' license [default: oss]
+      --version         Version of OpenSearch to download [default: 3.0.0}]
+      --base-path       Path containing cache/installations [default: /home/ubuntu/OpenSearch-Dashboards/.opensearch]
+      --install-path    Installation path, defaults to 'source' within base-path
+      --data-archive    Path to zip or tarball containing an OpenSearch data directory to seed the cluster with.
+      --password        Sets password for opensearch user [default: changeme]
+      -E                Additional key=value settings to pass to OpenSearch
+      --download-only   Download the snapshot but don't actually start it
+      --ssl             Sets up SSL on OpenSearch
+      --P               OpenSearch plugin artifact URL to install it on the cluster. 
+
+```
+$ yarn opensearch snapshot --version 2.2.0 -E cluster.name=test -E path.data=/tmp/opensearch-data --P org.opensearch.plugin:test-plugin:2.2.0.0 --P file:/home/user/opensearch-test-plugin-2.2.0.0.zip
+```
+
 **Warning:** Starting the Dashboards instance before or during the initialization of the OpenSearch Server can cause Dashboards to sometimes misbehave. Ensure that the OpenSearch server instance is up and running first before starting up the Dashboards dev server from the next step.
 
 ### Run OpenSearch Dashboards

RELEASING.md

@@ -1,3 +1,76 @@
 ## Releasing
 
-This project follows [OpenSearch project branching, labelling, and releasing](https://github.com/opensearch-project/.github/blob/main/RELEASING.md).
+This project follows [OpenSearch project branching, labelling, and releasing](https://github.com/opensearch-project/.github/blob/main/RELEASING.md).
+
+## Runbook
+
+### Overview
+
+The OpenSearch project releases versioned distributions of OpenSearch, OpenSearch Dashboards, and the OpenSearch plugins. This runbook details the steps involved in performing major, minor, and patch version releases for the OpenSearch Dashboards project; these steps need to be completed by the release managers (RM) assigned to each release. The RM is also responsible for updating the release status on the release tracking issues maintained on GitHub ([example](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2230)).
+
+\*Important Dates: https://opensearch.org/releases.html
+
+### Release Phase 1 - Preparation
+
+For major and minor releases, the OpenSearch build repository [maintainers](https://github.com/opensearch-project/opensearch-build/blob/main/MAINTAINERS.md) will create a release issue in the OpenSearch Dashboards repository ([example](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2230)) which links to the overall issue in the OpenSearch build repository ([example](https://github.com/opensearch-project/opensearch-build/issues/2447)). For patch releases, they will only create the overall issue in the OpenSearch build repository ([example](https://github.com/opensearch-project/opensearch-build/issues/2650)).
+
+The OpenSearch Dashboards release issue will be assigned to the RM who should prepare for the release by reviewing all listed tasks. They should also compare the current release issue to the issue of the previous release version to ensure that all new processes have been captured.
+
+The RM should review the [public roadmap](https://github.com/orgs/opensearch-project/projects/1) and confirm the release scope with other OpenSearch Dashboards [maintainers](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/MAINTAINERS.md) as well as the feature owners. Since release labels are intended to highlight the features and fixes meant for the upcoming release, the RM should verify that all issues and pull requests are labelled accordingly. For example, if current release version was v2.3.0, all features not ready for the release should be labeled as v2.4.0 or later by discussing with the feature owners. The RM should also check all PRs for the current release version to confirm that they are merged into the `main` branch and their backported PRs are merged with all CI passing.
+
+#### How to validate that merged commits have been properly backported?
+
+1. For any PRs merged to main, make sure it has backport labels ([example - Add v2.3.0 release notes](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2318))
+2. Backport PR is generated automatically by [opensearch-trigger-bot](https://github.com/apps/opensearch-trigger-bot)
+   ([example - [Backport 2.3] Add v2.3.0 release notes](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2333))
+3. If the backport PR is not generated by opensearch-trigger-bot after 24 hours, create it manually following this [instruction](https://github.com/opensearch-project/.github/blob/main/RELEASING.md#backporting)
+4. Ensure that all CI passed and it has two approvals. Then merge the PR.
+
+#### Prepare BWC data and update BWC versions
+
+Backward Compatibility Tests (BWC) are cypress tests that identify any obvious compatibility bugs with previous versions. The RM should generate test data and test locally following instructions [here](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/TESTING.md#backwards-compatibility-tests) and cut PR to include both generated data and version upgrade for automated build. (See example [PR](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2393/files))
+
+#### Cut release branch for major / minor release
+
+For major / minor release, the RM should cut the release branch from the parent branch, [following OpenSearch project branching](https://github.com/opensearch-project/.github/blob/main/RELEASING.md#opensearch-branching)
+
+### Release Phase 2 - Pre-Release
+
+The release process for OpenSearch is centralized. Jenkins workflows are in place to regularly find differences in the OpenSearch and OpenSearch Dashboards components and create new snapshots for those that have been updated. The RM should update the release branch version in the distribution manifest (see example [PR](https://github.com/opensearch-project/opensearch-build/pull/2586/files)) and increment the parent branch version (see example [PR](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2295/files)).
+
+#### Write release notes
+
+OpenSearch Dashboards maintains a [CHANGELOG.md](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/CHANGELOG.md) and verifies it as part of the PR checklist step. For the time being, the RM should create release notes PR with the label `doc`, referring to the `CHANGELOG.md` (see example [PR](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2318))
+
+### Release Phase 3 - Release testing
+
+#### Verify integration and BWC test results
+
+The automated integration test and BWC test are executed concurrently with the release artifacts build. The RM should examine the test results and assist in triaging the broken test case.
+
+Example build:
+
+x64: https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/995/
+
+arm64: https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/996/
+
+Example test results:
+https://opensearch-project.github.io/opensearch-dashboards-functional-test/site/?version=2.3.0&build_number=4104&test_number=996
+
+Note: change `arch` to match the operating system CPU architecture for which the build is being produced.
+
+#### Sanity test with tarball and docker image
+
+Once the release candidate artifacts are built, the RM should configure the OpenSearch cluster with OpenSearch Dashboards according to the [instructions in the OpenSearch build repo](https://github.com/opensearch-project/opensearch-build/issues/2447#issuecomment-1241406594) and produce sanity tests to identify broken functionalities caused by new features / code changes. If you find any, please file bug reports and assist in triaging the bugfix.
+
+### Release Phase 4 - Release Announcement
+
+Release artifacts and announcements will be available on https://opensearch.org/releases.html. Any website documentation changes will require a PR on the [OpenSearch documentation-website repo.](https://github.com/opensearch-project/documentation-website)
+
+### Release Phase 5 - Post-Release
+
+After a release is announced, OpenSearch build repository maintainers will trigger a job that creates a tag in each repository based on the commit hash and branch that was included in the release; the release tag could take a few hours to show up on GitHub.
+
+The RM should update the [release page](https://github.com/opensearch-project/OpenSearch-Dashboards/releases/) with the latest download URL and release notes after the release tag is created.
+
+If needed, the RM could conduct a retrospective review of the release, and publish their findings regarding any missed steps and process improvements.