Add explicit index template mappings and reporting config wiring for OpenSearchMetricsSink by Shivani-techno · Pull Request #2569 · opensearch-project/opensearch-migrations

Shivani-techno · 2026-03-30T04:36:03Z

Description

Adds explicit index template creation to OpenSearchMetricsSink so validation metrics
indices have consistent, optimized field mappings instead of relying on dynamic mapping.

Changes:

OpenSearchMetricsSink: Creates composable index template at startup with explicit
types (keyword for aggregation fields, date for timestamp, nested for comparisons,
text for request bodies with no length limit)
ReportingConfig: New YAML config parser for reporting framework settings
ShimMain: Added --reporting-config CLI parameter to enable validation reporting
ShimProxy/MultiTargetRoutingHandler: Wired MetricsReceiver into the request pipeline
to collect and publish validation metrics after each request
docker-compose.validation.yml: Added reporting config mount for shim-solr-primary
reporting-config.yaml: Sample configuration with placeholder credentials

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

akshay2000 · 2026-03-30T04:49:33Z

Please fix the DCO failure.

akshay2000 · 2026-03-30T04:50:24Z

TrafficCapture/SolrTransformations/docker/docker-compose.validation.yml

  shim-solr-primary:
    <<: *shim-base
+    volumes:
+      - transform-dist:/transforms:ro


Why was this change required?

The volume override is scoped to shim-solr-primary ( dual mode ) only, not the base. The transform-dist volume is re-declared because YAML merge replaces rather than appends — without it, the transform JS files wouldn't be mounted.

akshay2000 · 2026-03-30T04:51:28Z

TrafficCapture/SolrTransformations/docker/docker-compose.validation.yml

    <<: *shim-base
+    volumes:
+      - transform-dist:/transforms:ro
+      - ./reporting-config.yaml:/config/reporting-config.yaml:ro


Reporting config makes sense only when we are running in dual mode. Adding it here will mount the config on all the shims - it will largely remain unused.

akshay2000 · 2026-03-30T13:30:01Z

...onShim/src/main/java/org/opensearch/migrations/transform/shim/reporting/ReportingConfig.java

+    }
+
+    /** Simple YAML flattener — handles indentation-based nesting, strips comments. */
+    private static Map<String, String> flattenYaml(String content) {


We should really consider using a YAML parsing library. YAML is not just indent based line parsing.

Agreed. Raising the revision to use parsing library.

codecov · 2026-03-30T14:34:14Z

Codecov Report

❌ Patch coverage is 74.17840% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.67%. Comparing base (a9e040e) to head (108c361).
⚠️ Report is 280 commits behind head on main.

Files with missing lines	Patch %	Lines
...ransform/shim/reporting/OpenSearchMetricsSink.java	71.02%	21 Missing and 10 partials ⚠️
...opensearch/migrations/transform/shim/ShimMain.java	61.53%	8 Missing and 2 partials ⚠️
...ions/transform/shim/reporting/ReportingConfig.java	84.37%	2 Missing and 8 partials ⚠️
...ransform/shim/netty/MultiTargetRoutingHandler.java	71.42%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2569      +/-   ##
============================================
+ Coverage     72.09%   72.67%   +0.57%     
- Complexity       65       90      +25     
============================================
  Files           695      705      +10     
  Lines         32018    32728     +710     
  Branches       2714     2809      +95     
============================================
+ Hits          23084    23784     +700     
+ Misses         7694     7638      -56     
- Partials       1240     1306      +66

Flag	Coverage Δ
gradle	`69.02% <74.17%> (+0.78%)`	⬆️
node	`92.55% <ø> (+0.04%)`	⬆️
python	`76.67% <ø> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

akshay2000 · 2026-04-01T14:08:30Z

solrShimTestHarness/docker-compose.yml

      - discovery.type=single-node
      - DISABLE_SECURITY_PLUGIN=true
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=Admin_1234!
+      - OPENSEARCH_JAVA_OPTS=-Xms256m -Xmx256m


Why did we need to modify the existing OS container definition?

I added it because running two OpenSearch instances (destination + reporting) caused the destination to get OOM killed. But no need to do for the destination, so fixed it by removing the heap limit from the destination OS container and set the reduced heap on the reporting node.

@Shivani-techno take a look at your config in docker desktop for cpu/memory allocated to the docker runner, it may have caused the OOM you were seeing

akshay2000 · 2026-04-01T14:10:31Z

solrShimTestHarness/docker-compose.yml

      - "18080:8080"
    volumes:
      - transform-dist:/transforms:ro
+      - ../TrafficCapture/SolrTransformations/docker/reporting-config.yaml:/config/reporting-config.yaml:ro


Please create a reporting config for this harness specifically. The one in the other directory should serve only as a template.

akshay2000 · 2026-04-01T14:13:47Z

TrafficCapture/SolrTransformations/docker/docker-compose.validation.yml

      - opensearch=request:/transforms/solr-to-opensearch-request.js,response:/transforms/solr-to-opensearch-response.js
      - --watchTransforms

  # Mode 3: Dual-target, Solr primary — returns Solr response, validates against OpenSearch


Since we are modifying the test harness (aka. dev sandbox) we should not modify this file at all.

done, reverted back the changes

…porting node and Dashboards Signed-off-by: Shivani - <scivani@amazon.com>

AndreKurait · 2026-04-03T11:39:05Z

solrMigrationDevSandbox/docker-compose.yml

      retries: 30

+  opensearch-reporting:
+    image: opensearchproject/opensearch:3.3.0


Let's make this 3.5

AndreKurait · 2026-04-03T11:40:03Z

solrMigrationDevSandbox/docker-compose.yml

+      - discovery.type=single-node
+      - DISABLE_SECURITY_PLUGIN=true
+      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=Admin_1234!
+      - OPENSEARCH_JAVA_OPTS=-Xms256m -Xmx256m


Bump this to at least 1gb on Xmx, we've had issues with smaller opensearch clusters

AndreKurait

Please take a look at the tuple interface in the replayer. See #2605 which can be leveraged for high performance validation here.

AndreKurait · 2026-04-03T11:44:55Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+     * - headers use dynamic mapping since HTTP header values are always strings
+     *   and individual headers vary per request
+     */
+    private String buildIndexTemplateJson() {


Can we move this to preflight step prior to the shim starting up

AndreKurait · 2026-04-03T11:46:53Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+    private String buildIndexTemplateJson() {
+        return String.format("""
+            {
+              "index_patterns": ["%s-*"],


Do we have any test in here that validates if this works against the reporting opensearch version?

AndreKurait · 2026-04-03T11:51:25Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+                if (buffer.size() >= bulkSize) {
+                    List<ValidationDocument> batch = new ArrayList<>(buffer);
+                    buffer.clear();
+                    scheduler.execute(() -> sendBulk(batch));


This is going to be a bottleneck by single threading submit and waiting on the bulk request.

AndreKurait · 2026-04-03T11:53:17Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+        try {
+            synchronized (buffer) {
+                buffer.add(document);
+                if (buffer.size() >= bulkSize) {


Depending on number of documents without consideration for their size is troublesome. E.g. if each doc was 20MB this would be a 2GB request which opensearch would reject

Signed-off-by: Shivani - <scivani@amazon.com>

AndreKurait

This comment serves so i can use "Changes since reivew" post force-push's

…emplate, improve coverage Signed-off-by: Shivani - <scivani@amazon.com>

Signed-off-by: Shivani - <scivani@amazon.com>

AndreKurait · 2026-04-06T18:32:43Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+public class OpenSearchMetricsSink implements MetricsSink {
+
+    private static final Logger log = LoggerFactory.getLogger(OpenSearchMetricsSink.class);
+    private static final ObjectMapper MAPPER = new ObjectMapper();


There should be a mapper factory we should use

AndreKurait · 2026-04-06T18:35:04Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+        }
+    }
+
+    private long estimateDocSize(ValidationDocument document) {


Note, we have existing code in this project to do this without reserializing. See how RFS generates bulk documents

AndreKurait · 2026-04-06T18:36:17Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+                ndjson.append(MAPPER.writeValueAsString(doc)).append("\n");
+            }
+
+            var requestBuilder = HttpRequest.newBuilder()


We have existing code that does this in a robust way (e.g. applying GZIP client compression) ideally we would use the same underlying code for performance and maintainability

AndreKurait · 2026-04-06T18:37:24Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+                    .timeout(Duration.ofSeconds(30));
+
+            if (authHeader != null) {
+                requestBuilder.header("Authorization", authHeader);


Looks like this only supports basic auth, if you used our existing construcs in replay or RFS this could support sigv4

AndreKurait · 2026-04-06T18:37:56Z

.../src/main/java/org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSink.java

+     * Checks for partial failures in bulk response.
+     * Even if HTTP status is 200, individual documents may have failed.
+     */
+    void checkPartialFailures(String responseBody, int totalDocs) {


We already have exisitng logic in RFS which does this as well as a bisect on failure to only retry the failed docs, could be useful here

AndreKurait · 2026-04-06T18:39:17Z

...ture/transformationShim/src/main/java/org/opensearch/migrations/transform/shim/ShimMain.java

        public boolean watchTransforms;

+        @Parameter(names = {"--reporting-config"},
+            description = "Path to YAML configuration file for the validation reporting framework.")


are we getting value for the complexity of YAML here when the rest of the processes support JSON natively?

AndreKurait · 2026-04-06T18:40:18Z

...org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSinkIntegrationTest.java

+ * Validates index template creation, document indexing, and mapping against a real OpenSearch instance.
+ */
+@Testcontainers
+@Tag("longTest")


Use isolatedTest for testcontainers

AndreKurait · 2026-04-06T18:40:50Z

...org/opensearch/migrations/transform/shim/reporting/OpenSearchMetricsSinkIntegrationTest.java

+        .connectTimeout(Duration.ofSeconds(10)).build();
+
+    @Container
+    static final OpensearchContainer<?> opensearch = new OpensearchContainer<>("opensearchproject/opensearch:2.19.1")


Please see our helper methods for test containers which we use across our processes. Why is this using opensearch 2.19?

…config to JSON Signed-off-by: Shivani - <scivani@amazon.com>

jugal-chauhan · 2026-04-14T18:48:13Z

Checking in, are there more changes expected here ? Should we move this PR into draft until these changes are made ?

Shivani-techno requested review from AndreKurait, gregschohn, jugal-chauhan and sumobrian as code owners March 30, 2026 04:36

Shivani-techno had a problem deploying to migrations-cicd-require-approval March 30, 2026 04:36 — with GitHub Actions Error

akshay2000 requested changes Mar 30, 2026

View reviewed changes

Shivani-techno had a problem deploying to migrations-cicd-require-approval March 30, 2026 14:56 — with GitHub Actions Error

Shivani-techno force-pushed the validations__scivani branch from 5a22288 to 36cb74a Compare March 30, 2026 15:01

Shivani-techno had a problem deploying to migrations-cicd-require-approval March 30, 2026 15:01 — with GitHub Actions Error

Shivani-techno force-pushed the validations__scivani branch from 36cb74a to 4571c36 Compare March 31, 2026 04:44

Shivani-techno had a problem deploying to migrations-cicd-require-approval March 31, 2026 04:44 — with GitHub Actions Error

Shivani-techno force-pushed the validations__scivani branch from 4571c36 to 93f049a Compare March 31, 2026 04:47

Shivani-techno had a problem deploying to migrations-cicd-require-approval March 31, 2026 04:47 — with GitHub Actions Error

Shivani-techno had a problem deploying to migrations-cicd-require-approval March 31, 2026 09:21 — with GitHub Actions Error

Shivani-techno had a problem deploying to migrations-cicd-require-approval April 1, 2026 11:46 — with GitHub Actions Error

Shivani-techno force-pushed the validations__scivani branch from 3f4b1fc to ede0363 Compare April 1, 2026 11:52

Shivani-techno had a problem deploying to migrations-cicd-require-approval April 1, 2026 11:52 — with GitHub Actions Error

akshay2000 requested changes Apr 1, 2026

View reviewed changes

Shivani-techno force-pushed the validations__scivani branch from ede0363 to cb2dd17 Compare April 2, 2026 04:41

Shivani-techno had a problem deploying to migrations-cicd-require-approval April 2, 2026 04:41 — with GitHub Actions Error

Shivani-techno force-pushed the validations__scivani branch from 5088b3b to 920f22c Compare April 3, 2026 08:44

Shivani-techno had a problem deploying to migrations-cicd-require-approval April 3, 2026 08:44 — with GitHub Actions Error

Shivani-techno force-pushed the validations__scivani branch from 920f22c to 31122cd Compare April 3, 2026 08:50

Shivani-techno had a problem deploying to migrations-cicd-require-approval April 3, 2026 08:50 — with GitHub Actions Error

Integrate reporting into solrShimTestHarness with local OpenSearch re…

eb45cee

…porting node and Dashboards Signed-off-by: Shivani - <scivani@amazon.com>

Shivani-techno force-pushed the validations__scivani branch from 31122cd to eb45cee Compare April 3, 2026 09:18

Shivani-techno had a problem deploying to migrations-cicd-require-approval April 3, 2026 09:18 — with GitHub Actions Error

AndreKurait reviewed Apr 3, 2026

View reviewed changes

AndreKurait requested changes Apr 3, 2026

View reviewed changes

Fix spotless violations in ShimProxy and added tests

adf90f7

Signed-off-by: Shivani - <scivani@amazon.com>

AndreKurait reviewed Apr 3, 2026

View reviewed changes

Shivani - added 5 commits April 3, 2026 22:07

Bump OS 3.5, increase heap, async bulk, size-based flush, preflight t…

375a99e

…emplate, improve coverage Signed-off-by: Shivani - <scivani@amazon.com>

Fix SonarQube

d6e2125

Signed-off-by: Shivani - <scivani@amazon.com>

Added tests to increase the test coverage

568af40

Signed-off-by: Shivani - <scivani@amazon.com>

Add Testcontainers integration test for OpenSearchMetricsSink

ecafaa8

Signed-off-by: Shivani - <scivani@amazon.com>

Add unit tests to improve patch coverage for transformationShim

124fc2a

Signed-off-by: Shivani - <scivani@amazon.com>

AndreKurait reviewed Apr 6, 2026

View reviewed changes

Integrate RFS bulk infrastructure into OpenSearchMetricsSink, switch …

108c361

…config to JSON Signed-off-by: Shivani - <scivani@amazon.com>

Merge branch 'main' into validations__scivani

d973c42

Conversation

Shivani-techno commented Mar 30, 2026

Description

Uh oh!

akshay2000 commented Mar 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndreKurait left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndreKurait left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jugal-chauhan commented Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 30, 2026 •

edited

Loading