[BUG] Opensearch sink with index type trace-analytics-plain-raw won't select the correct index template

[andreufontb]

Describe the bug
With Data Prepper 2.15, when using a pipeline configured with index_type: otel-v2-apm-service-map, the index created in OpenSearch uses an incorrect template. As a result, document ingestion fails with mapping errors. Specifically, Data Prepper creates indices based on the legacy otel-v1-apm-span-index-template instead of the expected standard template for the configured index type otel-v1-apm-span-index-standard-template.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Data Prepper 2.15 with an OpenSearch backend
2. Configure the following pipeline:

otlp-pipeline:
  delay: "10"
  source:
    otlp:
      ssl: false
      port: 21893
  route:
    - logs: "getEventType() == \"LOG\""
    - traces: "getEventType() == \"TRACE\""
  sink:
    - pipeline:
        name: "otel-logs-pipeline"
        routes:
          - "logs"
    - pipeline:
        name: "otel-traces-pipeline"
        routes:
          - "traces"

otel-traces-pipeline:
  source:
    pipeline:
      name: "otlp-pipeline"
  sink:
    - pipeline:
        name: "traces-raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"

traces-raw-pipeline:
  source:
    pipeline:
      name: "otel-traces-pipeline"
  processor:
    - otel_traces:
  sink:
    - opensearch:
        hosts: [ "https://opensearch:443" ]
        insecure: true
        username: "${OPENSEARCH_USERNAME}"
        password: "${OPENSEARCH_PASSWORD}"
        index_type: "trace-analytics-plain-raw"

service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "otel-traces-pipeline"
  processor:
    - otel_apm_service_map:
        group_by_attributes: [telemetry.sdk.language]
        window_duration: 60s
        db_path: "data/otel-apm-service-map/"
  route:
    - otel_apm_service_map_route: 'getEventType() == "SERVICE_MAP"'
    - service_processed_metrics: 'getEventType() == "METRIC"'
  sink:
    - opensearch:
        hosts: ["https://opensearch:443"]
        insecure: true
        username: "${OPENSEARCH_USERNAME}"
        password: "${OPENSEARCH_PASSWORD}"
        index_type: otel-v2-apm-service-map
        routes: [otel_apm_service_map_route]
    - prometheus:
        url: "https://prometheus:443/api/v1/write"
        insecure: true
        authentication:
          http_basic:
            username: "${PROMETHEUS_USERNAME}"
            password: "${PROMETHEUS_PASSWORD}"
        threshold:
          max_events: 500
          flush_interval: 5s
        routes: [service_processed_metrics]

3. Make sure there are no previous span indexes already created
4. Observe in data preper logs the selected tempalte and then ingestion errors, on opensearch the index mapping is not the one should be.

Expected behavior
Data Prepper should select the correct index template corresponding to trace-analytics-plain-raw, and indices should be created with the proper mappings so that document ingestion succeeds without errors.

Environment (please complete the following information):

• OS: Ubuntu 24.04 LTS, Kubernetes 1.34 with opensearchproject/data-prepper:2.15.0 container.
• Version 2.15.0

Additional context
Opensearch is deployed using the operator

[JongminChung]

1. Set up the Index Template in OpenSearch Sink.

2. Get indexAlias by Alias in the IndexConfiguration.

• https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexConfiguration.java#L153-L155

3. It is registered as Alias under the same name and does not update the template because the two have the same version.

• https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexConstants.java#L49-L50
• https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java#L299-L306

Since it's the above situation, the index template is not automatically changed. So I think I need to update the index template manually.

We need to check with the data-prepper team to see if the above intention is correct.