[FEATURE] Support merging object-type fields when fetching the schema from the index

[penghuo]

Is your feature request related to a problem?
Currently, when the same object field (e.g., log) contains different inner fields in different indices, PPL randomly pick one, then PPL fails to resolve queries referencing any of them.

• with data

###
POST {{baseUrl}}/logs-test-ppl-json-plain/_doc
Content-Type: application/x-ndjson

{
  "@timestamp": "2025-05-05T09:33:00Z",
  "message": "text message 3",
  "log": {
    "file": "/var/log/msg"
  }
}

###
POST {{baseUrl}}/logs-test-ppl-json-default/_doc
Content-Type: application/x-ndjson

{
  "message": "json message 1",
  "@timestamp": "2025-05-13T09:55:00Z",
  "log": {
    "level": "ERROR"
  }
}

• run query

source = logs-* | where log.file= '/var/log/msg'

{
  "error": {
    "reason": "Invalid Query",
    "details": "can't resolve Symbol(namespace=FIELD_NAME, name=log.file) in type env",
    "type": "SemanticCheckException"
  },
  "status": 400
}

What solution would you like?
PPL should support merging object-type field definitions across indices (e.g., log.file, log.level) into a unified schema, so queries can reference valid fields even if they're only present in some indices.

[anasalkouz]

Is this issue PPL specific? Does SQL and DSL support it?

[penghuo]

Is this issue PPL specific? Does SQL and DSL support it?

Both for SQL and PPL.
DSL works well.

[penghuo]

What solution would you like?
PPL should support merging object-type field definitions across indices (e.g., log.file, log.level) into a unified schema, so queries can reference valid fields even if they're only present in some indices.

Merge solution does not scale. OpenSearch supports the _field_caps API, which returns merged field definitions across indices. It performs well with a small number of indices, but when querying over 1,000 indices, the latency can exceed 10 seconds.