ECS structure logging is not compatible with all collectors as it does not use the nested format

[kajh]

It would be nice with an option to get ecs logging as nested json like

{ "ecs":  {"version": "8.4"}, "service": {"name": "myservice"}}

instead of what we get today

{ "ecs.version": "8.4", "service.name" : "myservice"}

The reason for asking is that collecting and processing of logs would be easier.

We are using Spring Boot 3.4.4.

[nosan]

It would be nice with an option to get ecs logging as nested json like

I'm not sureif this is advisable, as it conflicts with the ECS Schema. Currently, ECS Schema does
not support a format such as "ecs": {"version":"8.11"}. Moreover, according to the ECS Schema, the
ecs.version field is required and must exist in all events, so supporting this functionality
would not provide any value from the user's perspective.

UPDATE: My previous statement was incorrect.

The reason for asking is that collecting and processing of logs would be easier.

For your specific case, you should create a custom implementation of JsonWriterStructuredLogFormatter with a format that suits exactly to your needs. This custom formatter can then be configured using the logging.structured.format.console property.

For more details, please refer to the Spring Boot documentation.

The following NestedElasticCommonSchemaStructuredLogFormatter might suit your needs:

package task.gh45063;

import ch.qos.logback.classic.pattern.ThrowableProxyConverter;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.classic.spi.IThrowableProxy;
import org.slf4j.event.KeyValuePair;
import org.springframework.boot.json.JsonWriter;
import org.springframework.boot.json.JsonWriter.PairExtractor;
import org.springframework.boot.logging.structured.ElasticCommonSchemaProperties;
import org.springframework.boot.logging.structured.ElasticCommonSchemaProperties.Service;
import org.springframework.boot.logging.structured.JsonWriterStructuredLogFormatter;
import org.springframework.boot.logging.structured.StructuredLoggingJsonMembersCustomizer;
import org.springframework.core.env.Environment;

import java.util.Objects;

class NestedElasticCommonSchemaStructuredLogFormatter extends
        JsonWriterStructuredLogFormatter<ILoggingEvent> {

    private static final PairExtractor<KeyValuePair> keyValuePairExtractor = PairExtractor.of(
            (pair) -> pair.key,
            (pair) -> pair.value);

    NestedElasticCommonSchemaStructuredLogFormatter(Environment environment,
            ThrowableProxyConverter throwableProxyConverter,
            StructuredLoggingJsonMembersCustomizer<?> customizer) {
        super((members) -> jsonMembers(environment, throwableProxyConverter, members), customizer);
    }

    private static void jsonMembers(Environment environment,
            ThrowableProxyConverter throwableProxyConverter,
            JsonWriter.Members<ILoggingEvent> members) {
        members.add("@timestamp", ILoggingEvent::getInstant);
        members.add("log").usingMembers((logMembers) -> {
                    logMembers.add("logger", ILoggingEvent::getLoggerName);
                    logMembers.add("level", ILoggingEvent::getLevel);
                }
        );
        members.add("process").usingMembers((processMembers) -> {
            processMembers.add("pid", environment.getProperty("spring.application.pid", Long.class))
                    .when(Objects::nonNull);
            processMembers.add("thread")
                    .usingMembers((threadMembers) -> threadMembers.add("name",
                            ILoggingEvent::getThreadName));
        });
        Service service = ElasticCommonSchemaProperties.get(environment).service();
        members.add("service").usingMembers((serviceMembers) -> {
            serviceMembers.add("name", service::name).whenHasLength();
            serviceMembers.add("version", service::version).whenHasLength();
            serviceMembers.add("environment", service::environment).whenHasLength();
            serviceMembers.add("node.name", service::nodeName).whenHasLength();
        });
        members.add("message", ILoggingEvent::getFormattedMessage);
        members.addMapEntries(ILoggingEvent::getMDCPropertyMap);
        members.from(ILoggingEvent::getKeyValuePairs)
                .whenNotEmpty()
                .usingExtractedPairs(Iterable::forEach, keyValuePairExtractor);
        members.add("error").whenNotNull(ILoggingEvent::getThrowableProxy)
                .usingMembers((throwableMembers) -> {
                    throwableMembers.add("type", ILoggingEvent::getThrowableProxy)
                            .as(IThrowableProxy::getClassName);
                    throwableMembers.add("message", ILoggingEvent::getThrowableProxy)
                            .as(IThrowableProxy::getMessage);
                    throwableMembers.add("stack_trace", throwableProxyConverter::convert);
                });
        members.add("ecs").usingMembers((ecsMembers) -> ecsMembers.add("version", "8.11"));
    }

}

{
  "@timestamp": "2025-04-10T14:35:20.081935Z",
  "log": {
    "logger": "task.gh45063.Gh45063Application",
    "level": "INFO"
  },
  "process": {
    "pid": 25264,
    "thread": {
      "name": "main"
    }
  },
  "service": {
    "name": "gh-45063"
  },
  "message": "Started Gh45063Application in 0.257 seconds (process running for 0.418)",
  "ecs": {
    "version": "8.11"
  }
}

[snicoll]

Thanks @nosan. Indeed @kajh we're not going to offer a format that's not backed by ECS

[rafaelma]

Hello, thanks for looking at this :)

ECS stores data in a nested format, our example will be saved as { "ecs": {"version": "8.4"} }, and the mapping definition of the ECS standard needed by e.g. elasticsearch or opensearch defines also the nested format. It’s important to note that the translation from { "ecs.version": "8.4" } to { "ecs": {"version": "8.4"} } before saving the data according to the ECS standard is done by the Elastic Agent using the decode_json_fields processor in the elastic agent ref: https://www.elastic.co/guide/en/fleet/current/decode-json-fields.html.

For reference, you can check the mapping representation of the ECS standard for our example here: https://github.com/elastic/ecs/blob/main/generated/elasticsearch/composable/component/ecs.json The ECS documentation https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html refers to attributes like ecs.version, but this is merely a simplified representation of a JSON nested attribute in the documentation for presentation purposes.

If you use another agent than the elastic-agent (e.g.fluent-bit) to process logs and spring delivers them as e.g. { "ecs.version": "8.4" }, you have to transform this to the nested version { "ecs": {"version": "8.4"} } before you can save it according to the standard.

Spring doesn't deliver the logs according to the ECS standard format and it will only work without problems if you collect these logs with the elastic-agent that will convert them to the right ECS format before sending them to storage.

regards,
Rafael Martinez Guerrero

[nosan]

I set up Elasticsearch via https://www.elastic.co/cloud and performed another round of testing.

It's important to clarify that Spring does not directly produce logs in the ECS standard; typically, logs require the Elastic Agent to properly transform them into ECS format before indexing.

However, in practice, the JSON format currently produced by ElasticCommonSchemaStructuredLogFormatter works seamlessly. Elasticsearch accepts them without issue, correctly parsing all fields.

Flat JSON example:

{
  "@timestamp": "2025-04-10T18:03:42.230354Z",
  "log.level": "ERROR",
  "process.pid": 30859,
  "process.thread.name": "main",
  "service.name": "gh-45063",
  "log.logger": "task.gh45063.Gh45063Application",
  "message": "MyError",
  "error.type": "java.lang.IllegalStateException",
  "error.message": "Error Test",
  "error.stack_trace": "java.lang.IllegalStateException: Error Test\n\tat task.gh45063.Gh45063Application.main(Gh45063Application.java:14)\n",
  "ecs.version": "8.11"
}

Additionally, Elasticsearch fully supports and parses nested JSON structures, like the one below:

Nested JSON example:

{
  "@timestamp": "2025-04-10T18:04:14.257693Z",
  "log": {
    "logger": "task.gh45063.Gh45063Application",
    "level": "ERROR"
  },
  "process": {
    "pid": 30877,
    "thread": {
      "name": "main"
    }
  },
  "service": {
    "name": "gh-45063"
  },
  "message": "MyError",
  "error": {
    "type": "java.lang.IllegalStateException",
    "message": "Error Test",
    "stack_trace": "java.lang.IllegalStateException: Error Test\n\tat task.gh45063.Gh45063Application.main(Gh45063Application.java:14)\n"
  },
  "ecs": {
    "version": "8.11"
  }
}

In summary, both flat and nested formats are compatible.

Screenshots

[rafaelma]

Thank you, @nosan, for investigating this issue further.

"...In summary, both flat and nested formats are compatible ....", as a source, but only when using Elasticsearch as the storage infrastructure because elasticsearch transforms flat format to the standard nested format before saving the log. If you examine the JSON code of the document saved by Elasticsearch after sending the flat version, you'll see it has been also converted into a nested json document to adhere to the standard.

UPDATE:

---

If elasticsearch works as opensearch this statement is not true: "If you examine the JSON code of the document saved by Elasticsearch after sending the flat version, you'll see it has been also converted into a nested json document to adhere to the standard"

OpenSearch accepts both flat and nested JSON with a nested mapping definition now, but the saved flat JSON document is not converted to nested format.

Check #45063 (comment) for full details.

---

The ECS standard is now utilized by many products following the merger of OpenTelemetry and Elastic standards to create a unified open standard. Refer to the following links for more information:

• https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/
• https://www.elastic.co/blog/ecs-elastic-common-schema-otel-opentelemetry-announcement

It would greatly enhance the Spring project if it could deliver logs in the ECS nested format. This would enable standardized log storage without the need to convert from flat to nested format before saving the log, when used with products other than Elasticsearch, such as Opensearch.

The conversion from flat to nested format to be able to save the standardized log, can be quite tedious and resource-intensive when processing millions of logs if you don't use elasticsearch.

regards

[nosan]

According to the ECS Reference:

The document structure should be nested JSON objects. If you use Beats or Logstash, the nesting of JSON objects is done for you automatically. If you’re ingesting to Elasticsearch using the API, your fields must be nested objects, not strings containing dots.

I also checked the ECS Logging Java repository elastic/ecs-logging-java and found they have a similar issues:

• AbstractEcsLoggingTest should log keys as nested objects, not with dotted key names elastic/ecs-logging-java#51
• Use nested JSON instead of dot strings elastic/ecs-logging-java#52
• Format logs in nested JSON elastic/ecs-logging-java#80
• Java Json format not consistent with ECS elastic/ecs-logging-java#102

However, the ECS Java logging library continues to use dot notation and relies on this processor afterward. It seems they chose to keep dot notation because they can't ensure that all fields are correctly nested especially considering fields provided through ILoggingEvent.getKeyValuePairs() or ILoggingEvent.getMDCPropertyMap(), since these may contain dots as well, and for example if you set a custom field using MDC like this MDC.put("geo.continent_name", "North America")
then according to the ECS Reference , the ElasticCommonSchemaStructuredLogFormatter should ideally convert this into nested JSON, as shown below:

{
  "geo": {
    "continent_name": "North America"
  }
}

Personally, if Spring Boot chooses to support the nested JSON format, I don't see any advantage in maintaining support for the current format, as only the nested JSON format fully complies with the ECS specification.

when used with products other than Elasticsearch, such as Opensearch.

I set up OpenSearch using Docker Compose and submitted several JSON documents to it. Both formats are supported by OpenSearch.

Screenshot

---

I hope I haven't overlooked anything, and I apologize once again for my earlier incorrect comment: #45063 (comment)