Logs API: not possible to use without SDK

[lmolkova]

Let's assume I'm building a logging bridge for one of many python non-standard loggers (e.g. structlog).

I should be able to create a library using logging (bridge) API alone.

So I should write a code like:

from opentelemetry._logs import get_logger_provider
from opentelemetry._logs import LogRecord as APILogRecord

logger = get_logger_provider().get_logger(__name__)

logger.emit(APILogRecord(body="Hello, World!", attributes={"foo": "bar"}))

It'd fail with

opentelemetry\sdk\_logs\_internal\export\__init__.py", line 97, in <lambda>
    formatter: Callable[[LogRecord], str] = lambda record: record.to_json()
                                                           ^^^^^^^^^^^^^^
AttributeError: 'LogRecord' object has no attribute 'to_json'

Probably related to #3552

---

The recommendation

1. Make it possible to (properly) use Logs API alone without touching SDK types
2. Add example and tests that use vanilla logs API (with note that it's for logging bridges, not end users) (update - this is not necessary anymore, logs API is now for everyone to use)
3. Call API and SDK log records differently to minimize confusion. E.g. SDK one could be ReadWriteLogRecord and should probably replace LogData (see Logs SDK: consider removing LogData and extending SDK LogRecord to have instrumentation scope #4313)
4. Consider using a pattern similar to tracing when users don't need to instantiate a new span. Instead they could call logger.emit(....) with parameters and the SDK implementation of logger will create SDK implementation of LogRecord

Part of open-telemetry/community#1751

[lmolkova]

This is part of the same issue - if I try to use SDK LogRecord directly like in this snippet

from opentelemetry._logs import LogRecord as APILogRecord
from opentelemetry._logs import get_logger_provider
from opentelemetry.sdk._logs import LogRecord as SDKLogRecord

...
resource = Resource.create({"service.name": "test logs"})
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(SimpleLogRecordProcessor(ConsoleLogExporter()))
set_logger_provider(logger_provider)

logger = get_logger_provider().get_logger(__name__)

with trace_provider.get_tracer("example").start_as_current_span("foo"):
    logger.emit(SDKLogRecord(body="Hello, World!", attributes={"foo": "bar"}))

I get into the following issues:

1. I have to provide timestamp even though it's optional

       formatter: Callable[[LogRecord], str] = lambda record: record.to_json()
                                                              ^^^^^^^^^^^^^^^^
     File "opentelemetry\sdk\_logs\_internal\__init__.py", line 230, in to_json
       "timestamp": ns_to_iso_str(self.timestamp),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "opentelemetry\sdk\util\__init__.py", line 27, in ns_to_iso_str
       nanoseconds / 1e9, tz=datetime.timezone.utc
       ~~~~~~~~~~~~^~~~~
   TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'
   

2. If I provide it (logger.emit(SDKLogRecord(timestamp=time_ns(), body="Hello, World!", attributes={"foo": "bar"}))) I don't get resource attributes set

   {
       "body": "Hello, World!",
       "severity_number": "None",
       "severity_text": null,
       "attributes": {
           "foo": "bar"
       },
       "dropped_attributes": 0,
       "timestamp": "2024-11-24T19:48:29.322928Z",
       "observed_timestamp": "2024-11-24T19:48:29.322928Z",
       "trace_id": "",
       "span_id": "",
       "trace_flags": null,
       "resource": {
           "attributes": {
               "telemetry.sdk.language": "python",
               "telemetry.sdk.name": "opentelemetry",
               "telemetry.sdk.version": "1.28.2",
               "service.name": "unknown_service"
           },
           "schema_url": ""
       }
   }

3. Also you can see that trace-context is not set despite having valid current span

[luccabb]

Probably related to #3552

When using the APILogRecord I get:

• Logs API: not possible to use without SDK #4319 on ConsoleLogExporter
• Logs: LogRecord not documented, usage attempt fails with an exception #3552 on OTLPLogExporter

[luccabb]

probably also part of the same issue, resource and scope attributes via the SDK are ignored:

from opentelemetry._logs import LogRecord, get_logger_provider, set_logger_provider, SeverityNumber
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LogRecord, LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.trace.span import TraceFlags

resource = Resource(attributes={"resource_attr": "value"})
logger_provider = LoggerProvider(resource=resource)
exporter = OTLPLogExporter(endpoint="http://localhost:4318/v1/logs", timeout=10)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
set_logger_provider(logger_provider)
logger = logger_provider.get_logger("test_logger", attributes={"scope_attr": "value"})
# also fails with:
# logger = get_logger_provider().get_logger("test_logger", attributes={"scope_attr": "value"})

logger.emit(
	LogRecord(
                body="Hello, World!", 
                attributes={"datapoint_attr": "value"},
                span_id=0, 
                trace_id=0, 
                trace_flags=TraceFlags.get_default(),
                severity_number=SeverityNumber.UNSPECIFIED,
	)
)

produces:

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "telemetry.sdk.language",
            "value": {
              "stringValue": "python"
            }
          },
          {
            "key": "telemetry.sdk.name",
            "value": {
              "stringValue": "opentelemetry"
            }
          },
          {
            "key": "telemetry.sdk.version",
            "value": {
              "stringValue": "1.23.0"
            }
          },
          {
            "key": "service.name",
            "value": {
              "stringValue": "unknown_service"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {
            "name": "test_logger"
          },
          "logRecords": [
            {
              "observedTimeUnixNano": "1739490609316028637",
              "body": {
                "stringValue": "Hello, World!"
              },
              "attributes": [
                {
                  "key": "datapoint_attr",
                  "value": {
                    "stringValue": "value"
                  }
                }
              ],
              "traceId": "",
              "spanId": ""
            }
          ]
        }
      ]
    }
  ]
}

[jhodnett2]

I worked around both of these issues. Pass the resource in the LogRecord like this:

LogRecord(
      body="Hello, World!", 
      attributes={"datapoint_attr": "value"},
      span_id=0, 
      trace_id=0, 
      trace_flags=TraceFlags.get_default(),
      severity_number=SeverityNumber.UNSPECIFIED,
      resource=logger.resource,
)

The implementation is probably not ideal, but both of the examples above got me to a working example.

The LoggingHandler code in the SDK does it like that.

[aabmass]

Yes we definitely need to sort this out and there will probably be some breaking changes before marking the logging API stable

[lmolkova]

I believe I had a prototype for this one - #4213, which is probably quite out of date, but demonstrates the proposal

[vytas7]

I guess it is not worth building anything with the Logs API yet?
Is there any reasonable workaround to this issue?

This is what I came up with but this is probably going to break at any time 🤔

span_context = span.get_span_context()

record = LogRecord(
    body='Using only the Logs API.',
    trace_id=span_context.trace_id,
    span_id=span_context.span_id,
    trace_flags=span_context.trace_flags,
    severity_number=SeverityNumber.WARN,
    severity_text="WARN",
    attributes={'foo': 'bar'},
)
# Attributes that only exist on SDK's flavour of LogRecord, and are apparently assumed to exist
record.resource = logger.resource
record.dropped_attributes = 0

logger.emit(record)

[alexmojaki]

Mildly related, the SDK LogRecord has this:

        self.resource = (
            resource if isinstance(resource, Resource) else Resource.create({})
        )

Resource.create({}) is unreasonably slow because it creates a thread pool. Demo:

import time

from opentelemetry.sdk._logs import LogRecord
from opentelemetry.sdk.resources import Resource

for resource in [None, Resource.create({})]:
    start = time.time()
    for _ in range(1000):
        LogRecord(resource=resource)
    end = time.time()
    print(f'Resource: {resource}')
    print(f'Time taken: {end - start:.6f} seconds')

Output:

Resource: None
Time taken: 4.610783 seconds
Resource: <opentelemetry.sdk.resources.Resource object at 0x115841ac0>
Time taken: 0.001270 seconds

[alexmojaki]

Consider using a pattern similar to tracing when users don't need to instantiate a new span. Instead they could call logger.emit(....) with parameters and the SDK implementation of logger will create SDK implementation of LogRecord

Agreed, the spec seems to imply that this is the way in https://opentelemetry.io/docs/specs/otel/logs/api/#emit-a-logrecord:

I don't think LogRecord is meant to be constructed at all.