OTEP: Context-scoped Attributes

[carlosalberto]

Add Context-scoped telemetry attributes which typically apply to all signals associated with a trace as it crosses a single service.

Resurrecting open-telemetry/oteps#207 while updating some changes that have happened since (e.g. changes in InstrumentationScope being a runtime concept rather than a compile-time one). Originally proposed by @Oberon00

Additionally:

1. A Processor alternative being discarded as we would need to add an additional pre OnStart call in order to support Samplers.
2. The open question whether Baggage should be included as part of stamping attributes to individual telemetry items (similar enough functionality).
3. We disable this functionality by default. Users have to opt-in to use this.

Java prototype for traces/logs: https://github.com/open-telemetry/opentelemetry-java/compare/main...carlosalberto:context-scoped-attrs-proto?expand=1

Summary

This OTEP aims to address various related demands that have been brought up in the past, where the scope of Resource attributes is too broad, but the scope of Span attributes is too narrow. For example, this happens where there is a mismatch between the OpenTelemetry SDK’s (and thus TracerProvider’s, MeterProvider’s) process-wide initialization and the semantic scope of a (sub)service.

The context-scoped attributes allows you to attach attributes to all telemetry signals emitted within a Context. Context-scoped attributes are normal attributes, which means you can use strings, integers, floating point numbers, booleans or arrays thereof, just like for Span or Resource attributes. Context-scoped attributes are associated with all telemetry signals emitted while the Context containing the Context-scoped attributes is active and are available to telemetry exporters. For spans, the context within which the span is started applies. Like other telemetry APIs, Context-scoped attributes are write-only for applications. You cannot query the currently set Context-scoped attributes, they are only available on the SDK level (e.g. to Samplers and SpanProcessors and exporters).

Context-scoped attributes should be thought of equivalent to adding the attribute directly to each single telemetry item it applies to.

[cijothomas]

we need to specify something about how it gets added to log/span/metrics. This OTEP says opt-in, but does not say anything specific beyond that. Is that intentionally left out in OTEP

[carlosalberto]

Clarified this to (very) succinctly mention that the Context-scoped attributes are added at telemetry creation/emit time.

[cijothomas]

resource cannot be used - this seems like a very specific limitation for the jvm example? Isn't it possible to create multiple Meter/Tracer/Logger providers, each with its own resource, for each independent application within the same process?

[carlosalberto]

Included this as an alternative mentioned in the Motivation section (albeit being a prohibitive one, given you would need to duplicate the pipeline).

[cijothomas]

exporters? That would be incorrect as a batching_processor can call exporter in its background thread, losing the original context.

[carlosalberto]

Clarified this. Context-scoped attributes only apply to telemetry at creation time.

[jmacd]

Samplers are able to capture context-scoped attributes and apply them for traces, logs have the context, and in metrics we discussed #4318. I see this context-scoped attributes as generic for MeasurementProcessor, in other words.

[cijothomas]

lets make it clear what is meant by disabled here. I guess you are referring to "Context-scoped attributes are not automatically added to spans/logs/metrics unless user has explicitly opted into it. Opt-in mechanism are in [link to the section where we talk about it.]"

[jack-berg]

I didn't see any text describing what an opt-in mechanism would look like. I expected it to take the form of config params on SDK TracerProvider, LoggerProvider, MeterProvider, or new built-in SpanProcessor, LogRecordProcessors, MeasurementProcessor (rejected 🙁) with corresponding config handles.

Did I miss something or is the opt-in mechanism TBD?

[cijothomas]

app-level data vs telemetry annotations - not sure of the differences between them.. In my view, the only difference between Baggage is that Baggage is out-of-proc+in-proc, whereas Context-scope Attributes is in-proc only.
(I can think of calling this as InProcOnlyBagagge as well)

What kind of data users put inside it - totally upto them. They can put data to affect control flow, telemetry attributes, feature-flags etc. OTel don't put any restrictions.

[carlosalberto]

Baggage only accepts strings. Also, if we were to go any Baggage way, we should offer automatic attachment of Baggage values at the SDK level, like the one defined by this OTEP.

[cijothomas]

I am not so sure about "write-only" aspect. Why restrict it, when Context itself supports reading:
specification/tree/main/specification/context#get-value

[carlosalberto]

I'd like to reduce the area if possible. If we get cases were it is valuable to expose this to the public, we can go ahead and do that.

[cijothomas]

I think a good way to describe this is - they differ in lifetime. Tracer/Meter etc. typically lives same time as the application itself. But context-scoped ties its lifetime to the context itself, so if the context is per-request, then it lives only that long. But if the context is app-start, then it lives entire application.

[carlosalberto]

Added a small paragraph to mention this.

[cijothomas]

this API section is merely talking about adding ScopedAttribtues to Context - so it is better to avoid the mentioning of Span case here (and span attributes)

[carlosalberto]

Actually I think examples are useful in order to be super clear. Changed this to "e.g. Span or LogRecordItem".

[dashpole]

Go has had a "labeler" concept specific to http that is similar to this proposal: https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp#ContextWithLabeler. I would love to see a more cohesive and generic design for this!

[dashpole]

I'm not sure I see the need for an API vs SDK split. As I understand it, this provides a way for the end user to "bypass" instrumentation libraries, and pass attributes to the SDK by including them in the context.

An instrumentation library shouldn't ever use of this API because it can already pass attributes to the SDK providing them directly to the API (which is more efficient, at least in Go). For users that want to avoid the overhead of these contextual calls, we could probably recommend checking to see if a global tracer/meter/logger provider are enabled.

[carlosalberto]

In that case, the API could expose Get AND Set operations for Context-scoped attributes, meaning anybody could read them (unlike attributes set on the Span). @cijothomas had already suggested this as well. As you are both maintainers and think it's fine to massage this functionality to go that way, I will update this OTEP to reflect this new direction.

[dashpole]

I think context-scoped attributes should not be added as metric attributes by default, given the impact of new attributes on cardinality. I would prefer if the default behavior was for the metrics SDK to treat them the same as default-disabled attributes provided to the Metrics API:

• These attributes are added as filtered_attributes on exemplars by default.
• These attributes can be opted-in to metric attributes using Views.

There are definitely cases where a user will specifically want the contextual attribute stored as a metric attribute by default. I would prefer if this was an optional parameter that could be passed when storing the attribute in context.

[carlosalberto]

I would prefer if this was an optional parameter that could be passed when storing the attribute in context.

I think we could add this with a flags attribute as a metric-only feature (as spans nor logs have such opt-in functionality, and hence a simple boolean wouldn't be so descriptive):

// API
void SetContextScopedAttributes(
  Context context,
  Attributes attributes,
  ContextScopeFlags flags);

enum ContextScopeFlags {
  NONE = 0,
  METRICS_OPT_IN = 1
}

// Usage from the metrics SDK
contextAttributes, flags = GetContextScopedAttributes(context);
if (flags & ContextFlags.METRICS_OPT_IN != 0) {
  // Conditional setting.
} else {
  // set them on metrics unconditionally.
}

Is this something close to what you have in mind?

[dashpole]

Yeah, more or less. In Go I was thinking it would just use the option pattern (like most of our parameters do):

// User code
ctx := context.Background()
attrs := attribute.NewSet(attribute.String("foo", "bar"))
ctx = contextual.ContextWithAttributes(ctx, attrs, contextual.WithAddToMetrics(true))

// Metrics SDK Code
attrs, config := contextual.AttributesFromContext(ctx)
if config.AddToMetrics() {
    // Add attrs directly to metric attributes
} else {
    // Add attrs to filtered attributes passed to exemplarReservoir.Offer.
}

[jack-berg]

I think context-scoped attributes should not be added as metric attributes by default, given the impact of new attributes on cardinality.

The key question for me is about configuration: just how granular / expressive should the configuration tools be. More granular mean more verbose, but a tool capable of solving more problems. Less granular means easier to configure, but less capable.

The PR currently proposes the least granular option:

file_format: "1.0"
# Add all context scoped attributes to all metrics, logs, traces within scope
context_scoped_attributes: true

On the other end of the spectrum, we might have something like:

file_format: "1.0"
tracer_provider:
  # include all context scoped attributes on spans
  context_scoped_attributes:
    included:
      - *
logger_provider:
  # include all context scoped attributes on logs
  context_scoped_attributes:
    included:
      - * 
meter_provider:
  # include all context scoped attributes excepted "foo" and "bar" on metrics
  context_scoped_attributes:
    included:
      - *
    excluded:
      - foo
      - bar   

The upside is that there's a built in mechanism to account for metrics cardinality issues, and any other expressiveness use cases that arise. The downside is that the common case of "enable all context scoped attributes on all signals" is more verbose:

file_format: "1.0"
tracer_provider:
  context_scoped_attributes:
    included:
      - *
logger_provider:
  context_scoped_attributes:
    included:
      - * 
meter_provider:
  context_scoped_attributes:
    included:
      - *

There's also a middle ground of providing a top level and signal specific config option:

file_format: "1.0"
# include all context scoped attributes on all signals
context_scoped_attributes:
  included:
    - *
tracer_provider: ...
logger_provider: ...
meter_provider:
  # override metrics context scoped attributes to include all attributes except foo and bar
  context_scoped_attributes:
    included:
      - *
    excluded:
      - foo
      - bar   

Given that declarative config already makes you repeat things like OTLP exporter configuration for each signal, I lean towards the middle ground approach. I would bet that omitting the additional granularity / expressive now would be temporary and we would end up adding it later anyway. The middle ground strikes a nice balance, allowing for relatively terse config for the common case while still accommodating signal-specific config use cases.

[XSAM]

@dashpole I am not sure metrics should be special-cased by default here.

The main value of this OTEP is a simple mental model: if an attribute is added to the active Context, it applies to telemetry produced in that Context.

Once metrics become the only exception, that model gets harder to explain and easier to misuse. Users will see the attribute on spans and logs and reasonably expect it to also be present on metrics from the same operation.

I agree metric cardinality is a real concern, but I don't think we should hard-code that concern into the default behavior. Cardinality is a consequence of which attributes the user chooses to put into context. That is already something users need to think about when they decide to use this API, just like they do today when adding metric attributes directly.

In other words, I think the right split is:

• semantics: context-scoped attributes apply consistently to all signals
• controls: users can disable/filter them for a specific signal.

That keeps the feature understandable while still giving metrics users the guardrails they need.

Also, for the motivating examples in this PR, http.app is exactly the kind of attribute that seems useful on metrics as well.

---

Besides, this seems like the more extensible design. If we special-case metrics now, then any future signal with similar cost/cardinality concerns may need another exception.

[dashpole]

In the vast majority of cases, someone will know if an attribute should apply to metrics or not at the point where they put it in context. http.app is a good example of something that we would want to make apply to metrics. But users often want to grab something from a header, or put an ID or something associated with the request. I think a lot of users will dump everything from baggage as attributes into the context. In those cases, I should be able to somehow indicate that the attribute isn't appropriate for metrics without having to do that via external configuration. I would at least like to see the ability to decide if an attribute is applicable to metrics at the point where it is put in context.

Cardinality is a consequence of which attributes the user chooses to put into context. That is already something users need to think about when they decide to use this API, just like they do today when adding metric attributes directly.

Most users lean on our instrumentation to make these decisions intelligently for them, and even we haven't always gotten it right. These contextual attributes will apply to all downstream metrics (incl. server and potentially multiple different clients), which can be hard to reason about

I could be persuaded on the default. I tend to prefer safer defaults, and think making users pass an additional option as an "are you sure" seems reasonable.

[XSAM]

But users often want to grab something from a header, or put an ID or something associated with the request. I think a lot of users will dump everything from baggage as attributes into the context.

Do you think users would think twice before adding unnecessary attributes if we make it explicit in this method’s documentation that these attributes will be propagated to all downstream signals and that cardinality should be taken into account?

Do you think it would be the user’s responsibility to use the API correctly once the documentation is clear?

If there is a future signal with a similar cardinality concern to metrics, what should we do with it? Do we exclude it by default, as we do for metrics?

---

I also think the semantic model should remain simple and future-proof: if an attribute is attached to the active Context, it applies to telemetry produced in that Context. This principle should remain unchanged, even if new signals are added in the future.

If we disable metrics by default, the model becomes: if an attribute is attached to the active Context, it applies to telemetry produced in that Context, except for metrics. That semantic meaning would then need to be revised again if a future signal with similar cardinality concerns is added: if an attribute is attached to the active Context, it applies to telemetry produced in that Context, except for metrics and that newly added signal.

[dashpole]

Do you think users would think twice before adding unnecessary attributes if we make it explicit in this method’s documentation that these attributes will be propagated to all downstream signals and that cardinality should be taken into account?

It isn't about necessary vs unnecessary. There are valid use-cases where someone wants to add a high-cardinality attribute to spans and logs but not metrics. If we only support contextual attributes on all signals at once, then people just won't be able to use it for things that are high-cardinality.

Do you think it would be the user’s responsibility to use the API correctly once the documentation is clear?

Of course. But we should also do our best to provide good defaults, and avoid giving users foot-guns.

If there is a future signal with a similar cardinality concern to metrics, what should we do with it? Do we exclude it by default, as we do for metrics?

It should have the same default behavior we choose for metrics in that case -- whatever we choose.

@XSAM can you help clarify your position to me? Are you OK with users being able to mark some attributes as "high cardinality" or "not for metrics" when adding them to the context? Are you just opposed to making that the default behavior?

[XSAM]

@dashpole

If we only support contextual attributes on all signals at once, then people just won't be able to use it for things that are high-cardinality.

I am not against disabling some signals when adding attributes to the context. I support offering the option to disable signals.

The semantic model I mentioned is just a way to think about the default behavior.

Are you OK with users being able to mark some attributes as "high cardinality" or "not for metrics" when adding them to the context?

Yes

Are you just opposed to making that the default behavior?

Yes

[dashpole]

Sounds good. I'm OK with the default being to add it to all signals.

[dashpole]

FYI @open-telemetry/go-maintainers I think whatever comes from this proposal is what should replace our otelhttp.Labeler.

[jack-berg]

standard attributes are expanded at this point to include complex types.

[jack-berg]

I think context-scoped attributes should not be added as metric attributes by default, given the impact of new attributes on cardinality.

The key question for me is about configuration: just how granular / expressive should the configuration tools be. More granular mean more verbose, but a tool capable of solving more problems. Less granular means easier to configure, but less capable.

The PR currently proposes the least granular option:

file_format: "1.0"
# Add all context scoped attributes to all metrics, logs, traces within scope
context_scoped_attributes: true

On the other end of the spectrum, we might have something like:

file_format: "1.0"
tracer_provider:
  # include all context scoped attributes on spans
  context_scoped_attributes:
    included:
      - *
logger_provider:
  # include all context scoped attributes on logs
  context_scoped_attributes:
    included:
      - * 
meter_provider:
  # include all context scoped attributes excepted "foo" and "bar" on metrics
  context_scoped_attributes:
    included:
      - *
    excluded:
      - foo
      - bar   

The upside is that there's a built in mechanism to account for metrics cardinality issues, and any other expressiveness use cases that arise. The downside is that the common case of "enable all context scoped attributes on all signals" is more verbose:

file_format: "1.0"
tracer_provider:
  context_scoped_attributes:
    included:
      - *
logger_provider:
  context_scoped_attributes:
    included:
      - * 
meter_provider:
  context_scoped_attributes:
    included:
      - *

There's also a middle ground of providing a top level and signal specific config option:

file_format: "1.0"
# include all context scoped attributes on all signals
context_scoped_attributes:
  included:
    - *
tracer_provider: ...
logger_provider: ...
meter_provider:
  # override metrics context scoped attributes to include all attributes except foo and bar
  context_scoped_attributes:
    included:
      - *
    excluded:
      - foo
      - bar   

Given that declarative config already makes you repeat things like OTLP exporter configuration for each signal, I lean towards the middle ground approach. I would bet that omitting the additional granularity / expressive now would be temporary and we would end up adding it later anyway. The middle ground strikes a nice balance, allowing for relatively terse config for the common case while still accommodating signal-specific config use cases.

[jack-berg]

Suggested change

	When Context-scoped attributes are enabled, they MUST be added to telemetry items
	When Context-scoped attributes are enabled, they MUST be added to telemetry items

[jack-berg]

Suggested change

	A similar problem occurs with `faas.name` and `cloud.resource_id`
	A similar problem occurs with `faas.name` and `cloud.resource_id`

[lmolkova]

Thanks for bringing it back!

[lmolkova]

why? it's a new API, nothing should create them today, nothing will be broken - users explicitly opt into them by using the API.

[jack-berg]

What if instrumentation (i.e. not the user) adds them?

[lmolkova]

it's a good question if instrumentations should be allowed to do it - #4931 (comment)

[lmolkova]

there should be no side-effects for existing users since nobody creates them today

[lmolkova]

does it include profiles in the future?

[jmacd]

I hope so! In Golang I'm aware of built-in runtime libraries that are meant for this sort of thing, like https://pkg.go.dev/runtime/trace#WithRegion. I think the ideal outcome would be that OTel profiling tools recognize OTel context-scoped attributes like this.

[lmolkova]

Suggested change

	Moreover, Tracer, Meter and LogEmitter instances typically have the same
	Moreover, Tracer, Meter and Logger instances typically have the same

[lmolkova]

Can we find a better / shorter name ? e.g. .NET Logger has StartScope method. I also would imagine the API that returns closeable / disposeable like

var scope = Context.StartScope(attrs);
// do stuff
scope.close();

in .NET or Java or

with start_scope(attrs):
   # do stuff

in Python.

I think we don't need to be prescriptive on what method returns in the OTEP and will polish the details in the spec PRs.

Some other suggestions: Context.addAttributes()

[lmolkova]

is it a problem at all? We already have 3 layers of attributes and effectively only use 2, in my mental model having 4th layer of attributes will bring only harm with no benefit.

So curious why we need to talk in such details about (what looks to me) a non-viable option

[lmolkova]

I believe a lot of context-scoped attributes would have high(ish) cardinality and I think we need different defaults for signals (on-by-default for spans and logs, off for metrics).

If we want to default to off for all signals consistently, then we need to start with per-signal configuration anyway.

[lmolkova]

I think it would be useful to mention similar concept that exists in logs (MDC in Java, .NET ILogger scopes) since they provide examples of API and we can also learn from years of usage and patterns people came up with.

[lmolkova]

this does not seem like a good use-case.

Imagine this

parent = tracer.startSpan("incoming request");

// scope1 should never change anything for parent since there could be a 
// lot of unrelated scopes under it

scope1 = Context.startScope({"operation.name" : "download data"});
child1 = tracer.startSpan("download");
// do stuff
child1.end();
scope1.close();


scope2 = Context.startScope({"operation.name" : "process data"});
child2 = tracer.startSpan("process");
// do stuff
child2.end();
scope2.close();

//  lifetime of context-scoped attributes is shorter than on 
// a parent span and there were many of them, so we should not update 
// parent
parent.end();

[lmolkova]

should instrumentation libs ever set context scoped attributes? I can find use-cases for it (e.g. when doing complex logical operations, it's useful to stamp parent operation name on all transport requests under it), but it means that a single bad instrumentation can create a lot of noise and cardinality issues on metrics.

If we never want instrumentations to use the API, then should it be SDK-only API?

[XSAM]

As an instrumentation lib author, I am happy to see this get implemented. So, I don't need to provide these methods as instrumentation options:

• https://pkg.go.dev/github.com/XSAM/otelsql#WithAttributesGetter
• https://pkg.go.dev/github.com/XSAM/otelsql#WithInstrumentAttributesGetter

[XSAM]

@dashpole I am not sure metrics should be special-cased by default here.

The main value of this OTEP is a simple mental model: if an attribute is added to the active Context, it applies to telemetry produced in that Context.

Once metrics become the only exception, that model gets harder to explain and easier to misuse. Users will see the attribute on spans and logs and reasonably expect it to also be present on metrics from the same operation.

I agree metric cardinality is a real concern, but I don't think we should hard-code that concern into the default behavior. Cardinality is a consequence of which attributes the user chooses to put into context. That is already something users need to think about when they decide to use this API, just like they do today when adding metric attributes directly.

In other words, I think the right split is:

• semantics: context-scoped attributes apply consistently to all signals
• controls: users can disable/filter them for a specific signal.

That keeps the feature understandable while still giving metrics users the guardrails they need.

Also, for the motivating examples in this PR, http.app is exactly the kind of attribute that seems useful on metrics as well.

---

Besides, this seems like the more extensible design. If we special-case metrics now, then any future signal with similar cost/cardinality concerns may need another exception.

[XSAM]

Similar to this https://github.com/open-telemetry/opentelemetry-specification/pull/4931/changes#r2971562882

This should work like the existing context API without an SDK, and it can't be a no-op.

[jmacd]

The list of alternatives makes me feel dizzy. Otherwise, looks good--thanks @carlosalberto!

[jmacd]

I hope so! In Golang I'm aware of built-in runtime libraries that are meant for this sort of thing, like https://pkg.go.dev/runtime/trace#WithRegion. I think the ideal outcome would be that OTel profiling tools recognize OTel context-scoped attributes like this.

[jmacd]

Samplers are able to capture context-scoped attributes and apply them for traces, logs have the context, and in metrics we discussed #4318. I see this context-scoped attributes as generic for MeasurementProcessor, in other words.

[jmacd]

Suggested change

	at the time of writing this OTEP there is no processor functionality for metrics.
	at the time of writing this OTEP there is no specified processor functionality for metrics.

Thinking of #4318

[dashpole]

PoC in Go: open-telemetry/opentelemetry-go@main...dashpole:opentelemetry-go:context_attrs_poc

It allows specifying whether the attributes are added as metric attributes at the point where you add them to the context, and lets views promote attributes added through the context as opt-in.

[pellared]

nit: this could be changed to [a mermaid diagram](nit: this could be changed to a https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-diagrams#creating-mermaid-diagrams)