OpenTelemetry

[mfosterw]

Description

I think it makes sense to offer observability through OpenTelemetry, at least for projects running through docker-compose (it should be possible without but I don't feel qualified to write the docs otherwise). This could potentially be an additional option alongside Sentry.

Rationale

I recently made a first stab at implementing OpenTelemetry in my cookiecutter project (see mfosterw/cookiestocracy#278) and even though there were some hiccups I think it's a really good way to get the type of observability typically reserved for managed deployments, for free.

[hey-granth]

The following architectural implementation will resolve this issue acc to me.

1. Configuration: Introduce prompt variable "use_opentelemetry": ["n", "y"] in cookiecutter.json.
2. Dependency Management: Inject opentelemetry-distro, opentelemetry-instrumentation-django, opentelemetry-instrumentation-celery, opentelemetry-instrumentation-psycopg, and opentelemetry-exporter-otlp into requirements/base.txt utilizing a Jinja conditional ({% if cookiecutter.use_opentelemetry == 'y' %}).
3. Instrumentation: Modify config/wsgi.py, config/asgi.py, and config/celery_app.py to initialize OpenTelemetry auto-instrumentation during the startup sequence.
4. Orchestration: Update docker-compose.local.yml and docker-compose.production.yml to instantiate an OpenTelemetry Collector and a visualization backend (Jaeger) upon feature activation.
5. Environment Configuration: Map standard OpenTelemetry environment variables (OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT) into Django and Celery service definitions via the .envs/ directory.
6. Interoperability: Implement configuration logic to enforce W3C Trace Context propagation when Sentry and OpenTelemetry are concurrently enabled.
7. Documentation: Append strict deployment instructions in the docs/ directory detailing collector endpoint configuration and local tracing UI access parameters.

please assign this to me @mfosterw .

[nobutakayamauchi]

Thanks for sharing this — the configuration-level observability approach makes a lot of sense.

One thing I’ve been thinking about recently is whether observability may eventually need to capture not only runtime signals (logs, metrics, traces), but also parts of the decision context that led to configuration or deployment choices.

For example:
• why a particular instrumentation strategy was chosen
• which alternatives were rejected
• environmental assumptions at the time of rollout

Today those tend to live only in PR discussions or human memory.

Do you see OpenTelemetry evolving toward capturing higher-level execution intent or decision context, or do you expect that layer to remain external to telemetry systems?

I’m curious how maintainers think about that boundary.

[mfosterw]

@hey-granth I am not a maintainer so I cannot assign issues, but you can just go ahead and open a pull request that mentions this if you (or Claude) have written the code. In the past I have found the maintainers more responsive to PRs than issues (understandably). Also fyi I used the grafana stack everywhere except the collector, so I have Tempo in place of Jaeger for storing traces, but I do not have a strong opinion on either.