Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 49 additions & 30 deletions content/en/docs/collector/internal-telemetry.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Internal telemetry
weight: 25
cSpell:ignore: alloc batchprocessor journalctl
cSpell:ignore: alloc batchprocessor journalctl otelgrpc
---

You can inspect the health of any OpenTelemetry Collector instance by checking
Expand Down Expand Up @@ -375,7 +375,7 @@ naming conventions, resulting in metric names that looked like
Versions 0.120.0 and later of the Collector use Prometheus 3.0 scrapers, so the
original `http*` and `rpc*` metric names with dots are preserved. The
[internal metrics](#lists-of-internal-metrics) on this page are listed in their
original form, such as`rpc.server.duration`. For more information, see the
original form, such as `rpc.server.call.duration`. For more information, see the
[Collector v0.120.0 release notes](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CHANGELOG.md#v01200).

Comment thread
tiffany76 marked this conversation as resolved.
### Lists of internal metrics
Expand Down Expand Up @@ -453,35 +453,54 @@ files in the repository.

#### Additional `detailed`-level metrics

| Metric name | Description | Type |
| ----------------------------------------------------- | ----------------------------------------------------------------------------------------- | --------- |
| `http.client.request.body.size` | Measures the size of HTTP client request bodies. | Counter |
| `http.client.request.duration` | Measures the duration of HTTP client requests. | Histogram |
| `http.server.request.body.size` | Measures the size of HTTP server request bodies. | Counter |
| `http.server.request.duration` | Measures the duration of HTTP server requests. | Histogram |
| `http.server.response.body.size` | Measures the size of HTTP server response bodies. | Counter |
| `otelcol_processor_batch_batch_`<br>`send_size_bytes` | Number of bytes in the batch that was sent. | Histogram |
| `rpc.client.duration` | Measures the duration of outbound RPC. | Histogram |
| `rpc.client.request.size` | Measures the size of RPC request messages (uncompressed). | Histogram |
| `rpc.client.requests_per_rpc` | Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs. | Histogram |
| `rpc.client.response.size` | Measures the size of RPC response messages (uncompressed). | Histogram |
| `rpc.client.responses_per_rpc` | Measures the number of messages sent per RPC. Should be 1 for all non-streaming RPCs. | Histogram |
| `rpc.server.duration` | Measures the duration of inbound RPC. | Histogram |
| `rpc.server.request.size` | Measures the size of RPC request messages (uncompressed). | Histogram |
| `rpc.server.requests_per_rpc` | Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs. | Histogram |
| `rpc.server.response.size` | Measures the size of RPC response messages (uncompressed). | Histogram |
| `rpc.server.responses_per_rpc` | Measures the number of messages sent per RPC. Should be 1 for all non-streaming RPCs. | Histogram |
| Metric name | Description | Type |
| ----------------------------------------------------- | --------------------------------------------------------------- | --------- |
| `http.client.request.body.size` | Measures the size of HTTP client request bodies. | Counter |
| `http.client.request.duration` | Measures the duration of HTTP client requests. | Histogram |
| `http.server.request.body.size` | Measures the size of HTTP server request bodies. | Counter |
| `http.server.request.duration` | Measures the duration of HTTP server requests. | Histogram |
| `http.server.response.body.size` | Measures the size of HTTP server response bodies. | Counter |
| `otelcol_processor_batch_batch_`<br>`send_size_bytes` | Number of bytes in the batch that was sent. | Histogram |
| `rpc.client.call.duration` | Measures the duration of outbound remote procedure calls (RPC). | Histogram |
| `rpc.server.call.duration` | Measures the duration of inbound remote procedure calls (RPC). | Histogram |

> [!NOTE]
>
> The `http*` and `rpc*` metrics are not covered by the maturity levels below
> since they are not under the Collector SIG control.
>
> The `otelcol_processor_batch_` metrics are unique to the `batchprocessor`.
>
> The `otelcol_receiver_`, `otelcol_scraper_`, `otelcol_processor_`, and
> `otelcol_exporter_` metrics come from their respective `helper` packages. As
> such, some components not using those packages might not emit them.
#### Ownership of emitted metrics

Some metrics are not owned by the Collector SIG and some are limited to certain
components.

**`http*`and `rpc` metrics**

These metrics are not under the Collector SIG's control, and as such, are not
covered by the maturity levels below.

**`rpc` metrics**

The Collector's internal RPC metrics come from the upstream
[`otelgrpc`](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/google.golang.org/grpc/otelgrpc)
instrumentation, which tracks the
[OpenTelemetry RPC semantic conventions](/docs/specs/semconv/rpc/rpc-metrics/).
The set of RPC metrics emitted by the Collector has changed across releases:

| Collector version | Emitted RPC metrics |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| v0.146.x and earlier | `rpc.client.duration`, `rpc.server.duration`, `rpc.*.request.size`, `rpc.*.response.size`, `rpc.*.requests_per_rpc`, `rpc.*.responses_per_rpc` |
| v0.147.0 | `rpc.client.call.duration`, `rpc.server.call.duration`, `rpc.*.request.size`, `rpc.*.response.size` (the `*_per_rpc` metrics are deprecated and no longer emitted) |
| v0.148.0 and later | `rpc.client.call.duration`, `rpc.server.call.duration` only |

RPC size metrics are not emitted by Collector v0.148.0 or later. The
[RPC semantic conventions v1.40.0](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.40.0)
deprecated them due to ambiguous definitions and inconsistent implementation.

**`otelcol_processor_batch_*` metrics**

These metrics are unique to the `batchprocessor`.

**`helper` package metrics**

The `otelcol_receiver_`, `otelcol_scraper_`, `otelcol_processor_`, and
`otelcol_exporter_` metrics come from their respective `helper` packages. As
such, some components not using those packages might not emit them.

### Events observable with internal logs

Expand Down
8 changes: 8 additions & 0 deletions static/refcache.json
Original file line number Diff line number Diff line change
Expand Up @@ -12535,6 +12535,10 @@
"StatusCode": 206,
"LastSeen": "2026-03-17T10:01:02.031229255Z"
},
"https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/google.golang.org/grpc/otelgrpc": {
"StatusCode": 206,
"LastSeen": "2026-04-18T20:56:37.309954588Z"
},
"https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/host": {
"StatusCode": 206,
"LastSeen": "2026-03-17T10:01:02.546647801Z"
Expand Down Expand Up @@ -16623,6 +16627,10 @@
"StatusCode": 206,
"LastSeen": "2026-03-12T10:02:23.33704249Z"
},
"https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.40.0": {
"StatusCode": 206,
"LastSeen": "2026-04-18T20:56:41.578659277Z"
},
"https://github.com/open-telemetry/semantic-conventions/tree/v1.26.0": {
"StatusCode": 206,
"LastSeen": "2026-03-18T09:57:31.902043431Z"
Expand Down
Loading