[bug][metrics OtelExporter] Invalid protobuf structure: proto: wrong wireType = 1 for field BucketCounts

[tebeco]

Package

OpenTelemetry.Exporter.OpenTelemetryProtocol

Package Version

Package Name	Version
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.12.0
OpenTelemetry.Extensions.Hosting	1.12.0
OpenTelemetry.Instrumentation.AspNetCore	1.12.0
OpenTelemetry.Instrumentation.Http	1.12.0
OpenTelemetry.Instrumentation.Runtime	1.12.0

here's my csproj lineup:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <UserSecretsId>agentless-otel-datadog</UserSecretsId>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Telemetry" Version="9.9.0" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.12.0" />
  </ItemGroup>

</Project>

Runtime Version

net9.0

Description

I'm seeing 400 in my logs when the Periodic Metrics exporter try to export Metrics to datadog

HttpClient gets Error 400 - "\b\u0003\u0012MInvalid protobuf structure: proto: wrong wireType = 1 for field BucketCounts\n"

it look like similar to this one:
open-telemetry/opentelemetry-collector#13727

see bellow for my repository link (inside that repo there's most of the link i've used to built it up for references)

Steps to Reproduce

The impossible part is ...

• Get a Datadog account
• ask to be whitelisted for their OTEL Metrics endpoint
  (if there's a need for a call or a session for debugging, or stuff to try, feel free to suggest)

The possible part:

• clone the repo: https://github.com/tebeco/AgentLessOtelDataDog
• set the API KEY as mentioned
• press F5
• Look at the logs
• Enable Exception breaking everything

Expected Result

OtelExporter, should managed to export metrics (i don't mean to be hard on that, i just don't know how to phrase it)

But I'm opening this ticket because I have no idea of the source

• My code
• My config
• one of the package here
• datadog side ingestion point (I have raised this ticket to their support already to be clear)

the reason I've decided to also open it here is because of open-telemetry/opentelemetry-collector#13727

Actual Result

no metrics are sent

Additional Context

I have raised this ticket to their support already to be clear as I don't know where the failure happens

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[martincostello]

Looks like DataDog have deployed v0.133.x (or affected parts thereof) to their hosted infrastructure, so the issue I found is causing your metrics to be rejected. If that’s the case, you need to create a support ticket with them to roll it back or roll forward to a version without the problematic change.

Another possible workaround, if they support it, is to change your SDK to export using gRPC instead of http/protobuf.

[martincostello]

For transparency, we had a similar issue at Grafana, which was reported here: grafana/grafana-opentelemetry-dotnet#270

[martincostello]

This was also already previously reported in #6464.

[Kielek]

@martincostello, do you think that we can improve OTLP exporter to work with this denser format? I suppose that we have some sub-optimal implementation?

[martincostello]

My understanding is that we're within behaving within the spec here (so it's not a bug), but the preferred implementation is the one that didn't get broken in v0.133.0.

We could change it, but I imagine the work to do that wouldn't be quick to implement and is probably equally as risky of accidentally breaking compatibility as the OTel Collector change was as both have custom protobuf implementations.

We'd probably want whatever solution for #6463 in place before attempting to do so.

[rajkumar-rangaraj]

I agree with @martincostello that once we have solid tests in place, we can safely fine-tune the implementation for wire efficiency (i.e. denser encoding, dropping redundancies, etc.).

[bogdandrutu]

@rajkumar-rangaraj this is not an optimization but an inconformity with the proto3 requirements. This is a bug not a "fine-tune".

[tebeco]

I just got a reply from support, which i'll adapt slightly for the ticket
note: I've asked their support to see of someone from the OTEL product team responsible for their ingest endpoint could come here to help every one understand what's happenning.

Support answer:

Following up on your question about the OTLP metrics failures (HTTP 400: Invalid protobuf structure - wrong wireType = 1 for BucketCounts), I checked with our OTEL team. Here’s what they confirmed:

No OTel Collector behind the intake. Our OTLP endpoint is our own HTTP server for OTLP, it does not run the OpenTelemetry Collector.
The endpoint currently uses go.opentelemetry.io/proto/otlp v1.7.0.
The OTLP proto is intended to be backward/forward compatible. Given that, the 400 you’re seeing indicates a .NET SDK/exporter compatibility bug with the payload it’s generating, rather than an incompatibility on our side.

I'm not sure how to proceed with this @bogdandrutu / @rajkumar-rangaraj / @Kielek
What I'm guessing is that go.opentelemetry.io/proto/otlp is about versionning the proto contract for OTLP
and either that version is bugged, or the current 1.12.0 of the dotnet implementation is bugged
issue is that I don't know how to assert any of that

I also have absolutly no idea how to setup a go server endpoint using go.opentelemetry.io/proto/otlp 1.7.0 just to test wire compatibility

[tebeco]

Sorry to ping you here @tigrannajaryan
but i found this:
open-telemetry/opentelemetry-proto#619
This was merged and released into 1.6.0

I'm wondering if the issue i have would be related to that or not ?

[martincostello]

go.opentelemetry.io/proto/otlp v1.7.0 is the component inside the collector that changed that caused the incompatibility (IIRC).

[martincostello]

No, I was wrong, it was pdata I think (or pdata was where the fix for that update happened): open-telemetry/opentelemetry-collector#13730