Skip to content

query: tenant header not forwarded on distributed Query gRPC fanout #8632

@agardiman

Description

@agardiman

Thanos, Prometheus and Golang version used:
Thanos: custom build from main (after patching tenant propagation); was reproduced on upstream main before patch.
Prometheus: N/A (issue in Thanos query path)
Go: go1.22 (dev env)

Object Storage Provider:
N/A (issue in query fanout, not storage)

What happened:
In distributed query mode, Thanos Query forwards PromQL to other Query nodes via the gRPC Query/QueryRange API. Even with --query.tenant-header configured (e.g. --query.tenant-header=X-Scope-OrgID), the downstream Query server does not see the tenant in gRPC metadata (metadata.FromIncomingContext is empty for THANOS-TENANT). Only the HTTP→StoreAPI path propagates the tenant.

What you expected to happen:
The tenant from the incoming HTTP header should be placed into gRPC metadata for all downstream Query gRPC calls so Query-compatible services can enforce tenancy and Mimir/Cortex-compatible headers are preserved.
How to reproduce it (as minimally and precisely as possible):

  1. Run Thanos Query in distributed mode with --query.tenant-header=X-Scope-OrgID.
  2. Send any /api/v1/query request with header X-Scope-OrgId: .
  3. On the downstream Query server (or a Query-compatible service), inspect metadata.FromIncomingContext(ctx) inside the gRPC handler: THANOS-TENANT key is missing.

Full logs to relevant components:
Not included; behavior is silent (metadata missing).

Anything else we need to know:
Tenant is stored on the HTTP context (tenancy.TenantKey) but never appended to outgoing Query gRPC metadata.
StoreAPI fanout already appends THANOS-TENANT via metadata.AppendToOutgoingContext; parity is missing for Query gRPC fanout.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions