Thanos, Prometheus and Golang version used:
Thanos: custom build from main (after patching tenant propagation); was reproduced on upstream main before patch.
Prometheus: N/A (issue in Thanos query path)
Go: go1.22 (dev env)
Object Storage Provider:
N/A (issue in query fanout, not storage)
What happened:
In distributed query mode, Thanos Query forwards PromQL to other Query nodes via the gRPC Query/QueryRange API. Even with --query.tenant-header configured (e.g. --query.tenant-header=X-Scope-OrgID), the downstream Query server does not see the tenant in gRPC metadata (metadata.FromIncomingContext is empty for THANOS-TENANT). Only the HTTP→StoreAPI path propagates the tenant.
What you expected to happen:
The tenant from the incoming HTTP header should be placed into gRPC metadata for all downstream Query gRPC calls so Query-compatible services can enforce tenancy and Mimir/Cortex-compatible headers are preserved.
How to reproduce it (as minimally and precisely as possible):
- Run Thanos Query in distributed mode with --query.tenant-header=X-Scope-OrgID.
- Send any /api/v1/query request with header X-Scope-OrgId: .
- On the downstream Query server (or a Query-compatible service), inspect metadata.FromIncomingContext(ctx) inside the gRPC handler: THANOS-TENANT key is missing.
Full logs to relevant components:
Not included; behavior is silent (metadata missing).
Anything else we need to know:
Tenant is stored on the HTTP context (tenancy.TenantKey) but never appended to outgoing Query gRPC metadata.
StoreAPI fanout already appends THANOS-TENANT via metadata.AppendToOutgoingContext; parity is missing for Query gRPC fanout.
Thanos, Prometheus and Golang version used:
Thanos: custom build from main (after patching tenant propagation); was reproduced on upstream main before patch.
Prometheus: N/A (issue in Thanos query path)
Go: go1.22 (dev env)
Object Storage Provider:
N/A (issue in query fanout, not storage)
What happened:
In distributed query mode, Thanos Query forwards PromQL to other Query nodes via the gRPC Query/QueryRange API. Even with --query.tenant-header configured (e.g. --query.tenant-header=X-Scope-OrgID), the downstream Query server does not see the tenant in gRPC metadata (metadata.FromIncomingContext is empty for THANOS-TENANT). Only the HTTP→StoreAPI path propagates the tenant.
What you expected to happen:
The tenant from the incoming HTTP header should be placed into gRPC metadata for all downstream Query gRPC calls so Query-compatible services can enforce tenancy and Mimir/Cortex-compatible headers are preserved.
How to reproduce it (as minimally and precisely as possible):
Full logs to relevant components:
Not included; behavior is silent (metadata missing).
Anything else we need to know:
Tenant is stored on the HTTP context (tenancy.TenantKey) but never appended to outgoing Query gRPC metadata.
StoreAPI fanout already appends THANOS-TENANT via metadata.AppendToOutgoingContext; parity is missing for Query gRPC fanout.