Query: exemplar proxy strips external label matchers in multi-tier query topologies

[jon-rei]

Thanos, Prometheus and Golang version used:

v0.40.1

What happened:

In a multi-tier Query topology (Query A → Query B → Sidecars), exemplar queries return results from all downstream sidecars regardless of label filters. Regular queries are not affected.

Query A receives {cluster="a", namespace="foo"}, strips cluster="a" before forwarding (because external labels: https://github.com/thanos-io/thanos/blob/main/pkg/exemplars/proxy.go#L97), and Query B receives {namespace="foo"}, it can no longer route to the correct sidecars and fans out to all of them.

This was introduced by #4123 (fix for #4116). The stripping is correct for Query → Sidecar (Prometheus cannot handle external labels), but breaks Query → Query → Sidecar because the intermediate Query node needs the matchers for its own store routing.

We are using the two-tiered queriers to have a separate query node which is doing mTLS to the sidecars running in different clusters.

What you expected to happen:

Only exemplars from sidecars matching cluster="a" are returned.

How to reproduce it (as minimally and precisely as possible):

1. Set up two Query tiers: Query A → Query B → Sidecars with different external labels (e.g. cluster="a", cluster="b")
2. Query exemplars through Query A: /api/v1/query_exemplars?query=my_metric{cluster="a"}
3. Observe exemplars from both cluster a and b in the response