compact: implement native histogram custom buckets downsampling

[bbondarenko]

Is your proposal related to a problem?

Compactor crashes with the following error:
panic: cannot reduce resolution to custom buckets schema

Describe the solution you'd like

Compactor support downsampling of native histograms custom buckets.

Describe alternatives you've considered

Disabling native histograms in prometheus.

Additional context

Prometheus: 3.9.1
Thanos : v0.41.0

Debug log:

ts=2026-02-26T18:13:18.445641724Z caller=streamed_block_writer.go:178 level=info msg="finalized downsampled block" mint=1770854400000 maxt=1772064000000 ulid=01KJDJBP3BKRF9RXC2Y7GAEQV1 resolution=300000
panic: cannot reduce resolution to custom buckets schema

goroutine 3325 [running]:
github.com/prometheus/prometheus/model/histogram.(*FloatHistogram).CopyToSchema(0x19c96d0eaff?, 0x0?)
	/go/pkg/mod/github.com/prometheus/prometheus@v0.308.0/model/histogram/float_histogram.go:154 +0x2e6
github.com/thanos-io/thanos/pkg/compact/downsample.(*histogramAggregator).add(0xc00146c390, {0x42d34f0?, 0xc00146c390?, 0xc0018dc3c0?})
	/app/pkg/compact/downsample/downsample.go:363 +0x45
github.com/thanos-io/thanos/pkg/compact/downsample.downsampleBatch({0xc000256f08?, 0x6066d20?, 0x60068?}, 0x493e0, {0x42d34f0, 0xc00146c390}, 0xc000fb6e70)
	/app/pkg/compact/downsample/downsample.go:843 +0x7f
github.com/thanos-io/thanos/pkg/compact/downsample.downsampleHistogramBatch({0xc000256f08?, 0x3386e00?, 0x164}, 0x493e0)
	/app/pkg/compact/downsample/downsample.go:455 +0x154
github.com/thanos-io/thanos/pkg/compact/downsample.downsampleRawLoop({0xc0032cc000?, 0xc000fb6ff8?, 0x1ec7033?}, 0x493e0, 0xc00316d200?, 0xc000fb6fd8, 0x3f09f30)
	/app/pkg/compact/downsample/downsample.go:692 +0x1aa
github.com/thanos-io/thanos/pkg/compact/downsample.DownsampleRaw({0xc0032cc000?, 0x164, 0x11aaa}, 0x493e0)
	/app/pkg/compact/downsample/downsample.go:648 +0x165
github.com/thanos-io/thanos/pkg/compact/downsample.Downsample({0x42de890, 0xc0009d8000}, {0x42a55a0, 0xc000930140}, 0xc0012894a0, {0x42eb020, 0xc001457a20}, {0x7ffe17a25284, 0x9}, 0x493e0)
	/app/pkg/compact/downsample/downsample.go:190 +0xaff
main.processDownsampling({0x42de890, 0xc0009d8000}, {0x42a55a0, 0xc000930140}, {0x4309568, 0xc00094b8f0}, 0xc0012894a0, {0x7ffe17a25284, 0x9}, 0x493e0, ...)
	/app/cmd/thanos/downsample.go:385 +0x845
main.downsampleBucket.func3()
	/app/cmd/thanos/downsample.go:265 +0x255
sync.(*WaitGroup).Go.func1()
	/usr/local/go/src/sync/waitgroup.go:239 +0x4a
created by sync.(*WaitGroup).Go in goroutine 83
	/usr/local/go/src/sync/waitgroup.go:237 +0x73

(Write your answer here.)

[RepoRonin]

Hi I have understood the issue, and decided to solve this.

Problem Analysis

The panic occurs in the histogramAggregator.add() method at line 363 when calling fh.CopyToSchema(h.schema). According to the stack trace, this happens because Prometheus's CopyToSchema method panics with "cannot reduce resolution to custom buckets schema" when trying to downsample native histograms with custom buckets.

The issue is:

• Prometheus 3.9.1 introduced native histograms with custom buckets (NHCB)
• Thanos v0.41.0 doesn't support downsampling these custom bucket histograms
• When the compactor tries to downsample a histogram with custom buckets, it calls CopyToSchema() which panics

Solution

The solution is to detect custom bucket histograms and skip downsampling them (or handle them appropriately). Custom bucket histograms have a negative schema value (specifically schema < 0 indicates custom buckets).

Here's how we can solve this:

• Check for custom buckets before processing — Custom bucket histograms have schema < 0
• Skip or handle them appropriately — Either skip downsampling or handle them separately
• Add proper error handling instead of letting it panic

The fix should be applied in multiple places:

• In histogramAggregator.add() — detect and handle custom buckets
• In minSchema() — ensure we don't try to use custom bucket schemas
• In DownsampleRaw() — filter out custom bucket histograms before processing

If my analysis is correct, give me the sign and I'll raise a PR

[RepoRonin]

I have raised a PR for this : #8699
Do check it.

[Br1an67]

I'd like to work on this. The fix adds a schema check in the downsample logic to skip resolution reduction for custom buckets (schema < 0) and use original data directly, preventing the runtime panic. PR: #8706