[Feature Request] Coordinator-level shard pruning using timestamp range metadata

[aallawala]

Problem

When a search request targets multiple indices (via wildcard patterns, aliases, or data streams), the coordinating node fans out a can_match probe to every shard across all matching indices, even when the query contains a narrow @timestamp range filter that clearly excludes most of them.

For time-series workloads with high index counts (e.g., daily indices with months of history, or data streams with many backing indices), this means:

• Unnecessary network round-trips to shards that cannot possibly match the query's time range
• Increased coordinator overhead assembling and waiting on responses from shards that will all return "no match"
• Higher tail latency on time-bounded queries, especially when older indices are on slower storage tiers (e.g., warm/UltraWarm nodes backed by S3)

The can_match phase mitigates this at the shard level using BKD tree min/max stats, but the coordinating node still has to send the probe and wait for the response from every shard. For a query spanning 1 hour across 365 days of daily indices, that's ~364 unnecessary network round-trips.

Proposed Solution

Store per-index @timestamp min/max bounds in IndexMetadata within cluster state. Since cluster state is already replicated to every node, the coordinating node can check these bounds locally — a pure memory read — and rewrite range queries on @timestamp into MatchNone for indices whose bounds don't overlap the query range. These shards are skipped entirely without any network probe.

Implementation outline

1. Track @timestamp range in IndexMetadata — maintain a min/max field range (e.g., an IndexLongFieldRange structure) that is updated as documents are indexed. Propagate via cluster state updates.

2. Coordinator rewrite phase in CanMatchPreFilterSearchPhase — before fanning out can_match probes, iterate over candidate shards and check the owning index's timestamp bounds against the query's range filter. Mark non-overlapping shards as pre-filtered and skip them.

3. Query rewrite integration — for shards that can be pruned, rewrite the query into MatchNoneQueryBuilder so existing search infrastructure handles them cleanly.

Scope considerations

• Indices without populated timestamp bounds (e.g., non-time-series indices) would fall through to the existing can_match path — no behavioral change.
• The event.ingested field could also benefit from the same treatment.
• This optimization is complementary to existing shard-level can_match and the Context Aware Segments RFC ([RFC] Context Aware Segments #18576).

Impact

This would significantly reduce search latency for the most common time-series query pattern: a narrow time range against a large index history. The benefit scales with index count — the more backing indices or daily indices that exist, the more shards are pruned without network cost.

[msfroh]

Track @timestamp range in IndexMetadata — maintain a min/max field range (e.g., an IndexLongFieldRange structure) that is updated as documents are indexed. Propagate via cluster state updates.

Hmm... we could generalize this to any field of the index and potentially extend it to other filter types (enum, bloom filter, etc). Essentially, it's similar to the skip indices that Parquet stores per row group (and ClickHouse stores per index block). Of course, for a more complicated dictionary, we'd probably need to move it out of IndexMetadata -- maybe IndexMetadata could hold a URL for the filter. Regardless, that can be a future problem. As long as we support min/max filters for any field and the concept of "filter type" (which could just be min/max for now), I think it's a good enough start.

Could we add a restriction that the index needs to be read-only? That would make it easier to handle invalidation of the skip indices. Otherwise we would need to invalidate the skip index whenever the index receives an add/update request. IMO, computing the skip index makes sense as part of ISM rollover.

[peternied]

@aallawala This seems like a great idea to improve search efficiency, love that you included a PR to help folks take a look at the problem space. Since this change is about improving performance - it will get a lot more attention if we can demonstrate the amount of improvement. Could you see about creating a test and giving us before/after numbers that we could inspect?

[RS146BIJAY]

@aallawala Thanks for creating this proposal. This idea was something which was discussed during initial Context Aware Segments design discussions as well that if it can be something which can added as an extension over context aware segments.

we could generalize this to any field of the index

+1 to @msfroh suggestions that we need to extend this behaviour to any feild and not just timestamp. While I do aggree timestamp is the most used field in log analytics query, we should not restrict the solution to a single feild.

Have some queries myself as well on the proposal. How are deciding that for which fields we will store the min/max bounds in the cluster state? Will this be for all fields or will we decide this on the basis of some query context (like based on fields customer is querying data most on)?

[jainankitk]

Thanks for the write-up @aallawala — the problem statement resonates. Fanning out can_match to hundreds of shards that obviously can't match a narrow time range is wasteful, and using cluster state (which every node already has) to short-circuit that is a natural fit.

A few things I'd love to understand better:

The big one — actively-written indices. For read-only/rolled-over indices, storing a static timestamp range in IndexMetadata is clean and safe. But for the current write index, the max timestamp is constantly moving. Keeping it current means frequent cluster state updates through the
cluster manager, which doesn't scale with high-ingest workloads. Batching (e.g., on refresh) introduces staleness, and a stale max bound means you can never safely prune the hot index — you'd risk skipping shards that actually have matching docs.

In practice this means the optimization can only reliably apply to indices where the range is fully known (no active writes). For the most common query ("last N minutes"), the hot index can never be pruned. The proposal should call this limitation out explicitly and scope the expected
impact accordingly.

Is the read-only-only win enough? Those indices already respond to can_match quickly (stable BKD stats, no contention). The value depends on scale — hundreds of cold indices where even cheap round-trips add up. Some benchmarks or estimates would help make the case.

Smaller items:

• Field shouldn't be hardcoded to @timestamp — should hook into the existing DataStream.TimestampField abstraction
• Consider pre-populating CanMatchSearchPhaseResults with canMatch=false for pruned shards rather than rewriting queries to MatchNone — simpler approach
• What's the rollout plan? Feature flag? Cluster setting?

Directionally this makes sense, just want to make sure the scope is realistic given the actively-written index constraint.

[Bukhtawar]

Nice to see this come up, I remember having discussed this in the past with @rishabhmaurya and @msfroh, also during CAS discussions with @RS146BIJAY

[aallawala]

Thanks everyone for the thoughtful feedback — really appreciate the engagement.

To clarify the intended scope: this optimization is exclusively targeted at read-only indices. Indices that are still actively receiving writes should not have timestamp bounds populated in IndexMetadata, and will fall through to the existing can_match implementation unchanged. The write index for a data stream is intentionally left out of this path — keeping a live max bound current would require frequent cluster state updates through the cluster manager, which doesn't scale with high-ingest workloads.

The expected benefit is precisely at the scale where this matters most: a narrow time-range query against a data stream or alias with many rolled-over backing indices. In a typical log workload with 90 days of daily indices, a "last 1 hour" query currently fans out can_match to ~89 indices that can be determined non-matching with a local memory read. That's the win being targeted.

On @msfroh and @RS146BIJAY's suggestion to generalize beyond @timestamp — fully agreed that the design should be field-agnostic from the start, with @timestamp as the first concrete use case.

---

I should also note that I'm not a maintainer of this project. I'd welcome guidance from the maintainers on whether this is the right direction, what the preferred implementation path looks like, and whether there's an existing initiative this should be folded into (e.g., the CAS work @Bukhtawar mentioned).

[msfroh]

I should also note that I'm not a maintainer of this project. I'd welcome guidance from the maintainers on whether this is the right direction, what the preferred implementation path looks like, and whether there's an existing initiative this should be folded into (e.g., the CAS work @Bukhtawar mentioned).

The very good news is that everyone who has commented on this issue is a maintainer and by the sounds of it, we love the idea! 🎉

My only concern with what you've proposed is that cluster state might not be the best place to store a big, complicated skip table for an index. The question of whether we want to store big, complicated skip tables is (IMO) still an open question, though. We can almost certainly get a lot of benefit from min/max values for one or two fields per index. Beyond that, we're probably looking at diminishing returns.

For an initial implementation, I think cluster state (or specifically index metadata within cluster state) is the right place to store this. I would just ask that it's implemented in a way that adds enough abstraction that in future we could potentially put (say) an object store pointer in the index metadata instead, without breaking backwards compatibility.