feat: DH-21522: allow column region optimizations in Predicate Pushdown filtering by lbooker42 · Pull Request #7666 · deephaven/deephaven-core

lbooker42 · 2026-02-10T18:02:02Z

This PR refactors predicate pushdown filtering for regioned column sources to support both table-location and per-column-region optimizations, enabling more granular (region-level) pushdown planning and execution.

Changes:

Introduces a new RegionedPushdownAction model (Location vs Region actions) and new regioned pushdown filter context types.
Updates regioned pushdown execution to run per-region and merge results, enabling column-region pushdown participation.
Refactors ParquetTableLocation pushdown logic from an internal enum to the new action-based API and updates the engine interfaces accordingly.

Code Coverage Summary:

ImmutableConstant[Type]Source
- makePushdownFilterContext / estimatePushdownFilterCost at 100%
- pushdownFilter effectively 100% (empty selection detection not tested, might not be reachable without dedicated test)

…delegate to the ColumnRegion for predicate pushdown.

…timizations.

github-actions · 2026-02-10T18:02:16Z

No docs changes detected for `9fe11c9`

Copilot

Pull request overview

This PR refactors predicate pushdown filtering for regioned column sources to support both table-location and per-column-region optimizations, enabling more granular (region-level) pushdown planning and execution.

Changes:

Introduces a new RegionedPushdownAction model (Location vs Region actions) and new regioned pushdown filter context types.
Updates regioned pushdown execution to run per-region and merge results, enabling column-region pushdown participation.
Refactors ParquetTableLocation pushdown logic from an internal enum to the new action-based API and updates the engine interfaces accordingly.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
extensions/parquet/table/src/test/java/io/deephaven/parquet/table/location/ParquetTableLocationTest.java	Removes the prior unit test that validated pushdown mode cost ordering.
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java	Implements location-level supported actions and action contexts for parquet pushdown planning/execution.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedPushdownHelper.java	Adds shared utilities for region-thread context and combining per-region pushdown results.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedPushdownFilterMatcher.java	Introduces the regioned action-based pushdown matcher API.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedPushdownFilterContext.java	Adds a regioned pushdown context carrying column definitions + rename mappings.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedPushdownFilterLocationContext.java	Extends the regioned context with access to the current table location.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedPushdownAction.java	Defines the new pushdown action abstraction (Location/Region) and related context interfaces.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedColumnSourceManager.java	Refactors manager-level pushdown scheduling/merging and exposes internals needed for region pushdown.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedColumnSourceBase.java	Refactors regioned column source pushdown to operate directly on regions with location-aware contexts.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/GenericColumnRegionBase.java	Adds default region-level pushdown orchestration combining region + location actions by cost.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegion.java	Makes column regions pushdown-capable via `RegionedPushdownFilterMatcher`.
engine/table/src/main/java/io/deephaven/engine/table/impl/sources/UnionSourceManager.java	Updates to use the new `BasePushdownFilterContext#filter()` accessor.
engine/table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java	Adds default action-based pushdown planning/execution for table locations.
engine/table/src/main/java/io/deephaven/engine/table/impl/locations/TableLocation.java	Switches `TableLocation` to the new `RegionedPushdownFilterMatcher` API.
engine/table/src/main/java/io/deephaven/engine/table/impl/PushdownResult.java	Adds a new cost constant for region-level single-value optimizations.
engine/table/src/main/java/io/deephaven/engine/table/impl/PushdownFilterMatcher.java	Provides default implementations for pushdown matcher methods.
engine/table/src/main/java/io/deephaven/engine/table/impl/BasePushdownFilterContext.java	Makes the base context abstract and introduces `filter()` accessor (encapsulation change).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…nderneath Unioned table to activate Column Region optimizations.

Copilot

Pull request overview

Copilot reviewed 64 out of 64 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rcaudy

Shipping comments and partial review.

cpwright

placeholder

rcaudy

Finished full pass.

rcaudy · 2026-04-22T20:30:35Z


+    private static final RegionedPushdownAction.Location ParquetRowGroupMetadata =
+            new RegionedPushdownAction.Location(
+                    () -> QueryTable.DISABLE_WHERE_PUSHDOWN_PARQUET_ROW_GROUP_METADATA,


Feels weird to centralize something that mentions Parquet on QueryTable.

…t. Documentation.

…ation' into nightly/DH-21522-parquettablelocation # Conflicts: # engine/table/src/main/java/io/deephaven/engine/table/impl/PushdownResult.java

rcaudy

.

…lumnSourceManager to store table definition instead of List<ColumnDefinition>

Copilot

Pull request overview

Copilot reviewed 94 out of 94 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

# Conflicts: # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/UnionSourceManager.java

rcaudy · 2026-04-25T00:03:57Z

+                final long regionFirstIncludedKey = maybeIt.peekNextKey();
+                final REGION_TYPE region = pageStore.lookupRegion(regionFirstIncludedKey);
+                final RowSequence regionRows = maybeIt.getNextRowSequenceThrough(region.maxRow(regionFirstIncludedKey));
+                final long regionFirstKey = regionFirstIncludedKey & pageStore.mask();


I'm pretty sure this is wrong. I think you should use io.deephaven.engine.page.Page#firstRow, as in final long regionFirstKey = region.firstRow(regionFirstIncludedKey);. I think you're getting the last possible row key, not the first.

lbooker42 added 2 commits February 5, 2026 14:48

Changed RegionedColumnSourceBase to handle single column filters and …

27978f0

…delegate to the ColumnRegion for predicate pushdown.

Lots more work, allowing column region and table locatiop pushdown op…

d44e00d

…timizations.

lbooker42 marked this pull request as draft February 10, 2026 18:02

lbooker42 changed the title ~~Nightly/dh 21522 parquettablelocation~~ DH-21522: allow column region optimizations in Predicate Pushdown filtering Feb 10, 2026

lbooker42 requested a review from Copilot February 10, 2026 18:03

lbooker42 self-assigned this Feb 10, 2026

Copilot started reviewing on behalf of lbooker42 February 10, 2026 18:04 View session

lbooker42 added NoDocumentationNeeded ReleaseNotesNeeded Release notes are needed labels Feb 10, 2026

Copilot AI reviewed Feb 10, 2026

View reviewed changes

lbooker42 added 9 commits February 10, 2026 10:39

Addressed copilot PR comments.

73adb07

Improved and consolidated RowKeyAgnosticChunkSource filtering.

3b67e34

Cleanup and spotless

163f4c7

Support for ColumnRegion Constant and Null regions and tests.

557eae2

Merge branch 'main' into nightly/DH-21522-parquettablelocation

43e895d

Updated the new test to use QueryTable disables rather than burying u…

8dc65e6

…nderneath Unioned table to activate Column Region optimizations.

Update test to reflect new reality.

0450746

Coordinated changes to allow Core+ optimizations.

e5e9e1f

Merge branch 'main' into nightly/DH-21522-parquettablelocation

5afc2ae

lbooker42 changed the title ~~DH-21522: allow column region optimizations in Predicate Pushdown filtering~~ feat: DH-21522: allow column region optimizations in Predicate Pushdown filtering Feb 24, 2026

lbooker42 marked this pull request as ready for review February 24, 2026 23:56

Final co-pilot suggestion.

fda03ee

lbooker42 requested review from Copilot and cpwright February 25, 2026 00:01

Copilot started reviewing on behalf of lbooker42 February 25, 2026 00:02 View session

Copilot AI reviewed Feb 25, 2026

View reviewed changes

lbooker42 added 3 commits February 24, 2026 16:15

Use createNull for new null regions.

4c8f2de

Minor changes from co-pilot.

49dcf57

Consolidation of repeated code.

3464b4a

rcaudy reviewed Apr 2, 2026

View reviewed changes

lbooker42 added 2 commits April 7, 2026 09:58

Addressed PR comments

c0f05c9

Updated pushdown costs, re-ordered and renamed.

bfa3430

cpwright reviewed Apr 15, 2026

View reviewed changes

lbooker42 added 2 commits April 22, 2026 10:25

Added columnLocation retrieval to column region as optional

c968f0c

Using region column location instead of the location-providing context.

ae4537d

cpwright reviewed Apr 22, 2026

View reviewed changes

Comment thread ...main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedPushdownFilterContext.java Outdated

rcaudy reviewed Apr 22, 2026

View reviewed changes

lbooker42 added 4 commits April 22, 2026 13:38

Using region column location instead of the location-providing contex…

2cf70b4

…t. Documentation.

Cleanup and documentation improvements.

425649a

Some PR comments addressed.

7050ce3

Merge remote-tracking branch 'origin/nightly/DH-21522-parquettableloc…

76b2d7e

…ation' into nightly/DH-21522-parquettablelocation # Conflicts: # engine/table/src/main/java/io/deephaven/engine/table/impl/PushdownResult.java

rcaudy reviewed Apr 22, 2026

View reviewed changes

lbooker42 added 5 commits April 22, 2026 18:16

Refactor RegionedPushdownAction, small fixes.

428d962

Add pushdown to sub-regions to static page stores, updated RegionedCo…

400e18f

…lumnSourceManager to store table definition instead of List<ColumnDefinition>

Test correction, update for correctness.

6a4b0d2

Better impl of region pushdown in static page store.

e93d2eb

spotless. grrr...

4600b74

lbooker42 requested a review from Copilot April 24, 2026 16:01

Copilot started reviewing on behalf of lbooker42 April 24, 2026 16:02 View session

Copilot AI reviewed Apr 24, 2026

View reviewed changes

rcaudy reviewed Apr 24, 2026

View reviewed changes

lbooker42 added 3 commits April 24, 2026 15:10

PR Comments.

eb442d6

Refactored ColumnRegion to have default estimationg, pushdown methods.

1533a62

Merge branch 'main' into nightly/DH-21522-parquettablelocation

27d7562

# Conflicts: # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/UnionSourceManager.java

rcaudy reviewed Apr 25, 2026

View reviewed changes

Corrected logic in PageStorePushdownHelper

9fe11c9

rcaudy approved these changes Apr 25, 2026

View reviewed changes

lbooker42 merged commit 90c5d1f into deephaven:main Apr 25, 2026
24 checks passed

github-actions Bot locked and limited conversation to collaborators Apr 25, 2026

Conversation

lbooker42 commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

No docs changes detected for 9fe11c9

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcaudy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cpwright left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rcaudy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcaudy Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

rcaudy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lbooker42 commented Feb 10, 2026 •

edited

Loading

github-actions Bot commented Feb 10, 2026 •

edited

Loading

No docs changes detected for `9fe11c9`