feat: DH-21493: pushdown support for sorted regions#7955
Open
lbooker42 wants to merge 49 commits intodeephaven:mainfrom
Open
feat: DH-21493: pushdown support for sorted regions#7955lbooker42 wants to merge 49 commits intodeephaven:mainfrom
lbooker42 wants to merge 49 commits intodeephaven:mainfrom
Conversation
…delegate to the ColumnRegion for predicate pushdown.
…nderneath Unioned table to activate Column Region optimizations.
…21493-sorted-pushdown
…t. Documentation.
…ation' into nightly/DH-21522-parquettablelocation # Conflicts: # engine/table/src/main/java/io/deephaven/engine/table/impl/PushdownResult.java
…lumnSourceManager to store table definition instead of List<ColumnDefinition>
…ation' into nightly/DH-21493-sorted-pushdown # Conflicts: # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedPushdownFilterLocationContext.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionByte.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionChar.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionDouble.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionFloat.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionInt.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionLong.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionObject.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionShort.java
# Conflicts: # engine/table/src/main/java/io/deephaven/engine/table/impl/BasePushdownFilterContext.java # engine/table/src/main/java/io/deephaven/engine/table/impl/BasePushdownFilterContextImpl.java # engine/table/src/main/java/io/deephaven/engine/table/impl/PushdownFilterMatcher.java # engine/table/src/main/java/io/deephaven/engine/table/impl/dataindex/DataIndexPushdownManager.java # engine/table/src/main/java/io/deephaven/engine/table/impl/locations/impl/AbstractTableLocation.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/UnionSourceManager.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegion.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegionByte.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegionChar.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegionDouble.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegionFloat.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegionInt.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegionLong.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegionObject.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/ColumnRegionShort.java # engine/table/src/main/java/io/deephaven/engine/table/impl/sources/regioned/RegionedPushdownAction.java # engine/table/src/test/java/io/deephaven/engine/table/impl/TestPartitionAwareSourceTableNoMocks.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetTableLocation.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionByte.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionChar.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionDouble.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionFloat.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionInt.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionLong.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionObject.java # extensions/parquet/table/src/main/java/io/deephaven/parquet/table/region/ParquetColumnRegionShort.java # extensions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableFilterTest.java # extensions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java
lbooker42
added
NoDocumentationNeeded
ReleaseNotesNeeded
Contributor
No docs changes detected for d8abd52 |
lbooker42
commented
Apr 27, 2026
|
|
||
| executedFilterCost = 0; | ||
| // Extract the effective filter and use it for populating the context. | ||
| final WhereFilter effectiveFilter = WhereFilterDelegating.maybeUnwrapFilter(filter); |
Contributor
Author
There was a problem hiding this comment.
This change is pretty significant, allowing wrapped/delegating filters to still be processed efficiently. Unfortunately, this means RowCapturingFilters row tracking is also often getting bypassed so a lot of tests had to be updated to show this.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 39 out of 39 changed files in this pull request and generated 20 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Author
|
An earlier PR (#7666) merged a change to the Page Stores that forces pushdown to the sub-regions of the store. This is good in many scenarios (e.g. dictionary) but not ideal for this scenario. It results in each sorted sub-region being searched even though they might have been skipped if we looked at the page store overall. Do we need a hint (maybe in the RegionedPushDownAction.Region) about whether we should apply the pushdown to sub-regions or to the top-level page-store. This might be the only case and it might not matter too much. We might someday have sorted subregions as part of a unsorted page-store so might this is over-optimizing? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds support for match and range filters to push down into binary search over sorted regions.