Skip to content

Commit 3665506

Browse files
peter-tothdongjoon-hyun
authored andcommitted
[SPARK-55692][SQL] Fix SupportsRuntimeFiltering and SupportsRuntimeV2Filtering documentation
### What changes were proposed in this pull request? This is a follow-up to #38924 clarify behaviour of scans with runtime filters. ### Why are the changes needed? Please see discussion at #54330 (comment). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This is a documentation change. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #54490 from peter-toth/SPARK-55692-fix-supportsruntimefiltering-docs. Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 7742e73 commit 3665506

File tree

2 files changed

+10
-9
lines changed

2 files changed

+10
-9
lines changed

sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeFiltering.java

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -49,10 +49,11 @@ public interface SupportsRuntimeFiltering extends SupportsRuntimeV2Filtering {
4949
* <p>
5050
* If the scan also implements {@link SupportsReportPartitioning}, it must preserve
5151
* the originally reported partitioning during runtime filtering. While applying runtime filters,
52-
* the scan may detect that some {@link InputPartition}s have no matching data. It can omit
53-
* such partitions entirely only if it does not report a specific partitioning. Otherwise,
54-
* the scan can replace the initially planned {@link InputPartition}s that have no matching
55-
* data with empty {@link InputPartition}s but must preserve the overall number of partitions.
52+
* the scan may detect that some {@link InputPartition}s have no matching data, in which case
53+
* it can either replace the initially planned {@link InputPartition}s that have no matching data
54+
* with empty {@link InputPartition}s, or report only a subset of the original partition values
55+
* (omitting those with no data) via {@link Batch#planInputPartitions()}. The scan must not report
56+
* new partition values that were not present in the original partitioning.
5657
* <p>
5758
* Note that Spark will call {@link Scan#toBatch()} again after filtering the scan at runtime.
5859
*

sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeV2Filtering.java

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,11 @@ public interface SupportsRuntimeV2Filtering extends Scan {
5353
* <p>
5454
* If the scan also implements {@link SupportsReportPartitioning}, it must preserve
5555
* the originally reported partitioning during runtime filtering. While applying runtime
56-
* predicates, the scan may detect that some {@link InputPartition}s have no matching data. It
57-
* can omit such partitions entirely only if it does not report a specific partitioning.
58-
* Otherwise, the scan can replace the initially planned {@link InputPartition}s that have no
59-
* matching data with empty {@link InputPartition}s but must preserve the overall number of
60-
* partitions.
56+
* predicates, the scan may detect that some {@link InputPartition}s have no matching data, in
57+
* which case it can either replace the initially planned {@link InputPartition}s that have no
58+
* matching data with empty {@link InputPartition}s, or report only a subset of the original
59+
* partition values (omitting those with no data) via {@link Batch#planInputPartitions()}. The
60+
* scan must not report new partition values that were not present in the original partitioning.
6161
* <p>
6262
* Note that Spark will call {@link Scan#toBatch()} again after filtering the scan at runtime.
6363
*

0 commit comments

Comments
 (0)