Skip to content

Commit 38e51eb

Browse files
yyanyycloud-fan
authored andcommitted
[SPARK-41398][SQL][FOLLOWUP] Update runtime filtering javadoc to reflect relaxed partition constraints
Update the javadoc for `SupportsRuntimeV2Filtering.filter()` and `SupportsRuntimeFiltering.filter()` to reflect the changes made in [SPARK-41398](https://issues.apache.org/jira/browse/SPARK-41398), which relaxed the constraint on partition values during runtime filtering. After that change, scans can now either: - Replace partitions with no matching data with empty InputPartitions, or - Report only a subset of the original partition values (omitting those with no data) The previous documentation stated that the "overall number of partitions" must be preserved, which is no longer required. The only constraint is that new partition values not present in the original partitioning cannot be introduced. ### What changes were proposed in this pull request? A javadoc update to follow up on #38924 ### Why are the changes needed? To make the java doc up to date for future implementer ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? compile and checkstyle ### Was this patch authored or co-authored using generative AI tooling? Yes - Claude Opus 4.5 Closes #54046 from yyanyy/spark-41398-update-javadoc. Authored-by: Yan Yan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent 3d1188a commit 38e51eb

File tree

2 files changed

+8
-5
lines changed

2 files changed

+8
-5
lines changed

sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeFiltering.java

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,10 @@ public interface SupportsRuntimeFiltering extends SupportsRuntimeV2Filtering {
5151
* the originally reported partitioning during runtime filtering. While applying runtime filters,
5252
* the scan may detect that some {@link InputPartition}s have no matching data. It can omit
5353
* such partitions entirely only if it does not report a specific partitioning. Otherwise,
54-
* the scan can replace the initially planned {@link InputPartition}s that have no matching
55-
* data with empty {@link InputPartition}s but must preserve the overall number of partitions.
54+
* the scan can either replace the initially planned {@link InputPartition}s that have no
55+
* matching data with empty {@link InputPartition}s, or report only a subset of the original
56+
* partition values (omitting those with no data). The scan must not report new partition values
57+
* that were not present in the original partitioning.
5658
* <p>
5759
* Note that Spark will call {@link Scan#toBatch()} again after filtering the scan at runtime.
5860
*

sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeV2Filtering.java

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,10 @@ public interface SupportsRuntimeV2Filtering extends Scan {
5555
* the originally reported partitioning during runtime filtering. While applying runtime
5656
* predicates, the scan may detect that some {@link InputPartition}s have no matching data. It
5757
* can omit such partitions entirely only if it does not report a specific partitioning.
58-
* Otherwise, the scan can replace the initially planned {@link InputPartition}s that have no
59-
* matching data with empty {@link InputPartition}s but must preserve the overall number of
60-
* partitions.
58+
* Otherwise, the scan can either replace the initially planned {@link InputPartition}s that
59+
* have no matching data with empty {@link InputPartition}s, or report only a subset of the
60+
* original partition values (omitting those with no data). The scan must not report new
61+
* partition values that were not present in the original partitioning.
6162
* <p>
6263
* Note that Spark will call {@link Scan#toBatch()} again after filtering the scan at runtime.
6364
*

0 commit comments

Comments
 (0)