Summary
3 Spark SQL tests fail due to DPP-related issues when native_datafusion scan is used in auto mode.
Failing Tests
DynamicPartitionPruningSuite: "static scan metrics" — scanOption.isDefined was falseExplainSuite: "explain formatted - check presence of subquery in case of DPP" — expected 2 subqueries, found 0- Multiple
TASK_WRITE_FAILED errors in DPP-related stages
Root Cause
nativeDataFusionScan already has a check for dynamic pruning filters (isDynamicPruningFilter), but these tests use DPP scenarios (e.g. static pruning, subqueries) that may bypass that check. The fallback logic may need to be broadened, or the plan structure produced by native_datafusion may not be compatible with DPP expectations.
Related
Discovered in CI for #3307 (enable native_datafusion in auto scan mode).
Summary
3 Spark SQL tests fail due to DPP-related issues when
native_datafusionscan is used in auto mode.Failing Tests
DynamicPartitionPruningSuite: "static scan metrics" —scanOption.isDefined was falseExplainSuite: "explain formatted - check presence of subquery in case of DPP" — expected 2 subqueries, found 0TASK_WRITE_FAILEDerrors in DPP-related stagesRoot Cause
nativeDataFusionScanalready has a check for dynamic pruning filters (isDynamicPruningFilter), but these tests use DPP scenarios (e.g. static pruning, subqueries) that may bypass that check. The fallback logic may need to be broadened, or the plan structure produced bynative_datafusionmay not be compatible with DPP expectations.Related
Discovered in CI for #3307 (enable native_datafusion in auto scan mode).