[native_datafusion] [Spark SQL Tests] Plan structure differences cause test failures

[andygrove]

Summary

4 Spark SQL tests fail because native_datafusion produces different plan nodes and partitioning info than expected.

Failing Tests

• ParquetV2Suite: "Fallback Parquet V2 to V1" — expects FileSourceScanExec or CometScanExec in plan, but native_datafusion uses CometNativeScan
• BroadcastJoinSuite: "broadcast join where streamed side's output partitioning is HashPartitioning" (x2) — UnknownPartitioning(8) instead of PartitioningCollection
• FileStreamSinkSuite: "self-union, DSv1, read via DataStreamReader API" / "self-union, DSv1, read via table API" — streaming query expects specific plan structure

Root Cause

native_datafusion uses CometNativeScan instead of CometScanExec/FileSourceScanExec and reports UnknownPartitioning instead of preserving the original partitioning information. Tests that inspect plan internals fail.

Related

Discovered in CI for #3307 (enable native_datafusion in auto scan mode).

[andygrove]

Reopening. Under Spark 4.0.1 with native_datafusion in auto scan mode, this test still fails:

• SubquerySuite: "SPARK-43402: FileSourceScanExec supports push down data filter with scalar subquery" — 0 did not equal 1 at SubquerySuite.scala:2699. The test's collect { case f: FileSourceScanLike => f; case c: CometScanExec => c } doesn't match CometNativeScanExec.