fix: Iceberg reflection for current() on TableOperations hierarchy

[karuppayya]

Which issue does this PR close?

Closes #3894.

Rationale for this change

Fix NoSuchMethodException from Iceberg Reflection

What changes are included in this PR?

Use findMethodInHierarchy instead of getDeclaredMethod so getTableMetadata and format-version reflection work when current() is declared on a superclass (e.g. BaseMetastoreTableOperations for Hive/Glue-style ops).

How are these changes tested?

• Added Units
• Tested by querying Hive backed Iceberg tables where this issue is present.

[karuppayya]

cc: @andygrove @mbutrovich @parthchandra
can any of you help review?
Also looks like workflow needs maintainer approval to progress(But remember that it used to work without this)

[parthchandra]

Is current the only method affected? operations (a few lines above this) is calling getDeclaredMethod also.
Also, are there other catalog implementations that could have similar issues but for other methods?

[karuppayya]

I think we could always call replace all occurences of getDeclaredMethod with findMethodInHierarchy
Especially, those that assume the superclass's depth seems fragile

      val tasksMethod = scan.getClass.getSuperclass
        .getDeclaredMethod("tasks")

      val filterExpressionsMethod = scan.getClass.getSuperclass.getSuperclass
        .getDeclaredMethod("filterExpressions")

[parthchandra]

@mbutrovich wdyt? Is there a potential performance impact here if we replace getDeclaredMethod with findMethodInHierarchy?

[mbutrovich]

I think I tried that at some point but it's been at least 6 months, open to trying again.

[parthchandra]

Thinking this over, I feel we can defer changing everything over to use findMethodInHierarchy until we hit an issue.

[karuppayya]

Sgtm

[parthchandra]

lgtm, pending ci

[parthchandra]

We've exposed a latent bug, perhaps. In CometScanRule.validateIcebergFileScanTasks, uri.getScheme returns null for the relative paths returned by InMemoryCatalog. We check

        if (scheme != null && !supportedSchemes.contains(scheme)) {
          unsupportedSchemes += scheme
        }

which allows such paths and this later fails.
@mbutrovich, any thoughts on how to handle this?

[karuppayya]

@mbutrovich ping!

[mbutrovich]

Thanks for improving the Iceberg support, @karuppayya! I took a look at this today:

The findMethodInHierarchy fix is correct, but it widens the set of scans Comet intercepts. Previously, getDeclaredMethod("current") failed for InMemoryCatalog's operations class (where current() is inherited), causing fallback to Spark. Now the reflection succeeds and Comet runs the scan natively.

The Iceberg CI failures (TestSparkReaderDeletes) happen because Comet's native reader (iceberg-rust/arrow-rs) doesn't expose the number of row deletes applied metric that these tests assert on. That metric is an Iceberg Java reader concept.

The root cause is CometScanRule.validateIcebergFileScanTasks line 809:

if (scheme != null && !supportedSchemes.contains(scheme))

InMemoryCatalog returns relative paths where URI.getScheme is null. The null check lets them through. Fix: treat null scheme as unsupported so these scans fall back.

if (scheme == null || !supportedSchemes.contains(scheme))

Same fix needed for the delete file scheme check on line 851.

[mbutrovich]

I’ll keep kicking CI :)

[karuppayya]

Fix: treat null scheme as unsupported so these scans fall back.

Rejecting scheme == null also blocks HadoopCatalog local paths (e.g. /tmp/warehouse/...) which have no URI scheme — this caused all CometIcebergNativeSuite tests to fail again.

Instead, added an explicit FileIO type check. (REST catalogs are unaffected(mentioned in the code comment) since the client-side table uses ResolvingFileIO, not InMemoryFileIO (see RESTSessionCatalog.DEFAULT_FILE_IO_IMPL))

Also added countDeletes() = false override in the Iceberg diffs for 3.5 (present already for 3.5) since iceberg-rust doesn't expose the delete count metric.

[mbutrovich]

I have minor style nits but don't want to send you through CI again. Thanks @karuppayya!