Implement zero-copy FFI transfer of batches from JVM to Native

[andygrove]

What is the problem the feature request solves?

Comet doesn't currently follow Arrow best practice when transferring batches from JVM to native code. Ownership is retained by the JVM after batches have been made available in FFI, requiring a copy to be made on the native side.

It would be better to pass ownership of batches to native rather than make copies.

This design was necessary originally due to the use of mutable Parquet buffers in the native_comet Parquet scan.

Describe the potential solution

Once #2152 is implemented, and once we have removed the native_comet scan (#2177) and all uses of mutable Parquet buffers, we should be able to implement zero-copy FFI.

Additional context

No response

[jatin510]

i would love to work on this issue @andygrove

[andygrove]

Thanks @jatin510. I think the next steps are:

• CometTestBase should no longer set the scan implementation to native_comet #2172 (in progress)
• Remove native_comet scan implementation #2177
• Remove mutable Parquet buffer use from native_iceberg_compat #2152