feat: expose DV metadata and payloads as Arrow streams

[ethan-tyler]

Description

Adds DeltaTable.deletion_vectors() -> RecordBatchReader returning one row per data file with a deletion vector.

Schema: filepath: utf8, selection_vector: list[bool] (true = keep, false = deleted).

Reuses the existing DataFusion replay path via replay_deletion_vectors(...). Results are deterministic and sorted by filepath.

Core changes:

• DeletionVectorSelection struct, DeltaScan::deletion_vectors(), shared
  scan_metadata_stream() helper to avoid drift between scan paths
• Replaced internal DV expect(...) with typed error propagation

Python binding:

• cloned_table_and_state() to avoid TOCTOU on table + snapshot
• Chunked Arrow batch output with non-nullable list items
• Preserves without_files guard behavior

Related Issue(s)

• Closes Expose deletion vectors information to Python  #4159

Documentation

cc @ion-elgreco

[ion-elgreco]

@ethan-tyler thanks for picking this up!

I think we can simplify the pr quite abit. For Python we only want to have a recordbatchreadwr with two columns: filepath, selection_vector<list[str]>

We only need a log replay as is done in here and then convert the dv_hashmap into a recordbatchreader:

https://github.com/delta-io/delta-rs/blob/main/crates%2Fcore%2Fsrc%2Fdelta_datafusion%2Ftable_provider%2Fnext%2Fscan%2Fmod.rs#L81-L82

[ethan-tyler]

@ion-elgreco - Agreed on simplifying the surface and it was actually my first pass. After implementing and validating, I saw real limitations with a filepath + selection_vector[list[bool]] boundary.

If this goes through DataFusion scan internals, that's an architectural mismatch for Polars and any other non-DF consumers.

• cross engine API stability couples to DF internals/refactors
• semantics defined by one engine's execution path, not a neutral Delta boundary
• Inherent maintenance cost increases for non-DF consumers

IMO for cross engine consumers- a protocol/kernel level DV boundary (or binding owned transform) is a better feature.

Since we're already in the internals, I'd rather preserve this core and simplify UX on top:

• I've done the complex parts with snapshot validation, descriptor parsing, bounded IO/concurrency, DV decoding, error handling.
• We add a filepath + dv_roaring_bytes is a thin wrapper on current output.
• filepath + selection_vector[list[bool]] can layer on top by decoding/expanding, keeping the compact backend as the efficient path.

Work is reusable either way. Happy to split into smaller PRs if scope is the concern. Lmk your thoughts and happy to discuss further.

[ion-elgreco]

If this goes through DataFusion scan internals, that's an architectural mismatch for Polars and any other non-DF consumers.

I dont really see how this is an issue. Consumers on the Python side receive a recordbatchreader including Polars. The dv hashmap is already materialized so we can actually even return an arrow table. How it's executed in rust should have no influence how Python consumers take in the deletion vectors

IMO for cross engine consumers- a protocol/kernel level DV boundary (or binding owned transform) is a better feature.

Since we're already in the internals, I'd rather preserve this core and simplify UX on top:

• I've done the complex parts with snapshot validation, descriptor parsing, bounded IO/concurrency, DV decoding, error handling.
• We add a filepath + dv_roaring_bytes is a thin wrapper on current output.
• filepath + selection_vector[list[bool]] can layer on top by decoding/expanding, keeping the compact backend as the efficient path.

The Python binding api surface should be minimal, this custom reader should never exist solely for the Python binding. That's why I'm suggesting to go through the replay_files.

[ethan-tyler]

ok that's fair, I still think there’s value in this approach in a different context but I’m aligned. I’ll shelf this and push the simplified version​​​​​​​​​​​​​​​​. Thanks for the discussion, appreciate you engaging on this

[ion-elgreco]

Why is the same thing done here?

[ethan-tyler]

lol right, missed this in my refactoring. Thanks for the call out, will fix it

[ion-elgreco]

Let's use large utf8 or utf8view to be on safer side

[codecov]

Codecov Report

❌ Patch coverage is 74.04580% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.62%. Comparing base (25b4968) to head (409f93f).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
python/src/lib.rs	61.46%	42 Missing ⚠️
...re/src/delta_datafusion/table_provider/next/mod.rs	77.21%	2 Missing and 16 partials ⚠️
...c/delta_datafusion/table_provider/next/scan/mod.rs	79.31%	3 Missing and 3 partials ⚠️
...elta_datafusion/table_provider/next/scan/replay.rs	95.55%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4168      +/-   ##
==========================================
- Coverage   76.63%   76.62%   -0.01%     
==========================================
  Files         166      166              
  Lines       46796    47030     +234     
  Branches    46796    47030     +234     
==========================================
+ Hits        35861    36036     +175     
- Misses       9211     9252      +41     
- Partials     1724     1742      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ethan-tyler]

@ion-elgreco is codecov a blocker on this one? Happy to add more tests if needed, just lmk.

[ion-elgreco]

@ion-elgreco is codecov a blocker on this one? Happy to add more tests if needed, just lmk.

Nope :)