feat: update to DataFusion 51, arrow 57, delta-kernel 0.18.0, pyo3 26, pyo3-arrow 0.14#3949
Merged
rtyler merged 39 commits intodelta-io:mainfrom Dec 2, 2025
Merged
feat: update to DataFusion 51, arrow 57, delta-kernel 0.18.0, pyo3 26, pyo3-arrow 0.14#3949rtyler merged 39 commits intodelta-io:mainfrom
rtyler merged 39 commits intodelta-io:mainfrom
Conversation
Signed-off-by: Stephen Carman <shcarman@gmail.com>
Signed-off-by: Stephen Carman <shcarman@gmail.com>
github-actions
bot
added
binding/python
Signed-off-by: DrakeLin <drakelin18@gmail.com> Signed-off-by: Stephen Carman <shcarman@gmail.com>
# Description
As of today, the Rust meta-crate re-exports the storage crates but never
calls their register_handlers helpers. That means every Rust binary
still has to remember to call deltalake::gcp::register_handlers(None)
(and the equivalents for S3/Azure/etc.) before using cloud URIs, even
though the Python bindings auto-register. This PR brings the meta-crate
to parity so DeltaOps::try_from_uri("gs://…") works out of the box when
the gcs feature is enabled.
# Problem
- Users of the deltalake crate must manually register each storage
backend before working with gs://, s3://, abfss://, etc.
- Forgetting the call leads to
DeltaTableError::InvalidTableLocation("Unknown scheme: gs"), which
blocks workflows like DataFusion writers on GCS.
- Docs/examples didn’t make it obvious when manual registration was
still required.
# Solution
- Add feature-gated ctor hooks in crates/deltalake/src/lib.rs that call
register_handlers(None) for AWS, Azure, GCS, HDFS, LakeFS, and Unity as
soon as their features are enabled.
- Pull in the lightweight ctor = "0.2" dependency so the hooks run at
startup.
- Add a small regression test that exercises
DeltaTableBuilder::from_uri("gs://…") with the gcs feature to guard
against regressions.
- Update the GCS integration docs and changelog to explain that the
meta-crate now auto-registers backends while deltalake-core users still
need to call the storage crates explicitly.
# Changes
- crates/deltalake/src/lib.rs: new #[ctor::ctor] modules for s3, azure,
gcs, hdfs, lakefs, and unity.
- crates/deltalake/Cargo.toml: add ctor dependency.
- crates/deltalake/tests/gcs_auto_registration.rs: new smoke tests for
gs:// URI recognition when the gcs feature is enabled.
- docs/integrations/object-storage/gcs.md & CHANGELOG.md: document the
auto-registration behavior.
# Testing
- cargo check -p deltalake --all-features
- cargo test -p deltalake --features gcs
- cargo test --test gcs_auto_registration --features gcs
- cargo build --example pharma_pipeline_gcs --features gcs,datafusion
# Documentation
- docs/integrations/object-storage/gcs.md
- CHANGELOG.md
---------
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Stephen Carman <shcarman@gmail.com>
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: Stephen Carman <shcarman@gmail.com>
…n and Rust Profiles for different use cases: - dev: Fast local development with good debug experience (Cargo defaults) - test: test builds with minimal debug info to save disk/RAM (custom) - release: Production Rust crates - maximum performance (Cargo defaults) - bench: For benchmarking with flamegraphs (cargo bench) - profiling: For performance profiling with release opts + debug info - ci: CI/CD optimized - fast builds, release-like performance - python-release: Python wheel builds - portable, reproducible (PyPI releases) Signed-off-by: Florian Valeye <florian.valeye@gmail.com> Signed-off-by: Stephen Carman <shcarman@gmail.com>
Signed-off-by: Florian Valeye <florian.valeye@gmail.com> Signed-off-by: Stephen Carman <shcarman@gmail.com>
…om_uri Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: Stephen Carman <shcarman@gmail.com>
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: Stephen Carman <shcarman@gmail.com>
--- updated-dependencies: - dependency-name: ctor dependency-version: 0.6.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Stephen Carman <shcarman@gmail.com>
This work was really cool when @houqp explored it. In 2025 we're not really reliant on dynamodb locking and at some point in the distant future maybe we'll not need a DynamoDBLogStore either. Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: Stephen Carman <shcarman@gmail.com>
--- updated-dependencies: - dependency-name: convert_case dependency-version: 0.9.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Stephen Carman <shcarman@gmail.com>
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: Stephen Carman <shcarman@gmail.com>
- Fix panic when get_add_actions() is called on tables with no add actions - Return empty RecordBatch with correct schema instead of panicking - Add unit test to verify get_add_actions() works after delete and vacuum Fixes delta-io#3918 Signed-off-by: Manish Sogiyawar <msogiyawar@vectra.ai> Signed-off-by: Stephen Carman <shcarman@gmail.com>
Signed-off-by: Manish Sogiyawar <msogiyawar@vectra.ai> Signed-off-by: Stephen Carman <shcarman@gmail.com>
See delta-io#3918 Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: Stephen Carman <shcarman@gmail.com>
… overwrite (delta-io#3912) # Description Remove references in docs that suggest using `partition_filters` for selectively overwriting partitions, which has been removed from the `write_deltalake` API. # Related Issue(s) fixes delta-io#3904 Signed-off-by: zyd14 <romerzs14@gmail.com> Signed-off-by: Stephen Carman <shcarman@gmail.com>
Fixes delta-io#3886 Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: Stephen Carman <shcarman@gmail.com>
Signed-off-by: Stephen Carman <shcarman@gmail.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3949 +/- ##
===========================================
+ Coverage 26.24% 74.34% +48.10%
===========================================
Files 124 152 +28
Lines 19824 39608 +19784
Branches 19824 39608 +19784
===========================================
+ Hits 5203 29448 +24245
+ Misses 14251 8823 -5428
- Partials 370 1337 +967 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Stephen Carman <shcarman@gmail.com>
# Conflicts: # crates/core/src/kernel/models/actions.rs
Signed-off-by: Stephen Carman <shcarman@gmail.com>
Signed-off-by: Stephen Carman <shcarman@gmail.com>
hntd187
changed the title
Signed-off-by: R. Tyler Croy <rtyler@brokenco.de>
rtyler
approved these changes
Nov 29, 2025
ethan-tyler
added a commit
to ethan-tyler/delta-rs
that referenced
this pull request
Jan 9, 2026
…, pyo3-arrow 0.14 (delta-io#3949) # Description A continuation from delta-io#3933 --------- Signed-off-by: Stephen Carman <shcarman@gmail.com> Signed-off-by: DrakeLin <drakelin18@gmail.com> Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com> Signed-off-by: R. Tyler Croy <rtyler@brokenco.de> Signed-off-by: Florian Valeye <florian.valeye@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Manish Sogiyawar <msogiyawar@vectra.ai> Signed-off-by: zyd14 <romerzs14@gmail.com> Co-authored-by: xiaolong <xiaolong@lakesail.com> Co-authored-by: DrakeLin <drakelin18@gmail.com> Co-authored-by: Ethan Urbanski <ethan@urbanskitech.com> Co-authored-by: R. Tyler Croy <rtyler@brokenco.de> Co-authored-by: Florian Valeye <florian.valeye@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Manish Sogiyawar <msogiyawar@vectra.ai> Co-authored-by: zyd14 <romerzs14@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A continuation from #3933