feat: extract log store into its own crate, deltalake-logstore

[abhiaagarwal]

Description

Extracts the logstore delta lake component into its own crate, making core take a dependency on it. This allows the aws, gcp, hdfs, etc implementations to rely exclusively on deltalake-logstore instead of depending on core, flipping the dependency graph and decreasing compile times (about a 30% decrease on my system).

Related Issue(s)

Progress towards #3899

Documentation

Still a WIP, but compiles locally without fixing the tests (which mostly require changing changing DeltaResult to LogStoreResult.

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[abhiaagarwal]

Timings of this branch:
timing-log-store.html

Timings of main:
timing-main.html

While the overall timing is the same (due to all the build targets such as tests and python still requiring deltalake_core), you can see that this drops deltalake_core's overall compile time from 20s -> 15s and enables it to be compiled 10s earlier due to increased parallelization. The biggest benefit is the improvement in rust-analyzer re-running on code changes, since the aws/gcp/etc (unfortunately, LakeFS wasn't possible) doesn't require a re-compile anymore.

(all after a cargo clean, running cargo build --all-targets --timings)

[codecov]

Codecov Report

❌ Patch coverage is 52.79720% with 270 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.77%. Comparing base (2e5f7b0) to head (7098406).

Files with missing lines	Patch %	Lines
crates/types/src/actions.rs	30.90%	114 Missing ⚠️
crates/mount/src/lib.rs	0.00%	29 Missing ⚠️
crates/catalog-unity/src/lib.rs	0.00%	26 Missing ⚠️
crates/logstore/src/logstore_impl.rs	69.49%	16 Missing and 2 partials ⚠️
crates/lakefs/src/logstore.rs	37.50%	14 Missing and 1 partial ⚠️
crates/mount/src/error.rs	0.00%	13 Missing ⚠️
crates/aws/src/logstore/default_logstore.rs	0.00%	7 Missing ⚠️
crates/core/src/table/mod.rs	73.91%	5 Missing and 1 partial ⚠️
crates/gcp/src/lib.rs	0.00%	6 Missing ⚠️
crates/lakefs/src/errors.rs	50.00%	5 Missing ⚠️
... and 15 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3913      +/-   ##
==========================================
- Coverage   73.88%   73.77%   -0.12%     
==========================================
  Files         152      151       -1     
  Lines       39426    39440      +14     
  Branches    39426    39440      +14     
==========================================
- Hits        29131    29097      -34     
- Misses       8971     9019      +48     
  Partials     1324     1324              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.