Buildable release builds

[findepi]

Problem description

From security standpoint it would be great to have reproducible (byte-for-byte) release builds, however this issue is not about this (but is a prerequisite thereof).
DataFusion downstream consumers consume DataFusion source code, via crates or forking.

Im both these situations it would be great to guarantee that a given release actually builds, not only at the moment of the release.

Observed

A fork of 43.0.0 release no longer passes tests today: https://github.com/findepi/datafusion/actions/runs/13130900468
A fork of 44.0.0 release no longer passes clippy today: https://github.com/findepi/datafusion/actions/runs/13132627770

Expected

A release compiles and works correctly (including passing test). Ideally it also passes clippy.
This is beneficial to anyone consuming a DataFusion code. People running a permanent fork likely mostly figured how to solve this already, but that would be very beneficial to anyone running a particular DataFusion version and wanting to fix a bug. Being able to check out code for the version they use and reproduce the bug there is really good first step, but this requires that the code works.

[findepi]

Proposed solution: commit Cargo.lock to the repository.
Lack of Cargo.lock in the repo gives some benefit (checking with latest versions of unpinned deps automatically). This can be provided with a CI job. I.e. make this a machine problem, not a human (consumer) problem.

[mbrobbel]

Proposed solution: commit Cargo.lock to the repository. Lack of Cargo.lock in the repo gives some benefit (checking with latest versions of unpinned deps automatically). This can be provided with a CI job. I.e. make this a machine problem, not a human (consumer) problem.

#14135

[alamb]

As @mbrobbel says, I agree it is time to put Cargo.lock int he repo:

• Discuss: Check in Cargo.lock file? #14135

[findepi]

Great. This probably would solve the biggest issue with re-buildable releases.
Some other stuff of similar kind is using pinned vs latest stable rust version.

[alamb]

Some other stuff of similar kind is using pinned vs latest stable rust version.

I think this can be set using a rust-toolchain.toml file, for example https://github.com/influxdata/influxdb/blob/main/rust-toolchain.toml

[toolchain]
channel = "1.84.0"
components = ["rustfmt", "clippy", "rust-analyzer"]

[findepi]

Let's do this. we upgrade compiler version manually anyway, so we can update one more place i believe.

[alamb]

Let's do this. we upgrade compiler version manually anyway, so we can update one more place i believe.

It would help us in InfluxDB too as it we have a patched version of DataFusion we keep around too.

I'll try and make a PR for it over the next few days if no one beats me to it

[alamb]

Here is a proposed PR:

• Specify rust toolchain explicitly, document how to change it #14655