[BUG] Datafusion adds `protoc` as build requirement

[andrross]

Describe the bug

PR #21057 has added a rust compilation step to building this project. I believe it assumes that protoc is available in the build environment. We have been playing whac-a-mole try to fix all the various build steps. Is there a better way to solve this? For example, the Java build that needs protoc uses a gradle plugin that "just works" out of the box with no additional requirements for the build environment.

Here are the currently failing steps I'm aware of:

• Nightly precommit: Nightly precommit failed #21121
• Dependabot workflow: https://github.com/opensearch-project/OpenSearch/actions/runs/24045804950/job/70144178299?pr=21139

I think a fix for both of these is to add a protoc install step in the GitHub Action setup. But is there a better way? If we fix these I won't really have confidence that there aren't more latent failures.

Related component

Build

To Reproduce

See failures

Expected behavior

The build should be simple without too many external requirements of the build environment

Additional Details

No response

[andrross]

@Bukhtawar @bharath-techie Can you take a look at this?

[andrross]

After a cursory investigation here are two options:

Use https://docs.rs/protobuf-src/latest/protobuf_src/. The substrait dependency supports this: substrait = { version = "=0.62.0", features = ["protoc"] }. However, that just trades one problem for another as the protobuf compilation requires CMAKE...

Use https://docs.rs/crate/protoc-bin-vendored/latest to pull a pre-built binary into the build

[andrross]

Here's another idea my AI agent suggested to use the existing binaries from Maven. I don't think this as written will work for windows though:

andrross@f1f28bd

[cwperks]

@andrross IMO the way to go would be #21125. See the discussion on there, there's a proposal to split off a separate CI check for sandbox.

[mch2]

+1 to splitting sandbox off to run in parallel - rust build also adds significant time, we can also set -DrustDebug=true to speed it up for pr builds.

[bharath-techie]

Yeah agree, with this, protoc and other dependencies will be confined within sandbox / plugins for now.