Skip to content

Factor out common datafusion types into another proto file#10649

Merged
mustafasrepo merged 19 commits intoapache:mainfrom
synnada-ai:feature/proto/common
May 27, 2024
Merged

Factor out common datafusion types into another proto file#10649
mustafasrepo merged 19 commits intoapache:mainfrom
synnada-ai:feature/proto/common

Conversation

@mustafasrepo
Copy link
Copy Markdown
Contributor

@mustafasrepo mustafasrepo commented May 24, 2024

Which issue does this PR close?

Closes #10477.

Rationale for this change

See issue body for the motivation.

This PR moves protobuf serialialization, deserialization codes for the common types in the Datafusion (e.g types that are under datafusion_common) into another proto crate named datafusion_proto_common (where its only dependency from datafusion is datafusion_common).
Note: This PR, is just reorganization of the existing code.

What changes are included in this PR?

Are these changes tested?

Existing tests should work

Are there any user-facing changes?

Copy link
Copy Markdown
Contributor

@ozankabak ozankabak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Even though this PR is merely and organizational change, it moves a non-trivial amount of code around, so it'd be good to get a careful review from at least one more person. @alamb or @andygrove, do you have time to take a look?

@comphead and @jayzhan211 it'd be good if you took a look too. Let's make sure not to miss anything

Comment thread datafusion/proto-common/Cargo.toml
Comment thread datafusion/proto-common/gen/src/main.rs
Comment thread datafusion/proto-common/src/common.rs Outdated
use datafusion_common::DataFusionError;

pub fn proto_error<S: Into<String>>(message: S) -> DataFusionError {
DataFusionError::Internal(message.into())
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internal_err! ?

Comment thread datafusion/proto-common/src/lib.rs Outdated
Copy link
Copy Markdown
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is 💪
Thanks @mustafasrepo
I left some minor comments which doesn't block the PR

Copy link
Copy Markdown
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM too -- thanks @mustafasrepo

cc @andygrove as we'll have to release the new crate as part of the next release

@mustafasrepo mustafasrepo merged commit 3194a98 into apache:main May 27, 2024
findepi pushed a commit to findepi/datafusion that referenced this pull request Jul 16, 2024
)

* Initial commit

* Minor changes

* Minor changes

* Minor changes

* Minor changes

* Move Column type

* Minor changes buggy

* Compiles

* Add new types to common

* Move scalarvalue

* Move new types to common

* Move additional fields

* Move common types

* Minor changes

* Change proto definition order

* Simplifications

* Minor changes

* REimport protobuf_common from proto

* Address reviews
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Seperate out common types from Datafusion Proto

4 participants