feat(arrrow/compute/expr): support substrait timestamp and decimal properly#418
Merged
zeroshade merged 5 commits intoapache:mainfrom Jun 30, 2025
Merged
Conversation
zeroshade
force-pushed
the
substrait-upgrade-temporal
branch
2 times, most recently
from
96b9321 to
b52370a
Compare
lidavidm
reviewed
Jun 21, 2025
| return arrow.FixedWidthTypes.Date64 | ||
| case sawDate32: | ||
| return arrow.FixedWidthTypes.Date32 | ||
| sawTimestampOrDate := zone != nil || sawDate32 || sawDate64 || sawDuration |
Member
There was a problem hiding this comment.
nit: this also includes duration
Member
Author
There was a problem hiding this comment.
Was following the existing CommonTemporal in arrow C++. Should we also change that function too?
Member
There was a problem hiding this comment.
Ah, probably, though I guess it's small enough to not really warrant its own PR...
Comment on lines
+372
to
+376
| if sawTime && sawTimestampOrDate { | ||
| // no common type possible | ||
| return nil | ||
| } | ||
|
|
Member
There was a problem hiding this comment.
This blocks (time AND any other type) but I would assume (duration AND any other type) is also out of the question?
Member
There was a problem hiding this comment.
Maybe all these checks could be combined into the three cases (saw timestamp or date and not time or duration) and (saw time and not anything else) and (saw duration and not anything else)?
lidavidm
approved these changes
Jun 23, 2025
zeroshade
force-pushed
the
substrait-upgrade-temporal
branch
from
6c03dfd to
2609888
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Fixes #404
Fixes #417
What changes are included in this PR?
Upgrades substrait-go to v4 and adds handling and support for PrecisionTime and PrecisionTimestamp, fixes substrait Decimal128Type handling.
Are these changes tested?
Yes, unit test is added.
Are there any user-facing changes?
only the new features being usable.
Relies on substrait-io/substrait-go#139 getting merged before this can get merged