DeltaTable.merge(): does a type mismatch between source and target raise an error?
#4369
Unanswered
loic-bellinger
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Context
The documentation states:
https://delta-io.github.io/delta-rs/usage/writing/
Observed behavior with
DeltaTable.merge()When merging data whose column type differs from the target table schema (e.g.
Int64source into aStringtarget column), no error is raised. The data is silently cast to match the target schema (seems similiar to that issue).Minimal repro :
Note:
error_on_type_mismatch=Trueis the default in.merge()but does not prevent this silent coercion.Additional finding
An error is raised only when the cast is impossible (e.g.
"abc"into anInt64column):The exception type is
Exceptionwrapping a cast error — notValueError.Question
Is this intentional? Does "schema differs" refer only to structural differences (column added/removed), not type changes on existing columns?
If type coercion is by design, it would be helpful to:
error_on_type_mismatchmore precisely - the name suggests it should prevent this, but it doesn't for castable typesSilent type coercion can corrupt data without any signal to the user.
Beta Was this translation helpful? Give feedback.
All reactions