Skip to content

[SPARK-56054][SQL] Add test for MERGE INTO schema evolution with aliased assignments#55239

Open
johanl-db wants to merge 2 commits intoapache:masterfrom
johanl-db:SPARK-56054-merge-into-schema-evolution-alias
Open

[SPARK-56054][SQL] Add test for MERGE INTO schema evolution with aliased assignments#55239
johanl-db wants to merge 2 commits intoapache:masterfrom
johanl-db:SPARK-56054-merge-into-schema-evolution-alias

Conversation

@johanl-db
Copy link
Copy Markdown
Contributor

@johanl-db johanl-db commented Apr 7, 2026

What changes were proposed in this pull request?

Follow up from #54891 in particular this comment.

Adds more tests covering schema evolution in MERGE INTO when the assignment contains an alias.
This change also undoes the fix from #54891. On closer inspection, this isn't needed in Spark as Spark already removes trivial aliases on nested field accessors in MERGE in resolveExprInAssignment, which is what the change aimed to allow. See this test added here covering trivial aliases implictily added on nested field access.

Delta doesn't strip aliases during resolution in that way and will require special handling, but that's up to Delta to implement in its custom resolution logic to replicate what Spark already does

How was this patch tested?

Adds tests covering MERGE INTO schema evolution with aliases in assignments

Copy link
Copy Markdown
Member

@szehon-ho szehon-ho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are all DataFrame API, wondering is there any equivalent in SQL? Be good to abstract it out if possible

If not, maybe we can we make a new file MergeIntoSchemaEvolutionExtraDataFrameTests to make them only run in DataFrame mode?

It is a bit of a mess now due to the inheritance patterns

@johanl-db
Copy link
Copy Markdown
Contributor Author

I moved the tests to a dedicated trait for dataframe tests.

There's no SQL equivalent, it's not possible to specify an alias in an assignment expression using SQL afaict: SET col = source.col AS other_col

@johanl-db johanl-db requested a review from szehon-ho April 9, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants