[Spark] Compatibility with schema evolution changes in Spark master

[johanl-db]

Description

Support for schema evolution in INSERT landed in Spark master: apache/spark#54488

A few changes are required to maintain compatibility in Delta:

• V2Write commands (AppendData, OverwriteByExpression, ..) now have an extra parameter withSchemaEvolution and introduce a few methods to implement for schema evolution for V2Write commands (e.p. writePrivileges)
• Spark doesn't set delta's writer option "mergeSchema" anymore, which was used as workaround until a withSchemaEvolution was introduced on write plan nodes. Instead, Spark now sets withSchemaEvolution directly, and a pre-resolution rule PropagateSchemaEvolutionWriteOption is added in Delta to set the writer option mergeSchema when withSchemaEvolution=true if it's not already explicitly set by the user.

How was this patch tested?

Existing tests

Does this PR introduce any user-facing changes?

No

[longvu-db]

"Spark doesn't set delta's writer option mergeSchema anymore, which was used as workaround until a withSchemaEvolution was introduced on write plan nodes"

=> So this is a fact, and I think this PR is doing something with this fact, could we also mention what that action is (adding a rule to add the mergeSchema option) and why we are taking that action (just compatibility I guess?)

[johanl-db]

"Spark doesn't set delta's writer option mergeSchema anymore, which was used as workaround until a withSchemaEvolution was introduced on write plan nodes"

=> So this is a fact, and I think this PR is doing something with this fact, could we also mention what that action is (adding a rule to add the mergeSchema option) and why we are taking that action (just compatibility I guess?)

I updated the PR description

[longvu-db]

LGTM