[Spark] Fix null expansion in MERGE with multiple clauses

[zhipengmao-db]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

This PR fixes null expansion issues in multi-clause MERGEs, when fields excluded in one MERGE clause are added to the final evolved schema by another MERGE clause, guarded by DELTA_MERGE_PRESERVE_NULL_SOURCE_STRUCTS.

Problem:
When using multiple UPDATE clauses with clauses in a single MERGE statement, NULL source structs were incorrectly expanded into non-NULL structs with all fields set to NULL.

Example:

Source table schema s: id: int, col struct<x: int, y: int>, extra: <val: int, val2: int>

Target table schema t: id: int, col struct<y: int, z: int>.

Source Table:

{id: 0, col: NULL, extra: NULL},
{id: 1, col: NULL, extra: NULL}

Target Table Before Query:

{id: 0, col: NULL},
{id: 1, col: NULL}

Query

MERGE WITH SCHEMA EVOLUTION INTO t
USING s
ON t.id = s.id
WHEN MATCHED AND id = 0 THEN UPDATE SET col = s.col
WHEN MATCHED AND id = 1 THEN UPDATE SET *

Target Table After Query - Current Behavior

{id: 0, col: NULL, extra: {val: NULL, val2: NULL},
{id: 1, col: NULL, extra: NULL}

Target Table After Query - Target Behavior

{id: 0, col: NULL, extra: NULL},
{id: 1, col: NULL, extra: NULL}

When a MERGE statement contains multiple UPDATE clauses with different clauses, e.g., one clause with UPDATE SET col = s.col and another with UPDATE *, we do not generate an action for extra for clause UPDATE SET col = s.col, while extra will be added to the final evolved schema by another clause UPDATE *.

We call generateUpdateOpsForNewTargetFields in PreprocessTableMerge to handle this case, which generates leaf-level operations for new target fields, causing null structs to be incorrectly expanded into non-null structs with null fields.

Solution:
Skip PreprocessTableMerge.generateUpdateOpsForNewTargetFields when the null expansion fix is enabled, since generateUpdateExpressions already handles evolved schema fields that do not have corresponding actions correctly by assigning them to NULL or default target expression.

How was this patch tested?

MergeIntoStructEvolutionNullnessMultiClauseTests.

Does this PR introduce any user-facing changes?

No. The change is protected by a flag disabled by default.

[tomvanbussel]

Thanks for fixing this long standing bug! LGTM.