Skip to content

fix(optimizer): preserve UNNEST struct-field columns - #7979

Open
fivetran-kwoodbeck wants to merge 4 commits into
mainfrom
optimizer/fix-numbering-unnest-canonicalize-internal-names
Open

fix(optimizer): preserve UNNEST struct-field columns#7979
fivetran-kwoodbeck wants to merge 4 commits into
mainfrom
optimizer/fix-numbering-unnest-canonicalize-internal-names

Conversation

@fivetran-kwoodbeck

@fivetran-kwoodbeck fivetran-kwoodbeck commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

canonicalize_internal_names was producing invalid SQL for struct field columns of an UNNEST. An UNNEST of structs exposes struct field names as physical columns, but the rule was treating them as internal handles.

Sample Input

SELECT t.id FROM t
JOIN (SELECT sid FROM UNNEST([STRUCT('x' AS sid), STRUCT('y')])) AS q
  ON q.sid = t.id;

Previous Output, invalid SQL: references non existent _c0

SELECT `_t1`.`id` AS `id` FROM `_t1` AS `_t1`
JOIN (SELECT `_c0` AS `_c0` FROM UNNEST([STRUCT('x' AS `sid`), STRUCT('y')])) AS `_t2` ON `_t2`.`_c0` = `_t1`.`id`;

Fixed Output: field name sid preserved

SELECT `_t1`.`id` AS `id` FROM `_t1` AS `_t1` 
JOIN (SELECT `sid` AS `_c0` FROM UNNEST([STRUCT('x' AS `sid`), STRUCT('y')])) AS `_t2` ON `_t2`.`_c0` = `_t1`.`id`;

Fix

In canonicalize_internal_names, added a preserve_names flag that treats an UNNEST + structs field column as physical. This is the same way is_base_source columns are already preserved. Scalar UNNEST element aliases are renameable definitions, so they still canonicalize like previously.

Behavior

source column result
UNNEST of struct (literal) struct field preserved
UNNEST of ARRAY column struct field preserved
scalar UNNEST / WITH OFFSET element alias _cN
correlated scalar UNNEST element alias _cN

@geooo109 geooo109 self-assigned this Jul 28, 2026
Comment thread sqlglot/optimizer/canonicalize_internal_names.py Outdated
@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

SQLGlot Integration Test Results

✅ All tests passed

Comparing:

  • this branch (sqlglot:optimizer/fix-numbering-unnest-canonicalize-internal-names @ sqlglot 6dfaa53)
  • baseline (main @ sqlglot 323f5a3)

Overall

main: 192414 total, 153545 passed (pass rate: 79.8%)

sqlglot:optimizer/fix-numbering-unnest-canonicalize-internal-names: 180220 total, 142393 passed (pass rate: 79.0%)

Transitions:
No change

Dialect pair changes: 0 previous results not found, 3 current results not found

✅ All tests passed

@fivetran-kwoodbeck
fivetran-kwoodbeck force-pushed the optimizer/fix-numbering-unnest-canonicalize-internal-names branch from e932791 to 8a47ac3 Compare July 28, 2026 15:34
and (element_type := seq_get(src.expressions[0].type.expressions, 0))
):
preserve_names = preserve_names or element_type.is_type(exp.DataType.Type.STRUCT)

@geooo109 geooo109 Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Input (bq):
SELECT s.sid FROM UNNEST([STRUCT('a' AS sid)]) AS s;
SELECT z.sid FROM UNNEST([STRUCT('a' AS sid)]) AS z

main (output for both queries):
SELECT `_c0`.`sid` AS `sid` FROM UNNEST([STRUCT('a' AS `sid`)]) AS `_c0`

PR  (output):
SELECT `s`.`sid` AS `sid` FROM UNNEST([STRUCT('a' AS `sid`)]) AS `s`
SELECT `z`.`sid` AS `sid` FROM UNNEST([STRUCT('a' AS `sid`)]) AS `z`

The granularity of the change should be column-level rather than source-level: only the struct field names need preserving, but the source-level flag also preserves the UNNEST's element alias (s/z above), which leaks them into the canonical form. Since both inputs are the same query, they should keep producing the same canonical output, as on main. cc: @georgesittas

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, and we should also add a test that does something like:

SELECT s FROM UNNEST([STRUCT('a' AS sid)]) AS s;

Because the source name now acts as both a source and a top-level column (i.e., part of the contract).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants