Skip to content

fix(lexer): normalize CRLF in template cooked values - #24912

Open
ShanonJackson wants to merge 1 commit into
fix/imp-lexer-hex-escapefrom
perf/imp-lexer-template-cr
Open

fix(lexer): normalize CRLF in template cooked values#24912
ShanonJackson wants to merge 1 commit into
fix/imp-lexer-hex-escapefrom
perf/imp-lexer-template-cr

Conversation

@ShanonJackson

@ShanonJackson ShanonJackson commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

A template body's line terminator sequences are normalized in both the TRV and the TV, so a raw CRLF or a lone CR in the source becomes a single LF in the cooked value. push_template was cooking with normalization switched off, on the reasoning that a consumer could redo it from the raw text, so on a CRLF checkout every multi-line template cooked to bytes the grammar says it should not have. oxc_parser does the normalization itself in template_literal_carriage_return, so the two disagreed.

Redoing it afterwards is not possible, which is the part worth spelling out. Once escapes are decoded a CR in the cooked bytes is ambiguous: it may have come from a raw CRLF, which has to collapse, or from a \r escape, which has to survive, and a template of \r\n must cook to CR LF while any blanket pass over the cooked bytes would flatten it to LF. The decision has to be made on the raw text before escapes are decoded, which is inside cook, so this flips the existing const-generic on rather than adding machinery. Only push_template changes, since identifier atoms cannot contain a raw CR and a raw CR in a string literal is already an error. The new tests go through push_template rather than cook_decode, because the decoder was always correct and already had a passing test while the wiring was what was wrong. No measurable cost, and across both corpora exactly one file's output changes, nanoplotx_components.tsx, by one byte, the single CRLF it has inside a template collapsing to LF.

A template body's line terminator sequences are normalized in both the TRV
and the TV, so a raw CRLF or a lone CR in the source becomes a single LF in
the cooked value. push_template was cooking with normalization switched off,
on the reasoning that a consumer could redo it from the raw text, so on a
CRLF checkout every multi-line template cooked to bytes the grammar says it
should not have.

Normalizing after the fact is not actually possible. Once escapes are
decoded a CR in the cooked bytes is ambiguous: it may have come from a raw
CRLF, which has to collapse, or from a `\r` escape, which has to survive.
A tagged template of `\r\n` must cook to CR LF, and any blanket pass over
the cooked bytes would flatten it to LF. The decision has to be made on the
raw text before escapes are decoded, which is inside cook, so this flips the
existing const-generic on rather than adding anything.

Only push_template changes. Identifier atoms cannot contain a raw CR, and a
raw CR in a string literal is already an error, so both keep the unnormalized
path. Tests go through push_template rather than cook_decode, since the
decoder was always correct and it was the wiring that was wrong; the escaped
`\r\n` case is kept as a permanent guard against re-deriving the naive fix.

No measurable cost. The extra needle is one compare folded into a scan that
is load-bound, and an interleaved A/B over the corpus moves nothing.
@ShanonJackson
ShanonJackson force-pushed the fix/imp-lexer-hex-escape branch from 2920b7c to fc822ee Compare July 25, 2026 23:29
@ShanonJackson
ShanonJackson force-pushed the perf/imp-lexer-template-cr branch from 655e71e to bc5a4d8 Compare July 25, 2026 23:30
@ShanonJackson
ShanonJackson marked this pull request as ready for review July 28, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants