fix(lexer): normalize CRLF in template cooked values - #24912
Open
ShanonJackson wants to merge 1 commit into
Open
fix(lexer): normalize CRLF in template cooked values#24912ShanonJackson wants to merge 1 commit into
ShanonJackson wants to merge 1 commit into
Conversation
A template body's line terminator sequences are normalized in both the TRV and the TV, so a raw CRLF or a lone CR in the source becomes a single LF in the cooked value. push_template was cooking with normalization switched off, on the reasoning that a consumer could redo it from the raw text, so on a CRLF checkout every multi-line template cooked to bytes the grammar says it should not have. Normalizing after the fact is not actually possible. Once escapes are decoded a CR in the cooked bytes is ambiguous: it may have come from a raw CRLF, which has to collapse, or from a `\r` escape, which has to survive. A tagged template of `\r\n` must cook to CR LF, and any blanket pass over the cooked bytes would flatten it to LF. The decision has to be made on the raw text before escapes are decoded, which is inside cook, so this flips the existing const-generic on rather than adding anything. Only push_template changes. Identifier atoms cannot contain a raw CR, and a raw CR in a string literal is already an error, so both keep the unnormalized path. Tests go through push_template rather than cook_decode, since the decoder was always correct and it was the wiring that was wrong; the escaped `\r\n` case is kept as a permanent guard against re-deriving the naive fix. No measurable cost. The extra needle is one compare folded into a scan that is load-bound, and an interleaved A/B over the corpus moves nothing.
ShanonJackson
force-pushed
the
fix/imp-lexer-hex-escape
branch
from
July 25, 2026 23:29
2920b7c to
fc822ee
Compare
ShanonJackson
force-pushed
the
perf/imp-lexer-template-cr
branch
from
July 25, 2026 23:30
655e71e to
bc5a4d8
Compare
ShanonJackson
marked this pull request as ready for review
July 28, 2026 08:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A template body's line terminator sequences are normalized in both the TRV and the TV, so a raw CRLF or a lone CR in the source becomes a single LF in the cooked value. push_template was cooking with normalization switched off, on the reasoning that a consumer could redo it from the raw text, so on a CRLF checkout every multi-line template cooked to bytes the grammar says it should not have. oxc_parser does the normalization itself in template_literal_carriage_return, so the two disagreed.
Redoing it afterwards is not possible, which is the part worth spelling out. Once escapes are decoded a CR in the cooked bytes is ambiguous: it may have come from a raw CRLF, which has to collapse, or from a \r escape, which has to survive, and a template of \r\n must cook to CR LF while any blanket pass over the cooked bytes would flatten it to LF. The decision has to be made on the raw text before escapes are decoded, which is inside cook, so this flips the existing const-generic on rather than adding machinery. Only push_template changes, since identifier atoms cannot contain a raw CR and a raw CR in a string literal is already an error. The new tests go through push_template rather than cook_decode, because the decoder was always correct and already had a passing test while the wiring was what was wrong. No measurable cost, and across both corpora exactly one file's output changes, nanoplotx_components.tsx, by one byte, the single CRLF it has inside a template collapsing to LF.