Skip to content

Commit bc5a4d8

Browse files
committed
fix(lexer): normalize CRLF in template cooked values
A template body's line terminator sequences are normalized in both the TRV and the TV, so a raw CRLF or a lone CR in the source becomes a single LF in the cooked value. push_template was cooking with normalization switched off, on the reasoning that a consumer could redo it from the raw text, so on a CRLF checkout every multi-line template cooked to bytes the grammar says it should not have. Normalizing after the fact is not actually possible. Once escapes are decoded a CR in the cooked bytes is ambiguous: it may have come from a raw CRLF, which has to collapse, or from a `\r` escape, which has to survive. A tagged template of `\r\n` must cook to CR LF, and any blanket pass over the cooked bytes would flatten it to LF. The decision has to be made on the raw text before escapes are decoded, which is inside cook, so this flips the existing const-generic on rather than adding anything. Only push_template changes. Identifier atoms cannot contain a raw CR, and a raw CR in a string literal is already an error, so both keep the unnormalized path. Tests go through push_template rather than cook_decode, since the decoder was always correct and it was the wiring that was wrong; the escaped `\r\n` case is kept as a permanent guard against re-deriving the naive fix. No measurable cost. The extra needle is one compare folded into a scan that is load-bound, and an interleaved A/B over the corpus moves nothing.
1 parent fc822ee commit bc5a4d8

1 file changed

Lines changed: 27 additions & 5 deletions

File tree

crates/oxc_lexer/src/lanes.rs

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -214,11 +214,7 @@ impl Lanes {
214214

215215
#[inline]
216216
pub fn push_template(&mut self, src: &[u8], bs: usize, be: usize) {
217-
// CRLF normalization is implemented and tested but wired off: the
218-
// extra `\r` needle costs ~2% on template-heavy sources, and the
219-
// parser facade cooks CR-bearing templates itself, so end-to-end
220-
// values are exact either way.
221-
let (sp, nes) = self.cook::<false, false>(src, bs as u32, be as u32);
217+
let (sp, nes) = self.cook::<false, true>(src, bs as u32, be as u32);
222218
// A NotEscapeSequence means cooked = None per the grammar; legality
223219
// depends on tagged-ness (parser context), so the span carries a
224220
// marker instead of a diagnostic.
@@ -1176,6 +1172,32 @@ unsafe fn cook_decode<const EMIT: bool, const CRLF: bool>(
11761172
mod tests {
11771173
use super::*;
11781174

1175+
fn push_tpl(body: &[u8]) -> Vec<u8> {
1176+
let mut padded = body.to_vec();
1177+
padded.extend_from_slice(&[0u8; 64]);
1178+
let mut l = Lanes::default();
1179+
l.push_template(&padded, 0, body.len());
1180+
let sp = l.templates[0];
1181+
let s = (sp.start & StringSpan::START_MASK) as usize;
1182+
let e = (sp.end_and_flags & StringSpan::END_MASK) as usize;
1183+
l.cooked[s..e].to_vec()
1184+
}
1185+
1186+
#[test]
1187+
fn push_template_normalizes_raw_cr() {
1188+
assert_eq!(push_tpl(b"a\r\nb"), b"a\nb");
1189+
assert_eq!(push_tpl(b"a\rb"), b"a\nb");
1190+
assert_eq!(push_tpl(br"a\r\nb\`c"), b"a\r\nb`c");
1191+
assert_eq!(push_tpl(b"a\r\nb\\`c"), b"a\nb`c");
1192+
}
1193+
1194+
#[test]
1195+
fn push_template_keeps_escaped_cr_lf() {
1196+
assert_eq!(push_tpl(br"\r\n"), b"\r\n");
1197+
assert_eq!(push_tpl(br"\r"), b"\r");
1198+
assert_eq!(push_tpl(b"\r\n"), b"\n");
1199+
}
1200+
11791201
fn cook(body: &[u8]) -> (Vec<u8>, bool) {
11801202
let mut padded = body.to_vec();
11811203
padded.extend_from_slice(&[0u8; 32]);

0 commit comments

Comments
 (0)