Commit bc5a4d8
committed
fix(lexer): normalize CRLF in template cooked values
A template body's line terminator sequences are normalized in both the TRV
and the TV, so a raw CRLF or a lone CR in the source becomes a single LF in
the cooked value. push_template was cooking with normalization switched off,
on the reasoning that a consumer could redo it from the raw text, so on a
CRLF checkout every multi-line template cooked to bytes the grammar says it
should not have.
Normalizing after the fact is not actually possible. Once escapes are
decoded a CR in the cooked bytes is ambiguous: it may have come from a raw
CRLF, which has to collapse, or from a `\r` escape, which has to survive.
A tagged template of `\r\n` must cook to CR LF, and any blanket pass over
the cooked bytes would flatten it to LF. The decision has to be made on the
raw text before escapes are decoded, which is inside cook, so this flips the
existing const-generic on rather than adding anything.
Only push_template changes. Identifier atoms cannot contain a raw CR, and a
raw CR in a string literal is already an error, so both keep the unnormalized
path. Tests go through push_template rather than cook_decode, since the
decoder was always correct and it was the wiring that was wrong; the escaped
`\r\n` case is kept as a permanent guard against re-deriving the naive fix.
No measurable cost. The extra needle is one compare folded into a scan that
is load-bound, and an interleaved A/B over the corpus moves nothing.1 parent fc822ee commit bc5a4d8
1 file changed
Lines changed: 27 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
214 | 214 | | |
215 | 215 | | |
216 | 216 | | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
| 217 | + | |
222 | 218 | | |
223 | 219 | | |
224 | 220 | | |
| |||
1176 | 1172 | | |
1177 | 1173 | | |
1178 | 1174 | | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
1179 | 1201 | | |
1180 | 1202 | | |
1181 | 1203 | | |
| |||
0 commit comments