fix(scraping): a nested <noscript> discards the entire page body - #2114
Open
Aitosoft wants to merge 1 commit into
Open
fix(scraping): a nested <noscript> discards the entire page body#2114Aitosoft wants to merge 1 commit into
Aitosoft wants to merge 1 commit into
Conversation
`<noscript>` may not nest. With scripting enabled its content is raw text, so
the parser never sees an inner `<noscript>` as an element and the OUTER element
is left unclosed — libxml2 then swallows every following node into it. Since
`LXMLWebScrapingStrategy._scrap` hands the raw serialised DOM straight to
`lhtml.document_fromstring`, the damage is done before there is a tree to fix.
A single WordPress lazy-load plugin that wraps the Google Tag Manager block in
a second `<noscript>` is enough to lose a whole site:
312,628 B rendered HTML -> 97 B cleaned_html -> 1 B markdown
with `status_code: 200` and `success: True`, so nothing downstream can tell.
`cleaned_html` is literally `<html><head><title>…</title></head></html>` — the
`<body>` is gone. I found this across 70 hosts in one corpus; it reproduced
byte-identically on the same URLs 3.5 months apart, because the plugin markup is
stable.
Six-line reproduction on unmodified v0.9.2:
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy
MIN = '''<html><head><title>T</title></head><body>
<noscript><iframe src="about:blank"></iframe><noscript><iframe src="about:blank"></iframe></noscript>
<h1>Contact</h1><p>Phone 010 123 4567</p>
<div><p>A second paragraph.</p></div>
</body></html>'''
LXMLWebScrapingStrategy().scrap('https://x/', MIN, word_count_threshold=1).cleaned_html
# nested: 42 B -> '<html><head><title>T</title></head></html>'
# single <noscript>: 187 B (well-formed noscript is harmless)
# no <noscript>: 188 B
Fix: `strip_noscript()`, applied at the top of `_scrap()` immediately before
parsing. Removal rather than unwrapping, because Crawl4AI renders with
JavaScript enabled — by definition `<noscript>` content is not what the page
displayed, and its scripted equivalent is already in the DOM. Two passes:
well-formed elements first (a non-greedy match ends at the first `</noscript>`,
which is where the browser's parser ends the outer element too), then any
unpaired tag left by truncated markup, so an unclosed `<noscript>` cannot
swallow a body either. Pages without the substring return the same object
unparsed, so the cost is one `in` check.
`grep noscript crawl4ai/content_scraping_strategy.py crawl4ai/utils.py` returns
nothing on develop — there is no existing `<noscript>` handling to conflict with.
Tests: tests/general/test_noscript_body_loss.py, 11 passed. Covers nested,
well-formed, unclosed, uppercase and attributed shapes, asserts nested and
well-formed become indistinguishable, and pins that a page whose only content
is inside `<noscript>` correctly stays empty.
pytest tests/general/test_noscript_body_loss.py -q # 11 passed
pytest tests/test_pr_1435_redirected_status_code.py -q # 7 passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
<noscript>may not nest. With scripting enabled its content is raw text, so aparser never sees an inner
<noscript>as an element and the outer elementis left unclosed — libxml2 then swallows every following node into it.
LXMLWebScrapingStrategy._scraphands the raw serialised DOM straight tolhtml.document_fromstring, so the damage is done before there is a tree thatcould be repaired downstream.
A single WordPress lazy-load plugin that wraps the Google Tag Manager block in a
second
<noscript>is enough to lose an entire page:with
status_code: 200andsuccess: True, so nothing downstream can tell thatanything went wrong.
cleaned_htmlis literally<html><head><title>…</title></head></html>— the whole<body>is gone. Themarkdown generator is innocent; it faithfully renders nothing.
I hit this across 70 hosts in one corpus, and it reproduced byte-identically
on the same URLs 3.5 months apart, because the plugin markup is stable.
Reproduction (6 lines, unmodified
develop)Confirmed by excision on the real captured DOM:
cleaned_html<noscript>tags<script>onlyFix
strip_noscript(), applied at the top of_scrap()immediately before parsing.Removal rather than unwrapping, because Crawl4AI renders with JavaScript
enabled: by definition
<noscript>content is not what the page displayed,and its scripted equivalent is already in the DOM. Dropping it loses nothing and
removes the whole malformed-nesting failure class.
Two passes, so this handles the tag rather than the specific nesting:
</noscript>,which is exactly where the browser's own parser ends the outer element too,
so the nested shape collapses correctly.
<noscript>cannot swallow a body either.Pages without the substring return the same object unparsed, so the cost on the
overwhelming majority of pages is one
incheck.grep noscript crawl4ai/content_scraping_strategy.py crawl4ai/utils.pyreturnsnothing on
develop— there is no existing<noscript>handling to conflictwith, which is also why this went unnoticed.
Tests
tests/general/test_noscript_body_loss.py— 11 tests. Covers nested,well-formed, unclosed, uppercase and attributed shapes; asserts that nested and
well-formed become indistinguishable (the point of the fix, as opposed to
special-casing the nested shape); and pins that a page whose only content is
inside
<noscript>correctly stays empty, since JavaScript was enabled.Known limitation
A
<noscript>literal inside a<script>string could in principle start amatch that ends at a later real
</noscript>. Any pre-parse repair carriesthis, and the alternative — a spec-compliant HTML5 parse of every document — is
a much larger change than this bug warrants. Not observed in the corpus above.
Happy to adjust the approach if you would rather see this handled elsewhere in
the pipeline.