fix(website): fold JS escapes out of the llms corpus - #1358
Conversation
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Read the whole diff. The fold itself is right, and the browser-order argument holds up: I enumerated every backslash escape that occurs inside a code-block across all 44 docs pages and the set is exactly \``, $, \and one}, none of which is a shape where JS cooking and a naive (.)` fold disagree. The accounted before-and-after does what it claims, and extending the walk mirror rather than weakening its assertion is the right instinct, since it turns the corpus walk itself into a second counterfactual.
Two things I want handled before this goes in, both about what the change leaves behind rather than what it does.
The first is that the two docs-page content fixes ship with nothing that would notice if they regressed. That matters more here than it normally would, because the whole argument for this PR is that the extractor now copies an authoring error into the corpus verbatim as a teaching sample. The next page that writes /\s+/g reproduces the exact defect being fixed, silently, on the live page and in the corpus.
The second is the residue. The table says 853 to 70 and stops there, and the AGENTS.md addition states an escape rule without saying where it stops applying. 50 of those 70 are the same defect in prose code spans, so the corpus still disagrees with the page it was generated from on 8 pages.
vivek7405
changed the title
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Re-read the prose fold and traced what it touches, and it turned up the thing the first pass missed: the fold now claims to be universal and is not.
extractPage has a FOURTH path that copies out of page source, the fallback description, and it is the majority path rather than an edge: only 8 of 44 pages declare metadata.description, so 36 fall back to their first <p>, and that one never folded. A paragraph would reach /llms-full.txt cooked while the same paragraph reached /llms.txt and the search index with its escape debris intact, which is the exact self-disagreement this PR says it removed. Nothing trips it today, so it is latent rather than a live regression, but it was left open while the docs asserted it closed.
The other two are the claims themselves. The prose count is off by unit, and the AGENTS.md entry reads as if no backslash survives anywhere, which is not true of the half of the corpus that never passes through this extractor at all.
vivek7405
left a comment
vivek7405
left a comment
There was a problem hiding this comment.
Read the description fold and what it claims. The behaviour is right and the numbers all check out, but the commit wrote a rule and left it unenforced, and it made the doc surface disagree with the file it documents.
The rule is "fold exactly once per path". I restored the fold at the metadata call site on top of this head, so that value folds twice, and all 23 tests stayed green. The fixture the comment credits for guarding it drives plainText alone, so it can only see a double fold inside one function, and a double fold here happens across two. No docs page carries a backslash in its description either, so the corpus walks have nothing to compare.
The other one is smaller but it is in the file this PR designates as its doc surface, and it contradicts a code comment the same commit wrote.
62fbdff to
07de54d
Compare
The llms corpus builder unescaped prose template holes but copied fenced
samples straight out of page source with only entity decoding applied. A
docs page body is a JS template literal, so a sample's backticks are
written \` and a literal hole \${, and the corpus carried that debris:
853 backslashes across 29 of 44 pages, including 5 form bindings that
taught `<form action=\${createPost}>` where the rendered page shows
`<form action=${createPost}>`. The corpus is the surface whose only
reader is an LLM, and invariant 12 governs exactly that shape.
The fold runs before the entity decode because that is the order a
browser applies them: JS cooks the literal first and the HTML parser
only ever sees cooked text. The two orders agree on every sample in the
repo and differ only where an escape splits an entity.
Two docs pages authored a regex with a single backslash, so the live
page already rendered `replace(/s+/g, '-')` and `/;s*/`. The fold is
faithful, so it would have shipped those broken samples to the corpus
verbatim; both are corrected here.
The fenced-sample fold left prose alone, so the corpus still taught
`html\`...\`` on 50 lines across 8 pages where the rendered page shows a
plain backtick. Prose is copied out of the same template literal as a
sample or a hole, so it folds the same way, and the module now has one
rule instead of three paths that disagree.
The fold runs after the two hole passes because those are what tell a
literal `\${x}` from a render-time `${x}`, and before the restore so
text they already folded cannot fold twice. The single entity decode
still comes last.
The two page regex fixes had nothing that would notice a regression,
which matters because the fold's whole point is that it now copies an
authoring error into the corpus verbatim. A letter after a backslash is
never meaningful in a template literal, so a test rejects one across
every docs page.
extractPage had a fourth path that copies out of page source without folding: the fallback description, which takes a page's first paragraph when it declares no metadata.description. That is the majority path, 36 of 44 pages, so a paragraph could reach /llms-full.txt cooked while the same paragraph reached /llms.txt and the search index with its escape debris intact. No page trips it today, which is why it survived the first pass; the docs claimed every path folded while three of four did. The fold moves inside plainText, which both description sites already go through, and comes off the metadata call site so neither folds twice. A second fold is destructive, since it eats a backslash an author wrote as an escaped one. The prose claim said 50 lines where 50 is the backslash count and 23 is the line count, in the two places a later reader would check it from. The AGENTS.md entry also read as if no backslash survives anywhere, when renderLlmsFull appends the repo-root skill markdown verbatim and those files are not template literals, so their single backslash is correct at source.
The commit that moved the description fold into plainText also wrote the rule that no path folds twice, and left that rule with nothing behind it: restoring the fold at the metadata call site keeps all tests green. The fixture the comment credited drives plainText alone, so it can only see a double fold inside one function, and no docs page carries a backslash in its description, so the corpus walks have nothing to compare either. extractPage is module-private and reads a real file, which leaves the call sites as the only thing to assert on. The AGENTS.md entry also lumped the title in with the description and said both fold inside plainText before the single decode. The title folds at its call site, never passes through plainText, and decodes nothing at all, because it is read out of a quoted metadata string rather than out of markup. The code comment beside it already said so, so the doc surface contradicted the file it documents.
07de54d to
430c4c9
Compare
1 participant
Closes #1344
The llms corpus builder copied text straight out of page source with only entity decoding applied. A docs page body is a JS template literal, so a backtick in it is written
\`and a literal hole\${, and the corpus carried that debris: 853 backslashes across 29 of 44 pages, including 5 form bindings that taught<form action=\${createPost}>where the rendered page shows<form action=${createPost}>. The corpus is the surface whose only reader is an LLM, and invariant 12 governs exactly that shape.#1331had already taught the two prose HOLE passes to fold, so this finishes the job on the other two paths: fenced samples, which is what #1344 asks for, and ordinary prose, which review turned up as the same defect from the same cause and which is fixed here rather than documented as a limit.What changed
website/lib/docs-llms.server.ts, the fenced-capture site: run the module's existingunescapeJsbeforedecodeEntities.website/lib/docs-llms.server.ts, prose: one fold pass over the body. It runs AFTER the two hole passes because those are what tell a literal\${x}from a render-time${x}, so folding first would erase the distinction and the dynamic-hole pass would drop the literal. It runs BEFORE the hole restore, so text those passes already folded cannot fold twice. The single entity decode still comes last, so the module's "strip at every stage, decode exactly once, at the end" rule holds verbatim.website/lib/docs-llms.server.ts, page title and description: the fold moves insideplainText, which both description sites already go through, and comes off the metadata call site so neither folds twice. A second fold is destructive, since it eats a backslash an author deliberately wrote as\\. This is the path review caught: only 8 of 44 pages declaremetadata.description, so 36 fall back to their first<p>, and that fallback never folded, which meant a paragraph could reach/llms-full.txtcooked while the same paragraph reached/llms.txtand the search index raw. No live page trips it, so the corpus is byte-identical and a fixture is what holds the rule.website/app/docs/backend-only/page.tsandwebsite/app/docs/websockets/page.tsauthored a regex with a single backslash, so the LIVE page already renderedreplace(/s+/g, '-')and/;s*/. The fold is faithful, so without this it would ship those broken samples to the corpus verbatim.website/app/docs/api-routes/page.ts:296was already correct and is the precedent. Both live pages were re-rendered throughrenderToStringand now show\s.website/test/ssr/docs-llms.test.ts: the corpus walk's mirror of the extractor gains the same fold, plus four fixtures and an authoring guard.website/AGENTS.md: thedocs-llms.server.tsinventory entry records the fold, its ordering, and where a backslash may still legitimately appear.Why escapes fold before entities decode
That is the order the browser applies them: JS cooks the template literal first, and the HTML parser only ever sees cooked text.
packages/core/src/html.jsstores thestringsarray as handed to it and never touches.raw, exactly as lit-html does, so the cooked array is what the renderer sees.Both orders were prototyped over the whole real corpus and produce byte-identical output, so the corpus cannot settle it. They differ on exactly one shape, an escape splitting an entity:
`&am\p;`cooks to&, which the parser then shows as&, while decoding first would ship&. No docs page carries that shape, which is why a fixture holds the rule. The reverse order can never help either, because nothing in the entity table yields a backslash, so decoding cannot manufacture an escape for the fold to eat.Accounted before-and-after
renderLlmsFull()atmainversus at this branch's head.grep -cF 'form action=\${'grep -cF 'form action=${'Every one of the 20 surviving backslashes is one an author wrote as
\\and meant: twocurlline continuations, three regexes (the two fixed here plusapi-routes), a'\n'in a string, a<\/p>in a regex, and 13 in the repo-root skill markdown thatrenderLlmsFullfolds in verbatim without passing it throughbodyToMarkdown, which is out of this change's reach by construction. Nothing in the docs-page half of the corpus carries escape debris any more.The 833 removed backslashes account to the escape table (512
\`, 265\$, 5\\, 2\sand 1\}inside fences, plus 50 in prose), minus the 2 the regex fixes put back. Those two regex lines read identically before and after, which is why they do not appear in the changed set:\s+in the old source and\\s+in the new one both reach the corpus as\s+, the difference being that only the new one also renders correctly on the live page.Test plan
website/test/ssr/docs-llms.test.ts: 24 pass. The corpus walk still compares 348 samples with 0 mangled, mirror extended rather than assertion weakened, so it stays well above its> 300gate.723073cbwith the change committed first and reverted through git, never by editing a sentinel into the source:plainTextfold: the description fixture reds.plainTextalone.cd website && npm test: 469 node tests pass, 84 browser tests pass across 8 files.cd website && npm run typecheck: clean.npx webjs check: all checks pass.npx webjs doctor: 11 passed, 2 warnings, 0 failed (both warnings pre-existing).node --test test/docs/llms.test.mjs: 11 pass, unchanged, it asserts nothing about escaping.website/test/lib/doc-headings.test.tsandwebsite/test/ssr/docs-search.test.ts: 11 pass with no edit. They consume this markdown and pin the fence predicate, and fence parity is unchanged.Repo-root
npm test: 4,143 pass, 5 fail. Those 5 are the documented linked-worktree failures (the twotest/bun/listener*tests and three elision assertions), not this change. Proven rather than asserted: a detached worktree atorigin/main, linked the same way, fails the identical 5 by name. They pass in the primary checkout and in CI.Layers that do not apply: browser and e2e, because this is a
.server.tsmodule whose output is served astext/plain, so nothing hydrates and the unit layer holds the exact bytes. Bun parity does not apply, checked against the hook rather than asserted:.claude/hooks/require-bun-parity-with-runtime-src.shscopes its runtime-sensitive match to^packages/([^/]+/src|editors/[^/]+/src|cli/lib)/, which nothing underwebsite/can match, and on the merits this is a pure string-to-string transform with no listener, serializer, crypto or stream surface. Smoke covers the example apps, not the marketing site. Dogfood: the website IS the app under change, and its own suite pluswebjs checkandwebjs doctorare reported above;examples/blogis untouched by awebsite/libchange.Docs surfaces
website/AGENTS.mdupdated, as above. That is the doc surface for this change.@webjsdev/*export, CLI flag,webjsconfig key,htmlhole prefix, lifecycle hook or app-author convention moved. RootAGENTS.md, the skill at.agents/skills/webjs/,README.md,CONVENTIONS.mdand the scaffold templates all stay untouched. The two docs-page edits are content corrections rather than doc-surface sync.Deliberately not in this change
One thing #1344 scoped out and I left alone, worth your call rather than mine: the fenced path copies a
${'...'}string-literal hole verbatim, sowebsite/app/docs/data-fetching/page.ts:36reaches the corpus as.fallback=${'${html`<p>Loading section...</p>`}'}where the rendered page shows.fallback=${html`<p>Loading section...</p>`}. About 40 fence lines carry that shape. It is a real and separate defect, it is not an escape problem, and fixing it would collide witha fenced sample keeps the interpolation holes and indentation the prose pipeline would eat, which depends on a genuine${children}surviving a fence verbatim.