One list, one key grammar, one fingerprint - #38
Open
zmaril wants to merge 4 commits into
Open
Conversation
The list of published tables was written out in five places: the column definitions, the parquet writer, the generated schema, the compaction Worker, and the site. None derived from another. When `phase_spans` arrived it had to be typed into each of them, and nothing would have failed if one had been missed -- which is not hypothetical: the site's copy has been missing it ever since. There was a reason no list existed. The ten tables are ten different types -- `Table<SessionRow>`, `Table<UnitRow>` -- so nothing could hold them together. `Published` forgets the row type, which is enough for everything that is not the writer, and `ALL` is the list. The generated schema and the compaction Worker read it. The site reads it too, through a generated TypeScript file that a test keeps in step; when it goes stale the test names the line rather than printing four kilobytes at you twice. The writer is the one that cannot: it pairs each table with the field of `Tables` holding its rows, so it must name both, and it is checked against `ALL` instead. Dropping a table from `ALL` now fails two tests rather than shipping. The site's copy is not replaced so much as inverted. It listed the seven tables the charts read in full, which is a list that must grow every time a chart learns to read something -- and did not. It now lists the two no chart reads, so a table added tomorrow is read in full without anyone remembering to say so. What renders today does not change: the old list had no effect, because the limit it was meant to raise was already the raised one. `tables.json` is byte-identical to the one this replaces, checked against a clean worktree of main. Its key order is a public interface -- readers index into it -- so a test asserts the order too. Also: `cargo test -p cratebank-core` failed to build an example needing `--features compress`, which reads as "this crate is broken" and is not. It declares the feature it needs now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361quprG7tUgaVFKAmKtJZ
The key an object gets was a `format!` in the ingest Worker and a separate parser in TypeScript on the site: one grammar, two implementations, nothing making them agree. It lives in core now, with the parser beside the writer and a test that one undoes the other. The schema segment reads `v1` rather than a sixteen-character hash. That hash looked like it partitioned staging -- the comment above it said so -- but compaction lists every staged object under `sessions/` whatever the segment says, and decides compatibility by reading each file's parquet metadata. The name never gated anything. What it did do was take its value from a header the caller controls, and put `DefaultHasher` output into a key space that outlives the toolchain that wrote it. Older keys still parse and still render. They are in the bucket, and their pages should keep opening. Sharing this with ingest meant giving it a dependency it deliberately did not have: it stores the body byte for byte and never opens it, and core brings a parquet implementation with it. So `parquet` is a feature now, and ingest takes core without it -- 1.03 MB of wasm to 1.04, where the ungated version measured about 1.5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361quprG7tUgaVFKAmKtJZ
A fingerprint decides whether two parquet files may have their row groups copied into one another, so it is a fact about the format rather than about any one file. It was computed with `DefaultHasher`, which the standard library explicitly does not promise is stable across Rust releases, and its output was part of a persistent key space. The day that algorithm changed, every client on a newer toolchain would have computed a different name for the same schema and quietly stopped agreeing with every client on an older one. It is FNV-1a written out here, so it is ours, and the fingerprints are recorded in schema-v1.txt where a test checks them. Renaming a column now fails with "units.parquet changed shape. If that was deliberate, bump payload::SCHEMA and update schema-v1.txt in the same commit" -- which turns a schema change from something noticed later into something someone decided. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361quprG7tUgaVFKAmKtJZ
`phase_spans` is the one table whose length grows with the size of a build rather than with its unit count, so it is the one that runs into the row limit the page decodes under. Running into that limit does not thin the chart evenly: the rows are stored grouped by unit, so what goes missing is whole units. On bun's release build the chart drew one of 223 units and looked complete. It says so now, the way the verdict does. The limit itself is a memory bound on the Worker and lifting it is not this function's job. Sampling at 499 Hz rather than 4999 already keeps a build this size under the limit, so the caveat is rare -- but "rare" is the wrong thing for a chart to be quietly wrong about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01361quprG7tUgaVFKAmKtJZ
cratebank previewRun automatically for a PR authored by @zmaril. OpenTofu planNo infrastructure changes. Plan outputcloudflare_r2_bucket.cratebank: Refreshing state... [id=cratebank]
cloudflare_workers_custom_domain.ingest[0]: Refreshing state... [id=6edef664db9c85623282947938570ffb8054b932]
cloudflare_workers_custom_domain.site["cratebank.io"]: Refreshing state... [id=41b6bb2dc82ba0dacaa13bb7056f0304a92e9065]
cloudflare_workers_custom_domain.site["www.cratebank.io"]: Refreshing state... [id=2426388fdde2ff5a2083a1458e423f27566fb994]
cloudflare_workers_route.data_root[0]: Refreshing state... [id=8392a042ec984865a4842bde55fa086c]
cloudflare_r2_custom_domain.data[0]: Refreshing state...
No changes. Your infrastructure matches the configuration.
OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for moving the viewer into Rust: before anything is ported, the facts that were written down more than once are written down once.
The list of tables lived in five places
The column definitions, the parquet writer, the generated schema, the compaction Worker, and the site. None derived from another. When
phase_spansarrived it had to be typed into each — and the site's copy never got it, which is why the phase-timeline chart drew one of bun's 223 units while looking complete.There was a reason no list existed: the ten tables are ten different types (
Table<SessionRow>,Table<UnitRow>), so nothing could hold them together.Publishedforgets the row type;ALLis the list.schema::tables_jsonandworker-compactderive from it.write::allcannot — it pairs each table with the field ofTablesholding its rows, so it must name both. It is checked againstALLinstead.The site's list is inverted rather than extended: it named the seven tables charts read in full, a list that has to grow whenever a chart learns to read something. It now names the two no chart reads, so a table added tomorrow is read in full without anyone remembering. Rendering is unchanged — the old list had no effect, because the limit it was meant to raise was already the raised one.
tables.jsonis byte-identical to before, checked against a clean worktree ofmain. Its key order is a public interface, so a test asserts that too.The key grammar existed twice
A
format!in the ingest Worker and a parser in TypeScript on the site. Nowcore::keys, parser beside writer, with a round-trip test.The schema segment reads
v1instead of a sixteen-character hash. That hash looked like it partitioned staging — the comment said so — but compaction lists everything undersessions/whatever the segment says, and decides compatibility by reading parquet metadata. It never gated anything. It did take its value from a caller-controlled header, and putDefaultHasheroutput into a persistent key space.Older keys still parse and render; they are in the bucket.
The fingerprint could have moved under us
DefaultHasheris explicitly not guaranteed stable across Rust releases, and its output was part of the key space. The day it changed, clients on a newer toolchain would compute a different name for the same schema and quietly stop agreeing with everyone else.It is FNV-1a written out now, and the fingerprints are pinned in
crates/core/schema-v1.txt. Renaming a column fails with "units.parquet changed shape. If that was deliberate, bumppayload::SCHEMAand update schema-v1.txt in the same commit."Features, with a number
Sharing the key grammar with ingest meant giving it a dependency it deliberately lacked — it stores the body byte for byte and never opens it, and core brings parquet. So
parquetis a feature, and ingest takes core without it: 1.03 MB → 1.04 MB of wasm, against roughly 1.5 MB ungated.Guards are verified to fail, not just to exist
ALL→ two tests failEach of the four commits builds and tests on its own.
cargo test -p cratebank-corealso works out of the box now; an example needed--features compressand failed the default build.Checks
core 39 · client 74 · worker 18 · straitjacket clean (18 files) · both Workers compile for wasm. Verified end to end against a cold bun release build: new keys write
schema=v1, legacy hex keys still render, all seven charts draw.🤖 Generated with Claude Code
https://claude.ai/code/session_01361quprG7tUgaVFKAmKtJZ