Skip to content

Resolve catalog supersede links against every row's effective assertion id - #306

Open
MaxGhenis wants to merge 4 commits into
codex/thesis-ledger-factsfrom
fix/catalog-effective-assertion-ids
Open

MaxGhenis wants to merge 4 commits into
codex/thesis-ledger-factsfrom
fix/catalog-effective-assertion-ids

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Status 2026-11-03. I rebased this onto the branch's new head fd83eec (#352), which leaves the ledger and catalog bytes unchanged. One commit since the second review, d80ade7, settles its nits:

  • The prefix-correction test asserts that all six plan lists are empty and that every series equals its committed row apart from unit.
  • The absorbed-row correction test runs with and without assertionVersion.
  • The properties emit their mutation and refusal classes as Hypothesis events, and a new test finds a journal for every class.
  • The stale statements in a test comment and in docs/catalog-curation-backlog.md now describe what the builder does.

At this head, tests/test_build_series_catalog.py gives 270 passed. I also ran the migration-shape check: the 253-line ledger plus 82 prototype unit corrections, built with this head's build_catalog. It gives no mints, supersedes, drops, revives or retire_pending, and every one of the 221 series keeps its UUID. 51 series change unit and no other series field moves. The third review is queued.

Status 2026-11-02. I adopted this PR; its authoring session is gone. I rebased it onto 5308a19. That dropped the spelling-pin commit, which the entity folds replaced, and passed the entity-fold seed to the live-ledger test. The catalog stays byte-identical.

Defect

scripts/build_series_catalog.py build_catalog built its assertion-version id set only from rows that carry assertionVersion.id. It then refused any correction whose supersedes names a pre-versioning row, which covers every row of the 128-line immutable prefix, with assertionVersion X supersedes unknown version Y.

receipt 0.6.2's append gate works differently. check_rows gives every row one effective id and reserves all of them. For a versioned row that is its explicit id, which must equal the recomputed av2 content address. For any other row it is that address. A correction must name the active version's effective id, so the gate accepts a correction that names a prefix row's address. effective_current_rows applies the same id rule through _effective_assertion_id.

The preconditions added in 3df42b9 were meant to mirror the gate ("standalone --observations runs are as safe as gate-validated ones"). The Thesis resolver runs this generator on every append: at ThesisInstitute/thesis@95ca41c8, ledger_catalog_refusals (scripts/resolve_pending.py:13815) and _regenerate_series_catalog (:13025, called from _prepare_release_files at :13178) run the branch-head build_series_catalog.py in write mode on the ledger plus the run's rows. If the base alone fails, the first screens nothing, and a non-zero exit in the second aborts the proposal. So one gate-accepted correction merged to the ledger would have made every later append fail.

Repro (2026-09-29). I took six synthetic corrections of the June 2026 Table A-19 prefix rows (ledger lines 119-124), each with supersedes set to that row's av2 address. receipt check_rows accepted them and the generator exited 1. Evidence: ~/chief-of-staff/state/thesis-resolution-backlog-2026-09-19/scratch/exp-0929/. With this change, both correction cases exit 0. The August-only case still refuses on its millions/thousands unit conflict, and this PR does not touch that. The base-unit migration removes the conflict.

Fix

check_assertion_versions(rows), extracted from build_catalog, now does four things:

  • It computes every row's effective id the way receipt does, via effective_assertion_version_id: the explicit assertionVersion.id, else check_thesis_facts_append.expected_assertion_version_id(row).
  • It reserves all of those ids for the duplicate check. Two identical unversioned rows now collide here as they already do at the gate, and so does a versioned row that reissues an unversioned row's address.
  • It resolves supersedes links against those ids.
  • It keeps the refusals for malformed ids and links, self-links and cycles unchanged.

A row whose content address cannot be computed (for example a non-object measure) is refused with a message instead of a traceback.

No ledger rows are added or changed. Max's 2026-10-21 ruling settles A-19 and supersedes d392 and d397's framing: Chronicle moves to base units, with the printed scale kept as metadata. That migration corrects 82 current rows, 14 of them prefix rows, so it needs this PR. A June 2026 A-19 row stays valid under it: its contract is in thousands, and Thesis converts to each forecast's registered unit.

Invariants (tested)

  1. Gate soundness. If receipt 0.6.2 check_rows accepts a journal, the generator's assertion-version preconditions accept it too. Property-tested over gate-valid journals built to include corrections of unversioned (prefix) rows. This is the direction the defect broke.

  2. Effective-id agreement. For every row, check_assertion_versions returns receipt's own _effective_assertion_id(row, APPEND_GATE_SPEC). This is a differential test against receipt's function, run on the generated journals and on the live ledger.

  3. Reason agreement ("refuses a link iff the gate does for that reason"). This holds on journals with at most one defect, whose explicit ids are their content addresses (the gate's first row rule). There the generator refuses for reason R exactly when the gate's refusal classifies as R. The reasons are duplicate id, self-link, unknown target and cycle. The classifier reads the gate's first refusal and places its target among the gate's own effective ids: the row itself, no row, or a row whose chain leads back to it.

    Some refusals belong to the gate alone, and there the generator must accept:

    • a link to a non-active, other-record or later version outside a cycle
    • a correction with no link

    The defects are unknown, self-link, cycle of two or three rows, arbitrary relink, unlink, duplicate row and A→B→A restore. At the committed 400 derandomized examples, --hypothesis-show-statistics shows the single-defect property reaching each generator refusal (self, duplicate, unknown, cycle) and acceptance, and test_the_mutation_strategy_reaches_every_class finds a journal for every class, gate-only included. Unversioned prefix-style rows appear in 74% of the journals.

  4. Refusal containment. With up to three defects, every generator refusal is also a gate refusal. When the gate refuses for a generator reason, the generator refuses too. This includes rows whose explicit id is rewritten away from their content address (the "reid" defect). The gate refuses those outright. The generator trusts explicit ids, so it can still refuse such a journal for a duplicate or orphaned link, which is why invariant 3 is not claimed for them.

  5. No-op on the live ledger. The rendered catalog is byte-identical before and after this change (sha256 1f365716d1bdd09285ba1e2f6a4e588ae9e92c6658a3b4bdceff311e376e8c27 at 5308a19, equal to the committed ledger/series_catalog.json). test_committed_catalog_is_current_and_valid pins it.

The generator's precondition set is deliberately a subset of the gate's. The gate's other rules stay the gate's alone: an explicit id must equal the content address, and a link must name the active version of the same record on an earlier line, so no version is superseded twice. A fork, where two corrections supersede one version, passes the generator and is refused at the gate.

Tests

tests/test_build_series_catalog.py, 270 passed locally at d80ade7 (Python 3.14, receipt 0.6.2):

  • test_live_ledger_reserves_the_gates_ids: the gate accepts the live ledger, the 128 prefix rows carry no assertionVersion, and the generator's ids equal the gate's.
  • test_correction_of_a_prefix_row_is_accepted_like_the_gate: the A-19 case on the live ledger. The gate accepts, the preconditions accept, and a full build_catalog leaves all six plan lists empty, every series equal to its committed row apart from unit, the six occupation series in millions, and the current row count from effective_current_rows. The corrections are synthetic and each supersedes its record's active version. Their archive digests are synthetic too, so no real correction can collide with them.
  • test_every_rows_effective_id_is_reserved: identical unversioned rows, a versioned reissue of an unversioned row's address, and A→B→A restores (from an unversioned and from a versioned A) are all refused, matching the gate's "restates".
  • test_link_refusals_fire_where_the_gate_refuses: unknown target, a near-miss address, a self-link and cycles of two and three rows.
  • Three Hypothesis properties for invariants 1-4 (400 examples each, derandomized, no example database). The adversarial review also re-ran them randomized at 5,000 examples each, and all passed.

Mutation check. I ran seven mutants of check_assertion_versions against the suite and every one fails it:

  • the original skip-unversioned-rows logic (6 tests fail)
  • unversioned ids not reserved
  • no duplicate check
  • no self check
  • no unknown check
  • no cycle check
  • a wrong content address

Thesis coupling (follow-up, not a blocker)

At ThesisInstitute/thesis@95ca41c8, bls_ledger_unit_conflict (scripts/resolve_pending.py:9058) builds its current view by dropping only rows whose explicit assertionVersion.id is superseded. So after a correction of a prefix row it still counts the June rows. Its docstring (:9073-9078) says that is deliberate and fails closed until "a reviewed change follows". docs/anchor-verifications.md (:1531-1541) already says the change that lifts the A-19 hold must update that guard.

After this merges, three things in Thesis describe the old generator: that docstring, the doc paragraph ("learns version ids only from rows that carry one ... exits 1"), and the comment in tests/test_bls_api_registrable.py::test_the_guard_follows_chronicles_supersede_links (:1100-1101). I'll open a Thesis follow-up for them. Nothing in Thesis parses the generator's messages.

Adversarial verification

A four-lens workflow ran against ab986a4. The integration lens replayed the resolver's two generator calls with the stage file set and refusal logic taken from Thesis:

  • With this PR, the corrected base passes, the registry is unchanged and _validate_catalog_binding passes.
  • With 598f394, the base fails ("supersedes unknown version") and every append would abort.
  • August A-19 rows in millions are still refused individually.

The review lens found no code bug. Its wording findings are fixed in ee04eac.

Earlier extras

The spelling pin and the hypothesis dependency this PR once carried are on the branch already: the entity folds replaced the pin, and the base's pyproject.toml lists hypothesis>=6.100,<7. This PR now changes only the builder, its tests and the backlog doc.

Neither ledger/** nor the gate surface is touched, so the append gate sees an unclassified-only change.

axiom: n/a: catalog-generator infrastructure, no policy change

🤖 Generated with Claude Code

MaxGhenis added a commit that referenced this pull request Sep 29, 2026
Review of #306 found wording that overstated the change: the gate-only
list in _gate_reason included an explicit id that is not the content
address, but the generator trusts explicit ids and can refuse such a
journal for a duplicate or orphaned link. The single-defect "iff" is now
stated for journals whose explicit ids are their content addresses, and
the containment property also rewrites ids ("reid") to hold the rest to
"every generator refusal is a gate refusal". check_assertion_versions
now says it enforces the part of the gate's rules the current view
relies on, names the gate-only rules (content address, active version,
earlier line, no version superseded twice), and says a duplicate needs
an exact restatement of addressed content, not just a restored value.
The receipt import comment says check_rows reserves the same id inline,
and the import-failure message names the content address too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 12, 2026
Answers the independent review of a803f08.

- Any change to the entity-fold seed is an identity change and needs
  --allow-remint --remint-note, even when it plans no registry event:
  retargeting a fold's survivor moved observations between two existing
  lineages with no ceremony (three-entity regression added).
- Undoing a fold now restores earlier same-dimension spelling merges. A row
  that matches nothing in its own slice rejoins a retired identity of its own
  entity whose names live on another slice's row, and those names move back
  off that row. Undoing all seven folds on the committed artifacts reproduces
  the pre-fold catalog exactly apart from the seed and registry digests;
  before this, the us.frb and us.census spellings revived as separate
  identities. The property test and an alias-bearing regression assert the
  exact round trip.
- A folded row's own entity is validated before the fold replaces it.
- The strict-JSON seed test checks duplicate keys, NaN and Infinity
  separately.
- The docstring says what the fold guarantees about corrections: an inert
  fold never fails. Superseding a pre-versioning row is refused by the
  builder whatever the folds say (a pre-existing limitation #306 addresses);
  a test shows a correction superseding a versioned absorbed row, with either
  entity, regenerates with the registry unchanged. A second test appends new
  prints of a folded series under both entities with the registry unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Nov 2, 2026
Review of #306 found wording that overstated the change: the gate-only
list in _gate_reason included an explicit id that is not the content
address, but the generator trusts explicit ids and can refuse such a
journal for a duplicate or orphaned link. The single-defect "iff" is now
stated for journals whose explicit ids are their content addresses, and
the containment property also rewrites ids ("reid") to hold the rest to
"every generator refusal is a gate refusal". check_assertion_versions
now says it enforces the part of the gate's rules the current view
relies on, names the gate-only rules (content address, active version,
earlier line, no version superseded twice), and says a duplicate needs
an exact restatement of addressed content, not just a restored value.
The receipt import comment says check_rows reserves the same id inline,
and the import-failure message names the content address too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the fix/catalog-effective-assertion-ids branch from ee04eac to fddd729 Compare November 2, 2026 23:49
MaxGhenis and others added 4 commits November 3, 2026 18:05
build_catalog collected assertion version ids only from rows that carry
assertionVersion.id, so a correction superseding a pre-versioning row
(every row of the 128-line immutable prefix) was refused as superseding
an "unknown version". receipt 0.6.2's append gate addresses such a row by
its recomputed av2 content address and accepts the correction, and the
Thesis resolver runs this generator on every append, so one
gate-accepted correction would have failed every later append.
Reproduced 2026-09-29 with six synthetic corrections of the June 2026
Table A-19 prefix rows.

check_assertion_versions now gives every row the gate's effective id
(the explicit id, else expected_assertion_version_id(row)), reserves
all of them for the duplicate check, and resolves links against them.
The self-link and cycle refusals are unchanged. Two identical
unversioned rows, and a versioned row reissuing an unversioned row's
address, are now duplicates here as they already are at the gate.

Tests: the live ledger gets the gate's ids (catalog bytes unchanged:
sha256 226174876fce...425a before and after); the A-19 correction case
passes the gate, the preconditions and a full build; duplicate, unknown,
self and two- and three-row cycle refusals match the gate's; Hypothesis
properties check gate-accepted journals pass, and on single-defect
journals the generator refuses for a reason iff the gate's refusal
classifies as that reason. hypothesis is added to the dev extras and
group; uv.lock gains only its entry (a full relock would have moved
pandas and policyengine-us a major version).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #306 found wording that overstated the change: the gate-only
list in _gate_reason included an explicit id that is not the content
address, but the generator trusts explicit ids and can refuse such a
journal for a duplicate or orphaned link. The single-defect "iff" is now
stated for journals whose explicit ids are their content addresses, and
the containment property also rewrites ids ("reid") to hold the rest to
"every generator refusal is a gate refusal". check_assertion_versions
now says it enforces the part of the gate's rules the current view
relies on, names the gate-only rules (content address, active version,
earlier line, no version superseded twice), and says a duplicate needs
an exact restatement of addressed content, not just a restored value.
The receipt import comment says check_rows reserves the same id inline,
and the import-failure message names the content address too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The branch added a refusal for regenerating without the seed once the
committed catalog declares it. A full build over the live ledger now has to
pass it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The prefix-row correction test now checks every identity-change list in
  the plan, retire_pending, revives and enrich_retires included, and that
  every series matches the committed catalog apart from its unit.
- The absorbed-row correction test also runs on an absorbed row written
  before assertion versioning, whose correction names its content address.
- The mutation strategy reports each mutation as applied or a no-op, and a
  new test finds a journal in every class the gate-agreement property
  compares. It fails for four classes when the strategy draws one kind.
- Two notes that still called prefix-row corrections refused now say what
  the builder does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the fix/catalog-effective-assertion-ids branch from d6b35c0 to d80ade7 Compare November 3, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant