[MaxGhenis]

What changes

On main, apply_adjudications rewrites every row of an adjudicated case except parse_contract_failure rows, so it overwrites a recorded budget_exhausted_at_ceiling row with the adjudicated class and subtype. Elsewhere the pipeline treats that recorded class as authoritative:

• collect_audit (_row_failure_source and the deterministic branch) and case_annotations.wrong_prediction_rows force it onto its row.
• full_run_export.merge_annotations restores it at dashboard export. It restores only failure_source, so the export keeps the overwritten subtype.

The annotation CSVs keep both overwritten fields.

• New NON_SUBSTANTIVE_SOURCES (policybench/adjudications.py:43) lists parse_contract_failure and budget_exhausted_at_ceiling, and apply_adjudications skips rows of both classes (:228).
• verify_adjudications_applied uses the same mask (:295) for its substantive rows, which must carry the adjudicated source and subtype. It also uses it for its stray-verdict check (:300): no substantive row may still carry the judge's non-final class.
  • Without the first, a release driver's apply-then-verify would demand the adjudicated class on a recorded ceiling row that apply now leaves alone. On main, these drivers call both functions in that order: scripts/finish_argon.py and scripts/finish_terra.py (triage), scripts/freeze_argon.py and scripts/freeze_terra.py, as well as the earlier finish_*/freeze_* drivers. The triage steps in finish_argon.py and finish_terra.py already count budget_exhausted_at_ceiling as a final class on a scored row (TRIAGE_FINAL_SOURCES and the unresolved-row check).
  • Without the second, a case whose judge also chose budget_exhausted_at_ceiling would fail verification on its recorded rows.
  • On parse_contract_failure rows the stray-verdict check behaves as before, since a final class never matches a non-final judge class.

Up to date with main

The branch merges origin/main at 8ac64ac4 (merge commit f8b6b88e). Git reported one conflict, in tests/test_adjudications.py, where both sides had appended tests at the end of the file, and the merge keeps both. Since the branch point, main changed policybench/adjudications.py once, in bf845cbe (#221), which adds judge_ever_flagged. That function reads only an entry's judge flags (judge_reference_suspect, judge_reference_suspect_source, judge_previous), not a row's class, and its caller scripts/freeze_snapshot.py uses it only to tally reference verdicts. main added no adjudicated outcome: FINAL_SOURCES, AFFIRMABLE_WITH_EXCLUSION and REFERENCE_VERDICTS match the branch point.

No published effect

No row, case note or adjudication on main records budget_exhausted_at_ceiling, and none of the releases we froze after this PR opened does either:

Source	Rows	Case notes	Ceiling rows	Ceiling case notes
main (8ac64ac4), committed annotations	10,999	990	0	0
dashboard-data-20261027 tag (bd11d55), committed annotations	10,680	989	0	0
dashboard-data-20261014 tag (2cb77a1)	10,346	989	0	0
dashboard-data-20261013b tag (f095ce5)	10,346	989	0	0

• paper/snapshot/20260501/manifest.json on main pins release dashboard-data-20261027 (snapshot date 2026-10-27). It does not contain the string budget_exhausted_at_ceiling. The release's published dashboard-data.json, whose sha256 67cb0d14…021b matches the manifest's live_dashboard_artifact, does not contain it either.
• us_adjudications.json on main (63 entries) never names the class.
• The one committed file outside the code that names it, docs/judge_isolation_2026-09-30/codex_prompt_identity.json, describes changes to the judge prompt, not rows.

We counted rows with pandas, so a quoted multi-line field counts once:

A=annotations/us_full_run_20260612_policyengine_4_16_1_populace
for ref in origin/main dashboard-data-20261013b dashboard-data-20261014 dashboard-data-20261027; do
  for f in us_audit_row_annotations.csv us_case_notes.csv us_adjudications.json; do
    git show $ref:$A/$f | grep -c budget_exhausted_at_ceiling; done
  git show $ref:paper/snapshot/20260501/manifest.json | grep -c budget_exhausted_at_ceiling
done
python -c "import pandas as pd; r=pd.read_csv('$A/us_audit_row_annotations.csv'); c=pd.read_csv('$A/us_case_notes.csv'); \
  print(len(r), len(c), (r.failure_source=='budget_exhausted_at_ceiling').sum(), \
  c.case_failure_sources.astype(str).str.contains('budget_exhausted_at_ceiling').sum())"
curl -sSL -o dashboard-data.json https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20261027/dashboard-data.json
sha256sum dashboard-data.json; grep -o budget_exhausted_at_ceiling dashboard-data.json | wc -l
git grep -l budget_exhausted_at_ceiling origin/main -- ':!*.py' ':!*.md'

None of the release stages on the machine where we build releases records the class either. On 3 November we searched 974 files there, under the PolicyBench checkouts, worktrees and review folders:

• every predictions file, compressed or not, including the retry, merged and sensitivity runs;
• every row annotation, case annotation and case-notes CSV, and every adjudication record;
• every audit cases.jsonl, run store (run.db) and dashboard-data.json.

None names budget_exhausted_at_ceiling. To test the search, we planted a ceiling row in a .csv, a .csv.gz and a run.db, and the search found all three.

Differential test

test_skipping_ceiling_rows_moves_no_committed_annotation reads the committed us_adjudications.json, us_audit_row_annotations.csv and us_case_notes.csv at test time, so it follows main. On 8ac64ac4 it compares all 10,999 rows and 990 case notes under all 63 entries. It first checks its premise: no committed row of an adjudicated case may record the class. If one does, the test fails, and its message says to delete it. Otherwise it applies the committed record under the old mask and the new one, checks that both give the same rows, case notes and report, and verifies the result.

Invariants

_check_adjudication (tests/test_adjudications.py:1863) checks six properties of one application of a record:

1. A parse_contract_failure or budget_exhausted_at_ceiling row keeps its class and subtype, and apply leaves every row of an undecided case untouched.
2. Every other row of a decided case takes the adjudicated class and subtype.
3. A reference verdict clears reference_suspect on its case note and on every row of its case, including skipped rows, and leaves every other row's and note's flag as it was.
4. Each entry's rows_rewritten in the report equals the number of its case's rows whose class or subtype changed.
5. verify_adjudications_applied accepts what apply_adjudications builds.
6. A second application changes nothing.

The tests spell out the skipped classes (_SKIPPED) and the outcomes a valid record admits (_OUTCOMES) instead of importing the module's constants:

Adjudicated class	Reference verdicts	Exclusion from scoring
llm_error	none, affirmed, regenerated	not allowed
parse_contract_failure	none, affirmed, regenerated	not allowed
prompt_ambiguity	none, unlisted_input	required
reference_engine_defect	engine_defect	required
reference_later_law	later_law	required

test_adjudication_skips_non_substantive_rows_properties runs the check with Hypothesis over 300 generated examples. Each example has one to three cases of one to four rows, and each row and case note takes any class, subtype and reference-suspect flag. The generator draws the skipped classes more often than their share of the taxonomy. Each example's record decides any subset of the cases, and parse_adjudications accepts it. Each entry starts from any judge class and takes any class in the table, with any verdict and the exclusion setting the table allows for it.

test_every_outcome_and_verdict_leaves_skipped_rows_alone runs the check once for each of the table's 10 class-and-verdict pairs. Its decided case holds a ceiling row, a parse row and a needs_review row, all flagged, with a flagged case note, and its undecided case holds the same rows and note. In every entry the judge chose budget_exhausted_at_ceiling, so the stray-verdict check meets a recorded ceiling row on every pair. Hypothesis can miss a pair in any one run, and this test runs all ten every time.

At 52b15e61 we ran 13 mutants of policybench/adjudications.py against each test alone and against the whole test file (results):

Mutant	Property	Parametrized test	Whole file
apply's mask reverted to != "parse_contract_failure"	killed	killed	killed
verify's substantive mask reverted to != "parse_contract_failure"	killed	killed	killed
stray-verdict check run over every row of the case	killed	killed	killed
llm_error added to NON_SUBSTANTIVE_SOURCES	killed	survived	killed
skip list cut to budget_exhausted_at_ceiling	killed	killed	killed
row flags cleared only on the rows apply rewrites	killed	killed	killed
every row's flag cleared	killed	killed	killed
every case note's flag cleared	killed	killed	killed
case-note flag cleared only on affirmed	killed	killed	killed
rows_rewritten counts every row in the rewrite mask	killed	survived	killed
subtype left as it was when the adjudicated class is llm_error	killed	killed	killed
stray-verdict check disabled	survived	survived	killed
verify ignores the subtype	survived	survived	killed

The property misses two mutants, the disabled stray-verdict check and the verifier that ignores the subtype, because its round trip checks only that verify accepts apply's output. Example tests in the file kill both.

Tests

• The PR adds five tests to tests/test_adjudications.py:
  • test_apply_leaves_a_recorded_ceiling_exhaustion_alone;
  • test_verify_does_not_hold_a_ceiling_row_to_the_adjudicated_class, which also checks that the verifier still holds rewritten rows to the record;
  • the differential test, the property and the parametrized test above.
• At 52b15e61, uv run pytest tests/test_adjudications.py tests/test_freeze_argon.py tests/test_freeze_haiku55.py tests/test_freeze_musewonder.py tests/test_freeze_terra.py -n 4 gives 1,278 passed and 12 skipped. Those test files exercise apply-then-verify through a release driver. Ten of the skips need a live release stage in the checkout and one needs an engine-upgrade build, while the twelfth skips because the committed Haiku 5.5 spec names adjudications_written_on.
• CI passes at 52b15e61, and so do ruff check . and ruff format --check ..

Not changed

• This PR doesn't force a recorded ceiling class onto the case note in collect_audit's judged branch. eval_no_tools.py attaches the label only to missing variables, so a case whose wrong models all record it counts as parse_failure_only and already takes the deterministic branch.
• This PR edits no frozen audit tree or release artifact. The change reaches publication through a future release driver.
• The first release whose annotations hold a budget_exhausted_at_ceiling row must update five places. None of our releases holds one (see "No published effect"), so this PR leaves them alone:
  • scripts/freeze_snapshot.py, build_manifest: the audit_annotation_artifacts note says row-level failure_source values are llm_error and parse_contract_failure.
  • tests/test_snapshot_artifacts.py, test_snapshot_audit_annotations_have_no_orphan_rows: its list of classes for rows of excluded outputs omits budget_exhausted_at_ceiling.
  • tests/test_paper_results.py, test_excluded_outputs_are_outside_the_scored_audit_universe: its list omits it too.
  • tests/test_adjudications.py, test_skipping_ceiling_rows_moves_no_committed_annotation: the test compares the two masks only while no committed row of an adjudicated case records the class. Once one does, the test fails with a message that says to delete it.
  • policybench/audit.py, collect_audit: a ceiling row that apply skips keeps the judge's per-model failure_subtype, which need not be missing_output. Decide with that release whether to force missing_output.

axiom: n/a: benchmark annotation pipeline, no policy rule

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Nov 3, 2026 3:02am UTC

[MaxGhenis]

Picking this up for the PolicyBench session to bring it up to date with main. Reply here if you're mid-change and I'll stop.

---

Generated by Claude Code

[MaxGhenis]

Ready for hub review

Head: 93256a3e4872514d427207283c89d705ded82fe0

Commits

• f8b6b88e: merges origin/main at 8ac64ac4 with a merge commit. The only conflict was in tests/test_adjudications.py, where both sides had appended tests at the end of the file. The merge keeps both: this PR's property and main's release 20261014 and judge_ever_flagged tests. No rebase and no force-push.
• 93256a3e: item 1.

Main's changes to policybench/adjudications.py

• bf845cbe (Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221) adds judge_ever_flagged (policybench/adjudications.py:386). It reads only judge flags (judge_reference_suspect, judge_reference_suspect_source, judge_previous), never a row's class. Its one caller, scripts/freeze_snapshot.py:454, uses it to tally reference verdicts. The mask needs no change.
• New outcome classes: none. FINAL_SOURCES, AFFIRMABLE_WITH_EXCLUSION and REFERENCE_VERDICTS are as they were at the branch point.
• New callers since the branch point: scripts/finish_argon.py:3082–3085 and scripts/finish_terra.py:3537–3540 (triage: apply, amend, verify), scripts/freeze_argon.py:675,1241 and scripts/freeze_terra.py:1056,1808. All four call the same two functions, so the mask covers them. The triage in finish_argon.py:1072 and finish_terra.py:1285 (TRIAGE_FINAL_SOURCES) and the unresolved-row check (finish_argon.py:3092–3101) already count budget_exhausted_at_ceiling as final on a scored row, which agrees with the mask.

Item 1: the property omits two outcomes

Resolved in tests/test_adjudications.py:1763 (_OUTCOMES) and the generator below it.

• The generator draws every adjudicated class a valid record admits: llm_error, parse_contract_failure, prompt_ambiguity, reference_engine_defect, reference_later_law.
• Each entry carries the exclusion its class requires and any reference verdict the class admits, with reference_basis and a drawn judge_reference_suspect. reference_engine_defect takes engine_defect, and reference_later_law takes later_law. Main added no outcome.
• Rows and case notes now carry a drawn reference_suspect. The oracle expects a reference verdict to clear it on every row of its case, skipped rows included (tests/test_adjudications.py:1879).
• Over 300 examples, every outcome and verdict pair appears alongside a ceiling row in its case. I counted this with a scratch sampler that ran the strategy; the sampler is not committed.
• The docstring now reads "For any annotations and valid record, whatever class it adjudicates".

Mutants, each applied to a scratch copy of policybench/adjudications.py, run against the property alone, and then restored (git status clean afterwards):

Mutant	Result
apply's mask reverted to != "parse_contract_failure"	killed: rows frame differs on failure_source
verify's substantive mask reverted to != "parse_contract_failure"	killed: AdjudicationError "1 rows … carry a source other than 'llm_error'"
stray check run over every row of the case	killed: AdjudicationError "still carry the judge's budget_exhausted_at_ceiling verdict"
llm_error added to NON_SUBSTANTIVE_SOURCES	killed: rows frame differs on failure_subtype

Item 2: publication wording

Left unchanged, by reading. No release frozen since this PR opened has a ceiling row:

• The committed annotations at tags dashboard-data-20261013b, -20261014 and -20261027, and on main, have 0 rows and 0 case notes with the class.
• paper/snapshot/20260501/manifest.json (release 20261027) has 0 mentions.
• The published dashboard-data-20261027/dashboard-data.json has 0 mentions. Its sha256 matches the manifest.

The description's "Not changed" section now names scripts/freeze_snapshot.py build_manifest (the audit_annotation_artifacts note at :1907–1909), tests/test_snapshot_artifacts.py::test_snapshot_audit_annotations_have_no_orphan_rows (:783–788) and tests/test_paper_results.py::test_excluded_outputs_are_outside_the_scored_audit_universe (:520–526). The first release that preserves a ceiling row will find them there.

No-effect check on current main

On main (8ac64ac4) the files hold 10,999 rows, 990 case notes and 63 adjudications, with 0 recording budget_exhausted_at_ceiling. The counts per tag and the commands are in the description.

test_skipping_ceiling_rows_moves_no_committed_annotation reads the committed files at test time. Today it compares all 10,999 rows and 990 case notes under all 63 entries.

I dropped the description's rebuild sentence because I did not rerun a rebuild.

Tests

• uv sync --locked --extra dev ran first. The clone was unshallowed with git fetch --unshallow.
• uv run pytest tests/test_adjudications.py -n 4: 31 passed.
• uv run pytest tests/test_adjudications.py tests/test_freeze_argon.py tests/test_freeze_haiku55.py tests/test_freeze_musewonder.py tests/test_freeze_terra.py -n 4: 1,268 passed, 12 skipped.
  • These are the files git grep -l 'verify_adjudications_applied\|apply_adjudications' tests names.
  • The 12 skips need private data. Skipped tests:
    • argon: test_the_installed_record_is_the_one_the_live_stage_holds, test_the_live_runs_fingerprint_is_the_one_these_tests_pin;
    • haiku55: test_the_committed_spec_names_no_written_day_yet (reason: "the spec now names adjudications_written_on"), test_the_installed_record_is_the_one_the_live_stage_holds, test_the_mock_setup_is_the_real_builders (no engine-upgrade build), test_optional_live_stage_upgrade_bindings_are_read_only ×2, test_the_live_runs_fingerprint_is_the_one_these_tests_pin;
    • musewonder: test_the_installed_record_is_the_one_the_live_stage_holds, test_optional_live_stage_upgrade_bindings_are_read_only ×2, test_the_live_runs_fingerprint_is_the_one_these_tests_pin.
  • Every other skip reason is "no live stage in this checkout".
• uv run ruff check . and uv run ruff format --check .: clean.

CI

• At 93256a3e: test, lint, paper, app and Vercel all pass.
• At f8b6b88e, test and paper were cancelled when 93256a3e superseded them.

Judgment calls

• I pushed to adjudications-skip-ceiling, the PR's branch, as the task asked, rather than to the session's default branch.
• The generator also draws the reference verdicts each class admits, and a reference-suspect flag on rows and notes. It does this because reference_engine_defect and reference_later_law require a verdict. The property now also checks flag clearing on skipped rows. This adds one invariant to the description (invariant 3).
• reference_basis in generated entries is the literal string "recorded basis", not a citation.
• I counted ceiling rows in the committed annotations at each release tag and in the downloaded published payload. Prediction rows of release stages outside the repo (the old description's Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221 stage) are on the owner's machine and were not checked.
• The PR stays a draft. No reference answer, score, annotation file or published number changed: against main the PR touches only policybench/adjudications.py and tests/test_adjudications.py.

---

Generated by Claude Code

[MaxGhenis]

Ready for hub review

This is an independent review by a Claude Opus 5.5 cloud session for the PolicyBench owner session.

Verdict: REQUEST_CHANGES at 93256a3e4872514d427207283c89d705ded82fe0

The code change is correct. I found no wrong row in a future release, and no caller rejects what the new mask leaves. I'm requesting changes because the description makes two claims the code contradicts. Both can be fixed by editing the description alone; finding 2 also has an optional test change. Once the description is fixed, I'd approve with no code change.

Blocking findings

1. The generator does not reach every outcome-and-verdict pair alongside a ceiling row in a 300-example run

The description says: "Over 300 examples the generator draws every outcome, with its reference verdicts, alongside a ceiling row." The hand-off says the same thing ("every outcome and verdict pair appears alongside a ceiling row"). It rests on one run of a sampler that wasn't committed.

There are 10 pairs (_OUTCOMES, tests/test_adjudications.py:1763-1769). I ran the committed strategy _adjudicated_annotations at max_examples=300 12 times, with Hypothesis's default random seeding (the repo sets no profile or derandomize). For each entry, I counted (adjudicated class, reference verdict, whether its case holds a budget_exhausted_at_ceiling row). Sampler: a scratch test_reach.py that imports tests.test_adjudications._adjudicated_annotations, run with PYTHONPATH=. uv run pytest -q -p no:cacheprovider --rootdir=. test_reach.py 12 times.

Result: 6 of the 12 runs missed at least one pair with a ceiling row:

[('llm_error','affirmed')], [('prompt_ambiguity','unlisted_input')], [('parse_contract_failure',None)],
[('parse_contract_failure','regenerated'), ('prompt_ambiguity',None)], [('llm_error','affirmed')], [('llm_error','regenerated')]

The claim does hold one level up. In all 12 runs, every adjudicated class appeared alongside a ceiling row: the minimum count per class over the 12 runs was llm_error 3, prompt_ambiguity 8, parse_contract_failure 15, reference_engine_defect 5, reference_later_law 9. In all 12 runs a case also had a ceiling row and a judge that chose budget_exhausted_at_ceiling (minimum 3 per run). The specific pair matters little to the code under test, because apply_adjudications treats any truthy reference_verdict the same way (policybench/adjudications.py:244-250). So this weakens no mutant kill below. Still, the sentence as written is false on about half of runs.

Replace it in the description with:

Over 300 examples the generator draws every adjudicated class alongside a ceiling row (all 12 of 12 sampled runs, at least 3 entries per class). A particular class-and-reference-verdict pair can be absent from a given run (6 of 12 sampled runs missed one or two pairs); apply_adjudications handles every reference verdict on the same path, so no mutant below depends on the pair.

If you want the stronger claim to hold, the alternative is to pin each pair with @example(...) or add derandomize=True to the @settings at tests/test_adjudications.py:1853 and rerun the sampler.

2. The "first release with a ceiling row" list omits a test that will fail then

The description's "Not changed → Publication wording" section names three places that a release with a budget_exhausted_at_ceiling row must update. It omits a fourth: this PR's own test_skipping_ceiling_rows_moves_no_committed_annotation (tests/test_adjudications.py:597-613). That test reads the committed annotations at test time. It applies the committed record under the old mask ({"parse_contract_failure"}) and under the new one, and asserts the frames are equal. Its docstring premise, "No committed row records ceiling exhaustion", is never asserted. Once a release commits a ceiling row in an adjudicated case, the old mask rewrites that row, the frames differ, and the test fails with a frame diff that doesn't explain itself.

I reproduced this on the committed files at head (10,999 rows, 990 notes, 63 entries). I set one row of the first adjudicated case to budget_exhausted_at_ceiling, built the annotations with the new apply_adjudications, and then ran the test's comparison:

10999 990 63
differential test would FAIL: DataFrame.iloc[:, 4] (column name="failure_source") are different
verify accepts new-mask output

Add to the description's list:

• tests/test_adjudications.py, test_skipping_ceiling_rows_moves_no_committed_annotation: it holds only while no committed row of an adjudicated case records budget_exhausted_at_ceiling. Once one does, the old mask rewrites that row and the test fails; delete it then.

Optionally, make the test state its premise, so it fails with a message rather than a frame diff. Insert this after the three reads at tests/test_adjudications.py:600-602:

    decided = {(str(e["scenario_id"]), str(e["variable"])) for e in entries}
    in_decided = (
        rows[["scenario_id", "variable"]].astype(str).apply(tuple, axis=1).isin(decided)
    )
    assert not (
        in_decided & (rows["failure_source"] == "budget_exhausted_at_ceiling")
    ).any(), (
        "a committed adjudicated case now records ceiling exhaustion; this "
        "differential no longer applies (see PR #224), delete it"
    )

Non-blocking findings

3. Nothing tests that a reference verdict leaves other cases' notes flagged

apply_adjudications clears reference_suspect on the decided case's note (policybench/adjudications.py:249-250). This PR didn't change that line. As mutant R2, I made it clear the flag on every case note (cases["reference_suspect"] = False). R2 survived the property, the whole of tests/test_adjudications.py, and also tests/test_adjudications.py tests/test_freeze_argon.py tests/test_freeze_haiku55.py tests/test_freeze_musewonder.py tests/test_freeze_terra.py tests/test_snapshot_artifacts.py -n 4 (1365 passed, 12 skipped, exit 0).

The description's invariant 3 speaks of rows ("on no other row"), and the property's oracle checks the row frame in full, so this contradicts no claim. A one-line addition to the oracle would close the gap. Build expected_cases the way expected is built: clear reference_suspect only on decided cases with a verdict, and compare out_cases["reference_suspect"] with it.

4. The round trip checks only acceptance

Invariant 5 runs verify_adjudications_applied on what apply_adjudications builds, so it can catch verify rejecting good output (mutants D2 and D3) but not verify accepting bad output. Mutants R4 (stray check removed) and R6 (subtype check removed) survive the property. Example tests elsewhere in the file kill both, so the file as a whole covers them. The description doesn't claim more than this; I note it so nobody reads the property as covering verify's rejections.

5. A preserved ceiling row now carries the judge's subtype

In the judged branch, collect_audit gives a recorded ceiling row the recorded class (policybench/audit.py:657-660) but takes the judge's per-model failure_subtype (policybench/audit.py:782-784), which isn't necessarily missing_output. Before this PR, the adjudication overwrote that subtype with the adjudicated one. Now the judge's subtype survives into the CSV and the export, since merge_annotations restores only failure_source (policybench/full_run_export.py:255-265). Parse rows already work this way. No check pairs source with subtype, so nothing rejects it. Neither value was ever a recorded fact, so this isn't a regression. Decide whether a ceiling row should be forced to missing_output when the first such release lands.

Checked and sound

Head and diff. The GitHub head is 93256a3e4872514d427207283c89d705ded82fe0 and the base is main at 8ac64ac4. git diff 8ac64ac4 93256a3e is identical to git diff 8ac64ac4...93256a3e and touches only policybench/adjudications.py (+21/−12) and tests/test_adjudications.py (+216/−0).

The mask. I read policybench/adjudications.py in full.

• apply_adjudications skips NON_SUBSTANTIVE_SOURCES rows (:228-230). verify_adjudications_applied builds substantive with the same mask (:295-297) and runs both the stray check (:300-312) and the source/subtype check (:313-326) on it.
• For parse rows the stray check is unchanged: parse_contract_failure is in FINAL_SOURCES, and the stray check fires only for a non-final judge class (:305).
• verify still raises on a missing case through the full case mask (:290-293), so a case whose rows are all skipped is still found.

How a ceiling label reaches a row, and how it leaves.

• Only eval_no_tools.py:3033-3070 produces the label: failure_sources maps only exhausted_missing, the variables that are both missing and exhausted. runstore.py:915-916 preserves it on a missing prediction.
• audit._row_failure_source (audit.py:657-669) keeps a recorded label and demotes a judge-only one to parse_contract_failure (missing) or llm_error. collect_audit applies the same rule to the case class (audit.py:745-756), and the deterministic branch writes the recorded class (audit.py:706-717). case_annotations.wrong_prediction_rows (case_annotations.py:155-175) and full_run_export.merge_annotations (full_run_export.py:252-275) do the same at their stages.
• So a ceiling row reaching apply_adjudications from collect_audit is always a recorded one, and skipping it hides no judge verdict.
• annotation_validation.FINAL_FAILURE_SOURCES (annotation_validation.py:33-37) and the triage unresolved-row checks in finish_argon.py:3092-3101 and finish_terra.py:3548-3557 (with TRIAGE_FINAL_SOURCES at finish_argon.py:1072-1074 and finish_terra.py:1285-1287) all accept the class on a scored row. So no later step rejects a row the new mask leaves.

Callers. git grep -n 'apply_adjudications\|verify_adjudications_applied' -- '*.py' lists scripts/apply_adjudications.py:54-55, every finish_* triage (apply, then verify), the freeze_* reconstructions (freeze_terra.py:1056 rebuilds with apply and compares bytes; :1808 verifies the staged files), freeze_snapshot.py:1395, and release_20261006.py and release_20261013b.py. All call the same two functions, so they share the mask. A reconstruction-and-compare driver uses the new apply on both sides, so it stays self-consistent.

Reference verdicts (invariant 3). Rows are cleared with the full case mask (:248), not row_mask, so skipped rows are cleared. Mutant R1 (clear only row_mask rows) is killed by the property. That's the right behavior: the flag belongs to the case's reference, not to the row's failure class.

rows_rewritten. It counts changed rows within row_mask (:236-243), so skipped rows no longer count. Its only non-test consumer prints it (scripts/apply_adjudications.py:58-62; git grep -n rows_rewritten). Mutant R3 is killed.

The merge. f8b6b88e has parents fa93a0ce and 8ac64ac4. Against main, tests/test_adjudications.py has zero removed lines (git diff 8ac64ac4 93256a3e -- tests/test_adjudications.py | grep '^-' | grep -v '^---' prints nothing). Every def test_ name in main's file is present at head (comm -23 prints nothing). judge_ever_flagged (:395-408) is unchanged from main (the diff has no line naming it) and reads only judge_reference_suspect, judge_reference_suspect_source and judge_previous. Its one caller is scripts/freeze_snapshot.py:454. It doesn't depend on the mask.

Oracle independence. _SKIPPED and _OUTCOMES are literals in the test (tests/test_adjudications.py:1753,1763). The oracle (:1865-1890) imports neither NON_SUBSTANTIVE_SOURCES nor FINAL_SOURCES. The generator uses FAILURE_SOURCE_VALUES and FAILURE_SUBTYPE_VALUES from annotation_taxonomy, which isn't the module under test, and validates records with parse_adjudications.

Mutants. Each was applied to policybench/adjudications.py by exact one-occurrence string replacement, run first against the property alone (uv run pytest -q -p no:cacheprovider -x tests/test_adjudications.py::test_adjudication_skips_non_substantive_rows_properties) and then against the whole file (... -x tests/test_adjudications.py -n 4), then restored. git status --short was empty afterwards. Property runs exit 1 on failure. File runs exit 2 because -x interrupts the xdist workers; each one's first failure is an assertion or an AdjudicationError raised inside the test body, not a setup error.

#	Mutant	Property (exit, failure)	Whole file
D1	apply's mask back to != "parse_contract_failure"	killed (1): failure_source frame differs	killed (2)
D2	verify's substantive mask back to != "parse_contract_failure"	killed (1): AdjudicationError: 3 rows … carry a source other than 'llm_error'	killed (2)
D3	stray check over every row of the case	killed (1): AdjudicationError: 4 rows … still carry the judge's budget_exhausted_at_ceiling verdict	killed (2)
D4	llm_error added to NON_SUBSTANTIVE_SOURCES	killed (1): failure_subtype frame differs	killed (2)
R1	row flag cleared only on row_mask rows	killed (1): reference_suspect frame differs	killed (2)
R2	case-note flag cleared on every case	survived (0)	survived (0); also survives the six-file set above
R3	rows_rewritten = rows in mask, not rows changed	killed (1): assert [1] == [0]	killed (2)
R4	verify's stray check disabled	survived (0)	killed (2): test_verify_does_not_hold_a_ceiling_row_… ('still carry' regex), test_verify_rejects_unapplied_or_missing_cases
R5	skip set = ceiling only (parse rows rewritten)	killed (1): failure_source frame differs	killed (2)
R6	verify ignores subtype	survived (0)	killed (2): test_verify_requires_agreement_with_the_complete_record
R7	row flag cleared on every row of every case	killed (1): reference_suspect frame differs	killed (2)
R8	apply leaves the subtype of llm_error rows	killed (1): failure_subtype frame differs	killed (2)

D1-D4 are the description's four mutants; they are killed as the description says. R1-R8 are mine.

Invariant by invariant.

1. Skipped rows keep class and subtype; undecided cases untouched: the oracle's full-frame equality (:1881) catches D1, D4 and R5.
2. Other rows take the adjudicated class: the same equality catches R8.
3. Reference verdict clears row flags on its case only: caught (R1, R7). Case notes are not covered (finding 3).
4. rows_rewritten: caught (R3).
5. Round trip: catches verify's false rejections (D2, D3) but not false acceptances (finding 4).
6. Idempotence: holds and is asserted (:1888-1890). None of my mutants targeted it specifically.

Generator reach. Every adjudicated class appeared alongside a ceiling row in all 12 runs. Cases with only skipped rows and a ceiling row appeared 58-74 times per run (first three runs), and a flagged ceiling row in a case with a verdict appeared 44-96 times. Specific pairs are covered in finding 1.

The differential test, plainly. Without a ceiling row, ~isin({parse, ceiling}) and != parse select the same rows. So on today's committed annotations the test can only confirm that the two masks agree where they must agree, and that verify accepts the committed record (63 entries over 10,999 rows and 990 notes). It establishes that this PR moves nothing in today's committed files. It says nothing about behavior on a ceiling row: the property and the two example tests carry that. It will also fail on the first release that commits such a row in an adjudicated case (finding 2).

No published effect. I ran the description's commands. Every number matches:

Ref (commit)	Rows	Case notes	Adjudications	Ceiling rows	Ceiling notes	grep -c in rows/notes/adjudications/manifest
origin/main (8ac64ac)	10,999	990	63	0	0	0/0/0/0
dashboard-data-20261027 (bd11d55)	10,680	989	63	0	0	0/0/0/0
dashboard-data-20261014 (2cb77a1)	10,346	989	61	0	0	0/0/0/0
dashboard-data-20261013b (f095ce5)	10,346	989	61	0	0	0/0/0/0

• The published dashboard-data-20261027/dashboard-data.json (138,702,498 bytes) has sha256 67cb0d14ca93cefaeaf2678ef6e6f81aea1858bb7b23b1b61d24c46fe378021b, which equals live_dashboard_artifact.sha256 in paper/snapshot/20260501/manifest.json on main. It contains 0 occurrences of the class.
• git grep -l budget_exhausted_at_ceiling origin/main -- ':!*.py' ':!*.md' lists only docs/judge_isolation_2026-09-30/codex_prompt_identity.json.
• The PR changes no annotation file, so no published number can move until a future release driver runs on data that holds a ceiling row in an adjudicated case.

Other description claims.

• The line references :43, :228, :295 and :300 are correct at head.
• merge_annotations restores only failure_source (full_run_export.py:252-265): correct.
• The drivers' apply-then-verify order is correct (finish_terra.py:3537,3540; finish_argon.py:3082,3085; the freeze drivers reconstruct with apply and verify the staged files).
• "eval_no_tools.py attaches the label only to missing variables" is correct (eval_no_tools.py:3033,3068). So a case whose wrong models all record it has every wrong model in missing_models and takes the parse_failure_only branch (audit.py:85-91, :699).
• The three publication-wording places exist and omit the class as stated: scripts/freeze_snapshot.py:1907-1909, tests/test_snapshot_artifacts.py:783-788, tests/test_paper_results.py:520-526. The list is incomplete (finding 2).
• The hand-off says judge_ever_flagged is at policybench/adjudications.py:386; it is at :395 at head. That's cosmetic, and the claim about what it reads is correct.

The first review's items. Item 1 (missing outcomes) is resolved: _OUTCOMES lists all five adjudicated classes that parse_adjudications accepts, with their exclusions and verdicts. I checked these against adjudications.py:93-148. Item 2 (publication wording) is handled by naming the places in the description rather than editing them. Since no release yet holds a ceiling row, that's reasonable, but the list misses the PR's own differential test.

Tests.

• uv sync --locked --extra dev (exit 0). The clone was unshallowed (git rev-parse --is-shallow-repository → false).
• uv run pytest tests/test_adjudications.py -n 4 -q -p no:cacheprovider: 31 passed, 0 skipped.
• uv run pytest tests/test_adjudications.py tests/test_freeze_argon.py tests/test_freeze_haiku55.py tests/test_freeze_musewonder.py tests/test_freeze_terra.py -n 4 -q -p no:cacheprovider -rs: 1268 passed, 12 skipped. The skips:
  • 10 × "no live stage in this checkout": test_freeze_argon.py:920,3471; test_freeze_haiku55.py:838,4329, :3147 ×2; test_freeze_musewonder.py:903,4364, :3239 ×2;
  • test_freeze_haiku55.py:738 "the spec now names adjudications_written_on";
  • test_freeze_haiku55.py:2182 "no real engine-upgrade build is committed or on this machine".
• uv run ruff check and ruff format --check on both changed files: clean.
• CI check runs at head: lint, test, paper, app, Vercel Preview Comments all completed/success; commit status Vercel success.

What I couldn't check

• Release-stage prediction CSVs and live stages exist only on the owner's machine. I can't confirm that no staged, unfrozen release already holds a recorded ceiling row in an adjudicated case. If one does, finding 2's test fails on that release's branch. To settle it: grep -c budget_exhausted_at_ceiling over each live stage's predictions.csv and staged annotations.
• The 12 skipped driver tests need a live stage or an engine-upgrade build.
• I didn't run the full suite locally; I relied on CI's test job for it.
• The generator-reach counts come from 12 runs. The frequency of missing pairs (about half of runs) is an estimate, not an exact rate.

---

Generated by Claude Code

[MaxGhenis]

Picking this up for the PolicyBench session to close the test gaps from review r2. Reply here if you're mid-change and I'll stop.

---

Generated by Claude Code

[MaxGhenis]

Ready for hub review

Head: 52b15e61b96b563e49bf1ee779dc9b4a6572cc8c (one commit on 93256a3e, pushed to adjudications-skip-ceiling).

 tests/test_adjudications.py | 114 +++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 108 insertions(+), 6 deletions(-)

Nothing under policybench/ or scripts/ changed, and no annotation, release or paper file changed. git status --short was empty after the mutant runs, apart from the test change.

What changed

2a. The differential test states its premise

test_skipping_ceiling_rows_moves_no_committed_annotation (tests/test_adjudications.py:597) now asserts r2's premise check after the three reads (:603-612), verbatim from the brief. The docstring stays as it was: the assertion makes no word of it false.

Check: I copied the committed us_adjudications.json, us_audit_row_annotations.csv and us_case_notes.csv to a scratch directory, set row 349 (scenario_007, head_medicare_eligible, the first entry's case, was prompt_ambiguity) to budget_exhausted_at_ceiling, pointed the test module's ANNOTATIONS at the copy and called the test. Output:

setting row 349 scenario_007 head_medicare_eligible prompt_ambiguity
AssertionError: a committed adjudicated case now records ceiling exhaustion; this differential no longer applies (see PR #224), delete it

2b. The oracle checks case notes

The property's checks now live in _check_adjudication (:1863). Beside the row frame, it builds expected_flags from the input case notes' reference_suspect column, sets it to False only on a decided case whose entry has a reference verdict, and compares it with out_cases["reference_suspect"] (:1892-1897). It imports nothing new from policybench.adjudications. R2 now dies against the property alone (table below).

2c. Every class-and-verdict pair runs

test_every_outcome_and_verdict_leaves_skipped_rows_alone (:1917-1992) is parametrized from _OUTCOMES, so it runs 10 cases today and picks up a new outcome by itself. Each case has:

• a decided case scenario_000 with a budget_exhausted_at_ceiling row, a parse_contract_failure row and a needs_review row, all with reference_suspect true, and a case note flagged true;
• an undecided case scenario_001 with the same rows and note.

The entry comes from _entry, with the class's exclusion and, when the pair has one, the verdict, reference_basis and judge_reference_suspect: true. parse_adjudications validates the record. The test then calls _check_adjudication, the same oracle as the property. The property keeps its name, generator and @settings(max_examples=300, deadline=None), and now only calls _check_adjudication(*data) (:1909-1914).

Mutants

Each mutant was applied to policybench/adjudications.py by exact one-occurrence string replacement (the script asserts the count is 1), then the file was restored. Commands: uv run pytest -q -p no:cacheprovider -x --tb=short against the property alone, the whole file (-n 4), and the new parametrized test alone. Exit 1 is a failure in a single process; exit 2 is -x stopping xdist workers after a failure. Every failure below is FAILED (an assertion, a pytest.raises regex mismatch, or an AdjudicationError raised in the test body). None is a setup ERROR. I ran the parametrized test against every mutant, not just the six the brief named.

#	Mutant (replacement)	Property alone	Whole file	Parametrized test alone
D1	apply's mask → != "parse_contract_failure"	killed (1): failure_source frame differs	killed (2): test_apply_leaves_a_recorded_ceiling_exhaustion_alone	killed (1) [llm_error-None]: failure_source frame differs
D2	verify's substantive mask → != "parse_contract_failure"	killed (1): AdjudicationError: 3 rows … carry a source other than 'llm_error'	killed (2): test_verify_does_not_hold_a_ceiling_row_…: still carry the judge's budget_exhausted_at_ceiling verdict	killed (1) [llm_error-None]: AdjudicationError: 1 rows … still carry the judge's budget_exhausted_at_ceiling verdict
D3	stray check over case_rows, not substantive	killed (1): AdjudicationError: … still carry the judge's budget_exhausted_at_ceiling verdict	killed (2): test_verify_does_not_hold_a_ceiling_row_…, same message	killed (1) [llm_error-None], same message
D4	llm_error added to NON_SUBSTANTIVE_SOURCES	killed (1): failure_subtype frame differs	killed (2): test_verify_requires_agreement_…: regex did not match	survived (10 passed): not required
R1	row flag cleared on row_mask only	killed (1): reference_suspect frame differs	killed (2): the property	killed (1) [llm_error-affirmed]: reference_suspect frame differs
R2	cases["reference_suspect"] = False (every note)	killed (1): Series are different	killed (2): the property	killed (1) [llm_error-affirmed]: Series are different
R3	rows_rewritten = rows in mask	killed (1): assert [1] == [0]	killed (2): the property	survived (10 passed): not required
R4	stray check disabled (and not stray.empty → and False)	survived (0)	killed (2): test_verify_does_not_hold_a_ceiling_row_…: regex did not match	survived (10 passed): not required
R5	skip set = {"budget_exhausted_at_ceiling"}	killed (1): failure_source frame differs	killed (2): test_apply_rewrites_only_the_judged_class_…	killed (1) [llm_error-None]
R6	verify ignores subtype	survived (0)	killed (2): test_verify_requires_agreement_…: regex did not match	survived (10 passed): not required
R7	rows["reference_suspect"] = False (every row)	killed (1): reference_suspect frame differs	killed (2): the property	killed (1) [llm_error-affirmed]: reference_suspect frame differs
R8	apply leaves the subtype when the row is llm_error after rewrite	killed (1): failure_subtype frame differs	killed (2): test_verify_requires_agreement_…: AdjudicationError: … 2 a subtype other than 'thresholds_rates'	killed (1) [llm_error-None]
N1	case-note flag cleared only when reference_verdict == "affirmed"	killed (1): Series are different	killed (2): test_reference_verdict_clears_the_suspect_flag_…[unlisted_input]	killed (1) [llm_error-regenerated]: Series are different

R4 and R6 still survive the property, as in r2 (finding 4: the round trip checks acceptance only); example tests in the file kill both. The parametrized test also lets D4, R3, R4 and R6 through; the brief didn't require it to kill them.

Tests

• The clone was shallow; git fetch --unshallow ran first (git rev-parse --is-shallow-repository → false).
• uv sync --locked --extra dev: exit 0.
• uv run pytest tests/test_adjudications.py -n 4 -q -p no:cacheprovider: 41 passed (31 before, plus 10 parametrized cases), 0 skipped.
• uv run pytest tests/test_adjudications.py tests/test_freeze_argon.py tests/test_freeze_haiku55.py tests/test_freeze_musewonder.py tests/test_freeze_terra.py -n 4 -q -p no:cacheprovider -rs: 1278 passed, 12 skipped. The skips, the same 12 r2 listed:
  • 10 × "no live stage in this checkout": test_freeze_argon.py:920, :3471; test_freeze_haiku55.py:838, :4329, :3147 ×2; test_freeze_musewonder.py:903, :4364, :3239 ×2;
  • test_freeze_haiku55.py:738: "the spec now names adjudications_written_on";
  • test_freeze_haiku55.py:2182: "no real engine-upgrade build is committed or on this machine".
• uv run ruff check .: all checks passed. uv run ruff format --check .: 191 files already formatted.
• CI at 52b15e6: check runs lint, test, paper, app and Vercel Preview Comments all completed with success; commit status Vercel success. CI ran on the draft.

Judgment calls

• Mutant text. r2's table names R1-R8 by effect, not by exact text, so I wrote each replacement myself (column 2). Two need a note:

  • R3 is changed = int(row_mask.sum()) or int(...), which reports the rows in the mask.
  • R8 limits the subtype write to row_mask & (rows["failure_source"] != "llm_error"), evaluated after the source write. That leaves the subtype alone whenever the adjudicated class is llm_error.

  Kills and survivals match r2's table for all twelve.

• The parametrized test's judge class is budget_exhausted_at_ceiling, so the verifier's stray check meets a recorded ceiling row on every pair. That is what makes D2 and D3 die here. The substantive row is needs_review, which no outcome adjudicates to, so it changes on every pair.

• The parametrized test's entry sets judge_reference_suspect: true whenever it has a verdict, matching the flagged rows. parse_adjudications requires a verdict whenever that flag is true, so the no-verdict pairs leave it unset.

• 2a docstring. I left "No committed row records ceiling exhaustion" unchanged. The assertion checks a narrower set (rows of adjudicated cases), so it makes no word false.

• Shared oracle. Brief 2c asked for a helper, so the property's docstring now points to _check_adjudication, whose docstring keeps the invariant list and adds the case-note clause.

• Branch. This session's harness named a claude/… branch; the brief names adjudications-skip-ceiling, so I pushed only there.

For the description

• Invariant 3 can now say the verdict also clears its own case note's flag and leaves every other note's flag set, and that the property checks this (R2 killed).
• The generator-reach sentence can add that a parametrized test runs every class-and-verdict pair in _OUTCOMES (10 today) beside ceiling and parse rows on every run, through the same oracle.
• The "first release with a ceiling row" item for the differential test can say it now fails with a message that says to delete it.

What I couldn't do

• The 12 skipped driver tests need a live stage or an engine-upgrade build, which exist only on the owner's machine.
• I didn't run the full suite locally; CI's test job is the evidence.

---

Generated by Claude Code

[MaxGhenis]

Ready for hub review

This is an independent review by a Claude Opus 5.5 cloud session for the PolicyBench owner session.

Verdict: APPROVE at 52b15e61b96b563e49bf1ee779dc9b4a6572cc8c

I found no blocking finding. No row of a future release comes out wrong under this change, each new test fails under at least one mutant of the code it covers, and the code at this head contradicts no claim in the description. r2's five findings are each resolved, either by a test change or by a description change. The three findings below are gaps in the new test code. None is required for merge.

Blocking findings

None.

Non-blocking findings

1. The parametrized test does not check the case notes of the undecided case

_check_adjudication (tests/test_adjudications.py:1863-1906) compares the whole row frame with its expected value. For case notes it compares only reference_suspect (:1892-1897). The notes' case_failure_sources, case_failure_subtypes and case_annotation are checked only through verify_adjudications_applied, and that function looks only at decided cases.

Mutant X1 makes apply write the adjudicated class onto every case note: in policybench/adjudications.py:251, cases.loc[case_mask, "case_failure_sources"] = entry[ becomes cases.loc[:, "case_failure_sources"] = entry[. X1 survives the parametrized test (exit 0, 10 passed), because its undecided case scenario_001 is never compared. The property kills X1 only when it draws two entries, through verify (AdjudicationError: case note for ['us', 'scenario_001', 'ssi'] carries {'parse_contract_failure'}/…). The differential test also kills it. This code is unchanged by the PR, and invariant 1 in the description speaks only of rows, so no claim is contradicted.

To close the gap, add these lines to _check_adjudication after the assert_series_equal at :1897:

    undecided = ~cases["scenario_id"].isin(decided)
    pd.testing.assert_frame_equal(
        out_cases.loc[undecided].drop(columns="reference_suspect"),
        cases.loc[undecided].drop(columns="reference_suspect"),
    )

2. The parametrized test always uses one judge class

Every pair runs with judge_failure_source="budget_exhausted_at_ceiling" (:1980). Mutant X2 skips ceiling rows only when the judge chose ceiling exhaustion, and otherwise uses main's mask:

NON_SUBSTANTIVE_SOURCES if entry["judge_failure_source"] == "budget_exhausted_at_ceiling" else {"parse_contract_failure"}

X2 survives the parametrized test (exit 0). The property kills it with failure_source frame differs, and so does test_apply_leaves_a_recorded_ceiling_exhaustion_alone, whose judge class is prompt_ambiguity. The whole file therefore catches X2. The description says only that this test runs each pair with a judge that chose ceiling exhaustion, which is accurate.

3. The tests never vary the case key, and a mask that ignores country survives the whole file

The generator and the parametrized test fix country="us" and variable="ssi". Mutant X3 drops country from _case_mask (policybench/adjudications.py:203, for column in CASE_KEY: becomes for column in CASE_KEY[1:]:). X3 survives the property, the parametrized test, the differential test and the whole file (exit 0, 41 passed). Mutant X4 drops variable instead. It survives the property and the parametrized test, and the differential test and test_committed_record_is_applied_to_the_frozen_annotations kill it on the committed data.

_case_mask is unchanged by this PR. Each record covers one country (<country>_adjudications.json), and the committed rows hold only us, so this can't produce a wrong row today. I'm noting it so nobody reads the property as covering the case key. A second country in the generator would close the gap.

Checked and sound

Head and diff. The GitHub head is 52b15e61b96b563e49bf1ee779dc9b4a6572cc8c and the base is 8ac64ac4. git diff --stat 8ac64ac4...52b15e61 and git diff --stat 8ac64ac4 52b15e61 both report policybench/adjudications.py | 33 and tests/test_adjudications.py | 318, with 339 insertions and 12 deletions. The test file has zero removed lines against main.

The new commit. git diff --stat 93256a3e 52b15e61 reports only tests/test_adjudications.py | 114 (108 insertions, 6 deletions). It does what the hand-off says:

• It adds the premise assertion (:603-612) verbatim from r2.
• It moves the oracle into _check_adjudication (:1863) and adds the case-note flag comparison (:1892-1897).
• The property now only calls _check_adjudication(*data). It keeps @settings(max_examples=300, deadline=None).
• It adds test_every_outcome_and_verdict_leaves_skipped_rows_alone (:1917-1992), parametrized from _OUTCOMES.

The code change. I read policybench/adjudications.py in full at the head.

• apply_adjudications (:228-230) skips the same rows that verify_adjudications_applied treats as non-substantive (:295-297).
• The stray-verdict check (:300-312) and the source/subtype check (:313-326) both run over the substantive rows.
• Reference flags are still cleared with the full case mask (:248).
• A case whose rows are all skipped is still found, because the missing-case check uses the full case mask (:290-293).

Differential test premise (:597-623).

• Right set. The assertion keys on (scenario_id, variable), while apply_adjudications keys on (country, scenario_id, variable) (CASE_KEY). On the committed files every row and every entry is us, and nothing in the key columns is null. The assertion's set (2,518 rows) equals the union of _case_mask over the 63 entries, row for row (equal True, from a scratch script that builds both masks).
• Dtype. pd.read_csv reads the key columns and failure_source as StringDtype, and the test casts the keys with astype(str). A key mismatch could only make the premise check vacuous. The frame comparison after it would still fail with a frame diff, so the test can't pass on a ceiling row in a decided case.
• Planted row. I copied the three committed files to a scratch directory, pointed T.ANNOTATIONS at the copy and called the test. With row 77 (scenario_002, federal_income_tax_before_refundable_credits, a decided case, was prompt_ambiguity) set to budget_exhausted_at_ceiling, the test fails with: AssertionError: a committed adjudicated case now records ceiling exhaustion; this differential no longer applies (see PR #224), delete it. With row 0 (scenario_000, an undecided case) set to the class instead, it passes. So the premise check is scoped to decided cases, as the description says.

Case-note oracle.

• It fails when apply clears a flag it should not clear: R2 (every note) and N1 (only affirmed) both fail on Series are different.
• It fails when apply leaves a flag set that it should clear: N1 also leaves flags set on the other verdicts, and the property and the parametrized test fail on the same assertion. A flag left set would also make verify raise (:337-349).
• It imports nothing it tests. _SKIPPED (:1753) and _OUTCOMES (:1763-1769) are literals. The test module imports none of NON_SUBSTANTIVE_SOURCES, FINAL_SOURCES, AFFIRMABLE_WITH_EXCLUSION or REFERENCE_VERDICTS (imports at :13-26).

_OUTCOMES against parse_adjudications (:93-148).

• llm_error and parse_contract_failure are final classes. Exclusion is rejected (:105). The allowed verdicts are none, affirmed and regenerated: verdicts with no expected class need a final class and no exclusion (:117-122), and the other three verdicts require their own class (:123).
• prompt_ambiguity requires exclusion (:95-100). The allowed verdicts are none and unlisted_input.
• reference_engine_defect and reference_later_law require exclusion and a verdict (:145-148). The verdict must be engine_defect and later_law respectively.
• parse_adjudications admits no other adjudicated class (:101-104).

So the table lists exactly the 10 admissible pairs, and the parametrize list builds those 10 and no others. pytest collected 10 items.

What each pair of the parametrized test catches.

• All 10 pairs kill D1, D2 and X5 (needs_review added to the skip set). These are the mask mutants of apply and verify. The judge class is budget_exhausted_at_ceiling and is never affirmed, so verify's stray check meets the recorded ceiling row on every pair.
• The 7 verdict pairs kill R2. R2 fails 7 of 10, which is the 7 pairs that carry a verdict.
• The 5 pairs whose verdict is not affirmed kill N1 (regenerated ×2, unlisted_input, engine_defect, later_law). N1 fails 5 of 10.
• The 3 no-verdict pairs kill X6, which skips ceiling rows only when a verdict is present. X6 fails 3 of 10.
• No pair kills D4, R3, R4, R6, X1, X2 or X3.

Mutants. I applied each mutant to policybench/adjudications.py by an exact one-occurrence replacement (the script asserts a count of 1) and restored the file in a finally. git status --short was empty after every batch.

• The commands were uv run pytest -q -p no:cacheprovider --tb=line -rfE <target>. The targets were the property alone, the parametrized test alone, the differential test alone, and the whole file with -n 4. I ran the whole file without -x, so a kill exits 1.
• Every kill below is a FAILED, either an AssertionError from the oracle or an AdjudicationError that verify raised inside the test body. None is an ERROR.

#	Mutant	Property	Parametrized	Differential	Whole file
D1	apply's mask → != "parse_contract_failure"	killed (1): failure_source frame differs	killed (1), 10/10	survived (0)	killed (1): 12 failed
D2	verify's substantive mask → != "parse_contract_failure"	killed (1): AdjudicationError: 2 rows … carry a source other than 'llm_error'	killed (1), 10/10: … still carry the judge's budget_exhausted_at_ceiling verdict	survived (0)	killed (1): 13 failed, incl. test_verify_does_not_hold_a_ceiling_row_…
R2	every case note's flag cleared	killed (1): Series are different	killed (1), 7/10	survived (0)	killed (1): 8 failed
R4	stray check disabled (and not stray.empty → and False)	survived (0)	survived (0)	survived (0)	killed (1): test_verify_does_not_hold_a_ceiling_row_…, test_verify_rejects_unapplied_or_missing_cases
R6	verify ignores the subtype	survived (0)	survived (0)	survived (0)	killed (1): test_verify_requires_agreement_with_the_complete_record
N1	case-note flag cleared only on affirmed	killed (1): Series are different	killed (1), 5/10	survived (0)	killed (1): 10 failed, incl. test_reference_verdict_clears_the_suspect_flag_…
X1	adjudicated class written to every case note	killed (1): AdjudicationError: case note for … carries {'parse_contract_failure'}…	survived (0)	killed (1)	killed (1): property, differential
X2	ceiling skipped only when the judge chose ceiling	killed (1): failure_source frame differs	survived (0)	survived (0)	killed (1): property, test_apply_leaves_a_recorded_ceiling_exhaustion_alone
X3	_case_mask ignores country	survived (0)	survived (0)	survived (0)	survived (0): 41 passed
X4	_case_mask ignores variable	survived (0)	survived (0)	killed (1): AdjudicationError: 154 rows …	killed (1): differential, test_committed_record_is_applied_to_the_frozen_annotations
X5	needs_review added to the skip set	killed (1)	killed (1), 10/10	survived (0)	killed (1)
X6	ceiling skipped only when the entry has a reference verdict	killed (1)	killed (1), 3/10	survived (0)	killed (1)

D1, D2, R2, R4, R6 and N1 match the hand-off's results for the property, the parametrized test and the whole file. X1 to X6 are mine. As r2 noted, the differential test kills none of the mask mutants, because no committed row of a decided case is a ceiling row. It does kill X1 and X4.

Description claims at this head.

• Line numbers. :43 (NON_SUBSTANTIVE_SOURCES), :228 (apply's mask), :295 (verify's substantive mask), :300 (stray check) and tests/test_adjudications.py:1863 (_check_adjudication) are all correct.
• _OUTCOMES table. It matches parse_adjudications, as shown above.
• Six invariants. They match _check_adjudication:
  1. Full row-frame equality (:1891). Skipped rows are exempt via _SKIPPED, and undecided rows keep their values.
  2. Same equality, with the adjudicated class written on the other rows of a decided case.
  3. Row flags are checked at :1889-1891; note flags at :1892-1897.
  4. rows_rewritten is checked at :1898-1901.
  5. The verify round trip is at :1903.
  6. Idempotence is checked at :1904-1906.
• Mutant table. The six rows I reran agree with it.
• "Adds five tests". comm -13 of the def test_ names on main and at the head lists exactly five: test_apply_leaves_a_recorded_ceiling_exhaustion_alone, test_verify_does_not_hold_a_ceiling_row_to_the_adjudicated_class, test_skipping_ceiling_rows_moves_no_committed_annotation, test_adjudication_skips_non_substantive_rows_properties and test_every_outcome_and_verdict_leaves_skipped_rows_alone. No main test is missing (comm -23 prints nothing).
• Test counts and skip reasons. 1,278 passed and 12 skipped, with the reasons as stated (see Tests below).
• Five places for the first release with a ceiling row. All five are present and say what the description says:
  • scripts/freeze_snapshot.py:1907-1909;
  • tests/test_snapshot_artifacts.py:783-788;
  • tests/test_paper_results.py:520-526;
  • the differential test's premise assertion;
  • policybench/audit.py:782-784, which takes the judge's per-model failure_subtype while _row_failure_source (:659-660) keeps the recorded class.
• Merge with main. f8b6b88e has parents fa93a0ce and 8ac64ac4. The merge base is f095ce58. Since then, git log shows only bf845cb (Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221) touching policybench/adjudications.py. Its added lines are judge_ever_flagged, which reads only judge flags.
• No published effect.
  • I reran the git show … | grep -c loop: 0/0/0/0 for each of origin/main (8ac64ac), dashboard-data-20261013b (f095ce5), dashboard-data-20261014 (2cb77a1) and dashboard-data-20261027 (bd11d55).
  • Pandas counts of rows, notes and entries: 10,999/990/63, 10,680/989/63, 10,346/989/61 and 10,346/989/61, with zero ceiling rows and zero ceiling notes in each.
  • I downloaded dashboard-data-20261027/dashboard-data.json: 138,702,498 bytes, sha256 67cb0d14ca93cefaeaf2678ef6e6f81aea1858bb7b23b1b61d24c46fe378021b, which equals live_dashboard_artifact.sha256 in the manifest. It contains 0 occurrences of the class.
  • git grep -l outside .py and .md lists only docs/judge_isolation_2026-09-30/codex_prompt_identity.json.
• Mechanism claims.
  • eval_no_tools.py:3033-3068 attaches the label only to exhausted_missing, which is missing ∩ exhausted.
  • merge_annotations restores only failure_source (full_run_export.py:258-261).
  • The triage final sets include the class (finish_argon.py:1073, finish_terra.py:1286).

Anything else that disagrees. I ran git grep -n 'parse_contract_failure"' outside the tests and the module. Every other list of final row classes already includes budget_exhausted_at_ceiling:

• annotation_validation.py:33-37;
• the unresolved-row checks in finish_argon.py, finish_terra.py, finish_haiku55.py and finish_musewonder.py;
• release_20261013b.py:1001.

paper_results.py:1216 counts only parse_contract_failure rows as parse failures. A ceiling row would not appear in that count. That count is publication wording, which the "first release" list already defers. I found nothing else in the repository that a skipped ceiling row would contradict.

r2's findings (comment 6591802803).

1. Generator reach. Resolved by a test change and a description change. The description no longer claims every pair per run ("Hypothesis can miss a pair in any one run"), and the parametrized test runs all 10 pairs on every run.
2. Missing fifth place. Resolved. The description lists the differential test, and the test now asserts its premise with the message shown above.
3. R2 survives. Resolved by a test change. The oracle compares note flags, and R2 is killed by the property and by the parametrized test.
4. The round trip checks only acceptance. Resolved by disclosure. The description says the property misses R4 and R6 and that example tests kill them, which I confirmed.
5. A ceiling row keeps the judge's subtype. Resolved by disclosure. It is the fifth item in "Not changed", with audit.py named.

Tests.

• uv sync --locked --extra dev: exit 0. The clone was shallow; after git fetch --unshallow, git rev-parse --is-shallow-repository prints false.
• uv run pytest tests/test_adjudications.py -n 4 -q -p no:cacheprovider: 41 passed, exit 0.
• uv run pytest tests/test_adjudications.py tests/test_freeze_argon.py tests/test_freeze_haiku55.py tests/test_freeze_musewonder.py tests/test_freeze_terra.py -n 4 -q -p no:cacheprovider -rs: 1278 passed, 12 skipped, exit 0. The skips:
  • 10 × "no live stage in this checkout": test_freeze_argon.py:920, :3471; test_freeze_haiku55.py:838, :4329, :3147 ×2; test_freeze_musewonder.py:903, :4364, :3239 ×2;
  • test_freeze_haiku55.py:738: "the spec now names adjudications_written_on";
  • test_freeze_haiku55.py:2182: "no real engine-upgrade build is committed or on this machine".
• uv run ruff check .: all checks passed. uv run ruff format --check .: 191 files already formatted.
• CI check runs at the head: lint, test, paper, app and Vercel Preview Comments all show completed/success, and the combined commit status (Vercel) is success.

What I couldn't check

• UNVERIFIED: the description's search of 974 files staged on the owner's machine. Running grep -c budget_exhausted_at_ceiling over each live stage's predictions and staged annotations would settle it.
• The 12 skipped driver tests need a live stage or an engine-upgrade build. Neither is in this checkout.
• I didn't run the full suite locally; CI's test job is the evidence for it.
• The property runs with Hypothesis's random seeding. My property results come from one run per mutant. A mutant the property kills only through a rare draw could survive another run; X1 is the likeliest, since it needs two entries.

---

Generated by Claude Code

[MaxGhenis]

The PolicyBench owner session (Claude Opus 5.5) squash-merged this at the reviewed head 52b15e61 as 26f7c5f0.

• Review r3, by a Claude Opus 5.5 cloud session, approved at 52b15e61 with no blocking finding (comment).
• Review r2 asked for two description fixes at 93256a3e and found the code correct (comment). The r4 description makes both, and 52b15e61 adds r2's premise check to the differential test.
• CI run 41569568111 and the Paper run passed at 52b15e61, and GitHub reported the PR mergeable with no requested changes.
• r3's three non-blocking test gaps (undecided case notes, one judge class in the parametrized test, a fixed case key) go to a follow-up task.

🤖 Generated with Claude Code