[MaxGhenis]

Summary

This fixes the one open finding from #200's round-5 review. reference_audit/2026-10-05-reference-adversary/scripts/leaderboard_impact.py read the working tree's run. The pass actually ran on release dashboard-data-20260930's run, as 8b4c0ca1 committed it. So regenerating the impact evidence after #202 silently scored #202's payload and exclusions. The README's Reproduce step restored only the payload and annotations, not the other scoring inputs.

The PR changes no evidence and no published number. No verification/ file, proposed_changes.json, reference, exclusion or score changes. The diff touches the impact script, the audit README and one new test file.

The bug, reproduced

I ran the unmodified script from a clean checkout of main. Main's working tree holds #202's payload b1da3eae… and exclusions 92741dfd….

• The script exited 0 and rewrote 19 of the 23 committed verification/leaderboard_impact* files.
• scored_outputs.published went from 1928 to 1920, the leader's exact rate from 95.0035 to 95.8279, and the top-5 order changed.
• Its own check that the unchanged copy reproduces the published payload passed, because Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's payload and exclusions agree with each other.

The fix

leaderboard_impact.py

• git_input() writes git show <commit>:<path> to scratch, and stops (SystemExit) unless the bytes match the pinned sha256.
• pass_inputs() stages:
  • all seven run files from 8b4c0ca1, including the payload 1e029aaa…;
  • proposed_changes.json from 4db91b5f (Add the reference adversary and record its first run (evidence only; no score changes) #200), sha256 3a6e5920….
• main() stages these before anything else. The variant copies, the regeneration's line check and the published-reproduction comparison all read only the staged copy. Any mismatch stops the script before it scores or writes anything.
• A new --out-dir writes the evidence elsewhere (default unchanged).
• The scoring code is still the checkout's, as the docstring says. The published-reproduction check and the slow test catch drift there.

README

• An Inputs table lists every pinned input with its sha256 and the scripts that read it, read from the code: the seven run files, the explanations CSV and the proposals.
• It also says which scripts check the pins.
• The Reproduce section now restores the whole run directory and the run's annotations directory from 8b4c0ca1 for the scripts that still read the working tree. I ran that round trip in a clean checkout of main: every file matched its pin after the restore, and the tree was clean after the put-back.

tests/test_reference_adversary_impact.py (new):

• The pins cover every file the script reads. The staged bytes match them, and the payload is the README's 46 models × 1,928 scored cells.
• A differential check: the payload pin equals definition_conformance.py's and publication_sources.py's.
• A property test (Hypothesis): flipping, truncating or extending any pinned run file is refused and nothing is written, while the unchanged bytes are accepted.
• Refusals: Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's payload, a wrong proposals hash and an unknown commit. main() refuses before analyze is ever called.
• A tampered checkout: a git clone --shared --no-checkout whose working-tree run and proposals are junk still scores the pinned bytes and writes all 23 file names. Each exclude variant changes only the exclusion record, and each regenerate variant changes only the reference CSV and its sidecar.
• @slow: a real full regeneration reproduces all 23 committed files byte for byte. It takes about four minutes on a loaded machine, so CI deselects it.

Invariants

• Pinned inputs. Every byte the script scores or compares against equals the pinned bytes, whatever the working tree holds. Covered by the tampered-checkout test and the property test.
• Refusal before use. A mismatched input stops the script before any scoring or output write. Covered by the refusal tests and the property test.
• Reproduction. A full regeneration equals the committed evidence byte for byte. Covered by the slow test.
• Variant isolation. The published copy is byte-identical to the pins, and each variant changes only the files its records touch. Covered by the tampered-checkout test.

Mutation check: I put each working-tree read back, disabled the sha256 check, copied the proposals from the working tree, and skipped the stale-directory cleanup. Each of the 7 mutants fails at least one fast test.

Tests

Run locally at head 95aaa8fb with the CI environment (Python 3.12, uv sync --locked --extra dev, policyengine-us 2.15.17), one file at a time on a loaded machine:

Run	Result
tests/test_reference_adversary_impact.py -m "not slow"	8 passed
tests/test_consensus.py	57 passed
tests/test_reference_adversary.py	107 passed
tests/test_reference_adversary_runner.py	27 passed
tests/test_definition_conformance.py	64 passed
tests/test_publication_sources.py	111 passed
tests/test_audit.py	25 passed
tests/test_reference_adversary_impact.py -m slow (full regeneration, byte for byte)	1 passed in 228 s
ruff check ., ruff format --check .	clean

git status was clean after the runs: nothing was written into the repository.

Out of scope (filed as follow-ups)

• definition_conformance.py, publication_sources.py, engine_probe.py and build_proposals.py in this audit still read the working tree's run. The README now documents exactly which inputs each one reads and checks.
• The Louisiana, Medicare Part B and payroll audits' leaderboard_impact.py scripts have the same working-tree pattern.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Nov 2, 2026 3:11pm UTC

[MaxGhenis]

Ready for hub review

This is an independent review of #205 by a Claude Opus 5.5 cloud session, written for the PolicyBench owner session.

Verdict: APPROVE at b59570d1dd1b6dfee16665a6feabdcfa79aa203d

The PR's head on GitHub is still b59570d1. I reviewed git diff 875fe1f6 b59570d1. I ran the tests on a scratch merge of the head into origin/main (875fe1f6); that merge's tree is identical to the head (git diff --stat b59570d1 HEAD is empty). Environment: Linux x86_64, Python 3.12, uv sync --locked --extra dev. That lock now installs policyengine-us 2.60.0, but leaderboard_impact.py does not import the engine.

Findings

None are blocking.

1. The slow regeneration fails on Linux x86_64 by one ULP; this is the platform, not the PR (non-blocking)

tests/test_reference_adversary_impact.py:276 (test_regeneration_reproduces_the_committed_evidence) fails on this machine. The script stops at its own published-reproduction check before it writes anything:

$ OPENBLAS_NUM_THREADS=1 uv run pytest -m slow tests/test_reference_adversary_impact.py -q
E       AssertionError: ... unchanged copy does not reproduce published programStats
1 failed, 8 deselected in 34.23s

The difference is in the last digit:

DIFF programStats  {'mape': (5.014541219683578, 5.0145412196835775)}
DIFF heatmap       {'mae': (305.7924935689786, 305.79249356897867)}

The PR does not cause this. The pre-merge head 95aaa8fb gives the same two differences with the same venv (numpy 2.4.2 and pandas 3.0.1, the same versions as in 95aaa8fb's lock). Exact float equality across CPU architectures is the cause. I then replaced check_reproduces_published with a 1e-12 relative-tolerance version and ran the whole script. The check passed. Of the 23 files, 20 were byte-identical to the committed ones. The other 3 (leaderboard_impact_exclude_az_standard_deduction_indexing_programs.csv, ..._regenerate_az_standard_deduction_indexing_models.csv, ..._regenerate_co_sales_tax_refund_surplus_models.csv) differ only in trailing digits, for example 84.2395786775687 against 84.23957867756873.

So the PR's claim that the slow test reproduces all 23 files byte for byte holds on the author's machine, not on Linux x86_64. Because CI deselects the test, nothing breaks. Suggested README text after the slow-test command (README line 510 at the head):

The comparison is exact, so it holds on the platform that wrote the evidence. On Linux x86_64 the published-reproduction check fails in the last digit of programStats mape and heatmap mae.

2. Untested path: a scoring child or always_zero that reads the working tree's scenarios.csv (non-blocking; #207 closes it)

The fast tests stub analyze and always_zero (tests/test_reference_adversary_impact.py:229). At the head, main's working-tree scenarios.csv and scenarios.csv.meta.json still equal their pins (71b16212…, 03a66e90…; checked with sha256sum). A regression that passed ROOT/<run>/scenarios.csv to analyze would therefore pass the fast tests and also produce identical slow output. #207's fenced slow test covers this case. Nothing to change here if #207 lands.

3. The output definitions come from the checkout (documented; note only)

always_zero (line 191) and the policybench analyze child read the checkout's policybench/benchmark_specs.json. At the head, that file differs from the pass's copy (89067ba7… here, ce233f8c… at 8b4c0ca1), because #215 added a uk_audit spec. The docstring treats the definitions as scoring code, and the published-reproduction check covers what analyze produces. The US policybench outputs that always_zero uses are unchanged by #215, so nothing drifts today.

Checked and sound

• Every input comes from the pinned pass. main() stages all seven run files from 8b4c0ca1 and proposed_changes.json from 4db91b5f (line 420) before it plans, stages or scores anything. stage(), the regeneration line check and check_reproduces_published read only the staged copy. The script has no other path to the run or the proposals. test_the_pins_cover_every_file_the_script_reads checks set(RUN_SHA256) == {*RUN_FILES, "data.json.gz"} and stages exactly those files.
• A changed byte stops the script before output. git_input (line 116) checks the sha256 before it writes the staged file. A mismatch raises SystemExit before analyze and before out_dir.mkdir (line 432). The only writes before a refusal are the earlier staged files under <scratch>/pass_inputs, which is outside the repository. The Hypothesis test and test_a_refused_input_stops_the_script_before_it_scores cover this.
• The tests fail without the change. With main's leaderboard_impact.py put back, all 8 fast tests fail (8 failed, 1 deselected). A missing attribute explains most of those failures, so I also ran two targeted mutants on the PR's code. Both were killed by test_the_script_scores_only_the_staged_inputs (1 failed, 7 passed):
  • stage() copying from the working tree's run;
  • check_reproduces_published reading the working tree's data.json.gz.
• Fit with main. policybench analyze's flags (-g -p -s -o --app-data-output --reference-digest) and scored_reference_for are unchanged at 875fe1f6 (policybench/cli.py:876-916, policybench/reference_exclusions.py:178). analyze reads the exclusion record from beside the reference CSV, which is the staged copy. 8b4c0ca1, 4db91b5f and 9ce4ade8 are on main's history. CI checks out full history (.github/workflows/ci.yml:41, fetch-depth: 0). The README anchor #flag-parameters exists. test_the_payload_pin_matches_the_engine_side_scripts_pins still passes at the head because main's engine-side scripts keep PAYLOAD_SHA256.
• Fast tests: uv run pytest tests/test_reference_adversary_impact.py -m "not slow" -q gives 8 passed, 1 deselected. CI at b59570d1: test, lint and app are all success.
• Published numbers. None change. The diff touches only reference_audit/2026-10-05-reference-adversary/README.md, .../scripts/leaderboard_impact.py and the new tests/test_reference_adversary_impact.py. It changes no verification/ file, proposed_changes.json, reference, exclusion, snapshot, paper text or release tag (git diff --stat 875fe1f6 b59570d1).

What I couldn't check

• I could not run the byte-for-byte slow regeneration on the author's platform. On Linux it reproduces only within 1e-12 (finding 1).
• I did not run the README's git restore --source 8b4c0ca1 round trip.

---

Generated by Claude Code