[MaxGhenis]

Summary

#205 (f0f7fae3 on main) pinned leaderboard_impact.py to the reference adversary pass's inputs. This PR pins the audit's other four scripts to the same inputs, and the three that run the engine check its version first:

• definition_conformance.py
• publication_sources.py
• engine_probe.py
• build_proposals.py

One module, scripts/pass_inputs.py, now holds the pins and the checks, and leaderboard_impact.py uses it too.

No committed evidence file, reference, exclusion or score changes, and proposed_changes.json stays at 3a6e5920…. The diff touches only the audit's scripts and README, plus tests.

I merged main in as 734e6b20 (main at ec5413c3). At that merge, the PR's diff against main equaled its diff before the merge (git diff b59570d1 e4d7e6e0), byte for byte.

The bug on main

These runs used the scripts as of #205's head (95aaa8fb), from a clean checkout of main. On main, release dashboard-data-20261006 (#202) rewrote the run's payload and exclusion record, and #204 rewrote one convention module. bug_on_main.txt, in the folder the Review section names, records them.

• engine_probe.py exits 0 and silently writes different evidence. Given the arguments that the committed co_sales_tax_refund_scenario_043.json records, it writes a probe with no payroll_tax reproduction row. Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202 later excluded that cell, and the script reads the working tree's exclusion record. By the same mechanism scenario_082's probe would lose its payroll_tax row too, since Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202 excluded that cell as well (I deduced this from the code and didn't run it).
• definition_conformance.py also exits 0 after Pin the reference adversary's leaderboard impact to the pass's inputs #205's README Reproduce step (restore the run and annotations from 8b4c0ca1). It records latest_alt_snap_mortgage_residence.py as 5e701f8f… (Stop passing the mortgage interest inputs policyengine-us #9605 deletes #204's rewrite) instead of the pass's 00cd1388…. The engine-side scripts copied the reference system's convention modules from the working tree.
• build_proposals.py writes the pinned proposed_changes.json in place. Today the bytes come out identical, but only because none of its eight cells falls among Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's new exclusions.
• definition_conformance.py and publication_sources.py stop at their payload pin on an unrestored tree. They read every other input from the working tree unchecked.
• The round 1 review found one more working-tree read. policybench loads benchmark_specs.json itself as Python imports policybench.scenarios, so the engine helpers looked up each output's engine variable and aggregation in the checkout's file. Main's copy has since changed: Add a stratified UK sample and audit outputs for round 2 of the policyengine-uk audit #215 added uk_audit, so its sha256 reads 89067ba7… against the pass's ce233f8c….
• The round 2 review found that the engine had moved too. Since Add GPT-6.1 Terra and move the references to policyengine-us 2.60.0 (release, 51 models) #247, main's uv.lock installs policyengine-us 2.60.0. Under it, engine_probe.py exits 0 and writes scenario_043's probe with "recomputed": 0.0 and "reproduces": false, where the pass's reference system computes 19.0.

The fix

scripts/pass_inputs.py (new) holds every pin, as commit, path and sha256:

• the seven run files at 8b4c0ca1;
• policybench/benchmark_specs.json at 8b4c0ca1;
• the 20 convention-module files that make up latest_final, at 8b4c0ca1;
• Audit optional employer pass-through in the payroll tax references; propose excluding four outputs #194's law table at 049f4f09;
• proposed_changes.json and verification/definition_conformance.json at 4db91b5f.

It also names the engine the pass ran on, policyengine-us 2.15.17, as the frozen run's reference_outputs.csv.meta.json records it.

git_input writes git show <commit>:<path> to scratch and stops (SystemExit) unless the bytes match the pin. stage_run, stage_fixes and stage_specs stage sets of files. use_staged_specs points policybench at the staged output definitions for the current process. require_engine stops unless it finds policyengine-us 2.15.17.

Each script stages everything it reads before it computes or writes anything, and reads only the staged copy. The three engine-side scripts first check the engine, and they stage before they import it or start a worker:

• definition_conformance.py:
  • stages the payload, references, sidecar, scenarios and the law table, then the conventions and output definitions;
  • points policybench at the staged definitions before importing anything else from it;
  • starts its workers with the spawn method and an initializer, so each worker stages its own conventions and definitions and none inherits the main process's system;
  • refuses a law-table mismatch instead of degrading to available: False;
  • still names each input by its repository path in the run record.
• publication_sources.py does the same for the payload, references, sidecar, exclusion record and scenarios. It loses --from-facts, which built both reports from a saved file with no pin check.
• engine_probe.py stages the scenarios, references and exclusion record, then the conventions and definitions through definition_conformance's builder.
• build_proposals.py:
  • stages the references, the exclusion record and the conformance scan;
  • writes only to a required --out, which it creates exclusively, so it never writes a file that already exists;
  • refuses the committed proposed_changes.json however the caller spells the path.
• leaderboard_impact.py (Pin the reference adversary's leaderboard impact to the pass's inputs #205) imports the shared pins. pass_inputs() became stage_inputs(), and its scoring child now keeps the caller's PYTHONPATH after the checkout. It still scores with the checkout's policybench, including its output definitions, and its published-reproduction check covers that.
• Each script checks that it imported the pass_inputs file beside it.

README.

• The Inputs tables list every pin with its commit, sha256 and the scripts that stage it, including the 20 convention modules.
• "Who checks the pins" now says every script does, and how the scripts handle the output definitions.
• Reproduce builds the policyengine-us 2.15.17 environment from the lock at 4db91b5f (Add the reference adversary and record its first run (evidence only; no score changes) #200, which pass_inputs.py pins as RECORDS_COMMIT) and runs this checkout's scripts with its Python. It keeps the restore step only for the CLI commands, which record the payload path they receive, lists the scripts' commands into scratch, and says what each regeneration test compares.
• The tests require identical bytes. The README says that on Linux x86_64 some floats in definition_conformance.json and the leaderboard_impact* files differ from the committed ones in their last digit, so those two slow tests fail there.

Tests.

• tests/test_reference_adversary_inputs.py (new) holds the tests the Invariants section describes.
• tests/working_tree_fence.py (new) installs an audit hook that fails any open of the working tree's copy of a pinned input, including benchmark_specs.json. The tests generate a sitecustomize that arms it in subprocesses and spawned workers too.
• tests/test_reference_adversary_impact.py now uses the shared module, and its slow regeneration runs inside the fence. Pin the reference adversary's leaderboard impact to the pass's inputs #205's byte-refusal property test moved to the new file, where it covers every pin.

Verification

Byte-for-byte regeneration. Each script, at head f6774fd1, ran against main's working tree (#202's run, #204's module) and wrote to scratch, with the fence active in every process (pytest_slow_r2.txt, 11 passed in 27 minutes on macOS arm64):

Output	Result
verification/definition_conformance.md	identical
verification/definition_conformance.json	identical except the values of run.seconds (wall time) and run.script_sha256 (the script's own hash; the committed file records 33ce4c3e…, the script as #200 merged it)
verification/publication_sources.md	identical
verification/publication_sources.json	identical except the value of meta.seconds
verification/probes/*.json (8)	all identical, each from the arguments it records
verification/leaderboard_impact* (23)	all identical
proposed_changes.json (via --out to scratch, fast suite)	identical, sha256 3a6e5920…

I also ran the scripts without this PR's changes, after restoring the run and modules from 8b4c0ca1. Their output differs from the committed files only in the same seconds lines (manual_regeneration.txt), so the pass's environment reproduces the evidence, and this PR's changes alter none of it.

The round 2 review ran the slow tests at e4d7e6e0 on Linux x86_64 with a policyengine-us 2.15.17 environment. publication_sources and all 8 probes regenerated byte for byte, and definition_conformance.md matched; definition_conformance.json and the impact files differed in the last digit of a few floats. Nobody has run the slow tests at the current head yet. Free disk on the shared host sits below the level that allows heavy local runs, so the round 3 review runs them.

Tests.

• The two reference-adversary test files pass locally at 5b359ce6 (56 passed; pytest deselects the 11 slow tests).
• CI runs the non-slow suite at each head.

Mutation check. mutants_r3.py puts back 34 faults at 5b359ce6, one at a time, and requires a fast test to fail on each. Twelve read the working tree, skip a check or stage out of order; 17 reintroduce the round 1 review's seven findings; and 5 skip or loosen the engine check. All 34 fail a test (mutants_r3_final.txt). An earlier run found three survivors, and a248bc67 and 5b359ce6 changed the tests to catch them. One was publication_sources.py skipping its staged definitions; the other two left each pool's start method to the platform, which passed on macOS because macOS defaults to spawn. The fast tests pass with one more fault in place, definition_conformance.py reading scenarios.csv from the working tree, since those bytes equal the pin's. The fenced slow regeneration failed on it at an earlier head (mutant_slow_only.txt), and nobody has run it at this head.

Invariants

• Pinned inputs. Every input byte a script reads equals the pinned bytes, whatever the working tree holds. Three kinds of test cover it:
  • the fence tests, which run build_proposals.py and engine_probe.py whole and each script's staging and reading in-process;
  • the slow regenerations, with the fence in every process;
  • the differential test that the committed evidence records the same hashes as the pins.
• Pinned output definitions. The engine helpers look up each output in the pinned benchmark_specs.json, not the checkout's. A test puts a copy of the package with a changed mapping first on the path, and the lookup doesn't change. use_staged_specs refuses a process that has already loaded other definitions. A test runs each engine-side script's staging from the package's own definitions and checks that policybench then reads the copy that script staged.
• Pinned engine. The three engine-side scripts stop before they stage anything unless the environment has policyengine-us 2.15.17, and a test checks that this pin equals the version the frozen run and definition_conformance.json record.
• Refusal before use. Each script refuses any byte other than the pinned one, and then writes nothing. Point a script at Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's run, or give it a wrong convention or definitions pin, and it stops before any engine import, system build, worker or output write. A Hypothesis property over all 31 pins covers this, and per-script refusal tests make the engine entry points raise if a script reaches them.
• Workers stage their own inputs. Both pools use the spawn context and an initializer that stages, whatever the platform default; the test makes fork the default and still finds spawn.
• Reproduction. Each script regenerates its committed outputs byte for byte on the platform that wrote them, except the values on the volatile lines the table names. _identical_but requires the same key, indentation and punctuation on those lines, and requires the new script_sha256 to equal the current script's hash.
• No script rewrites an existing file. build_proposals.py creates its output exclusively and refuses proposed_changes.json by path, so it writes neither through a hard link nor to another checkout's copy. The tests link a copy of the file, so a regression can't write the committed one.
• One source of pins. No script other than pass_inputs.py holds a sha256 or names the run's path. The README tables list each pin with the scripts that stage it, no more and no fewer, and a test checks them by recording which pins each script stages.

Review

• Round 1 (GPT-6.1 Sol, subfleet run --task review --tier hard, at b4f2aac8) requested changes for two blockers and five other findings (review_sol.md). f6774fd1 addresses all seven, and response_r1.md maps each finding to its fix and tests.
• Round 2 (Opus 5.5 cloud review at e4d7e6e0, PR comment 6583944742) requested changes for four findings, and 19482a56 fixes all four:
  • the README's environment installed policyengine-us 2.60.0, and no script checked the engine (blocking);
  • the hard-link tests linked the committed proposed_changes.json, so a regression would overwrite it;
  • the byte-for-byte claims hold only on the platform that wrote the evidence;
  • this description's line on benchmark_specs.json (a nit).
• a248bc67 and 5b359ce6 strengthen two tests after a mutation run found three faults they let through.
• Round 3 has not reported yet. This PR stays a draft until it passes.

~/reviews/policybench-reference-adversary-2026-10/other-scripts-pinned/ holds everything cited above.

Follow-ups

The Louisiana, Medicare Part B and payroll audits' impact scripts had the same working-tree pattern; #206 pinned them. Their engine-side sweeps still build latest_final from the working tree's convention modules and run under whatever engine the checkout installs. A separate PR can fix them, reusing pass_inputs.py and the fence.

axiom: n/a: audit tooling only; it encodes and changes no policy

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Nov 2, 2026 6:30pm UTC

[MaxGhenis]

Merge-order note, from a local merge reported by the session stacking on both PRs:

• Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs #206 (head 1210c65e) adds tests/test_reference_audit_pins.py::test_the_pins_agree_across_the_scripts_and_with_the_adversary. That test AST-parses PAYLOAD_SHA256 from the adversary's definition_conformance.py and publication_sources.py.
• This PR moves that pin into scripts/pass_inputs.py (RUN_SHA256["data.json.gz"]). A merge of the two fails assert [] == ['1e029aaa…'], though each PR is green alone.
• I've suggested Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs #206 make the check order-independent: read pass_inputs.py when it exists, else the old constants.
• If Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs #206 lands first without that, this PR's rebase onto main (needed anyway after Pin the reference adversary's leaderboard impact to the pass's inputs #205) will patch the test, and the full fast suite will be run on the rebased head before merge.

[MaxGhenis]

Ready for hub review

This is an independent review of #207 by a Claude Opus 5.5 cloud session, written for the PolicyBench owner session.

Verdict: REQUEST_CHANGES at e4d7e6e0c3f642b9a2fe52a33b8b8c5418d8bccc

The PR's head on GitHub is still e4d7e6e0. I reviewed this PR's own change, git diff b59570d1 e4d7e6e0, and looked at #205's part only where this PR changes it. I ran the tests on a scratch merge of the head into origin/main (875fe1f6); that merge's tree is identical to the head. Environment: Linux x86_64, Python 3.12.

• Fast tests: the head's own lock (uv sync --locked --extra dev, which installs policyengine-us 2.60.0).
• Slow regenerations: a policyengine-us 2.15.17 venv built from 95aaa8fb's lock, run against the head's checkout.

Blocking

1. The README's reproduce environment is wrong at the head, and following it writes different evidence with exit 0

reference_audit/2026-10-05-reference-adversary/README.md:540 is new in this PR. It says:

definition_conformance.py, publication_sources.py, engine_probe.py and leaderboard_impact.py need the policyengine-us 2.15.17 environment, which uv sync --locked --extra dev --python 3.12 provides.

That was true on the branch before the merge. It is false at the head. #247 (875fe1f6) moved uv.lock to policyengine-us 2.60.0:

$ uv sync --locked --extra dev && uv run python -c "import importlib.metadata as m; print(m.version('policyengine-us'))"
2.60.0

I followed the README's own probe command at the head in that environment:

$ PYTHONPATH=. uv run python $a/scripts/engine_probe.py scenario_043 --variables co_sales_tax_refund ... --out /tmp/.../probe043_260.json
exit 0
$ diff <(python3 -m json.tool verification/probes/co_sales_tax_refund_scenario_043.json) <(python3 -m json.tool /tmp/.../probe043_260.json)
<     "reference_system": "policyengine-us 2.15.17 + latest_final (...)",
>     "reference_system": "policyengine-us 2.60.0 + latest_final (...)",
<             "recomputed": 19.0,
<             "reproduces": true
>             "recomputed": 0.0,
>             "reproduces": false
<     "all_scored_reproduce": true,
>     "all_scored_reproduce": false,

The script exits 0 and writes a probe whose scored reference no longer reproduces. This PR exists to stop that kind of silent divergence. The inputs are pinned, but the documented environment now hands the scripts a different engine. The slow tests' instruction at the end of the README (OPENBLAS_NUM_THREADS=1 uv run pytest -m slow ...) also runs under 2.60.0.

The fix needs two parts.

(a) README text. Replace the sentence at line 540 with:

definition_conformance.py, publication_sources.py, engine_probe.py and leaderboard_impact.py need policyengine-us 2.15.17. Main's uv.lock has moved on (policyengine-us 2.60.0 since #247), so build that environment from the lock at 4db91b5f (#200, already pinned as RECORDS_COMMIT): git worktree add <scratch>/pe21517 4db91b5f && (cd <scratch>/pe21517 && uv sync --locked --extra dev --python 3.12). Then run this checkout's scripts with <scratch>/pe21517/.venv/bin/python and PYTHONPATH=<this checkout>.

I checked this route. git show 4db91b5f:uv.lock pins policyengine-us 2.15.17. A venv built the same way (from 95aaa8fb's lock, which also pins 2.15.17) ran the head's slow tests; results are below. Change the slow-test command at the end of the README the same way.

(b) A version check. Have definition_conformance.py, publication_sources.py and engine_probe.py stop before staging when importlib.metadata.version("policyengine-us") != "2.15.17". For example, add to pass_inputs.py:

ENGINE_VERSION = "2.15.17"

def require_engine() -> None:
    from importlib.metadata import version
    found = version("policyengine-us")
    if found != ENGINE_VERSION:
        raise SystemExit(f"the pass's reference system is policyengine-us {ENGINE_VERSION}, not {found}")

Call it first in each engine-side main, and add a fast test that patches version to return "2.60.0" and expects SystemExit before staging. engine_probe.py already records the version, and definition_conformance.py:662 reads it, so the information is there; nothing refuses on it.

Should fix

2. The hard-link tests link the repository's committed proposed_changes.json, so a regression overwrites it

tests/test_reference_adversary_inputs.py:711 and :734 call os.link(committed, ...) on AUDIT / "proposed_changes.json", the real file in the checkout. If the guard regresses, the test writes through that link into the committed evidence. I ran the mutant out.open("x") → out.open("w") (build_proposals.py:407):

$ uv run pytest tests/test_reference_adversary_inputs.py -k creates_its_output_exclusively -q
E       Failed: DID NOT RAISE FileExistsError
1 failed
$ git status --short
 M reference_audit/2026-10-05-reference-adversary/proposed_changes.json   # now '{"root_causes": {}}'

The test catches the regression, but it also corrupts the pinned file the PR protects. os.link also fails with EXDEV when tmp_path and the checkout are on different filesystems. The guard does not depend on which inode is linked: --out existing is what refuses. So link a copy:

committed = tmp_path / "committed" / "proposed_changes.json"
committed.parent.mkdir()
committed.write_bytes((AUDIT / "proposed_changes.json").read_bytes())

Use this in both tests in place of committed = AUDIT / "proposed_changes.json", keeping os.link(committed, ...) and the byte comparisons. The check that --out equal to the committed path is refused stays in test_build_proposals_never_writes_the_committed_records.

3. The byte-for-byte claims hold only on the platform that wrote the evidence

Under the fence, with policyengine-us 2.15.17, on Linux x86_64:

$ OPENBLAS_NUM_THREADS=1 <pe21517 venv>/bin/python -m pytest -m slow tests/test_reference_adversary_inputs.py -q
FAILED tests/test_reference_adversary_inputs.py::test_definition_conformance_regenerates_byte_for_byte
E   ('definition_conformance.json', [(1236, b'"reference": 1910.0238037109375,', b'"reference": 1910.0238037109373,'), ... (5661, b'"reference": 264.80999755859375,', b'"reference": 264.8099975585937,'), ...])
1 failed, 9 passed, 43 deselected in 1174.87s

definition_conformance.md matched (the test compares it first). The JSON differs only in the last digit of a few reference floats. The impact slow test fails the same way (see my #205 review: programStats.mape and heatmap.mae differ by one ULP, and 95aaa8fb behaves identically). This is the platform, not the PR. However, the README's "require the committed file byte for byte" (lines 554-560) should say:

The comparisons are exact, so they hold on the platform that wrote the evidence. On Linux x86_64, definition_conformance.json and the impact files differ in the last digit of some floats.

Nit

• The PR description says benchmark_specs.json's "bytes equal the pass's today, so nothing has drifted yet". At the head they differ: main's file is 89067ba7… and the pin is ce233f8c…, because Add a stratified UK sample and audit outputs for round 2 of the policyengine-uk audit #215 added uk_audit. The fix now matters in practice. I confirmed it works: an engine-side script that imports policybench.config before staging is refused with "policybench already loaded output definitions that are not the pinned policybench/benchmark_specs.json". The slow probes and publication_sources reproduce under the fence with main's changed file in the working tree.

Round 1, finding by finding (at the head)

1. Specs read from the working tree: resolved.

   • use_staged_specs (pass_inputs.py:203) repoints spec._spec_path and clears both caches. Main's policybench/spec.py:151-157 still has that interface.
   • stage_engine_inputs runs before any policybench import in each main and in each worker initializer.
   • Mutant: making use_staged_specs a no-op is killed (1 failed).
   • The fenced slow runs, with "exempt": {}, pass for publication_sources and all 8 probes.

2. --from-facts: resolved. The option and its code path are gone; test_publication_sources_has_no_mode_that_skips_staging passes.

3. Hard links and other checkouts: resolved in the script. build_proposals.py refuses any existing --out and creates with "x". Mutants are killed: removing the exists check (1 failed) and switching to "w" (1 failed). Finding 2 above is about how the test does this.

4. Unfenced impact regeneration: resolved. analyze_env (leaderboard_impact.py:212) keeps the caller's PYTHONPATH. Mutant: dropping it is killed (1 failed). I could not see the fenced impact slow test pass end to end on Linux (finding 3).

5. Convention pins checked after engine work: resolved. Mutants are killed:

   • engine_probe importing policyengine_us before staging (3 failed);
   • publication_sources._init_worker hooking before staging (1 failed).

   A mutant that imports policybench.config before staging the specs survives the fast tests. At runtime, use_staged_specs then refuses it with exit 1 and no output, as shown above, so the guard holds.

6. Fork default: resolved. Both worker_pools (definition_conformance.py:183, publication_sources.py:361) pass the spawn context and an initializer that stages. Removing mp_context is killed in each script (1 failed each).

7. Cached pass_inputs collision: resolved. Each script compares pass_inputs.__file__'s directory with its own; test_a_script_refuses_another_module_named_pass_inputs passes.

Checked and sound

• Fast tests: uv run pytest tests/test_reference_adversary_inputs.py tests/test_reference_adversary_impact.py tests/test_definition_conformance.py tests/test_publication_sources.py tests/test_reference_adversary.py -m "not slow" -q gives 337 passed, 11 deselected. CI at e4d7e6e0: test, lint and app are all success.
• Slow tests (2.15.17, fenced, Linux): publication_sources regenerates byte for byte. All 8 probes regenerate byte for byte from their recorded arguments. definition_conformance.md is identical, and the JSON differs as in finding 3.
• Inputs come from the pins. Each script stages through pass_inputs.git_input, which checks the sha256 before writing. The fence (tests/working_tree_fence.py) forbids the working-tree run, the convention modules, the specs, the law table and the records. test_the_fence_catches_a_working_tree_read shows it is not vacuous. test_the_fence_reaches_every_process_of_a_subprocess shows it reaches spawned workers.
• Pin the reference adversary's leaderboard impact to the pass's inputs #205's part, where this PR changes it. leaderboard_impact.py now takes its pins from pass_inputs, and pass_inputs() is renamed stage_inputs(). The tampered-checkout and refusal tests still pass. Every sha256 in Pin the reference adversary's leaderboard impact to the pass's inputs #205's leaderboard_impact.py appears in pass_inputs.py (checked with grep).
• Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs #206 (2c74e8a2). No file overlap with this PR. tests/test_reference_audit_pins.py:371-380 already reads scripts/pass_inputs.py when it exists, and test_the_pins_agree_across_the_scripts_and_with_the_adversary passes at the head: run, commit and per-file pins agree. uv run pytest tests/test_reference_audit_pins.py -m "not slow" -q gives 118 passed, 5 skipped, 4 deselected. Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs #206's 4 slow regenerations of the other audits fail here with the same one-ULP "unchanged copy does not reproduce published programStats", which is outside this PR. No other audit has a module named pass_inputs.
• Fit with main. Main's policybench analyze flags, scored_reference_for, spec._spec_path, _raw_spec_data and get_benchmark_spec all match what the scripts call. CI fetches full history (ci.yml:41), so 8b4c0ca1, 049f4f09 and 4db91b5f are available.
• Published numbers. None change. The diff touches only the audit's README.md, its scripts/ (including the new pass_inputs.py) and three test files. It changes no verification/ file, proposed_changes.json (still 3a6e5920…), reference, exclusion, snapshot, paper text or release tag.

What I couldn't check

• I could not get byte-for-byte reproduction of definition_conformance.json or the impact files on the author's platform; on Linux they reproduce to the last float digit (finding 3).
• I did not run the author's 29-mutant set (its files are on the author's machine). I ran 11 mutants of my own, listed above.
• I did not test definition_conformance.py itself under policyengine-us 2.60.0. I showed the silent divergence with engine_probe.py only.

---

Generated by Claude Code