[MaxGhenis]

On main, the CI test job runs pytest in one process. On 2 Nov it took 29 minutes 32 seconds at b0c8f2e, and at ec5413c it ran into the job's 30-minute limit, so GitHub cancelled it. This PR splits the tests across pytest-xdist workers, one per logical CPU (-n logical --dist loadfile), and xdist starts four on GitHub's runner. At f4c3c38, main's serial job took 22.3 minutes, and this PR's job on the merge with that commit took 12.3 minutes at 654c525 and 15.7 minutes at efcecfc. All three report 7,064 passed and 90 skipped.

• pyproject.toml, uv.lock: add pytest-xdist>=3.6 to the dev extra. The lock adds pytest-xdist 3.8.0 and execnet 2.1.2 and changes nothing else: git diff origin/main -- uv.lock shows no removed line, so no policyengine, policyengine-us, policyengine-uk, policyengine-core or litellm entry moves.
• tests/test_reference_audit_pins.py: parametrize two tests over sorted paths (see "Tests changed").
• .github/workflows/ci.yml: the test job runs uv run pytest -m "not slow" -n logical --dist loadfile --tb=short. uv sync installs psutil, so pytest-xdist 3.8.0 would count physical cores for -n auto; -n logical counts logical CPUs. Without -q, xdist prints the worker count (created: 4/4 workers). The comment on timeout-minutes: 30 gives the new measurements.
• CLAUDE.md: give the four-worker command for local runs, and say to run the suite from a full clone, since some tests read git history.

Times

Where	Mode	Runs	pytest time
CI, main (875fe1f, bd11d55, e0217ed)	serial	31 Oct, 26 Oct, 25 Oct	test job 17.3, 15.3, 19.9 min (pytest 16:43 on 31 Oct)
VM, 4 vCPU / 16 GB, Python 3.12	serial, unsorted parameters	1	34:10
VM	serial, sorted parameters	1	32:05
VM	-n 4 --dist loadfile	3	9:28, 9:39, 9:09
VM	-n 4 --dist load	1	10:08
CI, main at b0c8f2e (run 41486949933)	serial	2 Nov	test job 29.5 min (pytest 28:47); 6,900 passed, 90 skipped
CI, this PR (run 41490222217, merge eb99ee4 = 4491519 + b0c8f2e)	-n auto --dist loadfile	attempts 1, 2, 3 on 2 Nov	test job 12.7, 12.5, 16.2 min (pytest 12:00, 11:45, 15:39); 6,900 passed, 90 skipped each
CI, main at ec5413c (run 41495271792)	serial	2 Nov	GitHub cancelled it at the 30-minute limit (test job 30:01; pytest 29:22, 7,007 passed, 90 skipped)
CI, main at f4c3c38 (run 41524591677)	serial	2 Nov	test job 22.3 min (pytest 21:44); 7,064 passed, 90 skipped
CI, this PR (run 41526298328, merge 049f9a3 = 654c525 + f4c3c38)	-n logical --dist loadfile, 4 workers	2 Nov	test job 12.3 min (pytest 11:41); 7,064 passed, 90 skipped
CI, this PR (run 41529016470, merge 45de2e1 = efcecfc + f4c3c38)	-n logical --dist loadfile, 4 workers	2 Nov	test job 15.7 min (pytest 14:44); 7,064 passed, 90 skipped

Attempt 3 of run 41490222217 took about 3.5 minutes longer than attempts 1 and 2 on the same commit, and its elapsed time exceeded theirs by 18 to 42% at every progress mark from 10% to 100%. The job at efcecfc took 3.4 minutes longer than the job at 654c525 on the same tests, and its elapsed time exceeded that job's by 17 to 39% at those marks. In both comparisons the delay built up across the whole run, so we attribute it to a slower runner. At 16.2 minutes, attempt 3 finished 13.8 minutes inside the limit, so we kept timeout-minutes: 30.

We chose loadfile over load. It ran faster on the VM (9:09 to 9:39, against 10:08), and it keeps each test file in one worker, so a module-scoped fixture runs once per file, as in a serial run.

Invariant: same tests, same outcomes

The parallel job collects and runs the same tests as the serial job, and each test gets the same outcome.

• On main at 875fe1f and at 4491519 (this branch before it merged main), pytest -m "not slow" --collect-only -q collects 6,873 of 6,909 tests (36 deselected), and the sorted node ids match (diff prints nothing).
• With sorted parameters, pytest collects the tests in the same order under PYTHONHASHSEED 1, 2 and 3 (one md5 of the id list), as xdist requires of every worker.
• We compared per-test outcomes from --junitxml for the serial run with sorted parameters against parallel runs 2 and 3 (loadfile) and the load run. Each file holds 6,873 test cases; no node id appears in one and not the other, and no test's outcome differs.
• Every run on the VM, serial and parallel, gave 6,757 passed, 89 skipped and 27 failed.
• main gained Count only real request timeouts in the supervisor's concurrency backoff #183 and Probe card-less models under the card the onboarding gauntlet derives #189 after this branch started, so CI at 4491519 tested the merge with b0c8f2e. The serial test job on b0c8f2e and each of the three parallel jobs here report the same counts: 6,900 passed, 90 skipped and no failures, including the 27 tests that fail on the VM.
• In review r1, the reviewer merged this branch with main at 4e5a845 and ran the suite both ways on a 4-vCPU machine. The serial and parallel junit files held the same 7,097 tests with the same outcomes: 6,981 passed, 89 skipped, and the same 27 tests failed behind the reviewer's proxy.
• At f4c3c38, main's serial job and this PR's two parallel jobs on the merge with it all report 7,064 passed and 90 skipped.

The 27 VM failures all come from tests in tests/test_scenarios.py that reach scenarios_from_cps_frame → get_promptable_input_specs → make_us_microsimulation → materialize_certified_us_dataset. That last function calls hf_hub_download for the certified US dataset, and the VM's proxy refuses huggingface.co (403), so these tests fail on the VM serially and in parallel alike. CI reaches Hugging Face: at 875fe1f, where this branch started, main's test job passed 6,784 (6,757 + 27) with 89 skipped, and the parallel jobs on this PR passed all 27.

Tests changed

Test	Cause	Fix
tests/test_reference_audit_pins.py::test_any_refused_input_stops_the_impact_script_before_it_scores	Its parameters came from iterating a set of path strings (IMPACT[audit].inputs), whose order follows the per-process string hash seed. Each xdist worker collected the tests in a different order, and xdist refused to start ("Different tests were collected between gw1 and gw2").	Iterate sorted(IMPACT[audit].inputs). Sorting changes only the order of the node ids.
tests/test_reference_audit_pins.py::test_any_refused_input_stops_the_direct_script_before_it_writes	Its parameters came from iterating DIRECT[script].inputs, also a set.	Iterate sorted(DIRECT[script].inputs).

No other test failed only in parallel.

Where the test time goes

• From main on 8 Oct (ed4f89a) to 875fe1f on 31 Oct, where this branch started, test files grew from 53 to 91, def test_ functions from 1,090 to 3,608, and test code from 35k to 117k lines.
• In parallel run 1 on the VM, tests took 1,964 seconds summed across workers (--durations=0). test_freeze_terra.py took 443 s of that, followed by test_notes.py (149 s), test_freeze_gpt61sol.py (123 s), test_run_audit_claude.py (113 s), test_reference_audit_pins.py (93 s), test_freeze_haiku55.py (83 s), test_freeze_musewonder.py (81 s), test_tool_pilot_argon.py (78 s), test_finish_terra.py (71 s), test_finish_haiku55.py (56 s) and test_finish_musewonder.py (52 s).
• Seven of those files, the per-release freeze and finish files we added between 1 and 31 Oct, took 909 s together.
• The 40 slowest tests take about 557 s of the 34-minute serial run, and thousands of other tests share the rest.
• Under loadfile, one worker runs all of test_freeze_terra.py, about 7.4 minutes of work on the VM, so more cores can't bring the job under 7.4 minutes there. xdist queues files by test count rather than duration (--loadscope-reorder, on by default) and hands a worker its next file once two or fewer of its tests remain. On review r1's merge with main at 4e5a845, test_freeze_terra.py ranked seventh by test count, so with four workers it starts only after its worker finishes another file. Splitting it would lower the bound.

40 slowest tests, serial (VM, unsorted parameters)

#	Time	Phase	Test
1	69.73s	call	tests/test_freeze_terra.py::test_the_finalize_accepts_exactly_allowed_subsets_with_valid_checks
2	53.05s	call	tests/test_notes.py::test_no_board_model_declined_a_household_over_policyengine
3	36.56s	call	tests/test_reference_audit_pins.py::test_the_impact_script_scores_only_the_staged_inputs[reference_audit/2026-10-05-medicare-part-b]
4	35.70s	call	tests/test_engine_situation.py::test_every_situation_input_is_a_variable_of_the_pinned_engine[paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/scenarios.csv]
5	20.59s	call	tests/test_tool_pilot_argon.py::test_validator_accepts_only_known_names_on_their_entities
6	18.58s	call	tests/test_notes.py::test_release_20260929_note
7	17.45s	call	tests/test_tax_table_audit.py::test_the_independent_scorer_reproduces_the_release_under_each_convention
8	16.54s	call	tests/test_snapshot_artifacts.py::test_snapshot_deviation_audit_annotations_are_complete_and_final
9	14.20s	call	tests/test_call_provenance.py::test_household_classification_and_published_numbers
10	13.52s	call	tests/test_freeze_terra.py::test_rebound_adjudication_wording_reaches_applied_sentence_gate
11	13.03s	call	tests/test_tax_table_audit.py::test_the_independent_scorer_agrees_with_the_repo_scorer
12	11.47s	call	tests/test_supervisor.py::test_supervised_run_keeps_policyengine_out_of_workers
13	11.26s	call	tests/test_notes.py::test_america_gov_note_facts
14	10.64s	call	tests/test_tool_pilot_argon.py::test_score_counts_every_household_and_unparsed_outputs_as_misses
15	10.51s	call	tests/test_snapshot_artifacts.py::test_snapshot_audit_annotations_have_no_orphan_rows
16	10.46s	call	tests/test_tool_pilot_argon.py::test_arm_labels_follow_the_model
17	10.24s	call	tests/test_notes.py::test_release_20261010_note
18	10.02s	call	tests/test_notes.py::test_release_20261013_note
19	9.72s	call	tests/test_freeze_terra.py::test_the_finalize_accepts_publication_render_and_repin_sequence
20	9.59s	call	tests/test_freeze_terra.py::test_rebound_nonisolated_verdict_is_refused_at_export_and_freeze
21	9.35s	call	tests/test_sensitivity_evidence.py::test_doc_all_output_scores_recompute_on_the_frozen_references
22	9.21s	call	tests/test_freeze_terra.py::test_an_incumbent_prediction_change_is_refused_before_mutation[stage]
23	9.02s	call	tests/test_cost_latency.py::test_cost_is_present_exactly_when_recorded_usage_supports_it
24	8.97s	call	tests/test_reference_audit_pins.py::test_the_impact_script_scores_only_the_staged_inputs[reference_audit/2026-10-05-louisiana]
25	8.88s	call	tests/test_notes.py::test_release_20261006_note
26	8.50s	call	tests/test_finish_musewonder.py::test_install_references_is_idempotent_for_any_build
27	8.47s	call	tests/test_snapshot_artifacts.py::test_manuscript_bootstrap_point_estimates_reproduce_model_stats
28	8.33s	call	tests/test_finish_haiku55.py::test_install_references_is_idempotent_for_any_build
29	8.06s	call	tests/test_cost_latency.py::test_a_replay_can_only_lower_the_median
30	7.81s	call	tests/test_call_provenance.py::test_rows_record_each_responses_provenance_and_agree_with_the_ledger
31	7.55s	call	tests/test_uk_audit.py::test_a_later_stratum_does_not_change_earlier_draws
32	7.21s	call	tests/test_freeze_terra.py::test_rejudged_adjudicated_stage_passes_the_freeze
33	7.02s	call	tests/test_uk_audit.py::test_sample_is_deterministic
34	6.88s	call	tests/test_notes.py::test_categorical_asset_error_statements
35	6.61s	call	tests/test_snapshot_artifacts.py::test_frozen_impact_summaries_score_the_scored_reference
36	6.59s	call	tests/test_freeze_argon.py::test_a_stage_list_is_appended_to_release_20261013bs_amendments
37	6.53s	call	tests/test_reference_audit_pins.py::test_the_direct_script_reproduces_its_files_from_a_junk_checkout[reference_audit/2026-10-05-medicare-part-b/scripts/propose_exclusions.py]
38	6.35s	call	tests/test_tool_pilot_argon.py::test_arm1_rescoring_reproduces_the_published_board_row
39	6.20s	call	tests/test_freeze_terra.py::test_an_incumbent_prediction_change_is_refused_before_mutation[base]
40	6.20s	setup	tests/test_freeze_terra.py::test_repository_input_edits_after_export_are_refused_without_writes[pyproject.toml]

40 slowest tests, parallel (VM, -n 4 --dist loadfile, run 3)

#	Time	Phase	Test
1	60.17s	call	tests/test_notes.py::test_no_board_model_declined_a_household_over_policyengine
2	59.91s	call	tests/test_freeze_terra.py::test_the_finalize_accepts_exactly_allowed_subsets_with_valid_checks
3	35.00s	call	tests/test_reference_audit_pins.py::test_the_impact_script_scores_only_the_staged_inputs[reference_audit/2026-10-05-medicare-part-b]
4	31.90s	call	tests/test_engine_situation.py::test_every_situation_input_is_a_variable_of_the_pinned_engine[paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/scenarios.csv]
5	28.14s	call	tests/test_scenarios.py::test_generate_scenarios_count
6	22.14s	call	tests/test_tool_pilot_argon.py::test_validator_accepts_only_known_names_on_their_entities
7	19.69s	call	tests/test_notes.py::test_release_20260929_note
8	17.79s	call	tests/test_tax_table_audit.py::test_the_independent_scorer_reproduces_the_release_under_each_convention
9	13.07s	call	tests/test_call_provenance.py::test_household_classification_and_published_numbers
10	12.74s	call	tests/test_freeze_terra.py::test_rebound_adjudication_wording_reaches_applied_sentence_gate
11	11.86s	call	tests/test_tax_table_audit.py::test_the_independent_scorer_agrees_with_the_repo_scorer
12	11.59s	call	tests/test_tool_pilot_argon.py::test_score_counts_every_household_and_unparsed_outputs_as_misses
13	11.54s	call	tests/test_notes.py::test_america_gov_note_facts
14	11.39s	call	tests/test_tool_pilot_argon.py::test_arm_labels_follow_the_model
15	11.20s	call	tests/test_supervisor.py::test_supervised_run_keeps_policyengine_out_of_workers
16	10.99s	call	tests/test_notes.py::test_release_20261010_note
17	10.94s	call	tests/test_snapshot_artifacts.py::test_snapshot_deviation_audit_annotations_are_complete_and_final
18	9.85s	call	tests/test_notes.py::test_release_20261013_note
19	9.64s	call	tests/test_notes.py::test_release_20261006_note
20	9.43s	call	tests/test_finish_musewonder.py::test_install_references_is_idempotent_for_any_build
21	9.32s	call	tests/test_finish_haiku55.py::test_install_references_is_idempotent_for_any_build
22	8.78s	call	tests/test_snapshot_artifacts.py::test_manuscript_bootstrap_point_estimates_reproduce_model_stats
23	8.67s	call	tests/test_freeze_terra.py::test_rebound_nonisolated_verdict_is_refused_at_export_and_freeze
24	8.67s	call	tests/test_uk_audit.py::test_sample_is_deterministic
25	8.42s	call	tests/test_freeze_terra.py::test_the_finalize_accepts_publication_render_and_repin_sequence
26	8.31s	call	tests/test_freeze_terra.py::test_rebound_parse_flag_is_refused_at_export_and_freeze[False]
27	8.20s	call	tests/test_freeze_terra.py::test_rebound_parse_flag_is_refused_at_export_and_freeze[True]
28	7.89s	call	tests/test_snapshot_artifacts.py::test_snapshot_audit_annotations_have_no_orphan_rows
29	7.86s	call	tests/test_call_provenance.py::test_rows_record_each_responses_provenance_and_agree_with_the_ledger
30	7.82s	call	tests/test_notes.py::test_categorical_asset_error_statements
31	7.69s	call	tests/test_freeze_terra.py::test_rejudged_adjudicated_stage_passes_the_freeze
32	7.63s	call	tests/test_cost_latency.py::test_cost_is_present_exactly_when_recorded_usage_supports_it
33	7.56s	call	tests/test_uk_audit.py::test_a_later_stratum_does_not_change_earlier_draws
34	7.41s	call	tests/test_cost_latency.py::test_a_replay_can_only_lower_the_median
35	7.38s	call	tests/test_reference_audit_pins.py::test_the_impact_script_scores_only_the_staged_inputs[reference_audit/2026-10-05-louisiana]
36	6.79s	call	tests/test_snapshot_artifacts.py::test_frozen_impact_summaries_score_the_scored_reference
37	6.61s	call	tests/test_freeze_terra.py::test_a_run_state_treatment_off_the_registry_is_refused
38	6.52s	call	tests/test_freeze_argon.py::test_a_stage_list_is_appended_to_release_20261013bs_amendments
39	6.11s	call	tests/test_tool_pilot_argon.py::test_score_writes_a_strict_json_audit_and_flags_benchmark_searches
40	5.98s	call	tests/test_reference_audit_pins.py::test_the_direct_script_reproduces_its_files_from_a_junk_checkout[reference_audit/2026-10-05-medicare-part-b/scripts/propose_exclusions.py]

Test plan

• uv lock --check, ruff check ., ruff format --check .
• Serial suite on the VM with unsorted and with sorted parameters
• Three consecutive -n 4 --dist loadfile runs and one --dist load run, with per-test outcome comparison
• CI at 4491519: run 41490222217, attempts 1–3, all green
• Review r1 (Claude Opus 5.5) approved 4491519; 525d167 makes its suggested changes (-n logical, the timeout comment)
• CI at 654c525: run 41526298328, all green
• CI at efcecfc: run 41529016470, all green
• Review r2 (Claude Opus 5.5) at efcecfc: no blocking finding in the code, and this description makes the changes it asked for

axiom: n/a: CI test runner change, no policy logic

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Nov 2, 2026 9:33pm UTC

[MaxGhenis]

Ready for hub review

Head: 4491519e39d6049253197edafc0f111eaa087d3f

Items and how I resolved them

1. Ownership check. I listed the 25 open PRs into main and their changed files. No PR adds pytest-xdist. Add an unactivated v2 household prompt contract for issue 165 #173 touches ci.yml's test job, but only to add fetch-depth: 0, which main already has; it doesn't change the pytest command. I proceeded.
2. Branch. The branch starts at origin/main 875fe1f. Its name is claude/ci-parallel-tests-5i4gyq, not ci-parallel-tests: this cloud session is assigned that branch name and may push only there.
3. Baseline (VM, 4 vCPU / 16 GB, Python 3.12). --collect-only: 6873/6909 tests collected (36 deselected). Serial run: 34:23 wall (pytest 34:10), 6757 passed, 89 skipped, 27 failed. The 27 are tests/test_scenarios.py tests that download the certified US dataset from huggingface.co (materialize_certified_us_dataset → hf_hub_download); the VM's network policy refuses that host. The 40 slowest tests are in the PR description.
4. pytest-xdist. I added pytest-xdist>=3.6 to the dev extra and ran uv lock --upgrade-package pytest-xdist, which locked 3.8.0 and execnet 2.1.2. Local uv 0.11.32 also dropped pexpect's ptyprocess marker; I restored that line by hand, and uv lock --check accepts it. git diff origin/main -- uv.lock now holds only additions. No policyengine, policyengine-us, policyengine-uk, policyengine-core or litellm line moves.
5. Parallel safety. The first -n 4 --dist loadfile run stopped at start-up: workers collected test_reference_audit_pins.py tests in different orders. Two tests built their parameters from sets of path strings, whose order follows the per-process hash seed. I changed both to iterate over sorted paths (commit a79e57d). Nothing else failed only in parallel. No test is skipped, deselected, re-marked, weakened or put in an xdist_group.
6. Nothing lost. After the fix:
   • The sorted collected node ids match main's exactly.
   • The collected order is identical under PYTHONHASHSEED 1, 2 and 3.
   • I compared per-test outcomes from --junitxml between the post-fix serial run and parallel runs 2, 3 and load: 6873 cases in each, no id missing on either side, no outcome different.
   • The post-fix serial run gave the same 6757 passed / 89 skipped / 27 failed.
7. CI. The test job now runs uv run pytest -m "not slow" -n auto --dist loadfile --tb=short -q. timeout-minutes stays 30, with a rewritten comment giving main's three serial times and the VM times. lint, app and the other workflows are unchanged.
8. Draft PR and CI runs. Run 41490222217, attempts 1, 2 and 3 (two reruns through POST .../rerun; GitHub accepted both). All three tested merge eb99ee4 (this head + main b0c8f2e) and passed. Test job: 12.7, 12.5 and 16.2 min (pytest 12:00, 11:45, 15:39); 6900 passed and 90 skipped each time. The serial test job on main at b0c8f2e took 29.6 min (pytest 28:47) with the same counts.
9. PR description. It covers the VM and CI times, the invariant with the node-id and per-test comparison, the tests-changed table, and the 40 slowest tests before and after, with the growth analysis.

Tests and checks run

• VM serial, Python 3.13 (first try, before unshallowing): 29 failed, 18 errors. The errors came from the shallow clone, which can't read the git history the release gates need, and from Hugging Face. I discarded this run as a baseline.
• VM serial, Python 3.12, full history, main: 6757 passed, 89 skipped, 27 failed (Hugging Face), 34:10.
• VM -n 4 --dist loadfile, before the fix: collection mismatch, no tests run.
• VM -n 4 --dist loadfile, three runs in a row: 6757 / 89 / 27 each; 9:28, 9:39, 9:09.
• VM -n 4 --dist load: 6757 / 89 / 27; 10:08.
• VM serial after the fix: 6757 / 89 / 27; 32:05.
• uv lock --check, ruff check ., ruff format --check .: clean.
• CI: three green runs as above.

Judgment calls

• Python 3.12 on the VM. uv first built the venv on the VM's default Python 3.13. CI installs 3.12, so I rebuilt with --python 3.12 and took the baseline there.
• Full history. I unshallowed the VM clone to match CI's fetch-depth: 0, because the release-gate tests read files from old commits.
• Hugging Face failures. I left the 27 network-dependent failures on the VM as they are, added no credentials, and relied on CI, where they pass in parallel, to confirm them. They read a public dataset with no token. The task asked me to report any test that reaches the network; these do.
• loadfile over load. loadfile ran faster on the VM (9:09–9:39 against 10:08) and keeps module-scoped fixtures to one build per file, as in a serial run. Both modes gave identical per-test outcomes.
• Timeout kept at 30. Parallel CI ran 12.5–16.2 min. The serial job on today's main came 24 seconds from the limit, which this PR removes as a near-term risk.
• The ci.yml comment cites VM times, not this PR's CI times. Writing the CI times into the file would take another commit after the three runs, and that commit would start a fourth run on a new head. The CI times are in the PR description instead.
• Run 3's time. Run 3 took about 3.5 minutes longer than runs 1 and 2 on the same commit and test count. The logs don't say why; I report it as runner variance.
• main moved during the work. Count only real request timeouts in the supervisor's concurrency backoff #183 and Probe card-less models under the card the onboarding gauntlet derives #189 landed after I branched. I didn't merge or rebase; GitHub's merge ref covered them, and the PR reports as clean to merge.
• v2-microcosm-batches. I didn't touch it or any v2-* branch, and I didn't measure this change merged with it.

---

Generated by Claude Code

[MaxGhenis]

Ready for hub review

Independent review by a Claude Opus 5.5 cloud session for the PolicyBench owner session.

Verdict: APPROVE at 4491519e39d6049253197edafc0f111eaa087d3f

The PR's head on GitHub is still 4491519e. One brief value has moved: #252's head is now dc43c517 (one commit after 081727b9). I checked merges with both.

None of the findings below blocks the merge. Findings 1–3 propose comment or description text.

Findings

1. -n auto counts physical cores, so CI may start fewer workers than the comment says (non-blocking, UNVERIFIED)

.github/workflows/ci.yml:50-52 at the head:

      # One worker per core. --dist loadfile keeps each test file on one
      # worker; on the 4-core VM it ran faster than --dist load (10.1 minutes).
      - run: uv run pytest -m "not slow" -n auto --dist loadfile --tb=short -q

The description also says the job runs "on every runner core". In pytest-xdist 3.8.0, -n auto asks psutil first, for the physical core count (xdist/plugin.py:26-34):

    try:
        import psutil
    except ImportError:
        pass
    else:
        use_logical: bool = config.option.numprocesses == "logical"
        count = psutil.cpu_count(logical=use_logical) or psutil.cpu_count()

psutil is installed in CI: uv.lock lists it under ipykernel, policyengine and policyengine-core, and the synced environment has psutil 6.1.1. If the runner exposes 4 vCPUs as 2 cores with 2 threads each, -n auto starts 2 workers. If it exposes 4 cores, it starts 4.

I couldn't settle which applies. The CI logs don't show the worker count, because under -q xdist prints only bringing up nodes... (xdist/dsession.py:498-504, where the status line needs verbose >= 0). Attempt 1's job log shows bringing up nodes... twice at 17:21:21 and no created: N/N workers line. On my 4-vCPU session, psutil.cpu_count(logical=False), psutil.cpu_count() and len(os.sched_getaffinity(0)) all return 4, so I can't reproduce a difference here.

The speed-up doesn't settle it either. Serial main runs on nearly the same suite (6,784 to 6,900 passing tests) vary from 16:43 (875fe1f) to 28:47 (b0c8f2e) of pytest time, so a 2.4× ratio between one serial and one parallel run doesn't establish the worker count. (On my 4-vCPU session, with -n 4, the same comparison gave 43:26 / 11:11 = 3.9×.)

This doesn't make the job wrong: it passed three times and stays well inside the limit. But the comment states something the code may not do. One CI run of uv run python -c "import psutil, os; print(psutil.cpu_count(logical=False), os.cpu_count())" would settle it. Either of these changes would make the comment true whatever the runner reports:

      # One worker per logical CPU (-n auto would count physical cores, since
      # psutil is installed). --dist loadfile keeps each test file on one
      # worker; on the 4-core VM it ran faster than --dist load (10.1 minutes).
      - run: uv run pytest -m "not slow" -n logical --dist loadfile --tb=short -q

or keep -n auto and change the first comment line to # One worker per physical core (xdist uses psutil when installed).

2. The comment on timeout-minutes: 30 is out of date and doesn't match the description (non-blocking, text)

.github/workflows/ci.yml:33-37 at the head:

    # Run serially, the three latest test jobs on main took 17.3, 15.3 and
    # 19.9 minutes (2026-10-31, 10-26, 10-25). On a 4-core VM on 2026-11-02
    # the suite took 34.4 minutes serially and 9.2 to 9.7 minutes in three
    # runs with -n 4 --dist loadfile. Fail a hung job instead of holding a
    # shared runner for GitHub's 6-hour default.

• The 17.3, 15.3 and 19.9 figures match GitHub's records (table under "Checked and sound"). When 4491519 was committed (16:47 UTC on 2 Nov), they were the three latest. They aren't now. main's test jobs since then:

  Commit	Run	Test job
  c9f7269 (Count only real request timeouts in the supervisor's concurrency backoff #183)	41486939223	29:09, success
  b0c8f2e (Probe card-less models under the card the onboarding gauntlet derives #189)	41486949933	29:32, success
  f0f7fae (Pin the reference adversary's leaderboard impact to the pass's inputs #205)	41494610573	29:07, success
  7bbd92b (Record PolicyEngine provenance without importing policyengine #186)	41494665359	23:14, success
  ec5413c (Merge batch repair rounds cell by cell, like sync #180)	41495271792	cancelled at 30:01: the check run's annotation reads "The job has exceeded the maximum execution time of 30m0s", and the log shows the suite at [100%] at 18:22:45, about seven seconds before the job ended at 18:22:53

  So the serial job on main already hits this limit. That makes this PR more urgent than the description's "28 seconds short" suggests.

• "34.4 minutes serially" doesn't match the description's Times table, which gives 34:10 (34.2 min) before the fix and 32:05 (32.1 min) after. UNVERIFIED either way, since it is a VM figure.

• The comment leaves out the evidence that justifies keeping 30 minutes: the three CI attempts of the parallel job.

Suggested replacement for lines 33-37:

    # Serially, the test job on main took 29.5 minutes at b0c8f2e and hit
    # this limit at ec5413c (2026-11-02). With -n auto --dist loadfile it
    # took 12.7, 12.5 and 16.2 minutes (run 41490222217, attempts 1-3). Fail
    # a hung job instead of holding a shared runner for GitHub's 6-hour
    # default.

3. Description: three small corrections (non-blocking)

• Run 3. "Run 3 took about 3.5 minutes longer ... and its logs don't show why." That's accurate in that no line names a cause. But the logs do show the slowdown was spread evenly, not concentrated in one test. Elapsed seconds from the first progress line to each percentage mark:

  Attempt	10%	25%	50%	75%	90%	95%	100%
  1	17	87	180	476	594	663	700
  2	18	87	176	464	564	634	687
  3	23	108	214	614	764	859	917

  Attempt 3 runs 20–30% behind at every mark, while its checkout, pip install uv and uv sync steps took about as long as attempts 1 and 2 (20 s, 2 s, 3 s against 17–18 s, 2–5 s, 5–6 s). That points to a slower runner, not a test. Suggested sentence: "Run 3 was 20–30% slower at every progress mark, from 10% to 100%, with no single test holding it up, which points to a slower runner."

• The 7.4-minute floor. The bound is correct: one worker runs all of test_freeze_terra.py (LoadFileScheduling subclasses LoadScopeScheduling; xdist/scheduler/loadfile.py:10). But the job can't actually reach it. With --loadscope-reorder, on by default (xdist/plugin.py:131-134), xdist queues files by test count, not by duration (xdist/scheduler/loadscope.py:368-373). On today's merge, test_freeze_terra.py has 265 tests and is 7th by count. With 4 workers it goes out in the initial batch as some worker's second file, so it starts only after that worker finishes its first file. The floor is then "first file on that worker + 7.4 minutes". Suggested wording: "...so more cores can't bring the job under 7.4 minutes there, and in practice it starts after another file on its worker finishes."

• Base counts. "On main at 875fe1f and on this branch ... collects 6,873 of 6,909 tests (36 deselected)" is correct (below). Today's main merged with the head collects 7,097 of 7,135 (38 deselected), so the owner's rerun after syncing will report larger totals than the description.

4. Non-blocking notes for the owner

• CLAUDE.md:6 and :30, and README.md:60, tell contributors to run pytest / pytest -m "not slow". These still work. On a 4-core machine, pytest -m "not slow" -n 4 --dist loadfile took 11:11 for me against 43:26 serially. A one-line note in CLAUDE.md would save local runs time. Not needed for this PR.
• When the parallel suite runs from a shallow clone, it fails 22 more tests that need git history: test_build_references_upgrade.py, _1012.py, _1022.py ("cannot read ... at f108a95 (fetch full history)") and two test_notes.py bbce tests. My first parallel run hit this because my session's clone was shallow. The cause is the clone, not parallelism: CI's test job sets fetch-depth: 0 (ci.yml:44), and after git fetch --unshallow these tests passed in both modes.

Checked and sound

Item 1: the parallel suite on today's main. Scratch merge git checkout -B scratch-merge origin/main && git merge --no-ff --no-edit 4491519e... (origin/main = 4e5a8459, #251), uv sync --locked --extra dev --python 3.12: pytest-xdist 3.8.0, filelock 4.2.6, Python 3.12.3. Full history (after git fetch --unshallow).

• Parallel: uv run pytest -m "not slow" -n 4 --dist loadfile --tb=short -q -p no:cacheprovider --junitxml=par2.xml --durations=0 -rfE → 27 failed, 6981 passed, 89 skipped, 3 warnings in 671.85s (0:11:11).
• Serial: uv run pytest -m "not slow" --tb=short -q -p no:cacheprovider --junitxml=ser.xml --durations=0 -rfE → 27 failed, 6981 passed, 89 skipped, 38 deselected, 3 warnings in 2606.91s (0:43:26).
• I ran them one after the other. A first serial attempt was stopped at 81% by my session's 30-minute background limit, so I reran it in full. A first parallel run on a shallow clone is the one described in finding 4.
• Comparison of the two junit files by (classname, name): both files hold the same 7,097 test cases (none only in one), and no test's outcome differs (6,981 passed, 89 skipped, 27 failed in each). This confirms the description's "No other test failed only in parallel" on today's main.
• My session's proxy refuses huggingface.co (curl https://huggingface.co/api/models/gpt2 → CONNECT tunnel failed, response 403). The 27 failures in each run are the same 27 tests, all in tests/test_scenarios.py: test_adult_dependents_are_allowed_when_structure_is_clean, test_aggregate_net_worth_input_is_not_preserved, test_children_and_adults_split_by_age, test_conditional_inputs_are_not_preserved, test_formula_county_fields_are_not_prompted_or_sent_to_policyengine, test_generate_scenarios_count, test_generate_scenarios_deterministic, test_generate_scenarios_different_seeds, test_generate_scenarios_uses_loader, test_geographic_leaf_inputs_are_preserved, test_invalid_households_are_filtered, test_large_households_are_allowed_when_structure_is_clean, test_load_scenarios_from_manifest_round_trips, test_missing_bool_inputs_are_not_preserved, test_overtime_premium_input_is_sent_to_policyengine, test_pe_household_includes_cross_entity_inputs, test_promptable_inputs_are_pure_leaf_variables, test_richer_cps_inputs_are_preserved, test_scenario_manifest_exports_summary_fields, test_scenario_structure, test_scenarios_from_cps_frame_can_exclude_households, test_stated_usual_hours_reach_the_snap_work_tests (3 cases), test_total_income_ignores_deductions_and_hours, test_unstated_or_zero_hours_keep_the_engine_default (2 cases). In both junit files, every one of the 27 failure texts names huggingface.co/hf_hub_download with the proxy refusal, and diff of the two sorted FAILED lists prints nothing. The author saw the same 27.

Item 2: collection order.

• PYTHONHASHSEED={0,1,2,3,random} uv run pytest -m "not slow" --collect-only -q -p no:cacheprovider on the scratch merge: all five give 7,097 ids with md5 prefix 2d7223b27d08.
• Control: with sorted(...) removed from the two parametrizations in a scratch copy, --collect-only on tests/test_reference_audit_pins.py under seeds 1, 2 and 3 gave three different md5s (d8c833a1ddb3, 2b7f223738f7, 3bc27c3c8f32), so the check detects the defect the PR fixes. I restored the file with git checkout.
• I searched tests/, including the files Pin the reference adversary's leaderboard impact to the pass's inputs #205, Record PolicyEngine provenance without importing policyengine #186 and Merge batch repair rounds cell by cell, like sync #180 added or changed, for parametrizations and params= fixtures over sets, frozensets, set comprehensions, .keys()/.items()/.values() and globs. The two params= fixtures already sort (tests/test_reference_upgrade.py:180 params=sorted(TREES), tests/test_paper_results.py:546 params=sorted(RESULTS)). tests/test_notes.py:293 sorts its glob. The remaining .items()/.values() parametrizations (test_eval_no_tools.py:3320, test_finish_terra.py:3008, test_request_timeout_signature.py:120,127, test_cache.py:1145) iterate dicts, which keep insertion order. In test_reference_audit_pins.py, _refusals sorts its set (sorted(REWRITTEN & set(module.INPUTS)), sorted(path for path in module.INPUTS ...), lines 472 and 475), and the other parametrizations iterate the IMPACT/DIRECT/SPECS dicts. No pytest_generate_tests or pytest_collection_modifyitems hook exists in tests/.
• pytest-xdist 3.8.0 doesn't set a hash seed. grep -rni hashseed over xdist/ and execnet/ finds nothing. Workers start through execmodel.subprocess.Popen(args, stdout=PIPE, stdin=PIPE) (execnet/gateway_io.py:34), with no env=, so each worker inherits the controller's environment. Unless CI sets PYTHONHASHSEED (it doesn't), each worker randomizes str hashes on its own. xdist compares the workers' collections as ordered lists (xdist/report.py:18, if from_collection == to_collection, called from loadscope.py:409-421). So any future parametrization over a set of strings will stop the run with "Different tests were collected". The five-seed check above is the guard. A test or a conftest.py check could make it permanent, but this PR doesn't need one.

Item 3: state shared across workers.

• Empirical check: I touched a marker before parallel run 2 and ran find . -newer marker (excluding .venv, .git, __pycache__) and git status --porcelain after. No tracked file changed (git status empty). The only new paths were .hypothesis/ and an empty results/local/ (both git-ignored; results/local/ is the CLI's default output directory, policybench/cli.py:157 and others) plus __pycache__ directories. Run 1 gave the same result. Hypothesis writes its example database to ./.hypothesis/, shared by the workers. Hypothesis's directory database is meant for concurrent use, and no test failed from it. I didn't read Hypothesis's source to confirm its write protocol.
• os.chdir: none in tests outside a string fixture (test_fix_module_closure.py:71). Six files use monkeypatch.chdir, which is per process.
• Environment variables set without monkeypatch: test_model_response_window.py:154-164 sets TZ and calls time.tzset() with a try/finally restore. That is per process, so the same in serial and parallel. test_eval_no_tools.py:3060 sets LITELLM_LOCAL_MODEL_COST_MAP inside a code string run in a child interpreter.
• Fixed /tmp paths: none. Temporary paths come from tmp_path, tempfile.TemporaryDirectory() or tempfile.mkdtemp(...), all unique per call (test_consensus.py:744 mkdtemp(prefix="pb-pass-payload-") is never removed, but it can't collide). No test binds a port.
• Git: tests run git show, log, merge-base, rev-parse and ls-files against the shared checkout (read-only, no index lock). git add/commit run only in scratch repositories (test_finish_*.py, test_freeze_terra.py:1089-1090, with -C <root> under the test's temporary directory).
• Monkeypatched repository paths: e.g. test_freeze_gpt61sol.py:505 unlinks driver.JUDGE_PROVENANCE, which line 469 monkeypatches to a temporary record.
• Hugging Face: only policybench/policyengine_runtime.py:474-507 (materialize_certified_us_dataset) calls hf_hub_download. It is reached through make_us_microsimulation (policyengine_runtime.py:510-515), policybench/scenarios.py:477, 931 and policybench/population_weights.py:184. In both my runs every test that reached the download failed with the proxy's 403, and all 27 are in tests/test_scenarios.py. Under --dist loadfile that file runs on one worker, so two workers don't download at once in this suite. If a second file ever does, huggingface-hub 1.4.1 guards each blob with WeakFileLock(lock_path) (file_download.py:1214-1247), which wraps filelock.FileLock. I ran four spawn processes contending one WeakFileLock under filelock 4.2.6: each held it in turn ("max holders at once: 1", exit codes all 0).

Item 4: lock and pyproject.toml.

• pyproject.toml:29: "pytest-xdist>=3.6", inside dev = [ only. grep -n xdist pyproject.toml finds just that line.
• git diff 875fe1f 4491519e -- uv.lock: no removed lines. It adds the execnet 2.1.2 and pytest-xdist 3.8.0 package entries and one line each to the dev list and requires-dist. The header lines (version = 1, revision = 3, requires-python, resolution-markers) are unchanged.
• uv lock --check: passes at the head and on the scratch merge (Resolved 139 packages).
• CI's test job runs uv sync --locked --extra dev (ci.yml:49), so xdist is installed where -n runs.

Item 5: workflow. The lint and app jobs are unchanged. .github/workflows/paper.yml doesn't run pytest. No Makefile; CLAUDE.md and README.md mention plain pytest (finding 4). Worker count: finding 1.

Item 6: the test change.

• --collect-only -q at the merge base 875fe1f and at the head (each in its own worktree with PYTHONPATH set to it): both 6873/6909 tests collected (36 deselected). The sorted id lists are identical (diff prints nothing), and 52 diff lines differ in unsorted order (26 reordered ids).
• Mutation, impact script: with if digest != pinned: in reference_audit/2026-10-05-louisiana/scripts/leaderboard_impact.py:175 replaced by if False:, test_any_refused_input_stops_the_impact_script_before_it_scores[reference_audit/2026-10-05-louisiana-paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/data.json.gz] failed with AssertionError: stage ran after the inputs were refused. Restored, it passes. Killed.
• Mutation, direct script: with the same change at reference_audit/2026-10-05-medicare-part-b/scripts/propose_exclusions.py:142, test_any_refused_input_stops_the_direct_script_before_it_writes[reference_audit/2026-10-05-medicare-part-b/scripts/propose_exclusions.py-app/src/modelMeta.ts] failed with Failed: DID NOT RAISE SystemExit. Restored. Killed.

Item 7: merging.

• git merge-tree --write-tree origin/main 4491519e → 743458ef..., exit 0.
• git merge-tree --write-tree <#252 head dc43c517> 4491519e → 56b3415e..., exit 0. With 081727b9 (the brief's Move litellm to 1.95.0 (GHSA-g5ff-637f-6q2m) and accept it in the tool pilot's Gemini patch #252 SHA) → 6983955f..., exit 0.
• Three-way scratch merge (origin/main, then Move litellm to 1.95.0 (GHSA-g5ff-637f-6q2m) and accept it in the tool pilot's Gemini patch #252 at dc43c517, then this head): uv lock --check passes (Resolved 139 packages). The lock moves litellm 1.88.6 → 1.95.0 and adds execnet 2.1.2 and pytest-xdist 3.8.0, with no other name or version line changing against origin/main. I didn't run the suite on the three-way merge (below).

Item 8: the description.

• CI times, from actions/runs/<run>/attempts/<n>/jobs and the job logs:

  Claim	started → completed	pytest summary line
  875fe1f 17.3 min, pytest 16:43	12:05:37 → 12:22:55 = 17:18	6784 passed, 89 skipped, 36 deselected, 3 warnings in 1003.77s (0:16:43)
  bd11d55 15.3 min	23:37:54 → 23:53:14 = 15:20	—
  e0217ed 19.9 min	17:38:47 → 17:58:40 = 19:53	—
  b0c8f2e 29.5 min, pytest 28:47	17:00:32 → 17:30:04 = 29:32	6900 passed, 90 skipped, 36 deselected, 3 warnings in 1727.79s (0:28:47)
  attempt 1: 12.7 min, 12:00	17:20:28 → 17:33:11 = 12:43	6900 passed, 90 skipped, 3 warnings in 720.16s (0:12:00)
  attempt 2: 12.5 min, 11:45	17:34:12 → 17:46:42 = 12:30	6900 passed, 90 skipped, 3 warnings in 705.36s (0:11:45)
  attempt 3: 16.2 min, 15:39	17:47:06 → 18:03:20 = 16:14	6900 passed, 90 skipped, 3 warnings in 939.75s (0:15:39)

  All match. "29 minutes 32 seconds, 28 seconds short" matches. The check-runs for the head show test, lint, app and paper as success.

• Growth ed4f89a → 875fe1f: git ls-tree gives 53 → 91 tests/test_*.py files. git grep -c '^\s*def test_' over tests/*.py gives 1,090 → 3,608. Line counts of tests/**/*.py give 35,168 → 116,719. All match.

• The 909 s sum: 443+123+83+81+71+56+52 = 909, correct. The seven files were first added on 2026-10-01 (test_freeze_gpt61sol.py), 2026-10-10 (the haiku55 and musewonder freeze and finish files) and 2026-10-31 (the terra files), so "added between 1 and 31 Oct" holds. Per-file VM seconds: UNVERIFIED (my machine differs). My parallel run 2 shows the same pattern: --durations=0 summed to 2,451 s, with test_freeze_terra.py first (560 s), then test_notes.py 178, test_freeze_gpt61sol.py 162, test_run_audit_claude.py 130, test_reference_audit_pins.py 116, test_freeze_haiku55.py 111 and test_freeze_musewonder.py 107. The seven files summed to 1,170 s.

• "The 40 slowest tests account for about 557 s of the 34-minute serial run": UNVERIFIED for the VM. My serial run shows the same share: the 40 slowest phases sum to 684 s of 2,509 s of --durations=0 time (27%), against the description's 557 s of about 2,040 s (27%).

• VM wall times (34:10, 32:05, 9:28/9:39/9:09, 10:08) and "6,757 passed, 89 skipped and 27 failed": UNVERIFIED (VM). My runs on today's larger merge took 11:11 in parallel and 43:26 serially, and failed the same 27 Hugging Face tests.

• The 7.4-minute floor: 443 s = 7.4 min, and the reasoning holds as a lower bound (finding 3).

• "This PR puts no test in an xdist_group, and it skips, deselects, re-marks or weakens none": the diff touches only the two parametrize lines in tests (above). grep -rn xdist_group tests finds nothing.

What I couldn't check

• How many workers -n auto starts on GitHub's runner (finding 1). One CI line printing psutil.cpu_count(logical=False) would settle it.
• The 27 Hugging Face tests in tests/test_scenarios.py: the proxy blocks huggingface.co, so I never saw them pass. CI ran them (6,900 passed in each parallel attempt). I couldn't test concurrent downloads against the real hub, only the lock.
• The VM timings in the description (marked UNVERIFIED above).
• A parallel suite run on the three-way merge of origin/main, Move litellm to 1.95.0 (GHSA-g5ff-637f-6q2m) and accept it in the tool pilot's Gemini patch #252 and this head. I checked that it merges cleanly and that the lock resolves; the owner's CI rerun after syncing will cover the rest.
• Hypothesis's directory-database write protocol under concurrent writers (no failures observed; source not read).

---

Generated by Claude Code