[MaxGhenis]

Problem

PolicyEngine-US's Contrib other-shard-1 job runs under a 60-minute limit. On 2026-10-30 it timed out twice on a PR that changed nothing in the shard: its YAML step took 43:43 and 42:50, against 33:39 on a passing run. PolicyEngine/policyengine-us#10556 has since moved the contrib SSA YAML route from that shard to other-shard-3.

The YAML runner builds a full copy of the tax-benefit system for each new reform combination (_get_tax_benefit_system in policyengine_core/tools/test_runner.py), and on policyengine-us automatic garbage collection takes most of that build. On core master 617673ca under Python 3.14.7, the first combination of owners_draw.yaml set off 7 full collections in each of two runs, and collection spent 9.3 and 10.0 s of 12.1 and 13.1 s CPU builds. The copy added 15.7 million tracked objects to the 3.9 million the process held before it.

Change

On a cache miss, _get_tax_benefit_system runs the build (baseline.clone(), the reforms and the extensions) inside _automatic_gc_paused(collect):

• it records gc.isenabled() and disables automatic collection;
• when the build returns, the caller had collection on, and the call asked for collect, it runs one gc.collect();
• in a finally, it re-enables collection only if the caller had it on, so a build that raises, or a caller that had disabled collection, keeps the caller's setting.

_get_tax_benefit_system asks for collect only while the cache holds fewer than _FULL_COLLECTION_CACHE_LIMIT (3) systems, counting every baseline's entries. A miss past the limit still pauses collection and restores it, then leaves the new system to the automatic collector. A cache hit returns first and touches nothing. The change uses no gc.freeze() and leaves gc.set_threshold alone.

This version adds the limit. At 569c56a7 every successful miss forced a full collection, and each one walked every system the cache held, so a run of misses cost quadratic time. On the country template, 100 distinct parameter reforms took 5.9 s of CPU there against 0.07 s on its base (Python 3.14.7), with 100 full collections against none.

Why a limit

A forced full collection walks every tracked object in the process, so each cached system adds to its cost. CPython's automatic collector runs a full collection only once middle collections have moved into the oldest generation at least a quarter as many objects as survived the last full one. It applies that test in gc_select_generation (Python/gc.c, lines 1258 to 1304 at v3.14.7, and the same test at v3.13.9) and in gc_collect_generations (Modules/gcmodule.c at v3.11.15 and v3.12.14), so its full collections grow rarer as the heap grows. gc.collect() runs a full collection directly (gc_collect_impl in Modules/gcmodule.c, all four versions) and skips the test. It also merges the younger generations into the oldest, so the copy's objects skip the young and middle collections they would otherwise go through. That trade pays while the copy makes up a large share of the heap.

Choosing 3 (a model, not a PE-US measurement)

A second PE-US combination in one process passes the 4 GiB cap, so a model sets the constant. model_probe.py builds country-template systems through the real _get_tax_benefit_system. A callable reform gives each new system 2 million tracked objects (__slots__ instances chained to each other), allocated inside the build, where this head pauses collection. Padding on the baseline puts 0.86 million tracked objects in the heap before the first miss, 0.43 of a copy. That doubles the one-system heap's share of a PE-US copy (3.87 million against 17.3 million before #631) to stand in for CI's two-system heap. After each miss, a case allocates 375,000 short-lived objects in chunks of 3,000, running 113 to 116 young and 10 or 11 middle collections while promoting little to the oldest generation.

With a smaller pre-heap (0.3 of a copy, near the one-system heap's 0.22 to 0.25) and one miss, the model's master build ran 5 full collections and this head's 1, and master spent 4.0 times this head's collection CPU. On PE-US the same build ran 7 against 1, and master spent 3.0 to 3.3 times as much.

The table gives collection CPU over a run of misses on Python 3.14.7, from two rounds of one 32-miss process per arm, with full collections in brackets. The first n misses of a 32-miss process run the same collections as a process with n misses.

Misses	master 33dc9bb3	569c56a7	this PR (4baa215c)
1	0.57, 0.55 s (5)	0.13, 0.14 s (1)	0.13, 0.12 s (1)
2	0.94, 0.91 s (7)	0.30, 0.31 s (2)	0.29, 0.29 s (2)
4	1.75, 1.70 s (10)	0.78, 0.80 s (4)	0.84, 0.82 s (4)
8	3.19, 3.14 s (13)	2.23, 2.24 s (8)	1.94, 1.91 s (6)
16	5.97, 5.71 s (16)	7.11, 7.15 s (16)	3.87, 3.87 s (8)
32	11.12, 10.67 s (19)	23.56, 24.18 s (32)	8.34, 8.23 s (11)

Forcing a collection after every miss beat the automatic collector over the first four misses (0.20 s a miss against 0.29 s), broke even over misses 5 to 8 (0.36 against 0.34 s), and lost from the ninth on (0.53 against 0.33 s over misses 9 to 12). Across limits, at 16 and 32 misses:

Limit	0 (pause only)	1	2	3	4	5	6	8	none (569c56a7)
16 misses	4.87 s	3.95 s	4.52 s	3.87 s	4.13 s	4.58 s	3.95 s	4.16 s	7.11 s
32 misses	11.39 s	8.54 s	9.44 s	8.34 s	8.82 s	9.60 s	8.48 s	8.81 s	23.56 s

A limit of 3 gave the least collection CPU at both counts. Limits from 1 to 8 came within 20% of it, the pause alone cost 26% and 37% more, and no limit cost 1.8 and 2.8 times as much. A variant with equal-size systems put two padded systems of 1.75 million objects in the heap before the first miss and ran one process per count. There too this head spent the least from 8 misses on: 7.53 s at 32 misses, against master's 11.98 s and 569c56a7's 26.28 s.

Treat 3 as a practical bound, not a proven best value, because the ranking moves with the heap's shape. A variant padded with real template clones starts with about 2.2 copies of heap, against 0.43 above. Its first run favoured limits 0 to 2, but a rerun at 16 misses put limit 3 ahead of limit 0 (6.61 s against 6.93 s). Timings also move between runs: the 32-miss arm at limit 3 took 10.53 s in that rerun, with the same 11 full collections as the 8.34 s in the table. No evidence shows that 2 or 4 would do materially better on PE-US.

Invariants

Each invariant has a test in tests/core/tools/test_runner/test_gc_pause.py, which holds 22 tests.

1. Pausing changes no result. test_yaml_cases_have_the_same_outcomes_with_and_without_the_pause runs core's test_with_reform.yaml, test_with_reform_2.yaml and test_with_extension.yaml, plus three dotted-parameter cases (one failing on purpose), through the runner on fresh baselines, once with the pause and once with a null context in place of _automatic_gc_paused. The exit code, all 8 case outcomes and every computed value match, and every clone ran with collection off with the pause and on without it. Under policyengine-us, the 8 runs below and the 24 runs of the first version produced identical values and outcomes for each file, whatever the arm.
2. gc.isenabled() after a call equals its value before. A parametrized test covers a miss, a hit, and a clone, reform or extension that raises, each with collection on and off (10 cases). Another covers a miss past the limit with collection on and off. A Hypothesis test (200 examples, up to 25 calls each) sets the caller's state at random before every call.
3. The cache returns the identical object for a key it has seen, and builds a new system for any other key. The Hypothesis test checks the cache against a reference dict: a seen key returns the same object with no build and no gc call, and a new key builds a distinct system with the requested reforms and extensions.
4. A run of cache misses forces at most _FULL_COLLECTION_CACHE_LIMIT full collections. A miss runs one gc.collect(), a full collection of the oldest generation, only when the caller had collection on, the build returned, and the cache held fewer systems than the limit before the miss, counting every baseline's entries. A hit, a failed build, a caller with collection off, and a miss past the limit run none. The tests' spy records the generation each collection asks for, so gc.collect(1) fails them. A parametrized test sets the limit to 0, 1, 2, 3 and 5. It checks that the last forced collection falls on the miss that finds one system fewer than the limit cached, and that the next three misses run none. A second test fills the cache to the limit and checks that the next miss clones with collection off, runs no collection and restores the caller's setting. A third checks that systems cached for another baseline count toward the limit until that baseline goes away. The Hypothesis test checks the rule on every call and checks that no sequence forces more collections than the limit.

gc.disable() acts on the whole process, so two builds overlapping in different threads can leave collection on for a caller that entered with it off, but no production route overlaps them: YamlItem.runtest makes the only production call, and PE-US's batch threads each start a separate subprocess.

Mutants

Each mutant ran as a scratch copy against the new test file, one process each under a 2 GB cap and a 900 s timeout. All 13 died.

Mutant	Result	Killed by
Drop the finally	killed (4 failed)	state_after…before[{clone,reform,extension}_raises-True], …restores_the_callers_state_when_the_body_raises
Always re-enable instead of restoring	killed (7 failed)	state_after…before for a miss and the three raising builds with collection off, …body_raises, a_miss_past_the_limit…, Hypothesis cache test
Pause on cache hits too	killed (3 failed)	state_after…before[hit-True], a_cache_hit_leaves_collection_alone, Hypothesis cache test
Collect for a caller with collection off	killed (2 failed)	state_after…before[miss-False], Hypothesis cache test
No collection after the build	killed (8 failed)	…paused_then_collects_once, state_after…before[miss-True], misses_collect_only_while…[1,2,3,5], the_limit_counts…every_baseline, Hypothesis cache test
Collect after a failed build too	killed (4 failed)	state_after…before for the three raising builds with collection on, …body_raises
No pause	killed (5 failed)	…paused_then_collects_once, …body_raises, a_miss_past_the_limit…, Hypothesis cache test, YAML differential test
Collect in a nested finally (review)	killed (4 failed)	state_after…before for the three raising builds with collection on, …body_raises
gc.collect(1) (review)	killed (8 failed)	…paused_then_collects_once, state_after…before[miss-True], misses_collect_only_while…[1,2,3,5], the_limit_counts…every_baseline, Hypothesis cache test
Remove the limit (569c56a7's behaviour)	killed (8 failed)	misses_collect_only_while…[0,1,2,3,5], a_miss_past_the_limit…, the_limit_counts…every_baseline, Hypothesis cache test
Limit off by one (<=)	killed (8 failed)	misses_collect_only_while…[0,1,2,3,5], a_miss_past_the_limit…, the_limit_counts…every_baseline, Hypothesis cache test
Never collect	killed (8 failed)	…paused_then_collects_once, state_after…before[miss-True], misses_collect_only_while…[1,2,3,5], the_limit_counts…every_baseline, Hypothesis cache test
Count only this baseline's systems	killed (1 failed)	the_limit_counts…every_baseline

Measurements

The tables report CPU time, since wall times on the measuring lane swung up to threefold between rounds of the same arm.

Template probe

The review's sequence_probe.py 100 builds 100 distinct taxes.income_tax_rate reforms on the country template and keeps every system:

Python	Arm	CPU	Collection CPU	Full collections
3.13.9	master 617673ca	0.068 s	0.015 s	0
	569c56a7	6.146 s	6.043 s	100
	this PR 3464850c	0.239 s	0.182 s	3
3.14.7	master 617673ca	0.063 s	0.014 s	0
	569c56a7	5.933 s	5.849 s	100
	this PR 3464850c	0.234 s	0.178 s	3

This head exceeds master by 0.17 s on both versions, the cost of its three forced collections. 569c56a7 predates #631; against master 33dc9bb3 the probe gave master 0.095 s (3.13.9) and 0.072 s (3.14.7), and 4baa215c 0.235 s and 0.242 s.

PE-US, owners_draw.yaml's first combination (Python 3.14.7)

PE-US main at 4eeb4773 ran in its locked environment with update_itemization.sh's edit, through the first version's harness. Each process ran the file's first case through pytest.main with OpenFiscaPlugin and run_tests' arguments. It used the system policyengine_us builds at import as the baseline (a one-system heap), and read PE-US's and core's files from a preloaded bundle until the timed build. The two pairs on each master ran their arms back to back, in opposite orders.

Core	Arm	Build CPU	Collection CPU, build and first case	Full collections	Build + first case CPU	Peak RSS
after #631, pair 1	master 617673ca	13.05 s	10.02 s	7	13.76 s	2,950 MB
	this PR 3464850c	6.21 s	3.02 s	1	6.74 s	2,946 MB
after #631, pair 2	master 617673ca	12.08 s	9.36 s	7	12.65 s	2,948 MB
	this PR 3464850c	6.13 s	3.09 s	1	6.66 s	2,954 MB
before #631, pair 1	master 33dc9bb3	13.53 s	8.15 s	7	14.19 s	3,071 MB
	4baa215c	8.84 s	2.60 s	1	9.92 s	3,096 MB
before #631, pair 2	master 33dc9bb3	18.40 s	10.15 s	7	19.02 s	3,306 MB
	4baa215c	10.74 s	3.20 s	1	11.26 s	3,305 MB

• Build CPU fell 52% and 49% against current master, and 35% and 42% against 33dc9bb3. Collection CPU fell 67 to 70% in every pair, and every run computed the same 14 values and passed.
• Share nested metadata values between parameter clones #631 cut master's non-collection build CPU from 5.4 and 8.3 s to 3.1 and 2.7 s, so collection takes a larger share of what remains.
• After the build the process held 19.55 million tracked objects at current master and 21.17 million before Share nested metadata values between parameter clones #631, against 3.87 million after the import. The harness counted them after the timed build and took the count's 0.8 to 0.9 s out of "build + first case". The first pair before Share nested metadata values between parameter clones #631 ran without the count.
• 4baa215c differs from this head only in a comment and the merge of master.

Shard estimate (unmeasured)

These figures stay estimates until a core release reaches the job. Rebuilding the batches with test_batched.py's own functions at PE-US 4eeb4773, with SSA in other-shard-3 as #10556 routes it:

Shard	Subprocesses	Cases	Cache-miss builds	Builds past a subprocess's third	Estimated saving
other-shard-1 (per-subdirectory batches, 2 workers)	16	307	41	5	1.1 to 5.5 min
other-shard-3 (refundable-credit files, 2 workers, then SSA files, 1 worker)	21	117	73	25	3.3 to 9.7 min

The ranges assume the first version's 3.1 to 8 s saved per build: the low ends at 3.1 s with the two-worker routes fully overlapped, the high ends at 8 s with no overlap. Builds past a subprocess's third get the pause without a forced collection. In the model, misses past the third saved 17 to 19% of master's collection CPU, against 63 to 64% over the first three, so the high ends lean high. The first version's 28 subprocesses, 390 cases and 96 builds described the old routing, which ran SSA in other-shard-1.

First version's measurements (569c56a7, PE-US bdd08b8d, Python 3.14.7)

At its first combination each file's build behaves the same at this head, since the cache holds no system yet. Medians of three interleaved rounds, one-system heap:

File, first combination	Arm	Build CPU	Collection CPU	Full collections	Build + first case CPU
owners_draw	master 33dc9bb3	18.7 s	10.3 s	7	19.3 s
	569c56a7	11.1 s	3.2 s	1	11.7 s
owners_draw_credited_earnings	master	17.7 s	9.9 s	7	18.3 s
	569c56a7	10.5 s	3.1 s	1	11.0 s
owners_draw_net_earnings	master	18.2 s	10.2 s	7	18.8 s
	569c56a7	10.9 s	3.1 s	1	11.5 s

One interleaved pair with CI's two-system heap gave build CPU of 20.6 s against 12.2 s. Pausing without the collection (commit a92236561) moved the work into the next case: build plus first case took 37% and 20% more CPU than master in the same round. That version's summary.tsv holds 22 rows, and the two two-system results live only in their JSON files. Its 232-second import median covers the 20 one-system runs; across all 24 results the median falls to 205.7 s.

What this doesn't measure

• A second PE-US combination in one process passes the 4 GiB cap, so the limit rests on the model above.
• The PE-US pairs ran on Python 3.14.7, the version PE-US CI uses; only the template probe covers 3.13.9.
• No CI shard ran with this branch, so the shard figures stay estimates.

Tests and checks

• pytest tests/core/tools/test_runner/test_gc_pause.py: 22 passed.
• pytest tests/core/tools/test_runner tests/core/test_yaml.py tests/core/test_extensions.py tests/core/test_reforms.py tests/core/test_tax_benefit_system_cache_identity.py plus Share nested metadata values between parameter clones #631's two metadata-clone test files: 201 passed at 3464850c.
• ruff format --check . and ruff check .: clean.
• CI run 41174548954 at 3464850c: all checks passed.

Whichever of this PR and draft #569 lands second must wrap both of #569's build paths (its reform-free clone and its reformed build) in the pause, and count the systems its bounded cache holds (a reform-free system plus at most two reform systems, evicted oldest first) toward this limit.

impact: none on results: this PR changes when Python's collector runs, not what a simulation computes. An independent PE-US before-and-after run matched on all 14 compared values, and PE-US's own core tests passed against this head (782 passed).

axiom: n/a: core engine collection timing, with no policy change.

🤖 Generated with Claude Code

[MaxGhenis]

US + core hub merge audit, core#643 at 3464850c4e24e051062b5e02e106d94f00181cee (squash): Pause Python's collector during YAML test-runner system builds, and force a full collection only while fewer than three systems sit in the cache.

• Review: GPT-6.1 Sol, round 2 APPROVE at 3464850c4e with no P1 or P2 findings and SEMANTICS_CHANGE no (~/reviews/us-hub/reviews-gen/core-643/review-r2.md, evidence in r2-evidence/).
  • Sol reproduced the model: probe 100 reaches 3 full collections on Python 3.13.9 and 3.14.7.
  • Sol's PE-US pair on main 4eeb4773 under Python 3.14.7: build CPU fell from 11.452 s to 5.019 s (56.2%), collection CPU from 9.081 s to 2.784 s, and full collections from 7 to 1. RSS moved from 3,060 MiB to 2,973 MiB. All 14 compared values matched.
  • 201 tests passed. Mutants gc.collect(1), < to <= and a current-baseline-only count all failed tests (8, 8 and 1 failures).
• Round 2 P3s, both routed:
  • The limit's sensitivity to heap shape now sits in the body, with Sol's rerun figures and the wording "a practical bound, not a proven best value".
  • test_gc_pause.py has no interrupt coverage. A mutant with success-only restoration plus except Exception passes all 22 tests. The Stop the YAML test runner keeping every case's simulation and reform system #569 finish build now adds a BaseException test and the comment on heap-shape sensitivity at _FULL_COLLECTION_CACHE_LIMIT, because Stop the YAML test runner keeping every case's simulation and reform system #569 has to rework this same code. No user path reaches it short of a KeyboardInterrupt in a test run, so it doesn't hold this merge.
• Estimates in the body: the shard saving comes from these timings and stays an estimate until PE-US CI runs on a release with this change.
• Classification: CORE_SEMANTICS no. The PR changes when the collector runs, and no simulation output.
• PE-US core gate: PE-US's tests/core ran against this head, which already contains master 617673ca13, with PE-US main 5a6df2c085: 782 passed in 1:01:04. The run imported core from the gate tree.
• Gates checked live at merge:
  • every check passes (19 of 19 at the head, run 41174548954);
  • MERGEABLE;
  • not a draft;
  • no CHANGES_REQUESTED review;
  • head pinned.
• Axiom: n/a. This is core infrastructure, with no policy change.