Skip to content

Trace how calibration, sampling noise and rent imputation move state child SPM poverty rates - #1445

Draft
MaxGhenis wants to merge 6 commits into
mainfrom
state-child-spm-diagnosis
Draft

MaxGhenis wants to merge 6 commits into
mainfrom
state-child-spm-diagnosis

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Adds docs/evidence/us-state-child-spm-2026-11/, a diagnosis of why Populace US state child SPM rates miss Census P60-290 Table 18, with its scripts, state-level outputs and receipts. It changes no code and no data. Next steps sit in #1446.

What it finds

  • On the dense pool, calibration creates most of the systematic error, and most of the added state error lands on ASEC rows. Dense initial weights give a mean gap of 0.00 and a mean absolute gap of 1.78, against +3.62 and 4.33 at published weights. The initial-weight figures flatter the pool, because it shares households with Table 18. Allowing for that, state-specific error rises about 1.7 times, and a test with no shared households keeps the direction.
  • That result stops at the dense pool. On the default file, record selection sets the shift before calibration starts, and the ACS-local file stores no initial weights. The dense zero nets +2.20 on ASEC rows against -2.20 on PUF rows, and initial weights miss the administrative targets, so it doesn't argue for skipping calibration.
  • Noise accounts for most of the default file's state scatter. Three fits put it at 60%, 71% and 80%, and the first carries a 95% interval of about 10% to 89%.
  • Some state error predates calibration, and we cannot yet pin it on particular states. Louisiana has the largest joint miss at dense initial weights and on ACS rows, but a miss that large on both turns up by chance about one time in five (p = .19).
  • Rent lands on the wrong households. Of records that should carry rent, 89% do not hold their own. Place each ACS rent draw on its own household's head #1444 carries the fix. The fix nearly doubles modelled housing subsidies and barely moves state fit at published weights, and moves a state's reform effect by a median of 0.29 across 15 states. A few donor households drive the larger swings.
  • The default file is not the state path. policyengine.py routes state regions to the ACS-local file, which gives a mean gap of +0.88 and a mean absolute gap of 3.18, with a median state standard error of 2.98.

How to check it

receipts.json lists digests for the scripts, the outputs and every input outside the repo. Scripts in scripts/ ran on policyengine-us 2.90.2, cutting one state at a time. The README's reproduction section gives the order. Poverty stays out of the targets. Model figures come from microsimulation runs or labelled arithmetic on them, and Census comparisons use published tables or raw ASEC.

Any release of rebuilt data waits on Max: d2300 covers the rebuild and d2301 the ACS-share default.

axiom: n/a: evidence only, no policy logic.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits November 1, 2026 18:09
Scripts and a draft note on why Populace US state child SPM poverty rates miss
Census P60-290 Table 18. Figures still running carry slots in the README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fills the note from the 51-state runs: noise and systematic error, the
step decomposition, the rent assignment bug, ACS-local results, the
dashboard's exposure and the ordered fixes. Scripts, state-level outputs
and receipts sit beside it. Poverty stays a holdout throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits November 1, 2026 19:11
The review found the dense pool at initial weights fits Census far
better than at published weights, so weights, not missing records, carry
most of the systematic error. The note now leads with that, reports
noise's share with its interval, and reorders the fixes.

It also corrects the ablation labels, the null-test intervals, the
dashboard's build comparison, the PUF housing mechanism and the ACS rent
path. Clusters now merge an ASEC record with its PUF clone on every
file, and receipts pin input contents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The initial-weight columns share households with Table 18, so state-specific
error rises about 1.7 times, not double, and a test with no shared households
keeps the direction. Scope the weights finding to the dense pool, name
Louisiana alone among persistent misses, and call the fix order a judgement.

Also corrects the ablation, anchor and ASEC-PUF claims, adds the corrected-rent
reform pass, and stores the threshold swap and Kish figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stop naming Louisiana as a finding: one joint miss matches chance. Compare the
rent fix's moves like for like and trace Florida's swing to a few donor
households. Correct the shared-Census correlations for shared households, add
the same-pipeline row that anchors the penalty grid, and leave unrun reform
deltas empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The p of .53 answered whether any state clears 1.96 on both comparisons, not
whether a miss as large as Louisiana's beats chance. A permutation of the
largest joint z gives .19.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Hold: don't cite this PR's ACS-local figures yet. They rest on a wrong fill, and a rerun is pending.

The evidence scripts filled blank parent links in the ACS-local file with -1. PolicyEngine-US reads 0 as unknown (parent_1_id.py), and any nonzero id puts the whole household on the parent-link formulas (_parent_links.py). So -1 gave every donor household an absent parent it doesn't have.

Every blank sits on a donor: all 166,321 ASEC and PUF-channel people, in 57,240 households. They carry 70% of the weighted people and 68% of the weighted children. No ACS person holds a blank.

The deck's 51-state run uses fill 0 on the same file and model (PolicyEngine/policyengine-slides#95 at 99ee5b354). Comparing the two baselines gives this, as arithmetic on real runs:

Child SPM, ACS-local Fill -1 (this PR) Fill 0
California 19.81 20.37
Delaware 22.21 22.22
Missouri 16.712115 16.712122
The other 48 no change beyond 1e-6
Mean gap +0.88 +0.89
Mean absolute gap 3.18 3.19
Correlation .54 .54
Outside Census's margin 29 29

Census reads 17.9 for California, so its gap reads +2.47, not +1.91.

Still stale on the ACS-local file: standard errors, the ACS and donor splits, reform effects, the #1078 solves and the corrected-rent runs.

Not affected: everything on the default and dense files. The default file stores no parent columns, and the dense file stores no blanks.

Next: a full 51-state rerun under fill 0 replaces these figures. The host's release booking holds new microsim runs until the release passes its terminal gates, and Fleet ops asked me to wait for that. This PR stays in draft until the rerun lands.

@MaxGhenis
MaxGhenis marked this pull request as draft November 2, 2026 23:55
@MaxGhenis MaxGhenis changed the title Diagnose why state child SPM poverty rates miss Census Trace how calibration, sampling noise and rent imputation move state child SPM poverty rates Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant