Repository navigation
Conversation
The counted-once check no longer feeds NaN to set_input or fails a component for a head that is NaN at baseline. It sets each value that is not finite at baseline to 0 wherever it raises or holds a variable, judges no reading that rests on those values, names each variable not finite at baseline once (recorded as a finite: failure), and treats NaN as equal to NaN in the same place when it checks the reset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In the held measurements the check no longer sets a head that is not finite at baseline: it leaves the head to its formula and judges each household where the head came back at the value the measurement holds it at, NaN matching only NaN. A head that ignores the component is then held exactly everywhere, so it unjudges no household and fails no component. A head with no readable household gets no response or coefficient. Add tests for the issue's cases, the reset check, a Hypothesis property over random heads with extra heads not finite at baseline, and a differential against the check as it stood at 89bd11f. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep a 0.0 response from judge_total when it skips no household, as the existing differential on finite responses pins, and say in the module docstring that the 0s set for a raised component change values only in the households that hold them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a Hypothesis property that records every value the check sets while heads are NaN or infinite at baseline, and run case 2 on monthly heads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a regression for changes that cancel between units, group the held-heads assertion as guard.py computes it, and shorten the changelog and README text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
render.py keeps not_finite_note from this branch and withheld_text from main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed 9ee51800e0c6e2df9896eabec9aa5cc3bbceb24b: COMPLETE — no findings or material evidence gaps.
Baseline-invalid heads/components are named and remain blocking; held-path masks keep unrelated readable households judged. Restoration, entity/month mapping, suppressed small-cell counts and frozen-reference independence were checked.
All 420 focused checks passed on the head's locked Core 3.34.1. This covers the actual child head; no prospective merge with the newer #2850 parent was certified. Hosted checks are absent at this captured head, separately from these local passes.
Approval is deferred because the PR remains a draft; readiness, parent integration and applicable CI must be checked before approval.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22
left a comment
There was a problem hiding this comment.
Incremental review COMPLETE at 793ce35d0a504e79785621a5e8eabf6e7178850a, with no findings or material evidence gaps.
The #2850 merge-first hold is satisfied: its reviewed head, actual merge commit 451a9bc9 and the captured main base are ancestors of this child. The base retarget is therefore supported by the current integration, rather than inferred from the branch name.
The scoped change adopts the upstream own-target privacy API in measure.py; the other seven authored paths retain their bytes. 126 focused cases passed: 48 baseline guards, 46 measurement cases and 32 withheld-reason cases. The older 420 checks remain separate retained evidence. An initial reviewer filename typo produced zero test calls and was corrected once; no passing tests were repeated.
Applicable exact-head hosted checks are still pending, so this is a comment. Approval can be considered after they pass without repeating the completed engineering review.
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles; standard reform costings (change in gov_balance); standard reforms: change in each reform's target. Checks failing on both base and head (1)
Counted-once checkThe check raised each of 92 components by £1 per unit in 2026-27 and read every head of its totals household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A response that is not finite breaks every rule. The check measures a head that breaks a rule twice more: once holding every other head at its baseline value, and once holding every other head at its value after the first £1 while the component rises again. The head fails where either measurement still shows the response. The check cannot hold a head at a value that is not finite. In each unit where another head is not finite after the first £1, it holds that head at its baseline value, and the second measurement judges no household the unit belongs to. There a head fails wherever it first broke a rule, or wherever the check could not confirm whether its response contains the component. 0 failed and the check skipped 0.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
juaristi22
left a comment
There was a problem hiding this comment.
Approving unchanged current head 793ce35d0a504e79785621a5e8eabf6e7178850a. The completed review remains COMPLETE with no findings or material gaps, and all latest exact-head checks now pass.
Head, description, project/lock inputs and retained evidence were revalidated. The prior 126 focused passing cases are retained, with the older 420 checks kept separate; no tests or model calculations were repeated for this approval. The #2850 merge-first hold remains satisfied by the verified parent head/merge and current child ancestry.
Readiness, current activity, source/parent guards, required branch rules and exact-head CI were refreshed immediately before approval.
Fixes #2773
What and why
The counted-once check (
policyengine_uk/utils/aggregate_impact/guard.py) raises each component, holds the other heads at their baseline values, and reads each head's change from its baseline. A head that was already NaN at baseline broke it in four ways (the issue has the minimal case):ValueError: Unable to set value for variable "tax_a", as the input contains NaN values.tax_a moves, or is not finite, when tax_b doesandthe model's gov_tax moves by other than its heads, or is not finite, when tax_b does).np.array_equal(np.array([np.nan]), np.array([np.nan]))isFalse.With this change:
not_finite_at_baselinewhen a head, a model total or a component it raises is NaN or infinite at baseline: one entry each, with the head's id and label, the households with weight, and a message.measure.record_guard_checksrecords a head under its ownfinite:<head id>check and a total or double count underfinite:metrics, in the scopecounted_once:<year>:<variable>.compare.ALWAYS_BLOCKING_PREFIXESmakes both block, whatever the base did.judge_totaltakesunreadmasks: it judges no head in the households where that head is not finite at baseline, and no model total where the total or one of its heads is not. The check averages each head's response, and the model total's coefficient, over readable households, and omits it when none has weight._moved), and judges the household only where they match. A head that ignores the component therefore leaves every household judged and fails nothing. Where such a head does move with the component, a failure there stays and names it inheads_not_held_at_baseline.np.array_equal(..., equal_nan=True)). A value that turns NaN after the reset, or a NaN that moves, still counts as a change.record_guard_checksrecords it as skipped (ok=None), and the report saysnot checked.The issue's cases, run on each side:
tax_a, NaN at baselineValueErrortax_askipped ("the check could raise it only in households with no weight or where it is not finite at baseline");tax_bpasses;tax_aandgov_taxnamedtax_awhile measuringtax_c = tax_bValueErrortax_bfails fortax_conly: "gov_tax: tax_c moves when tax_b does in fewer than 10 households"tax_balonetax_aandgov_taxtax_aandgov_taxnamedFalseTrueEvery report today has heads and totals that are finite at baseline, and for those the published report does not change.
counted_once_base_reference.pyfreezes the check as it stood at 89bd11f:counted_once,record_guard_checks,publishable_guardand the report's_guardsection. A differential test asserts that the result, the recorded checks (ids, statuses, scopes, details), the published guard and the report text serialise to the same JSON, in the same order, on the guard tests' own models and on Hypothesis-drawn heads and weights. The new keys appear only when something is not finite at baseline.Invariants
These hold for every input:
counted_oncereturns, and every value it raises or holds a variable at is finite: the check sets a component's non-finite units to 0 and never sets a head that is not finite at baseline. Restoring the inputs afterwards puts back what the caller supplied, ±inf included.not_finite_at_baselinenames exactly the heads, model totals and raised components with a value that is not finite at baseline, each once. The key is absent when there are none. Each entry fails afinite:check, which blocks the comparison.model_totalproblem falls in a household where the total or one of its heads is not finite at baseline. No rule judges a household that holds a unit where the raised component is not finite at baseline.reset_reproduces_baselineis true exactly when each head's recomputed household array equals its first computation, NaN matching only NaN in the same place._settablereturns finite values unchanged at the byte level and 0 elsewhere.Test plan
uvx ruff@0.17.2 format --check,uvx ruff@0.17.2 checkand--select F811,F821on every changed Python file;git diff --checkchangelog.d/2773.fixed.mdNew tests,
policyengine_uk/tests/aggregate_impact/test_aggregate_impact_guard_baseline.py: 48 passed at 48f57f9.finite:scope, label and blocking; small counts dropped from the published guard and the report;set_inputis finite;counted_once_base_reference.py, plus eight named models and the twelve-household two-total model;_settable(invariant 7).Existing tests, each file run alone at e388e1f:
test_aggregate_impact_guard.pytest_aggregate_impact_measure.pytest_aggregate_impact_compare.pytest_aggregate_impact_childcare_provenance.pytest_household_subset.py::test_every_walk_up_to_an_ancestor_simulation_is_reviewedNo existing test's expected value changed.
core_counted_once, a helper intest_aggregate_impact_guard.py, now passes keyword options on tocounted_once.Before and after, with this branch's tests copied onto each tree:
ValueError. The 14 that pass are the 11 differential tests, which shows the frozen reference reproduces the base exactly, and 3 reset cases whose expected outcome the base already gives (inf against inf, a value turning NaN, inf against −inf)I applied each mutation to a scratch copy of the package and ran the new test file against it (
--rss-gb 2). The tests kill all eight:equal_nanset_input(at 0) instead of its formula_movedcounts NaN as differing from NaNrecord_guard_checksrecords nofinite:failureM1 ran against the file's first 44 tests, and M2 to M7 against its first 46. M8 replaces
_moved's unit comparison with a household-sum comparison, and both cases oftest_units_that_cancel_within_a_household_still_count_as_movedfail on it.Reproducer
Reproducer: policyengine_uk/tests/aggregate_impact/test_aggregate_impact_guard_baseline.py::test_a_head_nan_at_baseline_fails_no_component_it_ignores
axiom: n/a: a check on the model's aggregate report, not a rule of law
🤖 Generated with Claude Code