Repository navigation
Limit reform target-change checks to the reform's own target - #2852
Conversation
…its year withheld_reasons checked every costed reform's target, so a reform that moved another reform's target in 1 to 9 records lost its costing and target change, although the report publishes nothing about that target for the reform (#2775). It now takes the reform's own target and checks that one only; another reform's target counts as a part of gov_balance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t the target-less and missing-target cases Another reform's target counts as it would if no reform aimed at it: in gov_balance, and in the component conditions when it is a component. The docstrings now say so, privacy.py states which households the new disclosure path concerns, the balance test now tells the two rules apart, and two tests pin a reform with no target and a baseline without the reform's own target. measure_reforms reads the target it already holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aviour Add one test for own-target withholding in a later year and one for a reform aimed at gov_balance, which both killed a surviving mutant. Qualify the responsiveness bound in the privacy notes and remove fix history from durable comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles; standard reform costings (change in gov_balance); standard reforms: change in each reform's target. Checks failing on both base and head (1)
Counted-once checkEach of 92 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed 17774d3dfcfb02cd1ac614af60adc7190f7499f7. COMPLETE — no findings or material evidence gaps.
Reform target-change checks now use that reform's own target while preserving the separate balance/component conditions, missing-target failure handling and responsiveness checks. The API, comparison and rendered suppression reasons remain consistent with the actual issue #2775 contract.
All 246 focused checks passed on the head's locked Core 3.33.12 environment. The deliberately documented residual-inference exception was assessed against that reporting contract; this review does not certify general disclosure safety for combinations of aggregates.
Applicable exact-head CI, activity and readiness were refreshed before approval.
Fixes #2775
This pull request changes
measure.py,api.pyandprivacy.pyinpolicyengine_uk/utils/aggregate_impact/, the withheld-reasons tests and a changelog fragment. #1963 and the fix for #2759 (fix-2759-non-finite-responses) also changemeasure.py; git merges each with this branch without a conflict in the files it changes.What and why
withheld_reasonschecked every entry inbaseline["targets"], whichmeasure()fills with the target of every reform it costs. A reform that moved another reform's target in 1 to 9 records therefore lost that year's costing and target change, although the report publishes nothing about that target for the reform.withheld_reasonsandreform_withheldnow take the reform's own target as a required keyword,target.measure_reformspassesreform.responds.variableastarget, orNonewhenrespondsis absent. They check that target besidegov_balance, and no other reform's target. Another reform's target now counts as it would if no reform aimed at it: ingov_balance, and in the component conditions (suppressed_component_changed,small_component_left) when it is a partition component, as all six standard targets are. Each reason keeps its wording, the privacy helper (privacy.withheld_reason) keeps its name and import, and the tool's threshold stays at 10 records.What the report publishes for a reform, and what each figure reads
The issue asked for one check first: that no other published figure for the reform reads the other target. The tool records yearly metrics, checks and reform outcomes in
result.jsonfor each side, and the long-format report (api.to_report), the comparison (compare.py) and the pull request comment (render.py) carry it from there.reforms.<id>measure_reforms:weighted_total(reformed, "gov_balance")less the baseline'sgov_balance(gov_taxlessgov_spending), household-weighted: every tax and benefit in those lists, another reform's target among themreform_targets.<id>measure_reformssuppressedandwithheld_becauseon both figureswithheld_reasons,_record_reasonsgov_balancechange and the own target's change. Before this change: every costed reform's target, by nameresponsive:<id>verdict and detailmeasure_reforms,reforms.responsivenessreform_runs:<id>verdict and detailmeasurefinite:metrics_finite_metrics_checkreformsoutcomes inresult.jsonand the summarymeasure_reforms,compare._side_summaryrespondsand error textcompare.pysuppressedflags and reasons; flags usereport_spec.yaml'sreform_gbpthresholdsrender.pyprivacy.withheld_textof the reasons, and the checks abovereport_spec.yamlsets the horizon the reforms run over and thereform_gbpflag thresholds, and reads no data. The comparison's notes on non-finite figures and on figures measured on one side only, anddefinitions_sha256, read the same figures or the reform list.No figure reads another reform's target as a target. The costing reads it as a part of
gov_balance, which thegov_balancecondition (1 to 9 households) covers, as the issue says. The component conditions read it as they read any component, whichever reforms aim at it. Before this change asmall_target_changecould also name another reform's target; now it names only the reform's own. No figure keeps the old withholding.A disclosure path this widens
When
gov_balancecontains a reform's target, the costing less the target's part is the change in the rest ofgov_balance. The tool counts no records there, before or after this change.gov_taxorgov_spendinglists every standard reform's target (income_tax,national_insurance,vat,child_benefit,universal_credit,state_pension).test_another_reforms_target_counts_in_the_costing_and_withholds_nothingbuilds the case. The reform aimed at income tax raises it by £10 in all 20 households and the state pension, another reform's target, by £25 in three. The tool publishes a costing of £125 and a target change of £200, and their difference is the three pensions' £75.Before this change the tool withheld that year because the state pension, another reform's target, moved in three of its records. It never withheld a change in a variable no reform aims at this way: Housing Benefit or Pension Credit, say, which
gov_spendinglists and no standard reform targets. So the path already existed for those variables, and this change opens it for the six standard targets, in each reform that doesn't aim at them. The tool's suppression rule stays as it stands, andprivacy.pynow lists this path among its deliberate exceptions. A condition on the rest ofgov_balancewould close it for every variable; this pull request doesn't add one.Tests whose expected values move, and new tests
test_a_small_change_in_another_reforms_target_names_that_targetis nowtest_a_small_change_in_another_reforms_target_withholds_nothing. The reform aimed at income tax moves Pension Credit, another reform's target, in three records. Its reasons move from[small_target_change, pension_credit]to none, and its costing and target change publish at £200 each. Aggregate impact: another reform's target can withhold a reform's year #2775: the report publishes nothing about the other target for that reform.test_two_small_targets_are_both_namedis nowtest_of_two_small_targets_only_the_reforms_own_is_named. The reform now aims at Pension Credit, and Child Benefit is the other reform's target. Its reasons move from both targets to Pension Credit alone, for the same reason.test_a_year_is_withheld_exactly_when_it_has_a_reasonkeeps its expected values: its direct computation already checked only the reform's own target. It now measures with the other variables as other reforms' targets, asmeasure()does, and passestarget=.New tests. The first five fail on #2772, and the last two pin what it already does:
test_a_small_balance_change_withholds_whatever_other_targets_do: three balances still withhold the year, and the reason names only them, though Pension Credit, another reform's target, moves in three records too.test_a_reform_without_a_target_checks_no_other_reforms_target: a reform that names no target publishes its costing when another reform's target moves in three records.test_a_baseline_without_the_reforms_own_target_fails_the_reform: a baseline that lacks the reform's own target fails the reform (KeyError) instead of publishing an unchecked year.measure()always records every reform's target, so this guards direct callers ofmeasure_reforms.test_another_reforms_target_counts_in_the_costing_and_withholds_nothing: the case in the section above.test_a_year_is_withheld_exactly_when_its_own_target_or_balance_moves_in_few_records: the property below.test_a_small_target_change_withholds_a_year_after_the_first: one household's income tax and state pension each rise by £1 in 2027, no balance moves, and the tool withholds that year for the reform's own target.test_a_reform_aimed_at_gov_balance_gives_both_reasons: one balance moves, and the tool namesgov_balanceas the balance and as the target.The issue's cases map to these tests. The tool now publishes the reform aimed at income tax that moves Pension Credit in three records (the renamed test). It still withholds a reform that moves its own target in three records (
test_a_reform_that_changes_few_target_records_names_the_targetandtest_of_two_small_targets_only_the_reforms_own_is_named), and one that movesgov_balancein three households (test_a_reform_that_changes_few_balances_says_soandtest_a_small_balance_change_withholds_whatever_other_targets_do).Invariants
These invariants assume complete baseline inputs; a missing own-target entry fails the reform. For every reform and year:
withheld_reasonsreads only the reform's own entry inbaseline["targets"]. Other reforms' targets in the baseline change no reason, figure or check.gov_balancemoves by more thanCHANGE_GBPin 1 to 9 records that carry weight.reform_withheld_main_reference.py), given every target, withholds exactly when this rule does or another reform's target moves by more thanCHANGE_GBPin 1 to 9 records that carry weight.test_a_year_is_withheld_exactly_when_its_own_target_or_balance_moves_in_few_records(Hypothesis, 300 examples) checks 1 to 3 over which records and which of five targets move. It checks 3 differentially against the frozen rule.test_a_year_is_withheld_exactly_when_it_has_a_reasonchecks 1 and 4, and checks the reasons against a direct computation.Test plan
uvx ruff@0.17.2 format --check,uvx ruff@0.17.2 checkon the changed files,ruff check --select F811,F821 policyengine_uk,git diff --checkchangelog.d/2775.fixed.mdResults
Each file ran separately, with dataset credentials unset and pytest workers disabled (
-n 0).test_aggregate_impact_withheld_reasons.pytest_aggregate_impact_measure.pytest_aggregate_impact_compare.pytest_aggregate_impact_reforms.pycode_health/test_aggregate_impact_workflow.pytest_aggregate_impact_certified.pyThe issue's case also fails on
main(2da212a). A standalone copy of the reproducer, which imports only names thatmaindefines, fails onmainand on #2772 (the tool withholds the costing) and passes on this branch. The test file itself imports names that #2772 adds, so it can't load onmain.Six mutations of the change, each run against the test file, all fail tests. The first four ran against its first 30 tests:
measure_reformspassestarget=Nonetest_a_baseline_without_the_reforms_own_target_fails_the_reform; the run then hit its 1,500-second limit in the last testgov_balancetest_a_reform_aimed_at_gov_balance_gives_both_reasons(1 of 32)measure_reformspasses the target only in 2026test_a_small_target_change_withholds_a_year_after_the_first(1 of 32)Over 300 draws, the new property's strategy produced 32 worlds where the old rule withholds and this rule publishes, 24 where both publish and 244 where both withhold.
Reproducer: policyengine_uk/tests/aggregate_impact/test_aggregate_impact_withheld_reasons.py::test_a_small_change_in_another_reforms_target_withholds_nothing
axiom: n/a: changes how the aggregate impact report withholds small cells, which encodes no statute
🤖 Generated with Claude Code