[PavelMakarchuk]

Fixes #723

Problem

UK program_statistics came only from the fixed UK_PROGRAMS list (14 rows). The headline budgetary impact is the change in household_tax - household_benefits, which policyengine-uk builds from HOUSEHOLD_TAX_VARIABLES (27) and HOUSEHOLD_BENEFIT_VARIABLES (about 50). A change in any program outside the fixed list was in the headline with no row, so the apps' "Budgetary impact by programme" chart didn't add up and couldn't say why.

Example, live report 4738 (2027 UC freeze, policyengine 6.10.0 / policyengine-uk 2.132.6 / certified microcosm_uk_2024_25.h5): UC row £933.5m, headline £958.5m; the missing £25.0m is Scottish Child Payment, lost by Scottish families whose UC falls to zero. A PIP cut had no row at all.

Change

• uk_budget_component_programs(model_version) reads the two lists from policyengine-uk at run time and returns every component the model version defines that isn't already in UK_PROGRAMS, with is_tax from the list it came from (72 on policyengine-uk 2.151.0).
• configure_uk_budget_component_variables materializes them on both simulations, on each variable's own entity, through add_extra_variables (the same route as the labour supply variables).
• economic_impact_analysis returns all UK_PROGRAMS rows as before, then a row for each component the reform changes. Unchanged components are dropped, so outputs don't carry dozens of empty rows.

No hard-coded list to maintain: a program policyengine-uk adds to the totals gets a row with no change here. Existing rows, keys and fields are unchanged.

Testing

• New tests: the component list covers both totals beyond the fixed list (Scottish Child Payment and PIP as benefits, capital gains tax as a tax, no overlap with UK_PROGRAMS); components are materialized on their own entities; the analysis adds a row for a changed component (Scottish Child Payment, change −1,530) and none for an unchanged one (PIP); no component rows when nothing changes.
• Two existing test stand-ins for _validate_program_statistics_config now accept the new optional argument.
• End to end on this branch's bundled Microcosm UK data (2027, UC freeze): rows universal_credit £1,655.1m, scottish_child_payment £12.2m, tax_free_childcare −£2.3m, council_tax_benefit −£0.3m; the rows explain the headline (£1,664.7m) with a £0.0m gap. (The totals differ from the live report because main pins a newer model and dataset.)
• make format, make lint and the full test suite pass.

Notes for consumers

• is_tax on each row says whether it's a tax or a benefit. The app chart currently signs rows from its own list of benefit keys, so the simulation API should pass is_tax through to detailed_budget (Show an "Other programmes" bar when the programmes fall short of the total policyengine-app-v2#1451 shows any remaining gap as "Other programmes" meanwhile).
• Fixed rows that aren't parts of the totals are unchanged: vat and ni_employer (not in household_tax) and tax_credits (overlaps working_tax_credit + child_tax_credit). Summing every row can still double count; consumers that need rows adding to the headline should leave those three out.
• Related programs can move in opposite directions (for example extended_childcare_entitlement and targeted_childcare_entitlement with a personal allowance cut). Grouping is left to a follow-up.

🤖 Generated with Claude Code

[anth-volk]

Thanks, Pavel. Because of the Budget, building the rows from -uk's two lists is a good enough approach for now: every programme a reform changes gets its own row, and the rows now explain the headline. There's one problem to fix before this merges, and a longer-term change I'd like us to make afterwards.

Before merging. The v3 "Budgetary impact by programme" chart decides which rows are benefits from its own BENEFIT_PROGRAMS set, and sim-api's build_detailed_budget drops is_tax. Once sim-api deploys a .py release with this change, new benefit rows such as scottish_child_payment and pip would be drawn as taxes, with the wrong sign. On report 4738, Scottish Child Payment would show −£25m instead of +£25m, and with policyengine-app-v2#1451 in, an "Other programmes" bar of +£50m would make up the difference. #1451 fills the gap but can't correct the sign, so it doesn't cover this. Could you have sim-api pass is_tax into detailed_budget and have the chart sign rows by it, and land both before this merges?

Longer term. I'd like the list of UK budget programmes to live in one place, most likely a programme registry in -uk (like policyengine-us's programs.yaml) that household_tax and household_benefits are built from and that .py reads. This PR builds on two plain lists inside -uk's formula modules, which leaves us with:

1. Several lists that can disagree. We now have -uk's two lists, .py's UK_PROGRAMS and the app's BENEFIT_PROGRAMS, and the last of these is what causes the problem above. Of the 14 UK_PROGRAMS entries, 11 repeat -uk's, and vat, ni_employer and tax_credits aren't in the totals at all.
2. A dependency on -uk's internal file layout. .py imports HOUSEHOLD_TAX_VARIABLES and HOUSEHOLD_BENEFIT_VARIABLES from the household_tax and household_benefits variable modules, so moving or renaming either one breaks every UK economy analysis at import.
3. No metadata beside each name. There's no label, no tax/benefit flag (we infer it from which list a name is in) and no grouping, so the labels and grouping raised in #723 still have to be hard-coded downstream.
4. Totals the lists don't fully describe. abolish_council_tax removes programmes by name inside the formulas, and benefit uprating treats basic_income and state_pension separately. With those settings on, the rows won't match the headline.
5. Amounts mixed with reform-only changes. household_tax counts vat_change and the employer NI incidence terms rather than VAT and employer NI. That's part of why consumers have to know to leave out vat and ni_employer (and the overlapping tax_credits) for the rows to add up.
6. Silent changes between -uk releases. Between 2.151.0 and main, isa_cash_interest_flat_rate_charge and wales_childcare_offer were added and private_school_vat became private_school_vat_change. Because the lists are read at run time, the rows a report can show change with the -uk version, and nothing records that.

With one registry entry per programme (variable, label, tax or benefit, group, and whether it counts towards the headline):

• -uk could build both totals from it;
• .py could read it through a stable function rather than module internals;
• sim-api could pass the flags through;
• the app could drop BENEFIT_PROGRAMS.

I'd suggest tracking that in an issue for after the Budget.

[PavelMakarchuk]

Thanks, Anthony. Both parts are in place:

Before merging (the sign fix):

• Pass is_tax through to detailed_budget policyengine-sim-api#972 (fixes Pass is_tax through to detailed_budget so consumers can sign each program policyengine-sim-api#971): DetailedBudgetProgramOutput gains an optional is_tax, copied from each program-statistics row (null in outputs built before it). Golden OpenAPI regenerated; executor, gateway and contract tests pass.
• Show an "Other programmes" bar when the programmes fall short of the total policyengine-app-v2#1451: the chart now signs each row by is_tax when the output has it and keeps BENEFIT_PROGRAMS only for older outputs. A test covers report 4738's case: a flagged scottish_child_payment row is signed as +£25.0m spending saved, and the rows add up to £958.5m with no Other programmes bar.

Order: sim-api#972 and app#1451 merge and deploy first; this PR reaches sim-api after that.

Longer term: PolicyEngine/policyengine-uk#2702, for after the Budget. -uk already has programs.yaml (55 programmes with variable, name, category and agency, served as modelled_policies), so the issue proposes extending it with a budget role (tax / benefit / not in the totals) and a group, building household_tax and household_benefits from it, exposing a stable function for .py, and a test that the registry and the totals can't drift. It lists your six points.

[anth-volk]

Re-review of #724 at 1b570d433

Thanks, Pavel. This re-review covers three things:

• whether the before-merge point from the first review is resolved by Pass is_tax through to detailed_budget policyengine-sim-api#972 and Show an "Other programmes" bar when the programmes fall short of the total policyengine-app-v2#1451;
• whether those two have deployed;
• whether anything else that reads this output could break.

It also takes a fresh look at the PR's own code. The main findings were re-checked in the source.

The before-merge point: resolved for the v3 chart

• The flag reaches the v3 chart intact. is_tax passes through unchanged:
  • build_program_statistics writes it for both the fixed rows and the new component rows (src/policyengine/outputs/program_statistics.py:225).
  • sim-api's build_detailed_budget copies it into DetailedBudgetProgramOutput.is_tax (bool | None).
  • The v3 chart (app/src/components/flagship/report/charts/BudgetaryImpactByProgramSubPage.tsx) uses it when it is a boolean, and falls back to BENEFIT_PROGRAMS otherwise.
• The column type holds. After this PR's pd.concat, is_tax stays a bool column, and the records carry Python bool.
• Older stored outputs are fine. They have only the 14 fixed rows, and every fixed benefit is in BENEFIT_PROGRAMS, so the fallback signs them correctly.
• The Public API is unaffected. It passes detailed_budget through without reading it.
• Both companion PRs have deployed:
  • sim-api#972's "Deploy Simulation API" run (40679914475, 28 Oct 11:26 UTC) succeeded.
  • app-v2#1451 is in a successful Vercel production deployment. The current production deployment also contains #1456.
• Not confirmed: that policyengine.org serves that exact build, and that a live UK output carries is_tax.
• This PR is not in production yet. sim-api pins policyengine[models]==6.11.1.

New findings

1. A second app-v2 programme chart has no sign handling, and this PR adds more wrong rows to it (medium).

   • Where: policyengine-app-v2:app/src/pages/report-output/budgetary-impact/BudgetaryImpactByProgramSubPage.tsx (around 61–68) plots each row's raw difference. It is rendered from MigrationSubPage.tsx:231, which is the default tab for UK report outputs.
   • Example: after this PR, report 4738 would show Scottish Child Payment as −£25m instead of +£25m.
   • VAT counted twice: a VAT reform would show both vat and vat_change.
   • Employer NI: the employer NI incidence rows would appear beside ni_employer.
   • History: this chart already drew Universal Credit with the wrong sign, so the PR widens an existing defect rather than creating one. Neither #1451 nor #1456 touched this file.
   • Suggested fix: apply the v3 chart's is_tax signing and row exclusions here too, or route UK reports to the v3 chart, before this PR's release reaches sim-api.

2. Tax rows count every unchanged entity as both a winner and a loser (low; this predates the PR).

   • Where: src/policyengine/outputs/program_statistics.py:91 and :101. For taxes, winners use change_geq=-0.01 and losers use change_leq=0.01.
   • Effect: an entity whose tax doesn't change counts in both. This PR adds up to 24 more tax rows with these counts, such as capital_gains_tax, student_loan_repayments and vat_change.
   • Who is affected: detailed_budget doesn't carry the counts, so the charts aren't affected, but other readers of the programme statistics would be.
   • Fix: use change_leq=-0.01 for tax winners and change_geq=0.01 for tax losers.

3. The tests don't pin the PR's main claim (low). No test checks that the rows, leaving out vat, ni_employer and tax_credits, sum to Δhousehold_tax − Δhousehold_benefits. In addition:

   • Both analysis tests replace uk_budget_component_programs with a stub, so the real lists are covered only by three sampled names (tests/test_uk_program_statistics.py:312).
   • The analysis test checks only the set of program_name values. It doesn't check is_tax or change, which are the fields sim-api reads.

4. The output planner in sim-api's newer stage12_runtime pipeline doesn't add the new UK variables (low; not run).

   • stage12_runtime/output_planning.py repeats economic_impact_analysis's configuration calls but has no counterpart to configure_uk_budget_component_variables.
   • Once sim-api moves to a release with this PR, qualify_report_parity (stage12_parity.py, column_membership_mismatch) would flag about 72 extra columns on every UK comparison.
   • No workflow runs that comparison today, so production is unaffected.

5. A downstream script that sums every row would count VAT twice for a VAT reform (low; not verified against the version that repository uses). policyengine-30s/tools/uk/national.py sums all is_tax rows and all other rows except tax_credits, so it would add vat_change on top of vat.

Smaller notes

• The description says HOUSEHOLD_TAX_VARIABLES has 27 entries. policyengine-uk 2.151.0 has 28. The total of 72 components is right, and policyengine-uk main gives 74.
• The rows still won't match the total when abolish_council_tax or benefit_uprating is set, as noted in the first review.

Checked and found true

• The two policyengine-uk lists: there are no duplicates within or between them, on 2.151.0 or on main. The 11 fixed rows that appear in them have matching is_tax. Only vat, ni_employer and tax_credits sit outside the totals.
• Sums: at default parameters, both totals are plain sums of the lists, and the per-entity weights reproduce the household-level sums.
• Rows dropped and kept: zero-change component rows are dropped, and the fixed rows are always kept.
• Cached baselines: cached or deterministic UK baselines in sim-api that are missing the new columns are recomputed once rather than failing.
• Deployment code: the PR changes none, so there are no environment variables to check.

Not covered

• The PR's tests and a UK simulation, so the description's £0.0m gap was not reproduced.
• The runtime and memory cost of the roughly 72 extra columns.
• Rendering either chart.

🤖 Generated with Claude Code