Repository navigation
Conversation
The importer now reads the OBR's tobacco clearances (Table 4.11) and alcohol clearance forecasts (Table 4.12) and writes six per-head series under gov.economic_assumptions.yoy_growth.obr.per_capita, one per product group: clearance growth over one plus ons.population growth, to six decimals. Dataset projection grows household weights by ons.population, so weighted quantities grow with the clearances. Each quantity input declares its group's index and is mapped to its series in uprating_indices.yaml. The real-table fixtures gain the clearance sheets, with each kept sheet's XML and the shared strings unchanged. Fixes #2596. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ties The excise duties page now maps each quantity input to its OBR line, shows from the projection code why the series divide clearance growth by population growth, lists the Budget 2026 measures that reach the two duties, and compares the model's duty on the OBR's volumes with its receipts forecast. It records each source file, sheet, row and hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A clearance spec now refuses rates and an outturn of its own, which per_head_growth cannot use. The tail comment states the formula, which holds once population growth turns negative in 2055. The differential test also checks gov_balance. The README's unit rule names receipts tables only, and its refresh step lists every pin and the wine line's footnote change. The docs name the October 2024 tobacco rise and say which hashes are of the original downloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dices Merges #2761 at d08fbb6. Its growth_ratio, check_index, population_growth, growth_in_effect and per_head_growth replace this branch's copies, each defined once. ReceiptsSeriesSpec.per_head now covers the clearance specs, which sit under per_capita, so the renderer and summary write them per head through the shared helper. The clearance tail and comment basis stay. The importer rebuilds the merged yoy_growth.yaml byte for byte from the real October 2026 and March 2026 tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… head A line's row may hold only its label and a value under each year. The importer used to skip any other cell, so a "kg" cell beside "Other tobacco products" let the line borrow the thousand kg its non-cigarette total states, and growth read 999.0. A clearance table's year row holds only years, and the one cell that states a clearance line's unit must describe the line: its own label, the label of the total it breaks down (the row above "of which:"), or a note. ReceiptsGrowth carries each year's growth before rounding, and per_head_growth divides that, so a gambling base with its own outturn (previous_line None) grows per head by its base growth, not by its receipts or a growth rebuilt from levels and rates. The committed yoy_growth.yaml is unchanged byte for byte. The population test now covers all 13 per-head series: after the merge with #2761 the seven gambling bases move with population too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OBR states each clearance unit, splits Table 4.11's non-cigarette clearances, incorporates the Budget's measures in its forecast and lists them in its policy Table 1.1; tables, notes and forecasts no longer act. On the national construct the model's alcohol duty exceeds the OBR's receipts by 3.5% in 2027-28 and 0.3% in 2028-29 and first falls short in 2029-30, by 2.5%; tobacco duty falls short from 2025-26. The excise page said both fell short from 2027-28. The economic assumptions README says the per-head series keep the old population's values until the importer reruns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dices Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The importer reads every clearance unit the OBR states in a table against a closed list (sticks, kg, litres or hectolitres of alcohol or of product, at the scales it knows) and compares units by measure and size. It reads a unit as a line's in the line's label, its total's label, another line's label that names its product, or a note that names its product or no product. It refuses a line when such a cell states another unit or scale, when a note states two units it cannot tie to products, when a note or a line's cell holds a quantity word outside the list, or when it finds the unit only in a note about another product. Each spec names its products. Tests: both round-2 witnesses refused by message; notes about other products; respellings that agree; a Hypothesis property over labels and notes in two releases. Test docstrings drop reviewer names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README, the clearance spec's docstring and two test docstrings read as growth over one plus population growth; they now state one plus the growth, over one plus population growth, minus one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The importer no longer reads units or products from a clearance table's words. CLEARANCE_TABLE_TEXTS holds, for each table, the text of every cell in the March 2025, March 2026 and October 2026 tables, and the importer refuses any other text from the title's row to the next table's title (CLEARANCE_TABLE_ENDS), in every column, naming the table and the cell. Each line also needs the exact cell that states its unit. The unit parser, its product aliases and their tests go. New tests cover each witness class on the real tables, the list against the three releases, the receipts table below the alcohol clearances, and a property that inserts any text anywhere in a table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The excise duties guide and the economic assumptions README now say what the importer does: it interprets no words in the clearance tables, checks every cell against the texts someone has read, needs each line's unit cell, and names the table and cell it refuses. The README keeps the check that each alcohol line covers the same products in both EFOs, and the changelog states the projection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spec's error says which texts lack the unit cell, the docstrings name who acts, and the per-head docstrings write the ratio as one plus the aggregate growth over one plus population growth. The refusal test adds a note shaped like a table's title, and the property draws two such texts, since sheet_tables starts a table at one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The XLSX reader returned only the first text run of an inline string, so the clearance check compared a prefix of a formatted cell with its list. Inline and shared strings now share one helper that joins every <t>. validate_clearance_table checked the first block with the table's number and title while find_receipts_line could take a line from a later block with the same heading. It now refuses a sheet that repeats the heading and names the worksheet rows, which read_sheet_rows keeps as SheetRow.number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README and the guide now say the importer compares text cells from each clearance table's title through its configured end, and checks no cell above the title or below the end. They add the stop on a repeated heading and the reading of every formatting run, and split the guide's sentence on duty per unit cleared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docstrings only: their ASTs match the previous commit's once each docstring drops out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The closed list of clearance texts does not settle a line's unit: two rearranged copies of the real October 2026 Table 4.11 keep every text on the list and still put hand-rolled and other tobacco under the cigarette label's million sticks. The importer again refuses any text beside the years in the fiscal-year row over a clearance line, and requires each part of a total (part_of_total) to sit under an "of which:" row directly below the total's label, with only the total's other parts between (validate_clearance_part). _string_text now joins a string's own <t> and each run's <t>, so the <t> of a phonetic run (<rPh>) no longer joins a cell's text. An empty cell whose phonetic run spelt the alcohol receipts title could otherwise end the clearance table above an unread note. Tests: the two layouts and a second year row as refusal tests, the year row and a part's structure in the arbitrary-text property, phonetic runs in the formatting-run property, and a phonetic end title on the real Table 4.12. README and guide list both checks; four docstrings reflowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments, docstrings, the README step and the guide only: the importer and its tests parse to the same syntax trees as at 433b20f with docstrings left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validate_clearance_part now needs the total's row to hold its label and one or more numbers after it, and nothing else, so a row that holds only the label no longer counts as the total. The refusal names the total's label as receipts_label gives it. A parametrized test inserts such a row into each real release's tobacco clearances and checks the refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LwQq6L5XVCEvFyRi9rsktq
test_import_obr_forecasts.py and test_excise_quantity_projection.py each gain a module-scoped autouse fixture that clears real_clearance_rows and current_law after the module's tests, as test_test_module_caches_released requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LwQq6L5XVCEvFyRi9rsktq
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LwQq6L5XVCEvFyRi9rsktq
Replace the total's-label test with one parametrized over five layouts and the three releases: a label row without figures, a text after the total's figures, a text beside "of which:" and an empty row above or below it. Make the part check's message describe the whole layout it needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N2tyHRTCRBaS4PzLNYo6tN
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles; standard reform costings (change in gov_balance); standard reforms: change in each reform's target. Checks failing on both base and head (1)
Counted-once checkEach of 92 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
|
Ready for hub review Verdict: APPROVE WITH NITS at The last round's P2 and all three P3s are closed. The five I found one optional P3, on two docs that the delta didn't touch. CI on this head is red only because shard 2/6 hit the job's 75-minute timeout with no failing test. That isn't caused by this delta (details below). P3P3-1 (optional): two docs describe the layout less fully than the message and the new tests. with and make the same two insertions ("with no other cell", "its label, its figures and nothing else") in the README sentence. The README edit sits in the paragraph that conflicts with Previous round's findings
Checked and soundI ran everything in a detached worktree at
CI on this head (as found, not waited on)From
What I couldn't check
Generated by Claude Code |
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed 92c5fc930df77ef061388a380ac471f374766e39. COMPLETE — no findings or material evidence gaps.
The six current growth series match the independently checked March 2026 and October 2026 OBR workbook cells, units and footnotes. All 42 yearly per-head calculations match the committed values, and the retained XML in both current-source fixtures matches the original workbooks. The population adjustment correctly avoids counting population growth twice.
All 252 selected checks passed, including quantity projection through alcohol/tobacco duty, household tax, government tax/balance and household net income. The importer, reform, horizon and reset checks also passed. Applicable checks are green at this exact head.
The documentation clearly distinguishes OBR's conditional clearance forecasts from a static duty-rate reform and explains the fixed product mix/strength assumptions and the constructed-data limitations. This approval covers the current October-over-March projection; it does not certify the author's national revenue estimates or independently authenticate the older March 2025 compatibility fixture.
main carries #2853, which reads the national living wage forecast from OBR Table 1.13. In the economic assumptions README, this branch's clearance text ends step 6, #2853's national living wage step becomes step 7 and the horizon step step 8. In test_import_obr_forecasts.py, the merge keeps both branches' imports and fixtures, then main's national living wage tests and this branch's clearance tests, each block whole, so each branch's last Hypothesis test keeps its own decorators. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22
left a comment
There was a problem hiding this comment.
Re-reviewed current head 2df40a383506c309b7de4be9f4cf74ccf6ac7dfe after the merge from main. COMPLETE — no findings or material evidence gaps.
The resolutions preserve the clearance importer alongside main's importer, CLI options, fixtures, tests and documentation. The OBR source basis and clearance parameters are unchanged. All 58 selected repository tests passed, including projected quantities through duties, household tax, government balance and household net income, plus the shared importer/CLI checks.
This retains the forecast, product-mix and population-estimate qualifications in the earlier review. Approval is pending the current head's remaining CI checks; no repeat code review is needed if the reviewed inputs remain unchanged.
juaristi22
left a comment
There was a problem hiding this comment.
Approving current head 2df40a383506c309b7de4be9f4cf74ccf6ac7dfe now that its CI has passed. COMPLETE — no open findings or material evidence gaps.
The completed projection/importer/source review stands. The head, description, dependencies and reviewed source evidence are unchanged. The retained 58 passing checks and source/artifact bindings were verified; no tests or calculations were repeated. The forecast, product-mix and population-estimate qualifications from the earlier review remain. Applicable current-head checks pass.
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed the updated description at unchanged head 2df40a383506c309b7de4be9f4cf74ccf6ac7dfe. The existing approval stands; no new findings or material evidence gaps.
The engineering notes match the recorded commits: the merged NLW importer and README steps are retained, the test module has 167 distinct test-function names, and the named main/PR overlap checks agree with the revised notes. Those checks verify recorded merge facts, not the behavior of future merges.
The completed code/source assessment and retained 58 passing checks remain valid. No tests, model calculations, private population estimates or author mutation campaigns were repeated or newly certified. Current-head CI remains clear.
Fixes #2596
What and why
Alcohol and tobacco duty read eleven household quantity inputs, and none of them had an uprating index. Dataset projection therefore kept the base-year quantities in every later year, and the model charged each year's duty rates on them. In its October 2026 EFO the OBR forecasts 2031-32 cigarette clearances 47.2% below its March 2026 EFO's 2024-25 outturn. Household weights also grow by population, 1.54% over those years. A flat input's weighted total would therefore end 92.4% above the forecast clearances for cigarettes. The other lines would end 79.9% above (hand-rolled tobacco), 62.7% (other tobacco products), 12.2% (beer and cider), 10.3% (wine) and 8.3% (spirits).
This PR builds on #2761 (merged 2 Nov) and adds six per-head clearance series, one for each product group the OBR forecasts:
gov.economic_assumptions.yoy_growth.obr.per_capita.)cigarettes_per_weektobacco_clearances.cigaretteshand_rolling_tobacco_grams_per_weektobacco_clearances.hand_rolling_tobaccocigars_grams_per_week,other_tobacco_grams_per_week,heated_tobacco_grams_per_weektobacco_clearances.other_tobacco_productsbeer_litres,cider_litres,sparkling_cider_litresalcohol_clearances.beer_and_ciderwine_litres,other_fermented_litresalcohol_clearances.winespirits_litresalcohol_clearances.spiritsimport_obr_forecasts.pyaddsClearanceSeriesSpecand sixCLEARANCE_SERIES_SPECStoRECEIPTS_SERIES_SPECS, so the run that refreshes the receipts series rewrites these too. A clearance spec takes no rates and no outturn of its own.It finds each line by table title and row label, and links the outturn year to the previous EFO's outturn, as it does for the receipts series.
It interprets no words in the clearance tables.
CLEARANCE_TABLE_TEXTSlists each text, other than titles, numbers and fiscal years, in the March 2025, March 2026 and October 2026 tables: 7 for the tobacco clearances and 9 for the alcohol clearance forecasts. It compares a cell's text in lower case and single spaces, without a footnote number after a label's last letter, asfind_receipts_linereads row labels.It reads the whole text of each cell. The reader takes shared and inline strings through one helper,
_string_text, which joins a string's own<t>and the<t>of each formatting run, so a cell whose text sits in several runs reads whole. It leaves out the<t>of a phonetic run (<rPh>).read_sheet_rowsalso keeps each row's number on the worksheet (SheetRow.number). No cell in the three receipts releases changes value: they hold no inline strings and no phonetic runs, and the shared-string reader already joined every run.Before it reads a line's values, it compares the text cells of the line's table with the list, in every column, from the table's title through its configured end: the row above a title
CLEARANCE_TABLE_ENDSlists, or the sheet's last row. It skips the title cell, numbers and fiscal years. The OBR follows the alcohol clearances with the alcohol receipts forecasts, so the check leaves out their "£ billion" cell and note. The importer checks no cell above a table's title or below its end, and the economic assumptions README asks whoever runs it to read any note there.In both EFOs it stops before writing anything when:
unit_cell): "Cigarette clearances (million sticks)", "Non-cigarette clearances ('000s kg)" for both tobacco parts, or Table 4.12's "Note: All clearances are in thousands of hectolitres of alcohol.";part_of_total) does not sit in the OBR's layout (validate_clearance_part): an "of which:" row with no other cell directly above the line and the total's other parts, and directly above that row the non-cigarette total's row, which holds its label, its figures and nothing else. A row that holds the total's label and no figures does not count as the total, and an empty row on either side of "of which:" breaks the layout. The message names the whole layout the importer needs;Without these checks, if a note put the October 2026 cigarette forecasts in ten million sticks, with the figures divided by ten, the importer would read 2026-27 cigarette growth of -91.0% in place of -10.3%. If a second Table 4.12 heading below the alcohol receipts forecasts held the alcohol lines, with the spirits forecasts in ten thousand hectolitres, the importer would read 2026-27 spirits growth of -90.1% in place of -0.8%. If the OBR stated kg for other tobacco beside its line, the importer would read the line in the non-cigarette total's thousand kg, at growth of 999.0 for clearances that did not change.
A table can keep every text on the list and still put a line under another unit. Two rearranged copies of the real October 2026 Table 4.11 put hand-rolled and other tobacco under the cigarette label: one moves the label beside a second row of years over them, and one moves the cigarette line directly above "of which:". Both halve the two parts' figures. Without the two layout checks, the importer would read 2025-26 growth of -58.6% for hand-rolled tobacco in place of -17.1%, and -56.8% for other tobacco in place of -13.6%. A phonetic run can also spell "4.12 Alcohol receipts forecasts" in a cell with no text of its own. A reader that kept its
<t>would end the alcohol clearances at that cell, above a note in ten thousand hectolitres, and the importer would read the same -90.1% spirits growth.The importer writes every series under
per_capita, these six and Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761's seven gambling bases, through oneper_head_growth, which computes(1 + aggregate growth) / (1 + ons.population growth) - 1. It divides one plus each year's growth before rounding (ReceiptsGrowth.ratios) by one plus that year'sons.populationgrowth, subtracts one and rounds once to six decimals. It reads the population from theyoy_growth.yamlit rewrites. For a gambling base it divides the base's growth, receipts over the rate the model charges, whether the importer links the line to the previous EFO or to its own outturn.After the forecast it holds the clearances flat, as
road_fuel_volumeholds road fuel volume.yoy_growth.yamlfrom the October 2026 tables over the March 2026 ones, and a test rebuilds the file byte for byte from the real tables.gov.economic_assumptions.indices.obr.per_capita.*).uprating_indices.yamlmaps it to the series, so dataset projection to 2032 and Core's uprating after it use the same index.docs/engineering/excise-duties.md, with every source file, sheet, row and hash, and list the series and the refresh step in the economic assumptions README.Dataset projection keeps the strengths (
*_abv), draught shares andcigarette_price_per_packat their base-year values, since they measure ratios and a price, as the issue says.Why per head: how projection applies weights
From
policyengine_uk/data/economic_assumptions.py:extend_single_year_datasetcopies the base year into each later year.apply_single_year_upratingmultiplies each columnuprating_indices.yamllists by one plus its series' growth for that year.household_weightunderons.population.So, for household i in year t, with population growth p(t) and quantity series g(t):
With g(t) = (1 + c(t)) / (1 + p(t)) - 1, where c(t) is clearance growth, Q(t) = (1 + c(t)) × Q(t - 1), so the total follows the clearances. Growing quantities by c(t) itself would count population growth twice.
ons.populationgrows 1.54% from 2024-25 to 2031-32, so totals grown that way would end 1.54% above the clearances. In the OBR's outturn, cigarette clearances fell 9.97%, from 13,578.2 million sticks in 2024-25 (March 2026 EFO) to 12,224.6 million in 2025-26 (October 2026 EFO). Population grows 0.33% in 2025, so the importer writes 12,224.6 / 13,578.2 / 1.0033 - 1 = -0.102653.By 2031-32 the indices (2024 = 1) stand at:
Budget measures, and what they mean for reforms
The OBR incorporates the Budget's measures in its October 2026 forecast (EFO paragraph 3.2). In policy Table 1.1 it lists no change to either duty rate. It lists three measures there that change the two duties' receipts, each by less than £13 million a year:
The OBR still assumes rising duty rates. It attributes alcohol's receipts growth to "increases to the duty rate that offset an assumed average 1 per cent decline in alcohol consumption in each year" (paragraph 4.43). For tobacco it says "higher duty rates only partly offset falling consumption" (paragraph 4.44). So the OBR forecasts the clearances along its consumption trend, under its own path of rate rises.
The dataset projection reads no duty rate (
uprating_input_paths), so when a reform changes alcohol or tobacco duty rates, the model moves receipts through the rates alone and leaves quantities where they were. It applies no consumption response, as in a static costing, and a test checks it.Projected duty on one constructed household
A constructed household, not survey data, buys 300 litres of 5% beer, 60 of 4.5% cider, 20 of 7% sparkling cider, 80 of 12% wine, 10 of 40% spirits and 15 of 5% other fermented drinks a year. Each week it buys 70 cigarettes at £16 a pack, 25 g of hand-rolling tobacco, 5 g of cigars, 3 g of other tobacco and 4 g of heated tobacco. Its duty from the 2024 inputs, on #2761 at d08fbb6 (quantities flat) and on this branch (quantities per head):
Gap to the OBR's receipts forecast
We put the UK's 2024-25 clearances (March 2026 Tables 3.11 and 3.12) in a second constructed household, also not survey data, at weight 1, so its weighted duty covers the whole UK. We assume:
We compare its duty with the OBR's alcohol receipts by group (accrued, Tables 3.12 and 4.12, rows 16 to 18) and tobacco duties (cash, Tables 3.8 and 4.8, row 27).
The model's alcohol duty exceeds the OBR's receipts by 7.6% in 2024-25 and 7.9% in 2026-27, then by 3.5% in 2027-28 and 0.3% in 2028-29. It first falls short in 2029-30, by 2.5%, and by 6.8% in 2031-32. Its tobacco duty exceeds the OBR's 2024-25 receipts by 1.9%, falls short by 0.8% in 2025-26 and 0.4% in 2026-27, and then by more each year, to 18.0% in 2031-32.
In 2024-25, charging all beer and cider at the beer rate with no draught relief lifts that line 22.4% above its receipts. On a real dataset the model charges each household's own product mix, such as the one UK: impute alcohol, tobacco and gambling consumption so policyengine-uk's excise duties stop scoring zero microcosm#1359 will supply.
After 2026-27 the model holds its current-law rates: alcohol at its 1 February 2026 values, tobacco at its 1 October 2026 values and the default cigarette price at its 2024 average. The OBR assumes rising rates. Quantities move with the clearances in both, so the gap moves with duty per unit cleared:
On Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761, with quantities flat, the model charges later rates on 2024-25 volumes, and tobacco duty ends 54.6% above the forecast.
Invariants
These hold for every input:
compute_receipts_growth, inper_head_growthand through Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761'scheck_index_stays_positive.1 + aggregate clearance growthto within 5e-7 × (1 + population growth), and equals 1 after the forecast. Weights grow by population alone.per_head_growthdivides1 + aggregate growthbefore rounding by1 + population growth, subtracts one, and rounds once. For any growth and population growth, the product(1 + per-head value) × (1 + population growth)matches1 + aggregate growthwithin the stated tolerance, 5e-7 × (1 + population growth). A series that holds a duty's base, as Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761's gambling series do, grows per head by that base's growth, not its receipts'.st.text()or from unit-shaped strings, the importer refuses every line of the table. It names the cell, or finds no line or table where the text took a label's or the title's place. A listed text other than a line's label or a marker, outside the lines, stops every line when it lands in the fiscal-year row, and stops hand-rolled or other tobacco when it lands between that part and the non-cigarette total's row. Anywhere else the importer reads every line as before. An example test, independent of the draws, changes each release's tobacco clearances in five ways: it inserts a row that holds only the non-cigarette total's label directly above "of which:", writes "Cigarette clearances (million sticks)" after the total's figures or beside "of which:", or inserts an empty row directly above or directly below "of which:". In each case the importer refuses both parts with the part check's message and reads cigarettes as before. A second property splits any text of characters from U+0020 to U+D7FF into one to four formatting runs, with or without a phonetic run. The reader returns the runs' whole text and none of the phonetic run's, from an inline string and from a shared string alike. The importer stops a line with any cell beside its label and values, and stops every line of a table whose sheet repeats its heading. A test checks that each list equals the texts in the three releases' tables, so no unread or stale entry survives.ons.populationwithout rerunning it changes none of the 13 per-head series, so the six clearance series and seven gambling bases keep the values the importer wrote for the old population. Raising 2026 population growth from 0.2% to 0.3% then leaves weighted quantities 1.000998 times the clearances from 2026 on, and the byte-for-byte test fails. Rerunning the importer rewrites each per-head series' 2026 value and nothing else.Test plan
make format: run asuvx ruff@0.17.2 format --check .anduvx ruff@0.17.2 check .on the whole repository,uvx ruff@0.17.2 check --select F811,F821 policyengine_uk, andgit diff --check, all clean at0a68b3fcf.changelog.d/2596.fixed.mdNew tests:
test_excise_quantity_projection.py: 10 tests, 2 of them Hypothesis property tests (projection and reform).test_import_obr_forecasts.pygains 31 test functions over Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761 (84 at Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761's head0037137ad, 115 at0a68b3fcf), including four Hypothesis properties: rendered clearances, per-head rounding, one arbitrary text in a real clearance table, and a text in formatting runs. One parametrized test edits the real October 2026 tables in 33 ways. It changes units and scales, product labels, listed texts, headings, marker and year rows, and cells that should hold numbers. Another writes a note into the real Table 4.11 as two formatting runs, in three ways, each with a first run that would pass alone. A third moves rows of the real Table 4.11 in three ways that keep every text on the list and put the two tobacco parts under the cigarette label. A fourth changes the rows over the two tobacco parts in each of the three releases in five ways (TOTAL_LAYOUTS, 15 cases): a row that holds only "Non-cigarette clearances ('000s kg)" directly above "of which:", "Cigarette clearances (million sticks)" after the total's figures, the same text beside "of which:", and an empty row directly above or directly below "of which:". One test repeats Table 4.12's heading below the alcohol receipts forecasts, and another fills the empty cell above their title with a phonetic run that spells it. We added clearance tables to the synthetic workbooks, and the byte-for-byte test covers the clearance series.Results (local,
uv sync --frozen --extra dev, Hypothesis 6.170.11,-n 4,HUGGING_FACE_TOKENandPOLICYENGINE_UK_DEFAULT_DATASETunset): We ran every row below at0a68b3fcf,8f3f0f396or8d9763345, one file at a time.8f3f0f396changes only three lines of the economic assumptions README after8d9763345, and0a68b3fcfchanges only the part check's message in the importer and its tests after8f3f0f396. Before the total's-figures change, at7555d402d, the same environment rantest_import_obr_forecasts.pyto 192 passed and 1 failed in 113 seconds:test_any_text_no_one_has_read_stops_a_real_clearance_tabledrew a row that holds only the non-cigarette total's label above "of which:", which the part check accepted. Between7555d402dand0a68b3fcfonly the importer, its tests,test_excise_quantity_projection.py(a fixture that clears its cache), the economic assumptions README anddocs/engineering/excise-duties.mdchange.compute_receipts_growthandrender_receipts_growth_yaml8d9763345yoy_growth.yamlbyte for byte (120,408 bytes, SHA-256846cbe78c74ff9a1c94ce39c5e4b8bf275645b78943d3c2d84dda830099e24a8)test_import_obr_forecasts.py0a68b3fcftest_excise_quantity_projection.py0a68b3fcfcode_health(the whole directory)0a68b3fcfcode_health/test_changelog_fragments.py8d9763345towncrierinstalledcode_health/test_uprating_declarations.py8d9763345code_health/test_cached_arrays_not_written_in_place.py8d9763345test_parameter_descriptions.py8d9763345test_gambling_duty_base_projection.py8d9763345test_population_vintage.py8d9763345test_projection_horizon.py8d9763345policy/baseline/gov/hmrc/alcohol_duty/alcohol_duty.yaml8d9763345policy/baseline/gov/hmrc/tobacco_duty/tobacco_duty.yaml8d9763345policy/baseline/gov/hmrc/excise_integration.yaml8d9763345We left these files to CI's full suite:
test_uprating_inputs_reach_data.py,test_excise_schedules.py,test_energy_dataset_projection.py,test_corporate_receipts_statistics.py,test_long_run_joins.py,test_lagged_series.py,test_dated_reform_routes.py,test_ni_metadata_compat.py,test_rail_usage_uprating.pyandtest_road_fuel_volume_uprating.py. The repository's checkers pass at0a68b3fcf:.github/check_changelog_fragments.pyand.github/check_fix_pr.py(on this body).Differential against #2761 at d08fbb6: We projected two constructed datasets in this branch and in #2761: two households at weights 700 and 300 buying every product, through 2074, and the national construct above, to 2032.
Stored columns differ only in the eleven quantity inputs, from 2025 to 2074 (3,264 cell values compared).
We compared 16 outputs per year to 2032 (288 values). The two duties move, and
household_tax,gov_tax,household_net_incomeandgov_balancemove by the duty change, within £0.003. Real household net income, equivalised household net income and net income after housing costs move with net income. Income tax, NI, benefits, market income, HBAI net income, VAT and gambling duty stay identical. Other outputs that read net income move too; we compared only these 16.For the first household alone in 2025:
Through the dataset, the weighted volume of all 11 inputs follows the aggregate clearances in all 51 years from 2024 to 2074, to a maximum relative error of 2.06414×10⁻⁶ (561 checks). Through Core from a 2032 projection, it does so to 1.40668×10⁻⁵ (209 checks).
This branch adds 12 parameters to the tree, the six series and their indices, and changes or removes none. It changes the
upratingof the eleven inputs and no other variable.The differential ran at the branch's earlier heads; after them the branch changes the importer's checks, its tests and docs, and no series, input or parameter.
Mutations (each in its own copy of the branch): We applied each mutation alone and ran the tests in its row. We ran the older importer mutations through a harness that loads the importer and runs its test file without importing the rest of the package; unmutated, the importer passes all 43 clearance tests and all 7 per-head tests there. The clearance mutations ran at
65f3b3f4a. Later commits add the reader's run joining and row numbers, the repeated-heading refusal at the top ofvalidate_clearance_table, the spec'spart_of_totaland the two layout checks that run after it, and change the spec's error message, comments and docstrings; the walk and the checks those mutations change keep their code. The per-head mutations ran atade1df1ceand the three projection mutations atf0f3d37e4; no later commit changes the code they mutate or the assertions of the tests they ran. Five mutations of the XLSX reader and the heading check, and the walk mutation, ran at433b20fb7through the test file itself. The reader and heading mutations ran on the five tests that cover them, the list test, the real-pair test and the phonetic-run test (8 tests); the walk mutation on the 33 edits and the arbitrary-text property (34 tests). Unmutated, all 41 tests of those selections pass. The seven mutations of the layout checks, the total's row and the phonetic-run join ran at0a68b3fcfon the whole oftest_import_obr_forecasts.py(208 tests). Unmutated, all 208 pass.test_a_repeated_clearance_heading_stops_the_runtest_a_phonetic_run_cannot_end_a_clearance_table, whose empty first run the mutation reads as no valuetest_the_read_texts_are_the_texts_of_the_three_releases, since the real alcohol footnotes keep their numbers ("1", "2") in a run of their owntest_a_repeated_clearance_heading_stops_the_run<t>of a string, a phonetic run's included (the.//main:tjoin)test_a_string_reads_whole_in_any_formatting_runsandtest_a_phonetic_run_cannot_end_a_clearance_tableTOTAL_LAYOUTScases and the arbitrary-text propertylabel-without-figurescases and the arbitrary-text propertyor not all(NUMBER_RE.fullmatch(text) for text in total[1:]))text-on-total-rowcasesof_which[:1] != ["of which:"])text-on-of-which-rowcasesempty-row-above-of-whichandempty-row-below-of-whichcases in all three releasestest_a_real_table_without_its_unit_cell_refuses_the_lines_it_describestest_the_alcohol_receipts_table_stays_outside_the_clearance_table; the title-shaped note, added later, fails toogrowth_in_effectreturn zero before population's first entrytest_population_growth_before_its_first_entry_is_that_entryper_head_growth's index checktest_per_head_growth_refuses_an_index_that_rounds_to_zeroper_head_growth's division by populationtest_real_remote_gaming_base_per_head_rounds_onceOverlap:
0037137ad, whichmainalso carries. The two Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761 commits afterd08fbb6f1, the differential's base above, change documentation and one importer comment. The branch merges withmainataf9361114without conflicts.mainat 9dd44e1, which carries Read the national living wage forecast from OBR Table 1.13 #2853, the national living wage importer. In the economic assumptions README, this branch's clearance text ends step 6, the national living wage step follows as step 7 and the horizon step as step 8.test_import_obr_forecasts.pykeeps both branches' imports and fixtures, thenmain's national living wage tests and this branch's clearance tests whole: 167 tests, with no name repeated.yoy_growth.yaml, in the same three blocks where it conflicts withmain. Each of the five conflicts withmainin the same files as with this branch.Reproducer
Reproducer: policyengine_uk/tests/test_excise_quantity_projection.py::test_weighted_quantities_follow_the_clearances
On
origin/main(2da212a) it fails at the first projected year: weighted cigarette quantities grow by population (1.0033), where the OBR's clearances fall to 0.9003.axiom: n/a: this projects data inputs from an OBR forecast and changes no rule.
🤖 Generated with Claude Code