diff --git a/changelog.d/2596.fixed.md b/changelog.d/2596.fixed.md new file mode 100644 index 0000000000..b993355a07 --- /dev/null +++ b/changelog.d/2596.fixed.md @@ -0,0 +1 @@ +Dataset projection grows alcohol and tobacco quantity inputs with the OBR's forecasts of clearances per person, using one index per product group. The model charges later duty rates on those projected quantities. diff --git a/docs/engineering/excise-duties.md b/docs/engineering/excise-duties.md index 6c44e0db57..d9f834de19 100644 --- a/docs/engineering/excise-duties.md +++ b/docs/engineering/excise-duties.md @@ -203,6 +203,201 @@ rate: every later year stays one 1 April step behind. Reform classes and legacy reform dictionaries change the annual values after uprating and are not affected. +## Projecting alcohol and tobacco quantities + +Each alcohol and tobacco quantity input declares its product group's index, +and dataset projection grows it by the same group's series in +`gov.economic_assumptions.yoy_growth.obr.per_capita` +(`policyengine_uk/data/uprating_indices.yaml`). The importer writes each +series as the growth in the OBR's clearances of the group per person +(#2596): + +| Inputs | Series | OBR line | +| --- | --- | --- | +| `cigarettes_per_week` | `tobacco_clearances.cigarettes` | Table 4.11, "Cigarette clearances (million sticks)" | +| `hand_rolling_tobacco_grams_per_week` | `tobacco_clearances.hand_rolling_tobacco` | Table 4.11, "Hand rolled tobacco clearances" | +| `cigars_grams_per_week`, `other_tobacco_grams_per_week`, `heated_tobacco_grams_per_week` | `tobacco_clearances.other_tobacco_products` | Table 4.11, "Other tobacco products" | +| `beer_litres`, `cider_litres`, `sparkling_cider_litres` | `alcohol_clearances.beer_and_cider` | Table 4.12, "Beer and cider" | +| `wine_litres`, `other_fermented_litres` | `alcohol_clearances.wine` | Table 4.12, "Wine", which includes other fermented products | +| `spirits_litres` | `alcohol_clearances.spirits` | Table 4.12, "Spirits", which includes spirit-based ready-to-drinks | + +The OBR splits Table 4.11's non-cigarette clearances into hand-rolled tobacco +and other tobacco products only, so the cigar, other tobacco and heated +tobacco inputs share one series. Dataset projection keeps the strengths +(`_abv`), the draught shares and `cigarette_price_per_pack` at their +base-year values, since the first two measure ratios and the third a price. +A forward path for the cigarette price needs its own change. + +### Population + +`extend_single_year_dataset` copies the base year's dataset into each later +year, and `apply_single_year_uprating` multiplies each column that +`uprating_indices.yaml` lists by one plus its series' growth in that year. +The file lists `household_weight` under +`gov.economic_assumptions.yoy_growth.ons.population`. So, for household i in +year t, with population growth p(t) and a quantity series g(t): + +- the weight is w(i, t) = w(i, t - 1) × (1 + p(t)); +- the quantity is q(i, t) = q(i, t - 1) × (1 + g(t)); +- the weighted total is Q(t) = Σ w(i, t) × q(i, t) = (1 + p(t)) × (1 + g(t)) × Q(t - 1). + +With g(t) = (1 + c(t)) / (1 + p(t)) - 1, where c(t) is the growth in +clearances, Q(t) = (1 + c(t)) × Q(t - 1): the weighted total follows the +clearances. A series of c(t) itself would give +Q(t) = (1 + p(t)) × (1 + c(t)) × Q(t - 1) and count population growth twice. +`ons.population` grows 1.54% from 2024-25 to 2031-32, so totals grown that +way would end 1.54% above the clearances. + +In the OBR's outturn, cigarette clearances fell 9.97%, from 13,578.2 million +sticks in 2024-25 (March 2026 EFO) to 12,224.6 million in 2025-26 (October +2026 EFO). Population grows 0.33% in 2025, so the importer writes +12,224.6 / 13,578.2 / 1.0033 - 1 = -10.27% (-0.102653) for that year. + +### Years + +| Model year | Series | Source | +| --- | --- | --- | +| 2024 (2024-25) | Base year: 0, index 1 | The alcohol and tobacco duties start in model year 2024 | +| 2025 | Outturn | October 2026 Tables 4.11 and 4.12 (2025-26) over March 2026 Tables 3.11 and 3.12 (2024-25) | +| 2026 to 2031 | Forecast | October 2026 Tables 4.11 and 4.12 | +| 2032 on | Clearances held flat: 1 / (1 + p(t)) - 1 | As `road_fuel_volume` after the OBR's forecast | + +The OBR's years run from 1 April and the model's from 6 April. As for road +fuel, the importer stores the fiscal year that starts in year t at +t-01-01. +The indices, `gov.economic_assumptions.indices.obr.per_capita.*`, start at 1 +in 2024 and stand at: + +| Group | 2025 | 2026 | 2027 | 2028 | 2029 | 2030 | 2031 | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| Cigarettes | 0.897 | 0.803 | 0.721 | 0.660 | 0.599 | 0.558 | 0.520 | +| Hand-rolled tobacco | 0.826 | 0.769 | 0.712 | 0.652 | 0.610 | 0.584 | 0.556 | +| Other tobacco products | 0.861 | 0.815 | 0.781 | 0.715 | 0.679 | 0.653 | 0.615 | +| Beer and cider | 0.969 | 0.950 | 0.929 | 0.921 | 0.914 | 0.903 | 0.891 | +| Wine | 0.965 | 0.963 | 0.945 | 0.937 | 0.929 | 0.917 | 0.907 | +| Spirits | 0.984 | 0.974 | 0.956 | 0.950 | 0.944 | 0.933 | 0.924 | + +Past `PROJECTION_END_YEAR` (2032), policyengine-core scales each input by +the index it declares. + +The importer, `policyengine_uk/utils/import_obr_forecasts.py`, writes the +six series from `CLEARANCE_SERIES_SPECS` in the same run as the other +receipts series (see the steps for a new EFO in the economic assumptions +README), and reads `ons.population` from the `yoy_growth.yaml` it rewrites. +It interprets no words in the clearance tables. `CLEARANCE_TABLE_TEXTS` +lists each text, other than titles, numbers and fiscal years, in Tables 3.11 +and 3.12 of the March 2025 and March 2026 EFOs and Tables 4.11 and 4.12 of +October 2026, from each title down to the next table's title: the line +labels, the "Outturn" and "Forecast" markers, the "of which:" row, Table +4.12's note "All clearances are in thousands of hectolitres of alcohol" and +its footnotes. On Table 4.12's sheet the OBR follows the alcohol clearances +with the alcohol receipts forecasts. The importer ends the clearance table +at their title (`CLEARANCE_TABLE_ENDS`), so its check leaves out the +receipts table's "£ billion" cell and note. From each clearance table's +title through that configured end, or the sheet's last row, the importer +compares text cells in every column with the list. It skips the title cell, +numbers and fiscal years, and stops before writing anything when: + +- a cell holds any other text, such as a note that states another unit or + scale, a line for another product or a unit column's heading; +- the table lacks the cell that states a line's unit: the cigarette label's + "(million sticks)", the non-cigarette total's "('000s kg)" for hand-rolled + and other tobacco, or Table 4.12's note for alcohol; +- a text sits beside the years in the fiscal-year row over a line, such as + a unit column's heading or a line's label; +- hand-rolled or other tobacco does not sit under an "of which:" row + directly below the non-cigarette total's row, which holds its label and + figures, with only the total's other parts between; +- the OBR puts any cell beside a line's label and values; +- the sheet repeats the table's heading, since the importer would check the + text under one heading and could read a line under another. + +A table can keep every text on the list and still put a line under another +unit: a rearranged copy of the real October 2026 Table 4.11 can put the two +tobacco parts under the cigarette label's million sticks. The two layout +checks stop it. The importer reads every formatting run of a cell's text, +and leaves out any phonetic run. It names the table and each unread cell, +the text in the fiscal-year row, the missing total or the rows that repeat +the heading. Whoever runs it reads the text, converts the table's figures to +the listed units if it states another, and adds the text to the list. The +importer checks no cell above a table's title or below its end, so that +person reads any note there too. + +### Budget measures and reforms + +The OBR incorporates in its October 2026 forecast the measures the +government announced at Budget 2026 and since the March forecast (EFO +paragraph 3.2). It lists no change to an alcohol or tobacco duty rate in its +policy Table 1.1, and three measures that change the two duties' receipts, +none by more than £13 million a year: + +- measure 19, more generous Small Producer Relief from 1 February 2027: + alcohol duty £1.5 million lower in 2026-27 and £9.2 million to £9.9 + million lower a year after (the model leaves out Small Producer Relief); +- measure 45, HMRC's measures to close the tax gap: tobacco duty £0.7 + million higher in 2026-27 and £12.7 million higher in 2031-32, and alcohol + duty £0.3 million to £4.8 million higher; +- measure 95, the removal of the £135 customs duty relief on low-value + imports from October 2028: tobacco duty up to £0.8 million and alcohol + duty up to £0.4 million lower a year. + +The OBR still assumes rising duty rates. It expects alcohol receipts to +grow, "driven by increases to the duty rate that offset an assumed average 1 +per cent decline in alcohol consumption in each year" (paragraph 4.43), and +tobacco receipts to fall "as higher duty rates only partly offset falling +consumption" (paragraph 4.44). So the OBR forecasts the clearances along +that consumption trend, under rate rises it assumes rather than ones the +government announced at the Budget. + +The dataset projection reads no duty rate (`uprating_input_paths` in +`policyengine_uk/data/economic_assumptions.py`), so it grows quantities by +the indices whatever rates the model charges. When a reform changes alcohol +or tobacco duty rates, the model therefore moves receipts through the rates +alone and applies no consumption response. Under current law the model holds +the alcohol rates at their 1 February 2026 values, the tobacco rates at +their 1 October 2026 values and the default cigarette price at its 2024 +average, while the OBR forecasts its volumes with the rates it expects. + +### Receipts against the OBR's forecast + +On quantities that follow the clearances, the model's duty moves with its +own rates. A constructed dataset of one household at weight 1 that holds the +UK's 2024-25 clearances shows the effect. It treats beer and cider as beer +at 4% with no draught relief, wine and other fermented products as wine at +12%, spirits at 40%, other tobacco products as other smoking tobacco, and +cigarettes at the default price. On that mix the model's duty per unit +cleared rises 5.7% for alcohol by 2026-27 and 19.0% for tobacco by 2027-28. +The alcohol increases take effect in February 2025 and February 2026, and +the tobacco increases in October 2024, November 2025 and October 2026. Each +reaches its first full model year after 2024-25. After that, product mix +alone moves model duty per unit. The OBR's receipts per unit cleared keep +rising, by 22.1% for alcohol and 48.2% for tobacco by 2031-32. + +On this construct, the model's alcohol duty exceeds the OBR's receipts by +7.6% in 2024-25 and 7.9% in 2026-27, then by less each year: by 3.5% in +2027-28 and 0.3% in 2028-29. It first falls short in 2029-30, by 2.5%, and +by 6.8% in 2031-32. The model's tobacco duty exceeds the OBR's 2024-25 +receipts by 1.9%, falls short by 0.8% in 2025-26 and 0.4% in 2026-27, and +then by more each year, to 18.0% in 2031-32. Charging all beer and cider at +the beer rate with no draught relief lifts that line 22.4% above its +2024-25 receipts, and a dataset brings its own product mix. From 2027-28 +the model holds its rates while the OBR assumes rising ones, and each gap +moves with that difference. + +### Sources + +The series and the figures above come from these files. The parser fixtures +in `policyengine_uk/tests/fixtures/obr_receipts/` keep the receipts sheets +of the two receipts workbooks unchanged. + +| File | SHA-256 | Sheet and cells | +| --- | --- | --- | +| OBR October 2026 EFO, detailed forecast tables zip (`efo-october-2026-detailed-forecast-tables-zipfile.zip`) | `06ac908c69fce7d69dbfda4e16fc3b091a83bd8140ecd2f24990acd13d945fc2` | Holds the two workbooks below | +| October 2026 receipts tables, as published in the zip (`efo-october-2026-detailed-forecast-tables-receipts.xlsx`) | `d115705716d77b11c190385d41cb1924f30080c9331a8a386c04d4e65a368e95` | 4.11: rows 5 (cigarettes, unit in B5), 6 (non-cigarette, unit in B6), 8 (hand-rolled) and 9 (other tobacco products), columns C (2025-26 outturn) to I (2031-32); 4.12: rows 5 to 7 (beer and cider, wine, spirits), columns C to I, unit note B8, footnotes B9 and B10; 4.8: row 27 (tobacco duties, cash) and 4.12 rows 16 to 18 (alcohol receipts, accrued), for the comparison above | +| October 2026 policy tables (`efo-october-2026-detailed-forecast-tables-policy.xlsx`) | `cdcc2a06b100241bf3010bd159a4e5d43daef2348a3850797b1e88a308fd45fd` | 1.1: rows 95 (measure 19), 203 and 204 (measure 45), 446 and 447 (measure 95), 494 and 515 (totals for alcohol and tobacco duty) | +| October 2026 EFO (`economic-and-fiscal-outlook-october-2026-web-accessible.pdf`) | `9e0e9c340e35e603066f3124a614fcd68feadb3d5d10d1f76fb536e150e6225e` | Paragraphs 3.2, 4.43 and 4.44 | +| March 2026 receipts tables, as downloaded (`obr_efo_march_2026_receipts.xlsx`; the parser fixture of that name keeps only some of its sheets) | `f65b6cb7f96931d4abd85940794eb60f6f65afca8bbf16c74a4548bf92766a28` | 3.11: rows 5, 8 and 9, column C (2024-25 outturn); 3.12: rows 5 to 7, column C, and rows 16 to 18, column C (2024-25 receipts); 3.8: row 27, column D | + ## Population data follow-ups These formulas do not impute quantities or vehicle characteristics from the diff --git a/policyengine_uk/data/uprating_indices.yaml b/policyengine_uk/data/uprating_indices.yaml index c8d64451d8..ac650a2d71 100644 --- a/policyengine_uk/data/uprating_indices.yaml +++ b/policyengine_uk/data/uprating_indices.yaml @@ -199,6 +199,25 @@ gov.economic_assumptions.yoy_growth.obr.per_capita.gambling_duty_bases.bingo: - bingo_net_receipts gov.economic_assumptions.yoy_growth.obr.per_capita.gambling_duty_bases.lottery: - national_lottery_ticket_spending +# Alcohol and tobacco quantities grow with the OBR's clearances of their +# product group per person, as household weights grow by population (#2596). +gov.economic_assumptions.yoy_growth.obr.per_capita.tobacco_clearances.cigarettes: +- cigarettes_per_week +gov.economic_assumptions.yoy_growth.obr.per_capita.tobacco_clearances.hand_rolling_tobacco: +- hand_rolling_tobacco_grams_per_week +gov.economic_assumptions.yoy_growth.obr.per_capita.tobacco_clearances.other_tobacco_products: +- cigars_grams_per_week +- heated_tobacco_grams_per_week +- other_tobacco_grams_per_week +gov.economic_assumptions.yoy_growth.obr.per_capita.alcohol_clearances.beer_and_cider: +- beer_litres +- cider_litres +- sparkling_cider_litres +gov.economic_assumptions.yoy_growth.obr.per_capita.alcohol_clearances.wine: +- other_fermented_litres +- wine_litres +gov.economic_assumptions.yoy_growth.obr.per_capita.alcohol_clearances.spirits: +- spirits_litres gov.economic_assumptions.yoy_growth.obr.private_pension_index: - private_pension_income gov.economic_assumptions.yoy_growth.obr.lagged_average_earnings: diff --git a/policyengine_uk/parameters/gov/economic_assumptions/README.md b/policyengine_uk/parameters/gov/economic_assumptions/README.md index b12c72fc58..4c504ab099 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/README.md +++ b/policyengine_uk/parameters/gov/economic_assumptions/README.md @@ -36,6 +36,8 @@ GDP and population assumptions remain supported. | `business_rates.great_britain` | Business rates receipts, England, Scotland and Wales (cash) | EFO receipts, current receipts (cash basis): Table 4.8 in October 2026 | | `business_rates.northern_ireland` | Northern Ireland business rates receipts | EFO receipts, other taxes: Table 4.2 in October 2026 | | `per_capita.gambling_duty_bases.` | The base of each of the seven betting and gaming duties (receipts over the model's rate), per person | HMRC Betting and Gaming tables (2024-25, 2025-26), then EFO receipts, betting and gaming duties forecast breakdown: Table 4.10 in October 2026, over `ons.population` | +| `per_capita.tobacco_clearances.` | Cigarette, hand-rolled tobacco and other tobacco clearances per person | EFO receipts, tobacco clearances: Table 4.11 in October 2026 (3.11 in March 2026), over `ons.population` | +| `per_capita.alcohol_clearances.` | Beer and cider, wine and spirits clearances per person | EFO receipts, alcohol clearance forecasts: Table 4.12 in October 2026 (3.12 in March 2026), over `ons.population` | The four receipts series project the corporate SDLT and business rates statistics (`gov.hmrc.stamp_duty.statistics`, @@ -66,6 +68,21 @@ base. See for each duty's rates, the higher-stake share Machine Games Duty's rate assumes, and the approximations. +Dataset projection grows the alcohol and tobacco quantity inputs (#2596) +by the six clearance series under `per_capita`: +`tobacco_clearances.cigarettes`, `hand_rolling_tobacco` and +`other_tobacco_products`, and `alcohol_clearances.beer_and_cider`, `wine` +and `spirits`. From the OBR's clearances of each product group, the +importer writes +`(1 + aggregate clearance growth) / (1 + ons.population growth) - 1` to six +decimals. Dataset projection grows household weights by `ons.population`, +so the weighted quantities grow with the clearances. The importer links each +series' outturn year to the previous EFO's outturn, as it does for the four +receipts series, holds the clearances flat after the OBR's forecast, and +reads `ons.population` from the `yoy_growth.yaml` it rewrites. See +[household excise duties](../../../../docs/engineering/excise-duties.md#projecting-alcohol-and-tobacco-quantities) +for the mapping of inputs to groups, the arithmetic and the sources. + `national_living_wage` projects the minimum wage rates (`gov.hmrc.minimum_wage`) after the last rate set. The government sets each rate following the Low Pay Commission's recommendations, and the OBR @@ -511,8 +528,10 @@ the forecast levels, though not their growth. renumbering does not matter, and reads only that table's rows. It takes each workbook's release from its contents sheet, or else its file name or URL, and refuses receipts tables from another EFO or of unknown release. - Every selected table in both releases must declare `£ billion` in its - header (`£bn` and `£ bn` are also accepted, ignoring case and whitespace). + The importer requires `£ billion` in the header of every selected + receipts table in both releases, and also accepts `£bn` and `£ bn`, + ignoring case and whitespace. The OBR states each clearance line's unit + in a label or a note (below). Other units, including `£m`, and missing or ambiguous declarations stop the import before any file is written, so different cash magnitudes cannot be combined across releases. @@ -530,6 +549,43 @@ the forecast levels, though not their growth. EFO's policy tables for any new measure in the betting and gaming lines. Then update the published levels in `policyengine_uk/tests/test_gambling_duty_base_projection.py`. + The same run also refreshes the six alcohol and tobacco clearance series + under `per_capita` from the new EFO's tobacco clearances and alcohol + clearance forecasts, and the previous EFO's for the outturn year's base. + From each table's title through its configured end, the importer compares + text cells with `CLEARANCE_TABLE_TEXTS`, in both EFOs. The end falls on + the row above a title `CLEARANCE_TABLE_ENDS` lists, or on the sheet's + last row. It skips the title cell, numbers and fiscal years. It also + needs the cell that states each line's unit: million sticks in the + cigarette label, thousand kilograms in the non-cigarette total's label + and thousand hectolitres of alcohol in Table 4.12's note. It stops before + writing on any other text and names the table and the cell. Read that + text, convert the table's figures to these units if it states another, + and add the text to the list. A table of listed texts can still put a + unit over the wrong lines, so the importer also stops on any text beside + the years in the fiscal-year row over a line. It stops when hand-rolled + or other tobacco does not sit under an "of which:" row directly below the + non-cigarette total's row, which holds its label and figures, with only + the total's other parts between, and on any cell beside a line's label + and values. When a sheet repeats a + clearance table's heading, the importer stops and names the rows. It + checks no cell above a table's title or below its end, so read any note + there yourself. The list holds each EFO's footnotes as the OBR wrote + them, and the importer does not compare the products one EFO's line + covers with another's. Check that each alcohol line covers the same + products in both EFOs: in March 2026 the OBR added "Includes other + fermented products" to the wine line. The importer writes + `(1 + aggregate clearance growth) / (1 + ons.population growth) - 1`, + using the population in `yoy_growth.yaml`, so rerun it whenever that + series changes. Until then the six clearance series and the seven + gambling bases keep the values the importer wrote for the old population, + and the byte-for-byte test fails. Then update the published levels in + `policyengine_uk/tests/test_excise_quantity_projection.py` (`CLEARANCES`) + and `policyengine_uk/tests/test_import_obr_forecasts.py` + (`OCTOBER_2026_CLEARANCES` and `MARCH_2026_CLEARANCES`, renamed for the + new EFOs), the values those tests check by hand, the index table in + `docs/engineering/excise-duties.md`, and the real-table fixtures in + `policyengine_uk/tests/fixtures/obr_receipts/` (see its README). 7. The importer refreshes `national_living_wage` in the same run, from the new economy tables' national living wage table (Table 1.13 in October 2026), with `average_earnings` after it. Then replace the cut-down table in diff --git a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml index fd4de71acd..c3f6e030b7 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml +++ b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml @@ -1476,6 +1476,327 @@ obr: - title: OBR, Economic and fiscal outlook October 2026, detailed forecast tables, receipts, Table 4.5, divided by ons.population growth href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + # Alcohol and tobacco clearances per person (#2596), which project the + # household quantity inputs (data/uprating_indices.yaml). Each is the + # growth in the OBR's clearances of a product group over one plus + # ons.population growth, to six decimals: (1 + clearance growth) / + # (1 + population growth) - 1. Dataset projection grows household + # weights by ons.population, so the weighted quantity grows with the + # clearances. The outturn year's growth divides this EFO's outturn by the + # previous EFO's outturn for the year before. After the OBR's forecast + # the clearances hold flat, as road fuel volume does. + # policyengine_uk/utils/import_obr_forecasts.py rewrites these series + # from a new EFO's receipts tables and the ons.population growth below, + # and keeps the 2024 base year. + tobacco_clearances: + cigarettes: + description: Growth per person in the OBR's cigarette clearances, which projects each household's cigarettes_per_week. + values: + # Base year (2024-25): the alcohol and tobacco duties start in model year 2024 + 2024-01-01: 0.0000 + # Outturn (2025-26 over 2024-25) from OBR EFO October 2026 over March 2026, receipts Table 4.11 (March 2026: Table 3.11) "Cigarette clearances (million sticks)"; per head: clearance growth over one plus ons.population growth + 2025-01-01: -0.102653 + # Forecast (2026-2031) from OBR EFO October 2026, receipts Table 4.11; per head: clearance growth over one plus ons.population growth + 2026-01-01: -0.105009 + 2027-01-01: -0.102347 + 2028-01-01: -0.083831 + 2029-01-01: -0.092617 + 2030-01-01: -0.068657 + 2031-01-01: -0.06894 + # Beyond the OBR forecast: clearances held flat: 1 / (1 + ons.population growth) - 1 + 2032-01-01: -0.002195 + 2033-01-01: -0.002096 + 2034-01-01: -0.001896 + 2035-01-01: -0.001797 + 2036-01-01: -0.001697 + 2037-01-01: -0.001597 + 2038-01-01: -0.001498 + 2040-01-01: -0.001398 + 2042-01-01: -0.001298 + 2044-01-01: -0.001199 + 2045-01-01: -0.001099 + 2047-01-01: -0.000999 + 2048-01-01: -0.000899 + 2049-01-01: -0.000799 + 2050-01-01: -0.0006 + 2051-01-01: -0.0005 + 2052-01-01: -0.0003 + 2053-01-01: -0.0002 + 2054-01-01: 0.0000 + 2055-01-01: 0.0001 + 2056-01-01: 0.0003 + 2057-01-01: 0.0004 + 2058-01-01: 0.0005 + 2059-01-01: 0.0006 + 2060-01-01: 0.0007 + 2061-01-01: 0.000801 + 2064-01-01: 0.000901 + 2071-01-01: 0.001001 + metadata: + unit: /1 + label: Cigarette clearances growth per person + reference: + - title: OBR EFO October 2026 (detailed forecast tables, receipts, Table 4.11) + href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + - title: OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.11; 2024-25 outturn) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + hand_rolling_tobacco: + description: Growth per person in the OBR's hand-rolled tobacco clearances, which projects each household's hand_rolling_tobacco_grams_per_week. + values: + # Base year (2024-25): the alcohol and tobacco duties start in model year 2024 + 2024-01-01: 0.0000 + # Outturn (2025-26 over 2024-25) from OBR EFO October 2026 over March 2026, receipts Table 4.11 (March 2026: Table 3.11) "Hand rolled tobacco clearances"; per head: clearance growth over one plus ons.population growth + 2025-01-01: -0.17412 + # Forecast (2026-2031) from OBR EFO October 2026, receipts Table 4.11; per head: clearance growth over one plus ons.population growth + 2026-01-01: -0.069369 + 2027-01-01: -0.073763 + 2028-01-01: -0.083858 + 2029-01-01: -0.064999 + 2030-01-01: -0.041711 + 2031-01-01: -0.048922 + # Beyond the OBR forecast: clearances held flat: 1 / (1 + ons.population growth) - 1 + 2032-01-01: -0.002195 + 2033-01-01: -0.002096 + 2034-01-01: -0.001896 + 2035-01-01: -0.001797 + 2036-01-01: -0.001697 + 2037-01-01: -0.001597 + 2038-01-01: -0.001498 + 2040-01-01: -0.001398 + 2042-01-01: -0.001298 + 2044-01-01: -0.001199 + 2045-01-01: -0.001099 + 2047-01-01: -0.000999 + 2048-01-01: -0.000899 + 2049-01-01: -0.000799 + 2050-01-01: -0.0006 + 2051-01-01: -0.0005 + 2052-01-01: -0.0003 + 2053-01-01: -0.0002 + 2054-01-01: 0.0000 + 2055-01-01: 0.0001 + 2056-01-01: 0.0003 + 2057-01-01: 0.0004 + 2058-01-01: 0.0005 + 2059-01-01: 0.0006 + 2060-01-01: 0.0007 + 2061-01-01: 0.000801 + 2064-01-01: 0.000901 + 2071-01-01: 0.001001 + metadata: + unit: /1 + label: Hand-rolled tobacco clearances growth per person + reference: + - title: OBR EFO October 2026 (detailed forecast tables, receipts, Table 4.11) + href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + - title: OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.11; 2024-25 outturn) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + other_tobacco_products: + description: Growth per person in the OBR's clearances of other tobacco products, which projects each household's cigars_grams_per_week, other_tobacco_grams_per_week and heated_tobacco_grams_per_week. + values: + # Base year (2024-25): the alcohol and tobacco duties start in model year 2024 + 2024-01-01: 0.0000 + # Outturn (2025-26 over 2024-25) from OBR EFO October 2026 over March 2026, receipts Table 4.11 (March 2026: Table 3.11) "Other tobacco products"; per head: clearance growth over one plus ons.population growth + 2025-01-01: -0.13903 + # Forecast (2026-2031) from OBR EFO October 2026, receipts Table 4.11; per head: clearance growth over one plus ons.population growth + 2026-01-01: -0.053308 + 2027-01-01: -0.042058 + 2028-01-01: -0.083858 + 2029-01-01: -0.050588 + 2030-01-01: -0.039029 + 2031-01-01: -0.058055 + # Beyond the OBR forecast: clearances held flat: 1 / (1 + ons.population growth) - 1 + 2032-01-01: -0.002195 + 2033-01-01: -0.002096 + 2034-01-01: -0.001896 + 2035-01-01: -0.001797 + 2036-01-01: -0.001697 + 2037-01-01: -0.001597 + 2038-01-01: -0.001498 + 2040-01-01: -0.001398 + 2042-01-01: -0.001298 + 2044-01-01: -0.001199 + 2045-01-01: -0.001099 + 2047-01-01: -0.000999 + 2048-01-01: -0.000899 + 2049-01-01: -0.000799 + 2050-01-01: -0.0006 + 2051-01-01: -0.0005 + 2052-01-01: -0.0003 + 2053-01-01: -0.0002 + 2054-01-01: 0.0000 + 2055-01-01: 0.0001 + 2056-01-01: 0.0003 + 2057-01-01: 0.0004 + 2058-01-01: 0.0005 + 2059-01-01: 0.0006 + 2060-01-01: 0.0007 + 2061-01-01: 0.000801 + 2064-01-01: 0.000901 + 2071-01-01: 0.001001 + metadata: + unit: /1 + label: Other tobacco products clearances growth per person + reference: + - title: OBR EFO October 2026 (detailed forecast tables, receipts, Table 4.11) + href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + - title: OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.11; 2024-25 outturn) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + alcohol_clearances: + beer_and_cider: + description: Growth per person in the OBR's beer and cider clearances, which projects each household's beer_litres, cider_litres and sparkling_cider_litres. + values: + # Base year (2024-25): the alcohol and tobacco duties start in model year 2024 + 2024-01-01: 0.0000 + # Outturn (2025-26 over 2024-25) from OBR EFO October 2026 over March 2026, receipts Table 4.12 (March 2026: Table 3.12) "Beer and cider"; per head: clearance growth over one plus ons.population growth + 2025-01-01: -0.030668 + # Forecast (2026-2031) from OBR EFO October 2026, receipts Table 4.12; per head: clearance growth over one plus ons.population growth + 2026-01-01: -0.020398 + 2027-01-01: -0.021805 + 2028-01-01: -0.008938 + 2029-01-01: -0.007436 + 2030-01-01: -0.011936 + 2031-01-01: -0.012643 + # Beyond the OBR forecast: clearances held flat: 1 / (1 + ons.population growth) - 1 + 2032-01-01: -0.002195 + 2033-01-01: -0.002096 + 2034-01-01: -0.001896 + 2035-01-01: -0.001797 + 2036-01-01: -0.001697 + 2037-01-01: -0.001597 + 2038-01-01: -0.001498 + 2040-01-01: -0.001398 + 2042-01-01: -0.001298 + 2044-01-01: -0.001199 + 2045-01-01: -0.001099 + 2047-01-01: -0.000999 + 2048-01-01: -0.000899 + 2049-01-01: -0.000799 + 2050-01-01: -0.0006 + 2051-01-01: -0.0005 + 2052-01-01: -0.0003 + 2053-01-01: -0.0002 + 2054-01-01: 0.0000 + 2055-01-01: 0.0001 + 2056-01-01: 0.0003 + 2057-01-01: 0.0004 + 2058-01-01: 0.0005 + 2059-01-01: 0.0006 + 2060-01-01: 0.0007 + 2061-01-01: 0.000801 + 2064-01-01: 0.000901 + 2071-01-01: 0.001001 + metadata: + unit: /1 + label: Beer and cider clearances growth per person + reference: + - title: OBR EFO October 2026 (detailed forecast tables, receipts, Table 4.12) + href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + - title: OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.12; 2024-25 outturn) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + wine: + description: Growth per person in the OBR's wine clearances, including other fermented products, which projects each household's wine_litres and other_fermented_litres. + values: + # Base year (2024-25): the alcohol and tobacco duties start in model year 2024 + 2024-01-01: 0.0000 + # Outturn (2025-26 over 2024-25) from OBR EFO October 2026 over March 2026, receipts Table 4.12 (March 2026: Table 3.12) "Wine"; per head: clearance growth over one plus ons.population growth + 2025-01-01: -0.034606 + # Forecast (2026-2031) from OBR EFO October 2026, receipts Table 4.12; per head: clearance growth over one plus ons.population growth + 2026-01-01: -0.002797 + 2027-01-01: -0.018867 + 2028-01-01: -0.008149 + 2029-01-01: -0.008291 + 2030-01-01: -0.012466 + 2031-01-01: -0.011764 + # Beyond the OBR forecast: clearances held flat: 1 / (1 + ons.population growth) - 1 + 2032-01-01: -0.002195 + 2033-01-01: -0.002096 + 2034-01-01: -0.001896 + 2035-01-01: -0.001797 + 2036-01-01: -0.001697 + 2037-01-01: -0.001597 + 2038-01-01: -0.001498 + 2040-01-01: -0.001398 + 2042-01-01: -0.001298 + 2044-01-01: -0.001199 + 2045-01-01: -0.001099 + 2047-01-01: -0.000999 + 2048-01-01: -0.000899 + 2049-01-01: -0.000799 + 2050-01-01: -0.0006 + 2051-01-01: -0.0005 + 2052-01-01: -0.0003 + 2053-01-01: -0.0002 + 2054-01-01: 0.0000 + 2055-01-01: 0.0001 + 2056-01-01: 0.0003 + 2057-01-01: 0.0004 + 2058-01-01: 0.0005 + 2059-01-01: 0.0006 + 2060-01-01: 0.0007 + 2061-01-01: 0.000801 + 2064-01-01: 0.000901 + 2071-01-01: 0.001001 + metadata: + unit: /1 + label: Wine clearances growth per person + reference: + - title: OBR EFO October 2026 (detailed forecast tables, receipts, Table 4.12) + href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + - title: OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.12; 2024-25 outturn) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + spirits: + description: Growth per person in the OBR's spirits clearances, including spirit-based ready-to-drinks, which projects each household's spirits_litres. + values: + # Base year (2024-25): the alcohol and tobacco duties start in model year 2024 + 2024-01-01: 0.0000 + # Outturn (2025-26 over 2024-25) from OBR EFO October 2026 over March 2026, receipts Table 4.12 (March 2026: Table 3.12) "Spirits"; per head: clearance growth over one plus ons.population growth + 2025-01-01: -0.016504 + # Forecast (2026-2031) from OBR EFO October 2026, receipts Table 4.12; per head: clearance growth over one plus ons.population growth + 2026-01-01: -0.009907 + 2027-01-01: -0.0181 + 2028-01-01: -0.005966 + 2029-01-01: -0.006394 + 2030-01-01: -0.012237 + 2031-01-01: -0.009692 + # Beyond the OBR forecast: clearances held flat: 1 / (1 + ons.population growth) - 1 + 2032-01-01: -0.002195 + 2033-01-01: -0.002096 + 2034-01-01: -0.001896 + 2035-01-01: -0.001797 + 2036-01-01: -0.001697 + 2037-01-01: -0.001597 + 2038-01-01: -0.001498 + 2040-01-01: -0.001398 + 2042-01-01: -0.001298 + 2044-01-01: -0.001199 + 2045-01-01: -0.001099 + 2047-01-01: -0.000999 + 2048-01-01: -0.000899 + 2049-01-01: -0.000799 + 2050-01-01: -0.0006 + 2051-01-01: -0.0005 + 2052-01-01: -0.0003 + 2053-01-01: -0.0002 + 2054-01-01: 0.0000 + 2055-01-01: 0.0001 + 2056-01-01: 0.0003 + 2057-01-01: 0.0004 + 2058-01-01: 0.0005 + 2059-01-01: 0.0006 + 2060-01-01: 0.0007 + 2061-01-01: 0.000801 + 2064-01-01: 0.000901 + 2071-01-01: 0.001001 + metadata: + unit: /1 + label: Spirits clearances growth per person + reference: + - title: OBR EFO October 2026 (detailed forecast tables, receipts, Table 4.12) + href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + - title: OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.12; 2024-25 outturn) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + house_prices: description: House prices year-on-year growth. values: @@ -1622,7 +1943,8 @@ ons: # published. # The per_capita series under obr and orr are derived from this series; # re-derive them, and the petrol and diesel spending proxies, whenever it - # changes. + # changes. import_obr_forecasts.py rewrites the alcohol and tobacco + # clearance series from it. values: 2021-01-01: 0.0036 2022-01-01: 0.0105 diff --git a/policyengine_uk/tests/fixtures/obr_receipts/README.md b/policyengine_uk/tests/fixtures/obr_receipts/README.md index cf59073008..cc5562b490 100644 --- a/policyengine_uk/tests/fixtures/obr_receipts/README.md +++ b/policyengine_uk/tests/fixtures/obr_receipts/README.md @@ -3,10 +3,12 @@ These parser fixtures retain the original worksheet XML for Contents, Other taxes (3.2, or 4.2 in October 2026), Property transactions taxes: receipts by sector (3.7, or 4.7), Current receipts (on a cash basis: 3.9 in March 2025, -3.8 in March 2026, 4.8 in October 2026) and, from March 2026, Betting and -gaming duties forecast breakdown (3.10, or 4.10), plus the original shared -strings. The March 2025 tables have no betting and gaming breakdown. Only -the workbook's sheet list and relationships were reduced to those entries. The published unit cells, fiscal-year headers, markers, row labels and +3.8 in March 2026, 4.8 in October 2026), Tobacco clearances (3.11, or 4.11), +Alcohol clearance forecasts (3.12, or 4.12, a sheet that also holds the +alcohol receipts forecasts) and, from March 2026, Betting and gaming duties +forecast breakdown (3.10, or 4.10), plus the original shared strings. The +March 2025 tables have no betting and gaming breakdown. We reduced only the +workbook's sheet list and relationships to those entries. The published unit cells, fiscal-year headers, markers, row labels and numeric cells are unchanged. These are minimal archives for the importer's XLSX reader, rather than complete Excel documents with styles and charts. The October 2026 workbook titles sheet 4.8 "43.8 Current receipts (on a cash diff --git a/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_march_2025_receipts.xlsx b/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_march_2025_receipts.xlsx index 3e6ca8b1e8..2de2958e06 100644 Binary files a/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_march_2025_receipts.xlsx and b/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_march_2025_receipts.xlsx differ diff --git a/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_march_2026_receipts.xlsx b/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_march_2026_receipts.xlsx index 4ccc9c019b..b144090dcd 100644 Binary files a/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_march_2026_receipts.xlsx and b/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_march_2026_receipts.xlsx differ diff --git a/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_october_2026_receipts.xlsx b/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_october_2026_receipts.xlsx index fed8a21a1b..d4a90ce2d1 100644 Binary files a/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_october_2026_receipts.xlsx and b/policyengine_uk/tests/fixtures/obr_receipts/obr_efo_october_2026_receipts.xlsx differ diff --git a/policyengine_uk/tests/test_excise_quantity_projection.py b/policyengine_uk/tests/test_excise_quantity_projection.py new file mode 100644 index 0000000000..702fda11f7 --- /dev/null +++ b/policyengine_uk/tests/test_excise_quantity_projection.py @@ -0,0 +1,497 @@ +"""Alcohol and tobacco quantity inputs grow with the OBR's clearances per head +(#2596). + +Each input declares its product group's index under +``gov.economic_assumptions.indices.obr.per_capita``, and dataset projection +grows it by the same ``yoy_growth`` series, +(1 + aggregate clearance growth) / (1 + ons.population growth) - 1, from the +OBR's clearances of the group (October 2026 receipts Tables 4.11 and 4.12, +after the March 2026 EFO's 2024-25 outturn). Dataset +projection grows household weights by ``ons.population``, so the weighted +quantities grow with the clearances, and a duty-rate reform moves receipts +through the rate alone. +""" + +from functools import lru_cache +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest +import yaml +from hypothesis import given, settings +from hypothesis import strategies as st + +from policyengine_uk import CountryTaxBenefitSystem, Simulation +from policyengine_uk.data import UKSingleYearDataset +from policyengine_uk.data import economic_assumptions +from policyengine_uk.data.economic_assumptions import ( + extend_single_year_dataset, + uprating_input_fingerprint, +) +from policyengine_uk.model_api import Scenario + +pytestmark = pytest.mark.usefixtures("cloned_uk_tax_benefit_system") + +GROWTH = "gov.economic_assumptions.yoy_growth.obr.per_capita" +INDEX = "gov.economic_assumptions.indices.obr.per_capita" +POPULATION = "gov.economic_assumptions.yoy_growth.ons.population" +INPUTS = { + "tobacco_clearances.cigarettes": ("cigarettes_per_week",), + "tobacco_clearances.hand_rolling_tobacco": ("hand_rolling_tobacco_grams_per_week",), + "tobacco_clearances.other_tobacco_products": ( + "cigars_grams_per_week", + "heated_tobacco_grams_per_week", + "other_tobacco_grams_per_week", + ), + "alcohol_clearances.beer_and_cider": ( + "beer_litres", + "cider_litres", + "sparkling_cider_litres", + ), + "alcohol_clearances.wine": ("other_fermented_litres", "wine_litres"), + "alcohol_clearances.spirits": ("spirits_litres",), +} +GROUP = {name: group for group, names in INPUTS.items() for name in names} +# OBR clearances by product group, 2024-25 to 2031-32: the March 2026 EFO's +# 2024-25 outturn (receipts Tables 3.11 and 3.12, column C), then the October +# 2026 EFO's 2025-26 outturn and forecast (Tables 4.11 and 4.12, columns C to +# I). Million sticks, thousand kg, and thousand hectolitres of alcohol. +CLEARANCES = { + "tobacco_clearances.cigarettes": (13578.199084269218, 12224.568453264392, 10962.756413840059, 9853.5452739735829, 9042.8621730867635, 8222.5673694138313, 7677.1786518547042, 7165.065963684312), + "tobacco_clearances.hand_rolling_tobacco": (4028.4786869103718, 3338.0207129187247, 3112.6781722236924, 2886.8250342433803, 2649.2381871262742, 2482.242556381048, 2384.6513795861088, 2273.4315215751089), + "tobacco_clearances.other_tobacco_products": (696.58878604045401, 601.72119516629675, 570.78371235846851, 547.48868265410397, 502.43014658034485, 478.01479115950667, 460.50663098211788, 434.81308397816122), + "alcohol_clearances.beer_and_cider": (2168.2921615793966, 2108.7303043893562, 2069.8474469367434, 2027.3458030333193, 2012.6406194934405, 2001.8700847628832, 1982.9201100206114, 1962.5485885566063), + "alcohol_clearances.wine": (1619.9398031719702, 1569.041217813324, 1567.7826624097067, 1540.2033043369458, 1530.2485351419407, 1520.7475110216285, 1505.5439058920508, 1491.4031570887553), + "alcohol_clearances.spirits": (1289.5543778979199, 1272.4562775587838, 1262.3692523269856, 1241.1318815407608, 1235.8250379612948, 1230.5012764962146, 1218.4826638265554, 1209.5688671337984), +} # fmt: skip +FORECAST_YEARS = range(2024, 2032) +YEARS = range(2024, 2033) +ALCOHOL_RATES = "gov.hmrc.alcohol_duty.rates" +TOBACCO_RATES = "gov.hmrc.tobacco_duty.rates" +DUTIES = ("alcohol_duty", "tobacco_duty") + + +@lru_cache(maxsize=None) +def current_law(): + return CountryTaxBenefitSystem().parameters + + +@pytest.fixture(scope="module") +def parameters(): + return current_law() + + +@pytest.fixture(scope="module", autouse=True) +def release_current_law(): + # Drop the cached parameters once this module's tests are done, instead + # of keeping them for the rest of the test process. + yield + current_law.cache_clear() + + +def growth(parameters, path, year): + return parameters.get_child(path)(f"{year}-01-01") + + +def index(parameters, group, year): + return parameters.get_child(f"{INDEX}.{group}")(f"{year}-01-01") + + +def household_frame(year, households): + """Build a dataset with one adult in each household, from the given + household inputs. Pass each household as a dict of inputs; + ``household_weight`` defaults to 1.""" + count = len(households) + columns = sorted({name for household in households for name in household}) + return UKSingleYearDataset( + person=pd.DataFrame( + { + "person_id": range(1, count + 1), + "person_benunit_id": range(1, count + 1), + "person_household_id": range(1, count + 1), + "age": [40] * count, + } + ), + benunit=pd.DataFrame({"benunit_id": range(1, count + 1)}), + household=pd.DataFrame( + { + "household_id": range(1, count + 1), + "region": ["LONDON"] * count, + "tenure_type": ["OWNED_OUTRIGHT"] * count, + "council_tax": [0.0] * count, + "rent": [0.0] * count, + "household_weight": [1.0] * count, + **{ + name: [float(household.get(name, 0.0)) for household in households] + for name in columns + }, + } + ), + fiscal_year=year, + ) + + +def situation(inputs, year=2024): + return { + "people": {"person": {"age": {year: 40}}}, + "benunits": {"benunit": {"members": ["person"]}}, + "households": { + "household": { + "members": ["person"], + **{name: {year: value} for name, value in inputs.items()}, + } + }, + } + + +# A household that buys every product: 5% beer, 4.5% cider, 7% sparkling +# cider, 12% wine, 40% spirits and 5% other fermented drinks, and pays £16 a +# pack for its cigarettes. +SMOKER_AND_DRINKER = { + "beer_litres": 300.0, + "beer_abv": 0.05, + "cider_litres": 60.0, + "cider_abv": 0.045, + "sparkling_cider_litres": 20.0, + "sparkling_cider_abv": 0.07, + "wine_litres": 80.0, + "wine_abv": 0.12, + "spirits_litres": 10.0, + "spirits_abv": 0.40, + "other_fermented_litres": 15.0, + "other_fermented_abv": 0.05, + "cigarettes_per_week": 70.0, + "cigarette_price_per_pack": 16.0, + "hand_rolling_tobacco_grams_per_week": 25.0, + "cigars_grams_per_week": 5.0, + "other_tobacco_grams_per_week": 3.0, + "heated_tobacco_grams_per_week": 4.0, +} +QUANTITIES = {name: SMOKER_AND_DRINKER[name] for name in GROUP} + + +def test_every_quantity_input_declares_its_product_group_index(): + """Every alcohol and tobacco quantity input declares its product group's + index, and dataset projection grows it by the same group's series.""" + from policyengine_uk.system import system + + folder = Path(__file__).parents[1] / "variables/input/consumption" + quantities = { + path.stem + for path in [*folder.glob("alcohol/*.py"), *folder.glob("tobacco/*.py")] + if path.stem.endswith(("_litres", "_per_week")) + } + assert quantities == set(GROUP) + for name, group in GROUP.items(): + assert system.variables[name].uprating == f"{INDEX}.{group}", name + mapping = yaml.safe_load( + (Path(__file__).parents[1] / "data/uprating_indices.yaml").read_text() + ) + clearances = {key: value for key, value in mapping.items() if "clearances" in key} + assert clearances == { + f"{GROWTH}.{group}": sorted(names) for group, names in INPUTS.items() + } + + +def test_strengths_shares_and_the_cigarette_price_stay_as_they_are(): + """Strengths and draught shares measure ratios, and the cigarette price a + price, not a quantity (#2596 leaves its forward path out): none of them + declares an index or appears in the uprating map.""" + from policyengine_uk.system import system + + folder = Path(__file__).parents[1] / "variables/input/consumption" + others = { + path.stem + for path in [*folder.glob("alcohol/*.py"), *folder.glob("tobacco/*.py")] + } - set(GROUP) + assert others >= {"beer_abv", "wine_draught_share", "cigarette_price_per_pack"} + mapped = { + name + for names in economic_assumptions._uprating_index_map().values() + for name in names + } + for name in others: + assert system.variables[name].uprating is None, name + assert name not in mapped, name + + +def test_indices_are_positive_from_the_2024_base_year(parameters): + for group in INPUTS: + assert index(parameters, group, 2024) == 1 + for year in range(2024, 2040): + assert index(parameters, group, year) > 0, (group, year) + + +def test_each_series_is_its_clearances_per_head(parameters): + """Each year's growth equals, to six decimals, the clearances' growth + over one plus that year's population growth, the growth of household + weights.""" + for group, levels in CLEARANCES.items(): + assert growth(parameters, f"{GROWTH}.{group}", 2024) == 0 + for year in range(2025, 2032): + published = levels[year - 2024] / levels[year - 2025] + population = growth(parameters, POPULATION, year) + assert growth(parameters, f"{GROWTH}.{group}", year) == round( + published / (1 + population) - 1, 6 + ), (group, year) + # After the forecast the clearances hold flat. + for year in range(2032, 2074): + population = growth(parameters, POPULATION, year) + assert growth(parameters, f"{GROWTH}.{group}", year) == round( + 1 / (1 + population) - 1, 6 + ), (group, year) + + +def test_cigarettes_per_head_halve_by_2031(parameters): + """In its October 2026 EFO the OBR forecasts 2031-32 cigarette clearances + 47.2% below its March 2026 EFO's 2024-25 outturn. Population grows 1.5% over + those years, so clearances per head fall 48.0%, and the index stands at + 0.520.""" + levels = CLEARANCES["tobacco_clearances.cigarettes"] + assert levels[-1] / levels[0] - 1 == pytest.approx(-0.4723, abs=1e-4) + population = np.prod( + [1 + growth(parameters, POPULATION, year) for year in range(2025, 2032)] + ) + assert population - 1 == pytest.approx(0.0154, abs=1e-4) + assert index(parameters, "tobacco_clearances.cigarettes", 2031) == pytest.approx( + levels[-1] / levels[0] / population, abs=1e-5 + ) + assert index(parameters, "tobacco_clearances.cigarettes", 2031) == pytest.approx( + 0.520, abs=5e-4 + ) + + +def test_weighted_quantities_follow_the_clearances(parameters): + """Two households with different weights: dataset projection grows each + quantity per head and each weight by population growth, so every + weighted total grows with the OBR's clearances, and in 2032-33, after + the forecast, holds at 2031-32's. Growing quantities by the clearances + themselves would count population growth twice: 1.5% too much by + 2031-32.""" + households = [QUANTITIES, {name: 2 * value for name, value in QUANTITIES.items()}] + frame = household_frame(2024, households) + frame.household["household_weight"] = [1_000.0, 250.0] + projected = extend_single_year_dataset(frame, parameters, end_year=2032) + base = projected[2024].household + weights = {year: projected[year].household["household_weight"] for year in YEARS} + population = 1.0 + for year in YEARS: + if year > 2024: + population *= 1 + growth(parameters, POPULATION, year) + assert weights[year].sum() == pytest.approx(1_250 * population, rel=1e-12) + for name, group in GROUP.items(): + levels = CLEARANCES[group] + published = levels[min(year, 2031) - 2024] / levels[0] + total = (weights[year] * projected[year].household[name]).sum() + base_total = (weights[2024] * base[name]).sum() + assert total / base_total == pytest.approx(published, rel=1e-5), ( + name, + year, + ) + assert population - 1 > 0.015 + + +QUANTITY = st.floats(min_value=0, max_value=1e4, allow_nan=False) + + +# Each example projects a dataset through 2032. +@settings(max_examples=8, deadline=None, derandomize=True) +@given( + quantities=st.fixed_dictionaries({name: QUANTITY for name in GROUP}), + weight=st.floats(min_value=0.1, max_value=1e5, allow_nan=False), +) +def test_projected_quantities_are_the_base_times_the_index(quantities, weight): + """For any quantities and weight: dataset projection gives each year's + quantity as the 2024 quantity times its series' growth compounded, which + matches its index to the index's five decimals; Core's uprating of a 2024 + situation input (the other route to a later year) agrees; the weight + grows by population alone.""" + parameters = current_law() + frame = household_frame(2024, [quantities]) + frame.household["household_weight"] = [weight] + projected = extend_single_year_dataset(frame, parameters, end_year=2032) + simulation = Simulation(situation=situation(quantities)) + for name, group in GROUP.items(): + compounded = 1.0 + for year in YEARS: + if year > 2024: + compounded *= 1 + growth(parameters, f"{GROWTH}.{group}", year) + stored = projected[year].household[name].iloc[0] + assert stored == pytest.approx(quantities[name] * compounded, rel=1e-12) + ratio = index(parameters, group, year) / index(parameters, group, 2024) + assert compounded == pytest.approx(ratio, rel=1e-4), (name, year) + uprated = simulation.calculate(name, year)[0] + assert uprated == pytest.approx(quantities[name] * ratio, rel=1e-6) + population = 1.0 + for year in YEARS: + if year > 2024: + population *= 1 + growth(parameters, POPULATION, year) + stored = projected[year].household["household_weight"].iloc[0] + assert stored == pytest.approx(weight * population, rel=1e-12) + + +def test_a_later_year_charges_duty_on_the_uprated_quantity(parameters): + """A projected 2027 household pays the duty a household holding the 2027 + quantities pays, less than on its 2024 quantities: 2027 rates on fewer + cigarettes and drinks.""" + projected = Simulation(dataset=household_frame(2024, [SMOKER_AND_DRINKER])) + uprated = { + name: SMOKER_AND_DRINKER[name] + * index(parameters, group, 2027) + / index(parameters, group, 2024) + for name, group in GROUP.items() + } + held = {**SMOKER_AND_DRINKER, **uprated} + at_2027 = Simulation(situation=situation(held, year=2027)) + flat = Simulation(situation=situation(SMOKER_AND_DRINKER, year=2027)) + for duty in DUTIES: + duty_2027 = projected.calculate(duty, 2027)[0] + assert duty_2027 == pytest.approx(at_2027.calculate(duty, 2027)[0], rel=1e-4) + assert duty_2027 < flat.calculate(duty, 2027)[0] + # Cigarettes alone: 70 a week is 3,640 a year. Per head, clearances fall + # 10.3%, 10.5% and 10.2% in 2025-26, 2026-27 and 2027-28, an index of + # 0.721 in 2027, which leaves 2,624. + assert index(parameters, "tobacco_clearances.cigarettes", 2027) == 0.72092 + cigarettes = projected.calculate("cigarettes_per_week", 2027)[0] * 52 + assert cigarettes == pytest.approx(3_640 * 0.72092, rel=1e-5) + + +def rate_leaves(parameters): + """Return every alcohol and tobacco duty rate the duty formulas read, + without the wine strength easement's true-or-false switch.""" + leaves = [ + leaf + for node in (ALCOHOL_RATES, TOBACCO_RATES) + for leaf in parameters.get_child(node).get_descendants() + if not hasattr(leaf, "children") + ] + switches = [leaf.name for leaf in leaves if leaf.metadata["unit"] == "bool"] + assert switches == [f"{ALCOHOL_RATES}.wine_easement"] + return [leaf.name for leaf in leaves if leaf.name not in switches] + + +# Each example builds a reform, which reprocesses the parameter tree. +@settings(max_examples=4, deadline=None, derandomize=True) +@given( + factor=st.floats(min_value=0.1, max_value=3, allow_nan=False), + year=st.integers(min_value=2027, max_value=2031), + scale=st.floats(min_value=0.1, max_value=10, allow_nan=False), +) +def test_a_duty_rate_reform_moves_receipts_by_the_rate_alone(factor, year, scale): + """Scaling every alcohol and tobacco duty rate in a year from 2027-28, + when the current-law rates hold flat, leaves every projected quantity as + it was and multiplies each duty by the factor: each duty scales linearly + with its rates, and the dataset projection reads no rate.""" + parameters = current_law() + inputs = { + name: value * scale if name in GROUP else value + for name, value in SMOKER_AND_DRINKER.items() + } + changes = { + name: {str(year): parameters.get_child(name)(f"{year}-06-01") * factor} + for name in rate_leaves(parameters) + } + scenario = Scenario(parameter_changes=changes) + baseline = Simulation(dataset=household_frame(2024, [inputs])) + reform = Simulation(dataset=household_frame(2024, [inputs]), scenario=scenario) + assert uprating_input_fingerprint( + reform.tax_benefit_system.parameters + ) == uprating_input_fingerprint(baseline.tax_benefit_system.parameters) + for name in GROUP: + for period in (2024, year): + np.testing.assert_array_equal( + reform.calculate(name, period), baseline.calculate(name, period) + ) + for duty in DUTIES: + before = baseline.calculate(duty, year)[0] + assert before > 0 + assert reform.calculate(duty, year)[0] == pytest.approx( + before * factor, rel=1e-5 + ) + + +# Outputs that read neither duty, directly or through a total. +UNTOUCHED = ( + "income_tax", + "national_insurance", + "household_benefits", + "household_market_income", + "hbai_household_net_income", + "vat", +) + + +def test_the_change_touches_only_the_quantity_inputs(parameters): + """Compare with the projection without this change, which leaves the + quantity inputs out of the uprating map and out of Core's + fallback: every other stored input keeps its value in every year, the + quantity inputs differ by their index alone, and outputs differ only + through the two duties.""" + households = [ + {**SMOKER_AND_DRINKER, "remote_gaming_net_losses": 500.0}, + {"beer_litres": 50.0, "cigarettes_per_week": 0.0}, + ] + frame = household_frame(2024, households) + frame.person["employment_income"] = [30_000.0, 55_000.0] + frame.household["household_weight"] = [700.0, 300.0] + mapping = economic_assumptions._uprating_index_map() + unmapped = economic_assumptions._unmapped_uprating_indices + without_map = { + key: names for key, names in mapping.items() if "clearances" not in key + } + assert sum(len(names) for names in mapping.values()) - sum( + len(names) for names in without_map.values() + ) == len(GROUP) + with pytest.MonkeyPatch.context() as patch: + patch.setattr(economic_assumptions, "_uprating_index_map", lambda: without_map) + patch.setattr( + economic_assumptions, + "_unmapped_uprating_indices", + lambda uprating: { + name: index + for name, index in unmapped(uprating).items() + if name not in GROUP + }, + ) + before = extend_single_year_dataset(frame, parameters, end_year=2032) + after = extend_single_year_dataset(frame, parameters, end_year=2032) + for year in YEARS: + for table in ("person", "benunit", "household"): + old, new = getattr(before[year], table), getattr(after[year], table) + assert list(old.columns) == list(new.columns) + for column in old.columns: + if column in GROUP: + group = GROUP[column] + ratio = index(parameters, group, year) / index( + parameters, group, 2024 + ) + np.testing.assert_allclose( + new[column], old[column] * ratio, rtol=1e-4 + ) + np.testing.assert_array_equal(old[column], frame.household[column]) + else: + pd.testing.assert_series_equal(old[column], new[column]) + old_simulation = Simulation(dataset=before) + new_simulation = Simulation(dataset=after) + for year in YEARS: + changed = sum( + new_simulation.calculate(duty, year) - old_simulation.calculate(duty, year) + for duty in DUTIES + ) + if year > 2024: + assert changed[0] < 0 + for total in ("household_tax", "gov_tax", "gov_balance"): + difference = new_simulation.calculate( + total, year + ) - old_simulation.calculate(total, year) + np.testing.assert_allclose(difference, changed, atol=0.01) + net = new_simulation.calculate( + "household_net_income", year + ) - old_simulation.calculate("household_net_income", year) + np.testing.assert_allclose(net, -changed, atol=0.01) + for name in UNTOUCHED: + np.testing.assert_array_equal( + new_simulation.calculate(name, year), + old_simulation.calculate(name, year), + ) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 1a92947ee4..06868df038 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -1,9 +1,10 @@ import math import os +import re from contextlib import contextmanager from dataclasses import replace from datetime import date, timedelta -from functools import partial +from functools import lru_cache, partial from pathlib import Path from io import BytesIO from textwrap import indent @@ -12,7 +13,7 @@ import pytest import yaml -from hypothesis import event, example, given, settings +from hypothesis import assume, event, example, given, settings from hypothesis import strategies as st from policyengine_core.parameters import ParameterNode, config from policyengine_core.parameters.operations.uprate_parameters import ( @@ -27,18 +28,27 @@ from policyengine_uk.utils import import_obr_forecasts from policyengine_uk.utils.import_obr_forecasts import ( + ALCOHOL_CLEARANCES, + CLEARANCE_SERIES_SPECS, + CLEARANCE_TABLE_ENDS, + CLEARANCE_TABLE_TEXTS, + TOBACCO_CLEARANCES, RECEIPTS_SERIES_SPECS, SERIES_SPECS, STATUTORY_GAP_SPECS, _sheet_paths, + ClearanceSeriesSpec, ForecastGap, LevelAnchor, MACHINE_GAMES_HIGHER_STAKE_SHARE, + ReceiptsGrowth, + ReceiptsLine, ReceiptsSeriesSpec, machine_games_spec, average_earnings_path, average_earnings_years, build_model_index, + check_index, compute_level_growth, compute_receipts_growth, compound_model_index, @@ -46,13 +56,18 @@ format_gap, format_growth, get_yoy_growth_path, + growth_in_effect, infer_workbook_release, level_text, main, nearest_units, + per_head_growth, + population_growth, projected_rate_units, read_calendar_values, read_level_anchor, + read_sheet_rows, + receipts_label, read_workbook_sheets, render_forecast_gap_yaml, render_level_series_yaml, @@ -64,8 +79,6 @@ extract_september_cpi_from_receipts, infer_forecast_start_year, infer_release, - per_head_growth, - population_growth, update_forecast_gap_yaml, update_yoy_growth_yaml, ) @@ -526,6 +539,135 @@ def receipts_table( "bingo": "Bingo duty", "pool_betting": "Pool betting duty", } +# Synthetic clearances from each EFO's first (outturn) year, in each line's +# own unit: million sticks, thousand kg or thousand hectolitres of alcohol. +CLEARANCE_LEVELS = { + "cigarettes": (12_000.0, 10_800.0, 9_720.0), + "hand_rolling_tobacco": (3_300.0, 3_135.0, 2_978.25), + "other_tobacco_products": (600.0, 588.0, 576.24), + "beer_and_cider": (2_100.0, 2_079.0, 2_058.21), + "wine": (1_570.0, 1_585.7, 1_569.843), + "spirits": (1_270.0, 1_257.3, 1_244.727), +} +PREVIOUS_CLEARANCE_LEVELS = { + "cigarettes": (13_500.0, 12_150.0, 11_000.0), + "hand_rolling_tobacco": (4_000.0, 3_600.0, 3_400.0), + "other_tobacco_products": (700.0, 650.0, 640.0), + "beer_and_cider": (2_150.0, 2_120.0, 2_100.0), + "wine": (1_600.0, 1_590.0, 1_580.0), + "spirits": (1_300.0, 1_280.0, 1_270.0), +} + + +def clearance_table( + title: str, + first_year: int, + rows: list[tuple[str, tuple]], + notes: tuple[str, ...] = (), + outturn_years: int = 1, + start: int = 2, + unit: str | None = None, +) -> dict[int, list[str]]: + """Build a table in the layout of the OBR's Tables 4.11 and 4.12: title + in column B, the "Outturn" and "Forecast" markers on the next row, + fiscal years on the row after, then one row per line and any notes. A + clearance table has no unit cell, since the OBR states the unit in a + label or a note; ``unit`` adds one on a row of its own below the title, + as a receipts table has.""" + years = max(len(levels) for _, levels in rows) + columns = [column_letter(3 + i) for i in range(years)] + sheet = {start: [make_inline_cell(f"B{start}", title)]} + if unit is not None: + start += 1 + sheet[start] = [make_inline_cell(f"C{start}", unit)] + sheet[start + 1] = [ + *([make_inline_cell(f"C{start + 1}", "Outturn")] if outturn_years else []), + *( + [make_inline_cell(f"{columns[outturn_years]}{start + 1}", "Forecast")] + if outturn_years < years + else [] + ), + ] + sheet[start + 2] = [ + make_inline_cell( + f"{column}{start + 2}", f"{first_year + i}-{str(first_year + i + 1)[2:]}" + ) + for i, column in enumerate(columns) + ] + row = start + 3 + for label, levels in rows: + sheet[row] = [make_inline_cell(f"B{row}", label)] + [ + make_number_cell(f"{column}{row}", level) + for column, level in zip(columns, levels) + ] + row += 1 + for note in notes: + sheet[row] = [make_inline_cell(f"B{row}", note)] + row += 1 + return sheet + + +def tobacco_table(chapter, first_year, levels, outturn_years=1): + """Build Table 4.11 (3.11 to March 2026): cigarettes in million sticks, + then non-cigarette clearances in thousand kg and their "of which" split.""" + non_cigarette = tuple( + float(hand_rolled) + float(other) + for hand_rolled, other in zip( + levels["hand_rolling_tobacco"], levels["other_tobacco_products"] + ) + ) + return make_sheet( + { + 1: [make_inline_cell("A1", "Back to contents")], + **clearance_table( + f" {chapter}.11 Tobacco clearances", + first_year, + [ + ("Cigarette clearances (million sticks)", levels["cigarettes"]), + ("Non-cigarette clearances ('000s kg)", non_cigarette), + ("of which:", ()), + ("Hand rolled tobacco clearances", levels["hand_rolling_tobacco"]), + ("Other tobacco products", levels["other_tobacco_products"]), + ], + outturn_years=outturn_years, + ), + } + ) + + +def alcohol_table(chapter, first_year, levels, outturn_years=1, note=None): + """Build Table 4.12 (3.12 to March 2026): clearances in thousand + hectolitres of alcohol, which the OBR states in a note, then the duties' + receipts in £ billion under the same labels, which the importer must not + read for the clearance lines.""" + labels = { + "beer_and_cider": " Beer and cider", + "wine": " Wine1", + "spirits": " Spirits2", + } + clearances = clearance_table( + f"{chapter}.12 Alcohol clearance forecasts", + first_year, + [(labels[key], levels[key]) for key in labels], + notes=( + note or "Note: All clearances are in thousands of hectolitres of alcohol.", + "1 Includes other fermented products such as fruit ciders.", + "2 Includes spirit-based ‘ready-to-drinks’ such as alcopops.", + ), + outturn_years=outturn_years, + ) + receipts = clearance_table( + f"{chapter}.12 Alcohol receipts forecasts", + first_year, + [(labels[key], (4.0,) * len(levels[key])) for key in labels], + notes=("Note: All taxes are on an accrued basis.",), + outturn_years=outturn_years, + start=max(clearances) + 2, + unit="£ billion", + ) + return make_sheet( + {1: [make_inline_cell("A1", "Back to contents")], **clearances, **receipts} + ) def make_receipts_xlsx( @@ -538,11 +680,14 @@ def make_receipts_xlsx( sdlt_rows: list[tuple[str, tuple]] | None = None, chapter: int = 3, gambling_levels: dict[str, tuple] | None = None, + clearance_levels: dict[str, tuple] | None = None, + alcohol_note: str | None = None, ) -> bytes: """Receipts tables numbered in ``chapter``: 3 to March 2026, 4 from October 2026.""" levels = levels or RECEIPTS_LEVELS gambling_levels = gambling_levels or GAMBLING_LEVELS + clearance_levels = clearance_levels or CLEARANCE_LEVELS sheet_318 = make_sheet({2: [make_inline_cell("B2", "Other table")]}) sheet_319 = make_sheet( { @@ -645,6 +790,12 @@ def make_receipts_xlsx( ], outturn_years=0, ) + sheets[f"{chapter}.11"] = tobacco_table( + chapter, first_year, clearance_levels, outturn_years + ) + sheets[f"{chapter}.12"] = alcohol_table( + chapter, first_year, clearance_levels, outturn_years, alcohol_note + ) sheets[f"{chapter}.18"] = sheet_318 sheets[f"{chapter}.19"] = sheet_319 return make_workbook(sheets) @@ -660,6 +811,7 @@ def make_previous_receipts_xlsx(**overrides) -> bytes: "levels": PREVIOUS_RECEIPTS_LEVELS, "cash_table": "3.9", "northern_ireland_label": "Northern Ireland Business Rates", + "clearance_levels": PREVIOUS_CLEARANCE_LEVELS, **overrides, } ) @@ -863,6 +1015,33 @@ def test_small_gaps_are_written_in_fixed_point(value, text): - title: Kept href: https://example.com/kept """ +CLEARANCE_SERIES_FIXTURE = """ {key}: + description: {key} clearances growth per person. + values: + # Base year, which every run keeps + 2023-01-01: 0.0000 + # Older forecast + 2024-01-01: 0.0100 + 2031-01-01: 0.0200 + metadata: + unit: /1 + label: {key} clearances growth per person + reference: + - title: Old + href: https://example.com/old +""" +CLEARANCE_FIXTURE = ( + "".join( + f" {group}:\n" + + "".join( + CLEARANCE_SERIES_FIXTURE.format(key=spec.path[-1]) + for spec in CLEARANCE_SERIES_SPECS + if spec.path[1] == group + ) + for group in ("tobacco_clearances", "alcohol_clearances") + ) + + "\n" +) # Population growth changes in 2026 and 2030, so each year's per-head growth # must take that year's population growth out. POPULATION_FIXTURE = """ons: @@ -878,6 +1057,7 @@ def test_small_gaps_are_written_in_fixed_point(value, text): + LEVEL_SERIES_FIXTURE + RECEIPTS_FIXTURE + PER_CAPITA_FIXTURE + + CLEARANCE_FIXTURE + "".join(FLAT_SERIES_FIXTURE.format(key=key) for key in SERIES_KEYS[2:]) + POPULATION_FIXTURE ) @@ -1085,18 +1265,24 @@ def failing_replace(source, target): # Receipts growth series: stamp_duty_land_tax and business_rates under obr, -# which link their first year to the previous EFO's outturn, and the -# gambling_duty_bases under per_capita, which grow a duty's base per head -# from HMRC's outturn. +# which link their first year to the previous EFO's outturn, and under +# per_capita the gambling_duty_bases, which grow a duty's base per head from +# HMRC's outturn, and the clearance series, which link their first year as +# receipts do. -RECEIPTS_SPECS = tuple(spec for spec in RECEIPTS_SERIES_SPECS if spec.outturn is None) +RECEIPTS_SPECS = tuple( + spec + for spec in RECEIPTS_SERIES_SPECS + if spec.outturn is None and not isinstance(spec, ClearanceSeriesSpec) +) RECEIPTS_PATHS = {spec.path[-1]: spec.path for spec in RECEIPTS_SPECS} GAMBLING_PATHS = { spec.path[-1]: spec.path for spec in RECEIPTS_SERIES_SPECS if spec.outturn is not None } -ALL_RECEIPTS_PATHS = {**RECEIPTS_PATHS, **GAMBLING_PATHS} +CLEARANCE_PATHS = {spec.path[-1]: spec.path for spec in CLEARANCE_SERIES_SPECS} +ALL_RECEIPTS_PATHS = {**RECEIPTS_PATHS, **GAMBLING_PATHS, **CLEARANCE_PATHS} def _series(tree, path): @@ -1634,13 +1820,32 @@ def test_rendered_growth_compounds_the_published_levels(years, data): "great_britain": ("32.376814327646052", "34.051963932489258", "37.413586572144581", "38.240930947824609", "39.111850954748434", "42.419067764546902", "42.463193622519263"), "northern_ireland": ("0.38233350730688931", "0.39471664406441903", "0.40263364127921164", "0.41045270712298504", "0.41810894612429189", "0.42590403232705243", "0.43449009714292341"), } # fmt: skip +# Clearances as the OBR publishes them in the workbooks, from each EFO's outturn +# year: October 2026 Tables 4.11 (rows 5, 8 and 9) and 4.12 (rows 5 to 7) +# from 2025-26, and March 2026 Tables 3.11 and 3.12 from 2024-25. +OCTOBER_2026_CLEARANCES = { + "cigarettes": ("12224.568453264392", "10962.756413840059", "9853.5452739735829", "9042.8621730867635", "8222.5673694138313", "7677.1786518547042", "7165.065963684312"), + "hand_rolling_tobacco": ("3338.0207129187247", "3112.6781722236924", "2886.8250342433803", "2649.2381871262742", "2482.242556381048", "2384.6513795861088", "2273.4315215751089"), + "other_tobacco_products": ("601.72119516629675", "570.78371235846851", "547.48868265410397", "502.43014658034485", "478.01479115950667", "460.50663098211788", "434.81308397816122"), + "beer_and_cider": ("2108.7303043893562", "2069.8474469367434", "2027.3458030333193", "2012.6406194934405", "2001.8700847628832", "1982.9201100206114", "1962.5485885566063"), + "wine": ("1569.041217813324", "1567.7826624097067", "1540.2033043369458", "1530.2485351419407", "1520.7475110216285", "1505.5439058920508", "1491.4031570887553"), + "spirits": ("1272.4562775587838", "1262.3692523269856", "1241.1318815407608", "1235.8250379612948", "1230.5012764962146", "1218.4826638265554", "1209.5688671337984"), +} # fmt: skip +MARCH_2026_CLEARANCES = { + "cigarettes": ("13578.199084269218", "12254.443899171421", "10935.71615536125", "9796.6640404179016", "8866.6130178015883", "7995.729597841897", "7564.6566739628925"), + "hand_rolling_tobacco": ("4028.4786869103718", "3526.3857151341385", "3293.6576388022659", "2892.1697163272493", "2753.8285838812085", "2585.2813720012768", "2487.166773432516"), + "other_tobacco_products": ("696.58878604045401", "669.27631225428354", "677.85087256392069", "583.39376884631531", "632.00622388116062", "635.62283867654082", "648.80354856597478"), + "beer_and_cider": ("2168.2921615793966", "2083.3332688432952", "2047.9991564911497", "2055.0663286925724", "2069.6074004097741", "2085.2376704507528", "2101.0702511760765"), + "wine": ("1619.9398031719702", "1534.8113403538073", "1521.0993234521127", "1529.4264001780152", "1542.2932290667245", "1555.708517982323", "1569.3328887285095"), + "spirits": ("1289.5543778979199", "1239.4917691306655", "1236.8062704924209", "1240.0908662025893", "1247.2437066467319", "1254.8362045261438", "1262.8339376703782"), +} # fmt: skip def test_october_2026_tables_reproduce_the_committed_series(): - """yoy_growth.yaml holds exactly what the importer writes from the - October 2026 receipts tables (Tables 4.7, 4.8, 4.2 and 4.10) and the - March 2026 ones (3.7, 3.8 and 3.2): rerunning it on the committed file - changes nothing.""" + """yoy_growth.yaml holds, byte for byte, what the importer writes from the + October 2026 receipts tables (Tables 4.7, 4.8, 4.2, 4.10, 4.11 and 4.12) + and the March 2026 ones (3.7, 3.8, 3.2, 3.11 and 3.12): rerunning it on + the committed file changes nothing.""" committed = get_yoy_growth_path().read_text() rendered = render( committed, @@ -1651,6 +1856,7 @@ def test_october_2026_tables_reproduce_the_committed_series(): cash_table="4.8", chapter=4, gambling_levels=OCTOBER_2026_GAMBLING, + clearance_levels=OCTOBER_2026_CLEARANCES, ), previous=make_previous_receipts_xlsx( release="March 2026", @@ -1658,6 +1864,7 @@ def test_october_2026_tables_reproduce_the_committed_series(): levels=MARCH_2026_RECEIPTS, cash_table="3.8", northern_ireland_label="Northern Ireland business rates", + clearance_levels=MARCH_2026_CLEARANCES, ), release=("October", 2026), previous_release=("March", 2026), @@ -1676,6 +1883,7 @@ def test_october_2026_tables_reproduce_the_committed_series(): COMMITTED_RELEASES = {"release": ("October", 2026), "previous_release": ("March", 2026)} XLSX_MAIN = "http://schemas.openxmlformats.org/spreadsheetml/2006/main" XLSX_NS = {"m": XLSX_MAIN} +XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace" def mutate_real_receipts(workbook, table, unit, scale=1): @@ -4382,3 +4590,1702 @@ def test_a_written_level_series_keeps_every_index_positive(levels, earnings): assert rendered == unchecked model = model_index(written) assert {year: finished[year] for year in model} == model + + +# Alcohol and tobacco clearances per head (#2596): Tables 4.11 and 4.12 (3.11 +# and 3.12 to March 2026), whose growth over ons.population growth projects +# the household quantity inputs. + +CLEARANCE_SPECS = {spec.path[-1]: spec for spec in CLEARANCE_SERIES_SPECS} +# ons.population as POPULATION_FIXTURE holds it, in effect in each year. +FIXTURE_POPULATION = {2024: 0.01, 2025: 0.01, 2026: 0.02, 2027: 0.02, 2030: -0.005} + + +def test_clearance_lines_are_read_from_their_own_tables(): + """Each line comes from its clearance table, in its unit, not from the + receipts table below Table 3.12 under the same labels.""" + current = read_workbook_sheets(make_receipts_xlsx()) + previous = read_workbook_sheets(make_previous_receipts_xlsx()) + for key, spec in CLEARANCE_SPECS.items(): + line = find_receipts_line(current, spec) + before = find_receipts_line(previous, spec) + table = "3.11" if "tobacco" in spec.path[1] else "3.12" + assert (line.table, before.table) == (table, table) + assert line.levels == dict(zip((2024, 2025, 2026), CLEARANCE_LEVELS[key])) + assert before.levels[2023] == PREVIOUS_CLEARANCE_LEVELS[key][0] + assert (line.outturn_years, before.outturn_years) == ({2024}, {2023}) + assert find_receipts_line(current, CLEARANCE_SPECS["wine"]).label == "Wine" + + +def test_clearance_growth_is_per_head_of_the_population_the_file_holds(): + """The importer writes each year's value as one plus the clearances' + growth, over one plus that year's ons.population growth in the file it + writes, minus one; after the forecast it holds the clearances flat and + writes 1 / (1 + population) - 1.""" + rendered = render() + for key, path in CLEARANCE_PATHS.items(): + published = { + 2023: PREVIOUS_CLEARANCE_LEVELS[key][0], + **dict(zip((2024, 2025, 2026), CLEARANCE_LEVELS[key])), + } + expected = { + year: round( + published[year] + / published[year - 1] + / (1 + _in_effect(FIXTURE_POPULATION, year)) + - 1, + 6, + ) + for year in (2024, 2025, 2026) + } + assert _values(rendered, path) == { + 2023: 0.0, + **expected, + 2027: round(1 / 1.02 - 1, 6), + 2030: round(1 / 0.995 - 1, 6), + }, key + # Cigarettes fall 10% a year, 10.9% per head once population grows 1%. + assert _values(rendered, CLEARANCE_PATHS["cigarettes"])[2025] == -0.108911 + + +def test_clearance_comments_and_references_name_both_efos_and_the_population(): + rendered = render() + block = rendered[rendered.index(" cigarettes:") :] + block = block[: block.index(" hand_rolling_tobacco:")] + assert ( + " # Base year, which every run keeps\n" + " 2023-01-01: 0.0000\n" + " # Outturn (2024-25 over 2023-24) from OBR EFO March 2026 over " + 'March 2025, receipts Table 3.11 "Cigarette clearances (million ' + 'sticks)"; per head: clearance growth over one plus ons.population ' + "growth\n" + ) in block + assert ( + " # Beyond the OBR forecast: clearances held flat: " + "1 / (1 + ons.population growth) - 1\n" + # Per head the series falls while population grows and rises once it + # falls (2030 in the fixture). + " 2027-01-01: -0.019608\n" + " 2030-01-01: 0.005025\n" + ) in block + references = _series(yaml.safe_load(rendered), CLEARANCE_PATHS["wine"])["metadata"][ + "reference" + ] + assert [reference["title"] for reference in references] == [ + "OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.12)", + "OBR EFO March 2025 (detailed forecast tables, receipts, Table 3.12; " + "2023-24 outturn)", + ] + + +TOBACCO_TITLE = {"B": "4.11 Tobacco clearances"} +TOBACCO_HEADER = [ + {"C": "Outturn", "D": "Forecast"}, + {"C": "2025-26", "D": "2026-27"}, +] + + +def test_a_cigarette_line_in_another_unit_is_not_read(): + # The OBR states the unit in the cigarette label, so a new unit renames + # the row. + rows = [TOBACCO_TITLE, *TOBACCO_HEADER] + rows.append({"B": "Cigarette clearances (billion sticks)", "C": "12", "D": "11"}) + with pytest.raises(ValueError, match="found none"): + find_receipts_line({"4.11": rows}, CLEARANCE_SPECS["cigarettes"]) + + +def test_an_alcohol_table_in_another_unit_is_refused_before_any_write(tmp_path): + yoy, gap_dir = write_tree(tmp_path) + before = {path: path.read_bytes() for path in [yoy, *gap_dir.iterdir()]} + note = "Note: All clearances are in millions of litres of alcohol." + receipts = make_receipts_xlsx(alcohol_note=note) + with pytest.raises(ValueError, match=re.escape(f"not read: {note!r} in column B")): + run_main(tmp_path, make_test_xlsx(), "--yaml-path", str(yoy), receipts=receipts) + assert {path: path.read_bytes() for path in before} == before + + +def test_a_clearance_spec_needs_a_read_unit_cell_and_takes_no_rates_or_outturn(): + spec = CLEARANCE_SPECS["wine"] + with pytest.raises(ValueError, match="needs the cell that states its unit"): + ClearanceSeriesSpec(path=("per_capita", "x"), title=ALCOHOL_CLEARANCES, row="r") + with pytest.raises(ValueError, match=r"texts lack '\(tonnes\)'"): + replace(spec, unit_cell="(tonnes)") + # Each table's list holds its own cells. + with pytest.raises(ValueError, match="texts lack 'cigarette clearances"): + replace(spec, unit_cell=CLEARANCE_SPECS["cigarettes"].unit_cell) + with pytest.raises(ValueError, match="needs the cell that states its unit"): + replace(spec, title="alcohol receipts forecasts") + with pytest.raises(ValueError, match="takes no rates"): + replace(spec, rates=GAMBLING_SPECS[0].rates) + with pytest.raises(ValueError, match="takes no rates"): + replace(spec, outturn=GAMBLING_SPECS[0].outturn) + + +def test_real_october_2026_clearance_series(): + """Check the committed series by hand: the real October 2026 Tables + 4.11 and 4.12 over the real March 2026 Tables 3.11 and 3.12, per head of + the committed ons.population.""" + population = population_growth(get_yoy_growth_path().read_text()) + assert (population[2025], population[2026]) == (0.0033, 0.002) + growth = compute_receipts_growth( + REAL_OCTOBER_2026_RECEIPTS, + REAL_RECEIPTS[2026], + specs=CLEARANCE_SERIES_SPECS, + ) + per_head = { + path[-1]: per_head_growth(series, population).values + for path, series in growth.items() + } + # Cigarettes: 2025-26 outturn (October 2026) over 2024-25 outturn (March + # 2026), 10.0% fewer sticks, 10.3% fewer per head. + assert per_head["cigarettes"][2025] == round( + 12224.568453264392 / 13578.199084269218 / 1.0033 - 1, 6 + ) + assert per_head["cigarettes"][2025] == -0.102653 + assert per_head["hand_rolling_tobacco"][2025] == -0.17412 + # Beer and cider: 2026-27 over 2025-26, both from October 2026. + assert per_head["beer_and_cider"][2026] == round( + 2069.8474469367434 / 2108.7303043893562 / 1.002 - 1, 6 + ) + committed = get_yoy_growth_path().read_text() + for key, values in per_head.items(): + assert sorted(values) == list(range(2025, 2032)), key + stored = _values(committed, CLEARANCE_PATHS[key]) + assert stored[2024] == 0.0 + assert {year: stored[year] for year in values} == values, key + + +def test_a_population_change_moves_the_per_head_series_once_the_importer_reruns(): + """Editing ons.population leaves every per-head series as the importer + wrote it for the old population, so from 2026 the weighted cigarette + total runs 1.003 / 1.002 = 1.000998 times the clearances' path, and the + byte-for-byte test fails until someone reruns the importer. Rerunning + rewrites each per-head series' 2026 value, the six clearance series and + the seven gambling bases alike, and nothing else.""" + committed = get_yoy_growth_path().read_text() + edited = committed.replace( + " 2026-01-01: 0.0020\n 2027-01-01: 0.0013\n", + " 2026-01-01: 0.0030\n 2027-01-01: 0.0013\n", + ) + assert population_growth(edited)[2026] == 0.003 + for path in ALL_RECEIPTS_PATHS.values(): + assert _values(edited, path) == _values(committed, path), path + levels = [float(level) for level in OCTOBER_2026_CLEARANCES["cigarettes"]] + cigarettes = _values(edited, CLEARANCE_PATHS["cigarettes"])[2026] + weighted = (1 + cigarettes) * 1.003 / (levels[1] / levels[0]) + assert weighted == pytest.approx(1.003 / 1.002, abs=1e-6) + assert round(1.003 / 1.002, 6) == 1.000998 + + rendered = render( + edited, + current=REAL_OCTOBER_2026_RECEIPTS, + previous=REAL_RECEIPTS[2026], + **COMMITTED_RELEASES, + ) + growth = compute_receipts_growth(REAL_OCTOBER_2026_RECEIPTS, REAL_RECEIPTS[2026]) + moved = set() + for path in ALL_RECEIPTS_PATHS.values(): + before, after = _values(committed, path), _values(rendered, path) + if growth[path].spec.per_head: + assert after[2026] == round(growth[path].ratios[2026] / 1.003 - 1, 6) + assert after[2026] != before[2026] + assert {y: v for y, v in after.items() if y != 2026} == { + y: v for y, v in before.items() if y != 2026 + } + moved.add(path) + else: + assert after == before, path + assert moved == set(CLEARANCE_PATHS.values()) | set(GAMBLING_PATHS.values()) + assert len(moved) == 13 + + +CLEARANCE_LEVEL = st.floats(min_value=1, max_value=20_000, allow_nan=False) +# A year's clearances from half to twice the year before's, so the index stays +# well above the five decimals the model rounds it to. +CLEARANCE_GROWTH = st.floats(min_value=0.5, max_value=2, allow_nan=False) +POPULATION_GROWTH = st.floats(min_value=-0.02, max_value=0.03, allow_nan=False).map( + lambda value: round(value, 4) +) + + +@settings(max_examples=40, deadline=None, derandomize=True) +@given(years=st.integers(2, 7), data=st.data()) +def test_rendered_clearances_grow_the_weighted_total_with_the_clearances(years, data): + """For any clearances and population growth: in every year the household + quantity's growth times the household weight's growth equals the + clearances' growth, to the series' six decimals, and equals 1 after the + forecast; the series stays above -100%, so its index stays positive; a + second run changes nothing.""" + base, levels = {}, {} + for key in CLEARANCE_LEVELS: + level = data.draw(CLEARANCE_LEVEL) + base[key] = (level,) + series = [] + for _ in range(years): + level *= data.draw(CLEARANCE_GROWTH) + series.append(level) + levels[key] = tuple(series) + population = { + year: data.draw(POPULATION_GROWTH, label=f"population {year}") + for year in range(2024, 2036) + } + content = YOY_GROWTH_FIXTURE.replace( + POPULATION_FIXTURE, + "ons:\n population:\n values:\n" + + "".join( + f" {year}-01-01: {format_growth(value)}\n" + for year, value in population.items() + ), + ) + current = make_receipts_xlsx(clearance_levels=levels) + previous = make_previous_receipts_xlsx(clearance_levels=base) + rendered = render(content, current=current, previous=previous) + last = 2024 + years - 1 + for key, path in CLEARANCE_PATHS.items(): + values = _values(rendered, path) + published = { + 2023: base[key][0], + **dict(zip(range(2024, last + 1), levels[key])), + } + for year in range(2024, 2036): + growth = _in_effect(values, year) + assert growth > -1 + weighted = (1 + growth) * (1 + population[year]) + target = published[year] / published[year - 1] if year <= last else 1 + # Six-decimal rounding moves the product by at most 5e-7 times + # one plus the population growth. + assert abs(weighted - target) <= 5e-7 * (1 + population[year]) + 1e-12 + assert values[2023] == 0.0 + assert render(rendered, current=current, previous=previous) == rendered + + +# A unit beside a line must stop the run: the importer would otherwise read +# the line's values in the unit another cell states. + + +def mixed_unit_workbook(rows): + """Build a one-sheet workbook of inline strings.""" + table = rows[0]["B"].split()[0] + return make_workbook( + { + table: make_sheet( + { + number: [ + make_inline_cell(f"{column}{number}", str(value)) + for column, value in row.items() + ] + for number, row in enumerate(rows, start=1) + } + ) + } + ) + + +def test_a_unit_beside_a_clearance_line_is_refused_not_borrowed(): + """The previous release holds other tobacco in thousand kg, the current + one in kg in a cell beside the line, while its non-cigarette total still + states thousand kg. Read as one unit, the line has growth of 999.0; the + importer refuses it.""" + previous = mixed_unit_workbook( + [ + {"B": "3.11 Tobacco clearances"}, + {"C": "Outturn"}, + {"C": "2024-25"}, + {"B": "Non-cigarette clearances ('000s kg)", "C": 4000}, + {"B": "of which:"}, + {"B": "Other tobacco products", "C": 600}, + ] + ) + current = mixed_unit_workbook( + [ + {"B": "4.11 Tobacco clearances"}, + {"D": "Outturn", "E": "Forecast"}, + {"D": "2025-26", "E": "2026-27"}, + {"B": "Non-cigarette clearances ('000s kg)", "D": 4000, "E": 3900}, + {"B": "of which:"}, + {"B": "Other tobacco products", "C": "kg", "D": 600000, "E": 570000}, + ] + ) + spec = CLEARANCE_SPECS["other_tobacco_products"] + with pytest.raises(ValueError, match="not read: 'kg' in column C"): + compute_receipts_growth(current, previous, specs=(spec,)) + # Without the kg cell the same layout reads. + reconciled = mixed_unit_workbook( + [ + {"B": "4.11 Tobacco clearances"}, + {"D": "Outturn", "E": "Forecast"}, + {"D": "2025-26", "E": "2026-27"}, + {"B": "Non-cigarette clearances ('000s kg)", "D": 4000, "E": 3900}, + {"B": "of which:"}, + {"B": "Other tobacco products", "D": 600, "E": 570}, + ] + ) + growth = compute_receipts_growth(reconciled, previous, specs=(spec,))[spec.path] + assert growth.values == {2025: 0.0, 2026: -0.05} + assert per_head_growth(growth, {2025: 0.01, 2026: 0.01}).values[2025] == round( + 1 / 1.01 - 1, 6 + ) + + +def test_a_unit_note_beside_a_receipts_line_is_refused(): + """The importer refuses a cell beside a pound line's label and values too: + a "£ million" cell there would put its values in another magnitude.""" + rows = [ + {"B": "4.7 Property transactions taxes: receipts by sector"}, + {"C": "£ billion"}, + {"C": "Outturn", "D": "Forecast"}, + {"C": "2025-26", "D": "2026-27"}, + { + "B": "SDLT from residential property", + "C": "11.5", + "D": "12.0", + "E": "£ million", + }, + ] + spec = next(spec for spec in RECEIPTS_SPECS if spec.path[-1] == "residential") + with pytest.raises(ValueError, match=r"holds \{'E': '£ million'\} beside"): + find_receipts_line({"4.7": rows}, spec) + + +def put_unit_beside_real_line(workbook, table, label, unit, scale=1000): + """Shift the sheet's cells from column C, row 3, one column right, put + ``unit`` in C beside the line labelled ``label`` and scale that line's + values. Every other archive entry, including the shared strings, stays.""" + output = BytesIO() + with ZipFile(BytesIO(workbook)) as source, ZipFile(output, "w") as dest: + path = _sheet_paths(source)[table] + strings = [ + "".join(node.itertext()) + for node in ET.fromstring(source.read("xl/sharedStrings.xml")) + ] + sheet = ET.fromstring(source.read(path)) + lines = 0 + for row in sheet.findall("m:sheetData/m:row", XLSX_NS): + number = int(row.get("r")) + text = row.find("m:c[@t='s']/m:v", XLSX_NS) + line = text is not None and strings[int(text.text)] == label + lines += line + if number < 3: + continue + for cell in row.findall("m:c", XLSX_NS): + column = cell.get("r").rstrip("0123456789") + if len(column) == 1 and column >= "C": + cell.set("r", f"{chr(ord(column) + 1)}{number}") + value = cell.find("m:v", XLSX_NS) + if line and value is not None and cell.get("t") != "s": + value.text = repr(float(value.text) * scale) + if line: + cell = ET.SubElement( + row, f"{{{XLSX_MAIN}}}c", {"r": f"C{number}", "t": "inlineStr"} + ) + ET.SubElement( + ET.SubElement(cell, f"{{{XLSX_MAIN}}}is"), f"{{{XLSX_MAIN}}}t" + ).text = unit + assert lines == 1, (table, label, lines) + for entry in source.infolist(): + dest.writestr( + entry, + ET.tostring(sheet) if entry.filename == path else source.read(entry), + ) + return output.getvalue() + + +def test_real_tables_with_a_kg_cell_beside_a_line_are_refused(tmp_path): + """The helper writes other tobacco in kg beside its line in the real + October 2026 Table 4.11 and multiplies its values by 1,000. Reading them + in the total's thousand kg gives growth of 862.8112. The renderer refuses + it, and a run on the real March 2026 tables, changed the same way, stops + before it writes anything.""" + changed = put_unit_beside_real_line( + REAL_OCTOBER_2026_RECEIPTS, "4.11", "Other tobacco products", "kg" + ) + (line,) = [ + row + for row in read_workbook_sheets(changed)["4.11"] + if receipts_label(row.get("B")) == "other tobacco products" + ] + assert line["C"] == "kg" + assert line["D"] == repr( + float(OCTOBER_2026_CLEARANCES["other_tobacco_products"][0]) * 1000 + ) + with pytest.raises(ValueError, match="not read: 'kg' in column C"): + render( + get_yoy_growth_path().read_text(), + current=changed, + previous=REAL_RECEIPTS[2026], + **COMMITTED_RELEASES, + ) + yoy, gap_dir = write_tree(tmp_path) + before = {path: path.read_bytes() for path in [yoy, *gap_dir.iterdir()]} + with pytest.raises(ValueError, match="not read: 'kg' in column C"): + run_main( + tmp_path, + make_test_xlsx(), + "--yaml-path", + str(yoy), + receipts=put_unit_beside_real_line( + REAL_RECEIPTS[2026], "3.11", "Other tobacco products", "kg" + ), + previous_receipts=REAL_RECEIPTS[2025], + ) + assert {path: path.read_bytes() for path in before} == before + assert not list(yoy.parent.rglob("*.staged")) + # Shifted without the kg cell, the real table reads as before. + shifted = put_unit_beside_real_line( + REAL_OCTOBER_2026_RECEIPTS, "4.11", "Other tobacco products", "", scale=1 + ) + path = CLEARANCE_SPECS["other_tobacco_products"].path + assert ( + compute_receipts_growth( + shifted, REAL_RECEIPTS[2026], specs=CLEARANCE_SERIES_SPECS + )[path].values + == compute_receipts_growth( + REAL_OCTOBER_2026_RECEIPTS, + REAL_RECEIPTS[2026], + specs=CLEARANCE_SERIES_SPECS, + )[path].values + ) + + +REAL_OCTOBER_2026_SHEETS = read_workbook_sheets(REAL_OCTOBER_2026_RECEIPTS) + + +def edit_real_table(workbook, table, cells=(), scale=(), notes=()): + """Set each cell in ``cells`` to its text, number or, for None, nothing; + a tuple of texts gives one string with a formatting run for each, and an + element the inline string it holds. Add the cell and its row where the + sheet lacks them. Multiply the cells in ``scale`` by their factors, then + append ``notes`` in column B below the sheet's last row. Preserve every + other archive entry, including the shared strings.""" + cells, scale = dict(cells), dict(scale) + output = BytesIO() + with ZipFile(BytesIO(workbook)) as source, ZipFile(output, "w") as dest: + path = _sheet_paths(source)[table] + sheet = ET.fromstring(source.read(path)) + data = sheet.find("m:sheetData", XLSX_NS) + last = max(int(row.get("r")) for row in data) + for offset, note in enumerate(notes, start=1): + cells[f"B{last + offset}"] = note + + def cell(reference): + number = int(reference.lstrip("ABCDEFGHIJKLMNOPQRSTUVWXYZ")) + rows = {int(row.get("r")): row for row in data} + if number not in rows: + rows[number] = ET.Element(f"{{{XLSX_MAIN}}}row", {"r": str(number)}) + data.insert(sum(other < number for other in rows), rows[number]) + found = rows[number].find(f"m:c[@r='{reference}']", XLSX_NS) + if found is None: + found = ET.SubElement( + rows[number], f"{{{XLSX_MAIN}}}c", {"r": reference} + ) + return found + + for reference, factor in scale.items(): + target = cell(reference) + assert target.get("t") is None, reference + value = target.find("m:v", XLSX_NS) + value.text = repr(float(value.text) * factor) + for reference, value in cells.items(): + target = cell(reference) + for child in list(target): + target.remove(child) + target.attrib.pop("t", None) + if isinstance(value, str): + target.set("t", "inlineStr") + ET.SubElement( + ET.SubElement(target, f"{{{XLSX_MAIN}}}is"), f"{{{XLSX_MAIN}}}t" + ).text = value + elif isinstance(value, tuple): + target.set("t", "inlineStr") + target.append(formatting_runs("is", value)) + elif isinstance(value, ET.Element): + target.set("t", "inlineStr") + target.append(value) + elif value is not None: + ET.SubElement(target, f"{{{XLSX_MAIN}}}v").text = repr(float(value)) + for entry in source.infolist(): + dest.writestr( + entry, + ET.tostring(sheet) if entry.filename == path else source.read(entry), + ) + return output.getvalue() + + +def formatting_runs(tag, texts, reading=None): + """Build a ``tag`` string element, ``is`` or ``si``, with a formatting + run for each of ``texts``, the later runs bold, and with ``reading``, a + phonetic run (````) that holds it.""" + string = ET.Element(f"{{{XLSX_MAIN}}}{tag}") + for index, part in enumerate(texts): + run = ET.SubElement(string, f"{{{XLSX_MAIN}}}r") + if index: + ET.SubElement( + ET.SubElement(run, f"{{{XLSX_MAIN}}}rPr"), f"{{{XLSX_MAIN}}}b" + ) + ET.SubElement( + run, f"{{{XLSX_MAIN}}}t", {f"{{{XML_NAMESPACE}}}space": "preserve"} + ).text = part + if reading is not None: + string.append(phonetic_run(reading, len("".join(texts)))) + return string + + +def phonetic_run(reading, length): + """Build a phonetic run (````) that holds ``reading`` over the + first ``length`` characters of its string.""" + run = ET.Element(f"{{{XLSX_MAIN}}}rPh", {"sb": "0", "eb": str(length)}) + ET.SubElement(run, f"{{{XLSX_MAIN}}}t").text = reading + return run + + +def real_levels(workbook, key): + """Read the levels of ``key`` from its table alone in ``workbook``, and + from the real October 2026 table.""" + spec = CLEARANCE_SPECS[key] + table = "4.11" if spec.title == TOBACCO_CLEARANCES else "4.12" + return ( + find_receipts_line({table: read_sheet_rows(workbook, table)}, spec).levels, + find_receipts_line(REAL_OCTOBER_2026_SHEETS, spec).levels, + ) + + +# The importer interprets no words in a clearance table: each cell must hold a +# number, a fiscal year or a text someone has read (CLEARANCE_TABLE_TEXTS). + +TOBACCO_LINES = ("cigarettes", "hand_rolling_tobacco", "other_tobacco_products") +ALCOHOL_LINES = ("beer_and_cider", "wine", "spirits") +REAL_CLEARANCE_TABLES = { + (release, title): (workbook, f"{chapter}.{number}") + for release, workbook, chapter in ( + ("March 2025", REAL_RECEIPTS[2025], 3), + ("March 2026", REAL_RECEIPTS[2026], 3), + ("October 2026", REAL_OCTOBER_2026_RECEIPTS, 4), + ) + for title, number in ((TOBACCO_CLEARANCES, 11), (ALCOHOL_CLEARANCES, 12)) +} + + +@lru_cache(maxsize=None) +def real_clearance_rows(release, title): + """Read a real clearance table's sheet and find the table: its title's + row, the row of the next table's title or the sheet's end, and that + title in lower case without its number.""" + workbook, sheet = REAL_CLEARANCE_TABLES[release, title] + rows = read_sheet_rows(workbook, sheet) + titles = [" ".join(str(row.get("B") or "").split()) for row in rows] + (start,) = [ + i for i, text in enumerate(titles) if text.lower() == f"{sheet} {title}" + ] + for end in range(start + 1, len(rows)): + if re.match(r"\d+\.\d+ ", titles[end]): + return rows, start, end, titles[end].split(" ", 1)[1].lower() + return rows, start, len(rows), None + + +@pytest.fixture(scope="module", autouse=True) +def release_real_clearance_rows(): + # Drop the cached sheets once this module's tests are done, instead of + # keeping them for the rest of the test process. + yield + real_clearance_rows.cache_clear() + + +def unread(table, title, *cells): + """Write out the importer's refusal of real Table ``table`` for the + unread ``cells``, each as (text, column).""" + return ( + f"Table {table} (sheet {table}) holds text the importer has not read: " + + "; ".join(f"{text!r} in column {column}" for text, column in cells) + + ". Read each one. If it states another unit, convert the table's " + "figures to the units the read texts state. Then add the text, as " + f'receipts_label gives it, to CLEARANCE_TABLE_TEXTS["{title}"], or the ' + "title of a table that follows the clearances to CLEARANCE_TABLE_ENDS." + ) + + +def test_both_real_release_pairs_read_every_series(): + """The importer reads all 17 series, the six clearance lines among + them, from each real pair: March 2026 over March 2025 and October 2026 + over March 2026.""" + for current, previous in ( + (REAL_RECEIPTS[2026], REAL_RECEIPTS[2025]), + (REAL_OCTOBER_2026_RECEIPTS, REAL_RECEIPTS[2026]), + ): + growth = compute_receipts_growth(current, previous) + assert len(growth) == len(RECEIPTS_SERIES_SPECS) == 17 + assert set(CLEARANCE_PATHS.values()) <= set(growth) + + +def test_the_read_texts_are_the_texts_of_the_three_releases(): + """Each clearance table's list holds the text of every cell in the three + releases' tables, below the title and above the next table's title, but + numbers and fiscal years, and nothing else. The list of ends holds the + titles that follow them: the alcohol receipts forecasts, on Table 4.12's + sheet (3.12's in March), and nothing on Table 4.11's.""" + texts = {title: set() for title in CLEARANCE_TABLE_TEXTS} + ends = {title: set() for title in CLEARANCE_TABLE_TEXTS} + for release, title in REAL_CLEARANCE_TABLES: + rows, start, end, following = real_clearance_rows(release, title) + assert [value for value in rows[start].values() if value][1:] == [] + if following is not None: + ends[title].add(following) + for row in rows[start + 1 : end]: + for value in row.values(): + cell = " ".join(str(value or "").split()) + if not cell or re.fullmatch(r"\d{4}-\d{2}", cell): + continue + try: + float(cell) + except ValueError: + texts[title].add(receipts_label(cell)) + assert texts == { + title: set(listed) for title, listed in CLEARANCE_TABLE_TEXTS.items() + } + assert ends == { + title: set(listed) for title, listed in CLEARANCE_TABLE_ENDS.items() + } + assert ends[ALCOHOL_CLEARANCES] == {"alcohol receipts forecasts"} + + +# Each edit to the real October 2026 Table 4.11 or 4.12 below puts text no +# one has read in the table. Table 4.11 runs from row 2 to the sheet's end, +# so a note below the sheet's last row falls in it; Table 4.12's clearances +# run from row 2 to row 11, above the receipts forecasts' title. +OTHER_TOBACCO_NOTE = "Note: Other tobacco products {}" +FULL_NOTE = ( + "Note: Hand rolled tobacco clearances are in ('000s kg); other tobacco " + "products are in kg." +) +UNREAD_EDITS = { + # The OBR's cigarette forecasts in ten million sticks, and other tobacco + # in kilotonnes, with the figures to match. + "spelled-scale-real-release": ( + "4.11", + {}, + {f"{column}5": 0.1 for column in "DEFGHI"}, + ["Note: Cigarette forecasts shown in ten million sticks."], + ), + "new-unit-real-release": ( + "4.11", + {}, + {f"{column}9": 0.001 for column in "CDEFGHI"}, + [OTHER_TOBACCO_NOTE.format("shown in kilotonnes.")], + ), + **{ + f"unknown-note-{ending}": ("4.11", {}, {}, [OTHER_TOBACCO_NOTE.format(ending)]) + for ending in ( + "are in milligrams.", + "are in (milligrams).", + "shown in milligrams.", + "(milligrams).", + "shown in kilotonnes.", + "shown in tens of thousands of kilograms.", + ) + }, + "unknown-label-brackets": ( + "4.11", + {"B6": "Non-cigarette clearances (milligrams)"}, + {}, + [], + ), + **{ + f"product-negation-{product}": ( + "4.11", + {}, + {}, + [f"Note: {product} are in million sticks."], + ) + for product in ("Non-cigarette products", "Non-cigarettes") + }, + **{ + f"product-only-note-{product}": ( + "4.12", + { + "B8": f"Note: {product} clearances are in thousands of " + "hectolitres of alcohol." + }, + {}, + [], + ) + for product in ("Wine", "Sake", "swine") + }, + "full-note": ( + "4.11", + {"B6": "Non-cigarette clearances (kg)"}, + {f"{column}{row}": 1000 for column in "CDEFGHI" for row in (6, 9)}, + [FULL_NOTE], + ), + "competing-cigarettes": ( + "4.11", + {}, + {}, + [ + "Note: Outturn clearances use the unit in the label; forecasts are " + "in thousand sticks." + ], + ), + "competing-other-tobacco": ( + "4.11", + {}, + {}, + [OTHER_TOBACCO_NOTE.format("are in kg.")], + ), + # The listed units spelt another way. + "same-unit-thousand-kg": ( + "4.11", + {"B6": "Non-cigarette clearances (thousand kg)"}, + {}, + [], + ), + "same-unit-000s-hectolitres": ( + "4.12", + {"B8": "Note: All clearances are in '000s hectolitres of alcohol."}, + {}, + [], + ), + # A listed text with more words. + "extra-words-label": ( + "4.11", + {"B6": "Non-cigarette clearances ('000s kg) in tens"}, + {}, + [], + ), + "extra-words-note": ( + "4.12", + { + "B8": "Note: All clearances are in thousands of hectolitres of " + "alcohol. Spirits are in litres." + }, + {}, + [], + ), + # Units outside column B: in a new row, the "of which:" row, beside a + # line after its years, and in the title, marker and year rows. + "non-first-column-note": ( + "4.11", + {"E40": "Note: Cigarette forecasts shown in ten million sticks."}, + {}, + [], + ), + "non-first-column-of-which": ("4.11", {"K7": "kg"}, {}, []), + "after-the-years": ("4.11", {"J5": "ten million sticks"}, {}, []), + "title-row": ("4.11", {"D2": "Cigarettes in ten million sticks"}, {}, []), + "marker-row": ("4.11", {"K3": "Million sticks"}, {}, []), + "year-row": ("4.11", {"J4": "Unit"}, {}, []), + "alcohol-last-row": ( + "4.12", + {"B11": "Note: Spirits clearances are in litres."}, + {}, + [], + ), + # New lines: a product neither list names, and wine in Table 4.11. + "new-line-heated-tobacco": ( + "4.11", + {"B10": "Heated tobacco clearances", **{f"{c}10": 50 for c in "CDEFGHI"}}, + {}, + [], + ), + "new-line-sake": ( + "4.12", + {"B11": "Sake", **{f"{c}11": 10 for c in "CDEFGHI"}}, + {}, + [], + ), + "other-table-text": ("4.11", {"B10": "Wine"}, {}, []), + # Text where a number belongs. + "not-a-number": ("4.11", {"D5": "nan"}, {}, []), + # A note shaped like a table's title, which sheet_tables reads as one. + "title-shaped-note": ( + "4.11", + {}, + {}, + ["1.1 Note: Cigarette forecasts shown in ten million sticks."], + ), +} + + +@pytest.mark.parametrize("name", sorted(UNREAD_EDITS)) +def test_text_no_one_has_read_stops_the_real_tables(name): + """For each edit, the importer refuses every line of the edited table, + naming each unread cell, and the run stops before it renders anything; + the importer reads the other table's lines as before.""" + table, cells, scale, notes = UNREAD_EDITS[name] + changed = edit_real_table( + REAL_OCTOBER_2026_RECEIPTS, table, cells=cells, scale=scale, notes=notes + ) + title = TOBACCO_CLEARANCES if table == "4.11" else ALCOHOL_CLEARANCES + added = [(text, reference[0]) for reference, text in cells.items()] + expected = unread( + table, + title, + *[(text, column) for text, column in added if isinstance(text, str)], + *[(note, "B") for note in notes], + ) + sheets = {sheet: read_sheet_rows(changed, sheet) for sheet in ("4.11", "4.12")} + for key, spec in CLEARANCE_SPECS.items(): + if spec.title != title: + assert ( + find_receipts_line(sheets, spec).levels + == find_receipts_line(REAL_OCTOBER_2026_SHEETS, spec).levels + ), key + continue + with pytest.raises(ValueError) as refused: + find_receipts_line(sheets, spec) + assert str(refused.value) == expected, key + with pytest.raises(ValueError) as refused: + compute_receipts_growth( + changed, REAL_RECEIPTS[2026], specs=CLEARANCE_SERIES_SPECS + ) + assert str(refused.value) == expected + + +def test_the_scale_edits_change_the_growth_the_figures_give(): + """In the units the real tables state, the two rescaled tables' figures + give cigarette clearances growth of -91.0% in 2026-27, where the + published figures give -10.3%, and other tobacco growth of -99.9% in + 2025-26, where they give -13.6%.""" + + def growth(levels, key): + # 2025-26 over March 2026's 2024-25 outturn, then 2026-27 over 2025-26. + before = float(MARCH_2026_CLEARANCES[key][0]) + return [round(levels[0] / before - 1, 6), round(levels[1] / levels[0] - 1, 6)] + + for name, key, year, scaled, real in ( + ("spelled-scale-real-release", "cigarettes", 1, -0.910322, -0.103219), + ("new-unit-real-release", "other_tobacco_products", 0, -0.999136, -0.136189), + ): + table, _, scale, _ = UNREAD_EDITS[name] + changed = edit_real_table(REAL_OCTOBER_2026_RECEIPTS, table, scale=scale) + (line,) = [ + row + for row in read_sheet_rows(changed, table) + if receipts_label(row.get("B")) == CLEARANCE_SPECS[key].row + ] + levels = [float(line[column]) for column in "CD"] + published = [float(level) for level in OCTOBER_2026_CLEARANCES[key][:2]] + assert growth(levels, key)[year] == scaled, name + assert growth(published, key)[year] == real, name + + +def test_a_real_table_without_its_unit_cell_refuses_the_lines_it_describes(): + """The helper clears the non-cigarette total's label and, separately, + Table 4.12's note. Each leaves its lines, hand-rolled and other tobacco + or the three alcohol lines, without the cell that states their unit, + and the importer refuses them; it reads cigarettes as before.""" + for table, cell in (("4.11", "B6"), ("4.12", "B8")): + changed = edit_real_table(REAL_OCTOBER_2026_RECEIPTS, table, cells={cell: None}) + for key, spec in CLEARANCE_SPECS.items(): + if spec.title != ( + TOBACCO_CLEARANCES if table == "4.11" else ALCOHOL_CLEARANCES + ): + continue + if key == "cigarettes": + read, real = real_levels(changed, key) + assert read == real + continue + with pytest.raises(ValueError) as refused: + real_levels(changed, key) + assert str(refused.value) == ( + f"Table {table} (sheet {table}) lacks {spec.unit_cell!r}, the cell " + f"in which the OBR states the unit of {'.'.join(spec.path)}" + ) + + +def in_year_row(table, heading, key): + """Write out the importer's refusal of the ``key`` line of real Table + ``table`` with ``heading`` in the fiscal-year row over the line.""" + return ( + f"Table {table} (sheet {table}) holds {heading} in the fiscal-year row " + f"over the {'.'.join(CLEARANCE_SPECS[key].path)} line. A text beside " + "the years could give the line another unit, so the importer refuses " + "any text there." + ) + + +def not_under_total(table, key): + """Write out the importer's refusal of the ``key`` line of real Table + ``table``, a part that does not sit under its total.""" + spec = CLEARANCE_SPECS[key] + return ( + f"Table {table} (sheet {table}) lacks the layout that gives the " + f'{".".join(spec.path)} line its unit: an "of which:" row with no other ' + "cell directly above the line and the total's other parts, and directly " + "above that row the non-cigarette total's row, which holds its label " + f"({spec.unit_cell!r}, as receipts_label gives it), its figures and " + "nothing else. The line takes its unit from that total's label, so the " + "importer refuses any other layout." + ) + + +# A table of listed texts can still put a unit over the wrong lines. Each +# layout maps rows of the real October 2026 Table 4.11, columns B to I, to +# the real rows they take: "heading" for the cigarette label beside row 4's +# fiscal years, None for nothing. The helper halves the hand-rolled and +# other tobacco figures (real rows 8 and 9) wherever they land, so the two +# parts sit under the cigarette label's million sticks. +REAL_TOBACCO_ROWS = {row.number: row for row in REAL_OCTOBER_2026_SHEETS["4.11"]} +TOBACCO_LAYOUTS = { + # The cigarette label heads a second row of years over the parts, in + # place of "of which:", and the non-cigarette total moves below them. + "cigarette-unit-year-header": ({6: None, 7: "heading", 10: 6}, "year row"), + # The cigarette line moves directly above "of which:", and the + # non-cigarette total below the parts. + "cigarette-unit-of-which": ({5: None, 6: 5, 10: 6}, "total"), + # A second row of years, which the cigarette label heads, sits over the + # non-cigarette total, "of which:" and the parts, each a row lower. + "cigarette-unit-second-year-row": ( + {6: "heading", 7: 6, 8: 7, 9: 8, 10: 9}, + "year row", + ), +} + + +def tobacco_layout(name): + """Rewrite the real October 2026 Table 4.11 in the layout ``name``.""" + layout, _ = TOBACCO_LAYOUTS[name] + cells = {} + for target, source in layout.items(): + for column in "BCDEFGHI": + if source == "heading": + value = REAL_TOBACCO_ROWS[5 if column == "B" else 4][column] + elif source is None or REAL_TOBACCO_ROWS[source].get(column) is None: + value = None + elif column == "B": + value = REAL_TOBACCO_ROWS[source][column] + else: + value = float(REAL_TOBACCO_ROWS[source][column]) + value /= 2 if source in (8, 9) else 1 + cells[f"{column}{target}"] = value + halved = { + f"{column}{row}": 0.5 + for row in (8, 9) + if row not in layout + for column in "CDEFGHI" + } + return edit_real_table( + REAL_OCTOBER_2026_RECEIPTS, "4.11", cells=cells, scale=halved + ) + + +@pytest.mark.parametrize("name", sorted(TOBACCO_LAYOUTS)) +def test_a_layout_that_puts_the_parts_under_another_unit_stops_the_run(name): + """Each layout keeps every text on the list and the non-cigarette + total's label in the table, but sets the hand-rolled and other tobacco + lines under the cigarette label's million sticks. In thousand kg, the + unit March 2026 states for them, the halved figures give 2025-26 growth + of -58.6% for hand-rolled tobacco and -56.8% for other tobacco, and the + published figures -17.1% and -13.6%. The importer reads cigarettes as + before and refuses both parts. It names the fiscal-year row when that + row holds the label, and the missing total otherwise. The run stops + before it renders anything.""" + _, check = TOBACCO_LAYOUTS[name] + changed = tobacco_layout(name) + rows = read_sheet_rows(changed, "4.11") + texts = { + receipts_label(value) + for row in rows[2:] + for value in row.values() + if receipts_label(value) + and not re.fullmatch(r"\d{4}-\d{2}|[\d.]+", receipts_label(value)) + } + assert texts <= CLEARANCE_TABLE_TEXTS[TOBACCO_CLEARANCES] + assert CLEARANCE_SPECS["hand_rolling_tobacco"].unit_cell in texts + for key, read, real in ( + ("hand_rolling_tobacco", -0.585697, -0.171394), + ("other_tobacco_products", -0.568094, -0.136189), + ): + (line,) = [ + row + for row in rows + if receipts_label(row.get("B")) == CLEARANCE_SPECS[key].row + ] + before = float(MARCH_2026_CLEARANCES[key][0]) + assert round(float(line["C"]) / before - 1, 6) == read, key + published = float(OCTOBER_2026_CLEARANCES[key][0]) + assert round(published / before - 1, 6) == real, key + sheets = {"4.11": rows, "4.12": REAL_OCTOBER_2026_SHEETS["4.12"]} + expected = {} + for key in TOBACCO_LINES: + if key == "cigarettes": + assert ( + find_receipts_line(sheets, CLEARANCE_SPECS[key]).levels + == find_receipts_line( + REAL_OCTOBER_2026_SHEETS, CLEARANCE_SPECS[key] + ).levels + ) + continue + expected[key] = ( + in_year_row("4.11", {"B": REAL_TOBACCO_ROWS[5]["B"]}, key) + if check == "year row" + else not_under_total("4.11", key) + ) + with pytest.raises(ValueError) as refused: + find_receipts_line(sheets, CLEARANCE_SPECS[key]) + assert str(refused.value) == expected[key], key + with pytest.raises(ValueError) as refused: + render( + get_yoy_growth_path().read_text(), + current=changed, + previous=REAL_RECEIPTS[2026], + **COMMITTED_RELEASES, + ) + assert str(refused.value) == expected["hand_rolling_tobacco"] + + +# Ways to change the rows over the parts of a real tobacco clearances table, +# each given the rows and the place of the "of which:" row. In all three +# releases, column N of the total's row and column C of the "of which:" row +# are empty, so each layout adds a text and overwrites none. +TOTAL_LAYOUTS = { + # A row that holds the total's label and none of its figures. + "label-without-figures": lambda rows, of_which: rows.insert( + of_which, {"B": "Non-cigarette clearances ('000s kg)"} + ), + # The total's row holds another text after its figures. + "text-on-total-row": lambda rows, of_which: rows[of_which - 1].update( + {"N": "Cigarette clearances (million sticks)"} + ), + # The "of which:" row holds another cell. + "text-on-of-which-row": lambda rows, of_which: rows[of_which].update( + {"C": "Cigarette clearances (million sticks)"} + ), + "empty-row-above-of-which": lambda rows, of_which: rows.insert(of_which, {}), + "empty-row-below-of-which": lambda rows, of_which: rows.insert(of_which + 1, {}), +} + + +@pytest.mark.parametrize("layout", sorted(TOTAL_LAYOUTS)) +@pytest.mark.parametrize("release", ["March 2025", "March 2026", "October 2026"]) +def test_a_part_needs_its_total_directly_above_of_which(release, layout): + """Change the rows over the parts of the real tobacco clearances in one + way. The importer refuses hand-rolled and other tobacco with the part + check's message and reads cigarettes as before.""" + _, sheet = REAL_CLEARANCE_TABLES[release, TOBACCO_CLEARANCES] + real, start, end, _ = real_clearance_rows(release, TOBACCO_CLEARANCES) + (of_which,) = [ + i for i in range(start, end) if receipts_label(real[i].get("B")) == "of which:" + ] + rows = [dict(row) for row in real] + TOTAL_LAYOUTS[layout](rows, of_which) + for key in TOBACCO_LINES: + spec = CLEARANCE_SPECS[key] + if key == "cigarettes": + assert ( + find_receipts_line({sheet: rows}, spec).levels + == find_receipts_line({sheet: real}, spec).levels + ) + continue + with pytest.raises(ValueError) as refused: + find_receipts_line({sheet: rows}, spec) + assert str(refused.value) == not_under_total(sheet, key), key + + +def test_the_alcohol_receipts_table_stays_outside_the_clearance_table(): + """On the real October 2026 sheet 4.12 the OBR follows the alcohol + clearances with the alcohol receipts forecasts, which hold "£ billion" + and "Note: All taxes are on an accrued basis.". The importer ends the + clearance table at their title. It reads the clearance lines with a note + or "£ million" in the receipts table, and refuses them with the same + note in the clearance table's last row, or under a receipts title + CLEARANCE_TABLE_ENDS does not name, since the clearance table then runs + to the sheet's end.""" + rows, start, end, following = real_clearance_rows( + "October 2026", ALCOHOL_CLEARANCES + ) + assert (start, end, following) == (1, 11, "alcohol receipts forecasts") + assert rows[12]["C"] == "£ billion" + assert rows[18]["B"] == "Note: All taxes are on an accrued basis." + note = "Note: Spirits clearances are in litres." + for cells in ({"B22": note}, {"J13": note}, {"C13": "£ million"}): + changed = edit_real_table(REAL_OCTOBER_2026_RECEIPTS, "4.12", cells=cells) + for key in ALCOHOL_LINES: + read, real = real_levels(changed, key) + assert read == real, (cells, key) + changed = edit_real_table(REAL_OCTOBER_2026_RECEIPTS, "4.12", cells={"B11": note}) + with pytest.raises(ValueError) as refused: + real_levels(changed, "wine") + assert str(refused.value) == unread("4.12", ALCOHOL_CLEARANCES, (note, "B")) + retitled = "4.12 Alcohol receipts forecasts (accrued)" + changed = edit_real_table( + REAL_OCTOBER_2026_RECEIPTS, "4.12", cells={"B12": retitled} + ) + with pytest.raises(ValueError) as refused: + real_levels(changed, "wine") + assert str(refused.value) == unread( + "4.12", + ALCOHOL_CLEARANCES, + (retitled, "B"), + ("£ billion", "C"), + ("Note: All taxes are on an accrued basis.", "B"), + ) + + +def unread_text(text, title): + """Decide from the lists, without the importer, whether it must refuse + ``text`` in a clearance table titled ``title``. The test leaves out + blanks, numbers, fiscal years and table titles a list names.""" + label = receipts_label(text) + if not label or label in CLEARANCE_TABLE_TEXTS[title]: + return False + if re.fullmatch(r"\d{4}-\d{2}", label): + return False + try: + float(label) + return False + except ValueError: + pass + heading = re.match(r"(\d+\.\d+)\s+(.+)$", " ".join(text.split())) + return not ( + heading + and receipts_label(heading.group(2)) + in {*CLEARANCE_TABLE_ENDS[title], TOBACCO_CLEARANCES, ALCOHOL_CLEARANCES} + ) + + +UNIT_SHAPED = st.one_of( + st.sampled_from( + [ + "kg", + "'000s kg", + "(kg)", + "(tonnes)", + "Unit", + "£ billion", + "million sticks", + "(million sticks)", + "thousands of hectolitres of alcohol", + "Of which: cigars", + "Cigarette clearances (billion sticks)", + "Non-cigarette clearances (thousand kg)", + "Heated tobacco clearances", + "Note: Non-cigarettes are in million sticks.", + "Note: Sake clearances are in thousands of hectolitres of alcohol.", + "Note: All clearances are in millions of litres of alcohol.", + # Shaped like a table's title, so sheet_tables starts a table there. + "1.1 Note: Cigarette forecasts shown in ten million sticks.", + "4.13 Tobacco clearances in ten million sticks", + ] + ), + st.builds( + "{}{}{}{}".format, + st.sampled_from( + [ + "", + "(", + "Note: ", + "Note: All clearances are in ", + "Note: Cigarette forecasts shown in ", + "Note: Other tobacco products are in ", + ] + ), + st.sampled_from(["", "ten ", "tens of ", "thousand ", "'000s ", "million "]), + st.sampled_from( + [ + "kg", + "sticks", + "million sticks", + "kilotonnes", + "tonnes", + "litres", + "hectolitres of alcohol", + "litres of product", + ] + ), + st.sampled_from(["", ".", ")"]), + ), +) + + +@settings(max_examples=200, deadline=None, derandomize=True) +@given(table=st.sampled_from(sorted(REAL_CLEARANCE_TABLES)), data=st.data()) +def test_any_text_no_one_has_read_stops_a_real_clearance_table(table, data): + """Put one text in a real table, in an empty cell of any row from its + title through the table's end, in any column, or on a new row. Any text + outside the table's list, from st.text() or shaped like a unit, stops + every line of the table: the importer names the cell, or finds no line + or table where the text took a label's or the title's place. A text from + the list, other than a line's label or a marker, in a cell outside the + lines, stops every line when it sits in the fiscal-year row, and a part + of a total when it sits between the part and the total's row; the + importer names the year row or the missing total. Anywhere else it + changes nothing: the importer reads every line as before.""" + release, title = table + _, sheet = REAL_CLEARANCE_TABLES[table] + real, start, end, _ = real_clearance_rows(release, title) + rows = [dict(row) for row in real] + keys = [key for key, spec in CLEARANCE_SPECS.items() if spec.title == title] + listed = data.draw(st.booleans(), label="listed") + if listed: + labels = {CLEARANCE_SPECS[key].row for key in keys} | {"outturn", "forecast"} + text = data.draw( + st.sampled_from(sorted(CLEARANCE_TABLE_TEXTS[title] - labels)), label="text" + ) + else: + text = data.draw(st.one_of(st.text(min_size=1, max_size=40), UNIT_SHAPED)) + assume(unread_text(text, title)) + year_row = next( + i + for i in range(start, end) + if any(re.fullmatch(r"\d{4}-\d{2}", str(v or "")) for v in rows[i].values()) + ) + line_rows = { + i + for i in range(year_row + 1, end) + if any(re.fullmatch(r"[\d.]+", str(v or "")) for v in rows[i].values()) + } + column = data.draw(st.sampled_from("ABCDEFGHIJKLM"), label="column") + if data.draw(st.booleans(), label="new row"): + lowest = year_row + 1 if listed else start + 1 + place = data.draw(st.integers(lowest, end), label="row") + rows.insert(place, {column: text}) + else: + empty = [ + i for i in range(start, end) if not str(rows[i].get(column) or "").strip() + ] + if listed: + empty = [ + i for i in empty if i not in line_rows and (i != start or column > "B") + ] + assume(empty) + place = data.draw(st.sampled_from(empty), label="row") + rows[place][column] = text + # The real rows of each line and of the total that states a part's unit. + # A new row at ``place`` sits above the real row there. + real_row = { + receipts_label(real[i].get("B")): i + for i in range(start, end) + if real[i].get("B") + } + for key in keys: + spec = CLEARANCE_SPECS[key] + if listed: + if place == year_row: + event("listed: fiscal-year row") + expected = in_year_row(sheet, {column: text}, key) + elif ( + spec.part_of_total + and real_row[spec.unit_cell] < place <= real_row[spec.row] + ): + event("listed: between a part and its total") + expected = not_under_total(sheet, key) + else: + event("listed: read") + assert ( + find_receipts_line({sheet: rows}, spec).levels + == find_receipts_line({sheet: real}, spec).levels + ) + continue + with pytest.raises(ValueError) as refused: + find_receipts_line({sheet: rows}, spec) + assert str(refused.value) == expected + continue + with pytest.raises(ValueError) as refused: + find_receipts_line({sheet: rows}, spec) + if str(refused.value).startswith(f'Expected one "{spec.row}" row'): + event("unread: no line or table") + else: + event("unread: named") + assert str(refused.value) == unread(sheet, title, (text, column)) + + +# A spreadsheet can split a cell's text into formatting runs. The check must +# see the whole text: each first run below passes alone, as a listed text, a +# number or a fiscal year. +TEN_MILLION_STICKS = "Note: Cigarette forecasts shown in ten million sticks." +FIRST_RUNS = { + "rich-listed-prefix": "of which:", + "rich-numeric-prefix": "1", + "rich-fiscal-year-prefix": "2026-27", +} + + +@pytest.mark.parametrize("name", sorted(FIRST_RUNS)) +def test_the_check_reads_every_formatting_run_of_a_cell(name): + """The helper divides the real October 2026 cigarette forecasts by ten + and writes B10 of Table 4.11 as one inline string in two runs: the + first ("of which:", "1" or "2026-27"), then " Note: Cigarette forecasts + shown in ten million sticks." The reader returns both runs, and the + importer refuses every tobacco line, naming the whole cell, and stops + before it renders anything.""" + runs = (FIRST_RUNS[name], " " + TEN_MILLION_STICKS) + changed = edit_real_table( + REAL_OCTOBER_2026_RECEIPTS, + "4.11", + cells={"B10": runs}, + scale={f"{column}5": 0.1 for column in "DEFGHI"}, + ) + with ZipFile(BytesIO(changed)) as archive: + sheet = ET.fromstring(archive.read(_sheet_paths(archive)["4.11"])) + cell = sheet.find("m:sheetData/m:row/m:c[@r='B10']", XLSX_NS) + assert [ + run.findtext("m:t", namespaces=XLSX_NS) + for run in cell.findall("m:is/m:r", XLSX_NS) + ] == list(runs) + rows = read_sheet_rows(changed, "4.11") + (note,) = [row["B"] for row in rows if row.number == 10] + assert note == "".join(runs) + expected = unread("4.11", TOBACCO_CLEARANCES, (note, "B")) + for key in TOBACCO_LINES: + with pytest.raises(ValueError) as refused: + find_receipts_line({"4.11": rows}, CLEARANCE_SPECS[key]) + assert str(refused.value) == expected, key + with pytest.raises(ValueError) as refused: + render( + get_yoy_growth_path().read_text(), + current=changed, + previous=REAL_RECEIPTS[2026], + **COMMITTED_RELEASES, + ) + assert str(refused.value) == expected + + +def runs_workbook(texts, reading=None): + """Build a one-sheet workbook that holds ``texts`` in A1 as an inline + string and in A2 as a shared string, each with a formatting run for each + text, and their join in A3 as an inline string of one ````. With + ``reading``, each string ends in a phonetic run that holds it.""" + sheet = make_sheet( + { + 1: [ + '' + + ET.tostring(formatting_runs("is", texts, reading), encoding="unicode") + + "" + ], + 2: ['0'], + 3: [make_inline_cell("A3", "")], + } + ) + sheet = ET.fromstring(sheet) + joined = sheet.find("m:sheetData/m:row/m:c[@r='A3']/m:is/m:t", XLSX_NS) + joined.text = "".join(texts) + if reading is not None: + sheet.find("m:sheetData/m:row/m:c[@r='A3']/m:is", XLSX_NS).append( + phonetic_run(reading, len(joined.text)) + ) + strings = ET.Element(f"{{{XLSX_MAIN}}}sst") + strings.append(formatting_runs("si", texts, reading)) + output = BytesIO() + with ZipFile(BytesIO(make_workbook({"1": ET.tostring(sheet)}))) as source: + with ZipFile(output, "w") as dest: + for entry in source.infolist(): + dest.writestr(entry, source.read(entry)) + dest.writestr("xl/sharedStrings.xml", ET.tostring(strings)) + return output.getvalue() + + +STRING_TEXT = st.text(st.characters(min_codepoint=0x20, max_codepoint=0xD7FF)) + + +@settings(max_examples=100, deadline=None, derandomize=True) +@given( + texts=st.lists(STRING_TEXT, min_size=1, max_size=4), + reading=st.one_of(st.none(), STRING_TEXT), +) +def test_a_string_reads_whole_in_any_formatting_runs(texts, reading): + """However a spreadsheet splits a text into formatting runs, the reader + returns the whole text, from an inline string and from a shared string + alike, as it does from one ````. It leaves out the text of any + phonetic run (````) the string holds.""" + rows = read_sheet_rows(runs_workbook(texts, reading), "1") + assert [(row.number, row["A"]) for row in rows] == [ + (number, "".join(texts)) for number in (1, 2, 3) + ] + + +TEN_THOUSAND_HECTOLITRES = ( + "Note: Spirits forecasts shown in ten thousand hectolitres of alcohol." +) + + +def test_a_phonetic_run_cannot_end_a_clearance_table(): + """The helper writes B11 of the real October 2026 sheet 4.12, empty in + the published sheet, as an inline string with no text of its own and a + phonetic run that holds "4.12 Alcohol receipts forecasts". It puts a + note in ten thousand hectolitres in C11 and divides the spirits + forecasts by ten. A reader that kept the phonetic run would end the + clearance table at B11, above the note, and the importer would read + 2026-27 spirits growth of -90.1% in place of -0.8%. This reader returns + B11 empty, so the table runs to the real title in row 12, and the + importer refuses every alcohol line and names the note.""" + changed = edit_real_table( + REAL_OCTOBER_2026_RECEIPTS, + "4.12", + cells={ + "B11": formatting_runs("is", ("",), "4.12 Alcohol receipts forecasts"), + "C11": TEN_THOUSAND_HECTOLITRES, + }, + scale={f"{column}7": 0.1 for column in "DEFGHI"}, + ) + rows = read_sheet_rows(changed, "4.12") + (row,) = [row for row in rows if row.number == 11] + assert (row["B"], row["C"]) == ("", TEN_THOUSAND_HECTOLITRES) + (spirits,) = [line for line in rows if line.number == 7] + assert round(float(spirits["D"]) / float(spirits["C"]) - 1, 6) == -0.900793 + expected = unread("4.12", ALCOHOL_CLEARANCES, (TEN_THOUSAND_HECTOLITRES, "C")) + for key in ALCOHOL_LINES: + with pytest.raises(ValueError) as refused: + find_receipts_line({"4.12": rows}, CLEARANCE_SPECS[key]) + assert str(refused.value) == expected, key + with pytest.raises(ValueError) as refused: + compute_receipts_growth( + changed, REAL_RECEIPTS[2026], specs=CLEARANCE_SERIES_SPECS + ) + assert str(refused.value) == expected + + +# A sheet that repeats a clearance table's heading must stop the run: the +# importer would check the text under one and could read a line under the +# other. +def repeat_alcohol_heading(workbook, sheet): + """Clear the lines of the real alcohol clearance table on ``sheet``, rows + 5 to 7, and repeat its rows 2 to 10 below the sheet's last row, so a + second copy of its heading follows the alcohol receipts forecasts. + Divide the repeated spirits forecasts by ten and put a note in ten + thousand hectolitres below the repeated rows. Return the workbook and + the second heading's row.""" + rows = read_sheet_rows(workbook, sheet) + shift = rows[-1].number + 1 + cells = {f"{column}{number}": None for column in "BCDEFGHI" for number in (5, 6, 7)} + for row in rows: + if not 2 <= row.number <= 10: + continue + for column, value in row.items(): + if not str(value or "").strip(): + continue + try: + value = float(value) + except ValueError: + pass + else: + if row.number == 7 and column > "C": + value /= 10 + cells[f"{column}{row.number + shift}"] = value + cells[f"B{11 + shift}"] = TEN_THOUSAND_HECTOLITRES + return edit_real_table(workbook, sheet, cells=cells), 2 + shift + + +def repeated(table, row): + """Write out the importer's refusal of real Table ``table`` with its + heading in rows 2 and ``row``.""" + return ( + f"Table {table} (sheet {table}) repeats its heading, in rows 2 and " + f"{row}. The importer would check the text under one and could read a " + "line under another, so keep one heading." + ) + + +def test_a_repeated_clearance_heading_stops_the_run(tmp_path): + """The helper clears the lines of the real October 2026 Table 4.12 and + repeats the table below the alcohol receipts forecasts, under a second + "4.12 Alcohol clearance forecasts" heading in row 53. It divides the + repeated spirits forecasts by ten and states ten thousand hectolitres in + a note below them. Each line then sits once, under the second heading, + and the first keeps the table's listed note. In thousand hectolitres, + the repeated figures give 2026-27 spirits growth of -90.1%, and the + published figures -0.8%. The importer refuses every alcohol line and + names both headings' rows, and the renderer gets no growth. A run on the + real March 2026 tables, changed the same way, stops before it writes + anything. The importer reads the tobacco lines as before.""" + changed, row = repeat_alcohol_heading(REAL_OCTOBER_2026_RECEIPTS, "4.12") + assert row == 53 + rows = read_sheet_rows(changed, "4.12") + (spirits,) = [line for line in rows if line.number == 58] + assert receipts_label(spirits["B"]) == "spirits" + published = [float(level) for level in OCTOBER_2026_CLEARANCES["spirits"][:2]] + assert round(float(spirits["D"]) / float(spirits["C"]) - 1, 6) == -0.900793 + assert round(published[1] / published[0] - 1, 6) == -0.007927 + sheets = {"4.11": REAL_OCTOBER_2026_SHEETS["4.11"], "4.12": rows} + for key, spec in CLEARANCE_SPECS.items(): + if spec.title == TOBACCO_CLEARANCES: + assert ( + find_receipts_line(sheets, spec).levels + == find_receipts_line(REAL_OCTOBER_2026_SHEETS, spec).levels + ), key + continue + with pytest.raises(ValueError) as refused: + find_receipts_line(sheets, spec) + assert str(refused.value) == repeated("4.12", 53), key + # Rows built as plain dicts carry no number, so the refusal counts their + # places: row 53 sits 47th among the rows the sheet stores. + with pytest.raises(ValueError) as refused: + find_receipts_line( + {"4.12": [dict(line) for line in rows]}, CLEARANCE_SPECS["spirits"] + ) + assert str(refused.value) == repeated("4.12", 47) + with pytest.raises(ValueError) as refused: + render( + get_yoy_growth_path().read_text(), + current=changed, + previous=REAL_RECEIPTS[2026], + **COMMITTED_RELEASES, + ) + assert str(refused.value) == repeated("4.12", 53) + march, row = repeat_alcohol_heading(REAL_RECEIPTS[2026], "3.12") + yoy, gap_dir = write_tree(tmp_path) + before = {path: path.read_bytes() for path in [yoy, *gap_dir.iterdir()]} + with pytest.raises(ValueError) as refused: + run_main( + tmp_path, + make_test_xlsx(), + "--yaml-path", + str(yoy), + receipts=march, + previous_receipts=REAL_RECEIPTS[2025], + ) + assert str(refused.value) == repeated("3.12", row) + assert row == 38 + assert {path: path.read_bytes() for path in before} == before + assert not list(yoy.parent.rglob("*.staged")) + + +# per_head_growth divides each series' own growth, a duty's base for a line +# with rates, whether the importer links the line to the previous EFO or to +# its own outturn. + +REMOTE_GAMING = next( + spec for spec in RECEIPTS_SERIES_SPECS if spec.path[-1] == "remote_gaming" +) + + +def test_per_head_growth_divides_the_base_growth_the_series_holds(): + """Remote Gaming Duty receipts double and its base, receipts over the + rate, grows 10%, while population grows 1%. per_head_growth returns + 1.10 / 1.01 - 1 for the base's growth per head, with or without a + previous line, not 2 / 1.01 - 1 (receipts) or the growth the levels and + the model's rates give.""" + line = ReceiptsLine( + table="4.10", + label="Remote Gaming Duty", + levels={2025: 100.0, 2026: 200.0}, + outturn_years=frozenset(), + ) + series = ReceiptsGrowth( + values={2025: 0.0, 2026: 0.10}, + line=line, + previous_line=None, + spec=REMOTE_GAMING, + ) + population = {2025: 0.01, 2026: 0.01} + assert per_head_growth(series, population).values == { + 2025: round(1 / 1.01 - 1, 6), + 2026: 0.089109, + } + previous = ReceiptsLine( + table="3.10", + label="Remote Gaming Duty", + levels={2024: 100.0}, + outturn_years=frozenset({2024}), + ) + assert ( + per_head_growth(replace(series, previous_line=previous), population).values[ + 2026 + ] + == 0.089109 + ) + + +def test_real_remote_gaming_base_per_head_rounds_once(): + """The real line has its own outturn and no previous line. Each year the + importer divides one plus the base's growth before rounding by one plus + population growth, subtracts one and rounds once, as the committed + series holds it.""" + growth = compute_receipts_growth( + REAL_OCTOBER_2026_RECEIPTS, REAL_RECEIPTS[2026], specs=(REMOTE_GAMING,) + )[REMOTE_GAMING.path] + assert growth.previous_line is None + population = population_growth(get_yoy_growth_path().read_text()) + per_head = per_head_growth(growth, population) + for year, ratio in growth.ratios.items(): + assert per_head.values[year] == round( + ratio / (1 + growth_in_effect(population, year)) - 1, 6 + ) + assert per_head.ratios[year] == ratio / (1 + growth_in_effect(population, year)) + stored = _values(get_yoy_growth_path().read_text(), REMOTE_GAMING.path) + assert {year: stored[year] for year in per_head.values} == per_head.values + + +@settings(max_examples=60, deadline=None, derandomize=True) +@given( + ratios=st.lists( + st.floats(min_value=0.5, max_value=2, allow_nan=False), min_size=1, max_size=8 + ), + population=st.lists(POPULATION_GROWTH, min_size=8, max_size=8), +) +def test_per_head_growth_times_population_growth_is_the_series_growth( + ratios, population +): + """For any growth before rounding and any population growth, + (1 + per-head growth) × (1 + population growth) matches one plus the + aggregate growth to within the six-decimal rounding, 5e-7 × + (1 + population growth). The per-head series rounds its own ratios, and + for a series without its ratios per_head_growth divides one plus its + six-decimal values instead.""" + years = range(2025, 2025 + len(ratios)) + by_year = dict(zip(years, ratios)) + population_by_year = dict(zip(range(2025, 2033), population)) + values = {year: round(ratio - 1, 6) + 0.0 for year, ratio in by_year.items()} + line = ReceiptsLine("4.12", "Wine", {}, frozenset()) + spec = CLEARANCE_SPECS["wine"] + series = ReceiptsGrowth(values, line, None, spec, by_year) + per_head = per_head_growth(series, population_by_year) + for year, ratio in by_year.items(): + growth = 1 + population_by_year[year] + assert ( + abs((1 + per_head.values[year]) * growth - ratio) <= 5e-7 * growth + 1e-12 + ) + assert per_head.values[year] == round(per_head.ratios[year] - 1, 6) + 0.0 + rounded = per_head_growth(replace(series, ratios=None), population_by_year) + assert rounded.values == { + year: round((1 + value) / (1 + population_by_year[year]) - 1, 6) + 0.0 + for year, value in values.items() + } + + +def test_a_series_growth_before_rounding_must_round_to_its_values(): + line = ReceiptsLine("4.12", "Wine", {}, frozenset()) + with pytest.raises(ValueError, match="does not round to the growth"): + ReceiptsGrowth({2025: 0.01}, line, None, CLEARANCE_SPECS["wine"], {2025: 1.02}) + with pytest.raises(ValueError, match="does not round to the growth"): + ReceiptsGrowth( + {2025: 0.01}, + line, + None, + CLEARANCE_SPECS["wine"], + {2025: 1.01, 2026: 1.0}, + ) + + +def test_population_growth_before_its_first_entry_is_that_entry(): + """The model backdates a parameter's first value, so a year before + ons.population's first entry takes that entry, not zero: wine flat in + 2025 falls 1 / 1.02 - 1 per head when population grows 2% in its first + entry, 2026.""" + assert growth_in_effect({2026: 0.02, 2030: -0.005}, 2025) == 0.02 + assert growth_in_effect({2026: 0.02, 2030: -0.005}, 2029) == 0.02 + assert growth_in_effect({2026: 0.02, 2030: -0.005}, 2031) == -0.005 + before = ReceiptsLine("3.12", "Wine", {2024: 100.0}, frozenset({2024})) + current = ReceiptsLine("4.12", "Wine", {2025: 100.0}, frozenset({2025})) + series = ReceiptsGrowth( + {2025: 0.0}, current, before, CLEARANCE_SPECS["wine"], {2025: 1.0} + ) + assert per_head_growth(series, {2026: 0.02}).values == {2025: -0.019608} + + +def test_per_head_growth_refuses_an_index_that_rounds_to_zero(): + """Clearances that fall from 1,000,000 to 4.55 leave a raw index of + 0.00001 at five decimals; divided by 1.03 for population, the per-head + index rounds to zero, so the importer refuses the series.""" + before = ReceiptsLine("3.12", "Wine", {2024: 1e6}, frozenset({2024})) + current = ReceiptsLine("4.12", "Wine", {2025: 4.55}, frozenset({2025})) + ratio = 4.55 / 1e6 + series = ReceiptsGrowth( + {2025: round(ratio - 1, 6) + 0.0}, + current, + before, + CLEARANCE_SPECS["wine"], + {2025: ratio}, + ) + check_index("raw", series.values) + assert round(1 + series.values[2025], 5) == 0.00001 + with pytest.raises(ValueError, match="too small to grow from"): + per_head_growth(series, {2025: 0.03}) diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index ab7f5a5b11..70e8600b05 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -228,16 +228,33 @@ def extract_economy_workbook_bytes( return Path(workbook_name).name, archive.read(workbook_name) +STRING_TEXT = f"{{{WORKBOOK_NS['main']}}}t" +STRING_RUN = f"{{{WORKBOOK_NS['main']}}}r" + + +def _string_text(item: ET.Element) -> str: + """Join a shared or inline string's own ```` and the ```` of each + formatting run (````), in order, so the reader returns the whole text + of a string split into runs. It skips any other child, such as a + phonetic run (````), whose ```` holds a reading of the text, not + more of it.""" + parts = [] + for child in item: + if child.tag == STRING_TEXT: + parts.append(child.text or "") + elif child.tag == STRING_RUN: + parts.extend( + node.text or "" for node in child.findall("main:t", WORKBOOK_NS) + ) + return "".join(parts) + + def _load_shared_strings(archive: ZipFile) -> list[str]: if "xl/sharedStrings.xml" not in archive.namelist(): return [] root = ET.fromstring(archive.read("xl/sharedStrings.xml")) - values: list[str] = [] - for item in root.findall("main:si", WORKBOOK_NS): - parts = [node.text or "" for node in item.iterfind(".//main:t", WORKBOOK_NS)] - values.append("".join(parts)) - return values + return [_string_text(item) for item in root.findall("main:si", WORKBOOK_NS)] def _sheet_paths(archive: ZipFile) -> dict[str, str]: @@ -265,22 +282,37 @@ def _cell_value(cell: ET.Element, shared_strings: list[str]) -> str | None: if cell_type == "s" and value_node is not None: return shared_strings[int(value_node.text)] if cell_type == "inlineStr" and inline_node is not None: - text_node = inline_node.find(".//main:t", WORKBOOK_NS) - return text_node.text if text_node is not None else "" + return _string_text(inline_node) if value_node is not None: return value_node.text return None -def read_sheet_rows(xlsx_bytes: bytes, sheet_name: str) -> list[dict[str, str | None]]: +class SheetRow(dict): + """Hold a worksheet row as ``read_sheet_rows`` reads it: each cell's + text by column letter, and ``number``, the row's number on the + worksheet.""" + + def __init__(self, number: int): + super().__init__() + self.number = number + + +def read_sheet_rows(xlsx_bytes: bytes, sheet_name: str) -> list[SheetRow]: + """Read the rows a worksheet stores, in order. A worksheet can skip + rows, as October 2026's sheet 4.12 skips row 33, so a row's place in the + list can differ from its number.""" with ZipFile(BytesIO(xlsx_bytes)) as archive: shared_strings = _load_shared_strings(archive) sheet_path = _sheet_paths(archive)[sheet_name] sheet = ET.fromstring(archive.read(sheet_path)) rows = [] + number = 0 for row in sheet.findall(".//main:sheetData/main:row", WORKBOOK_NS): - values: dict[str, str | None] = {} + # A row without a number takes the one after the row before it. + number = int(row.attrib.get("r", number + 1)) + values = SheetRow(number) for cell in row.findall("main:c", WORKBOOK_NS): ref = cell.attrib["r"] column = "".join(ch for ch in ref if ch.isalpha()) @@ -677,14 +709,155 @@ class ReceiptsSeriesSpec: @property def per_head(self) -> bool: - """Whether the series sits under ``per_capita``. Dataset projection - grows household weights by ``ons.population``, so - ``render_receipts_growth_yaml`` writes such a series' growth over one - plus that growth (``per_head_growth``, ``per_head_tail``), and the - weighted total grows with the line in every year.""" + """Return whether the renderer writes this series under + ``per_capita``. Dataset projection grows household weights by + ``ons.population``, so ``render_receipts_growth_yaml`` writes such a + series as (1 + aggregate growth) / (1 + ons.population growth) - 1 + (``per_head_growth``, ``per_head_tail``), and the weighted total grows + with the line in every year.""" return self.path[0] == "per_capita" +@dataclass(frozen=True) +class ClearanceSeriesSpec(ReceiptsSeriesSpec): + """Describe a clearance line the importer uses to grow household + quantity inputs per head (#2596). + + The OBR counts clearances in quantities, not pounds, so a clearance + table has no "£ billion" cell. ``unit_cell`` holds the text of the cell + in which the OBR states the line's unit, as ``CLEARANCE_TABLE_TEXTS`` + lists it. In both EFOs the importer requires that cell and refuses any + text in the table that the list lacks (``validate_clearance_table``). + For a part of a total (``part_of_total``), the OBR states the unit in + the total's label, and the importer requires that label and the total's + figures on the row directly above the "of which:" row over the line + (``validate_clearance_part``). + Dataset projection grows household weights by ``ons.population``, so + ``render_receipts_growth_yaml`` writes + (1 + aggregate clearance growth) / (1 + ons.population growth) - 1 + (``per_head_growth``), and the weighted total grows with the + clearances. After the OBR's forecast the importer holds the clearances + flat, as ``road_fuel_volume`` holds road fuel volume. + """ + + unit_cell: str = "" + part_of_total: bool = False + + def __post_init__(self): + name = ".".join(self.path) + if self.unit_cell not in CLEARANCE_TABLE_TEXTS.get(self.title, ()): + raise ValueError( + f"{name}: a clearance line needs the cell that states its " + f"unit, and its table's read texts lack {self.unit_cell!r}" + ) + # The importer links clearance outturns to the previous EFO's outturn. + if self.rates or self.outturn is not None: + raise ValueError( + f"{name}: the OBR counts clearances in quantities, so a " + "clearance line takes no rates and links to the previous EFO, " + "not its own outturn" + ) + + +TOBACCO_CLEARANCES = "tobacco clearances" +ALCOHOL_CLEARANCES = "alcohol clearance forecasts" +# The cells in which the OBR states each clearance unit: the cigarette label +# for cigarettes, the label of the non-cigarette total that Table 4.11 (3.11 +# in March 2026) splits into hand-rolled and other tobacco, and Table 4.12's +# (3.12's) note for alcohol. +TOBACCO_STICKS = "cigarette clearances (million sticks)" +TOBACCO_KILOGRAMS = "non-cigarette clearances ('000s kg)" +ALCOHOL_HECTOLITRES = "note: all clearances are in thousands of hectolitres of alcohol." +# The text of every cell from each clearance table's title down to the next +# table's title, in Tables 3.11 and 3.12 of the March 2025 and March 2026 +# EFOs and Tables 4.11 and 4.12 of October 2026, but the title, numbers and +# fiscal years. Someone read each one. The importer compares a cell's text as +# receipts_label gives it: lower case, in single spaces and without a +# footnote number after a label's last letter, so " Wine1" gives "wine". It +# refuses any other text (validate_clearance_table), so it reads no unit or +# product from words no one has read. Add a text only after reading it, and +# when it states another unit, convert the table's figures to the units these +# texts state first. +CLEARANCE_TABLE_TEXTS = { + TOBACCO_CLEARANCES: frozenset( + { + "outturn", + "forecast", + TOBACCO_STICKS, + TOBACCO_KILOGRAMS, + "of which:", + "hand rolled tobacco clearances", + "other tobacco products", + } + ), + ALCOHOL_CLEARANCES: frozenset( + { + "outturn", + "forecast", + "beer and cider", + "wine", + "spirits", + ALCOHOL_HECTOLITRES, + "1 includes spirit-based ‘ready-to-drinks’ such as alcopops.", + "1 includes other fermented products such as fruit ciders.", + "2 includes spirit-based ‘ready-to-drinks’ such as alcopops.", + } + ), +} +# The titles, as row_title gives them, of the tables that may follow each +# clearance table on its sheet. The importer checks a clearance table from +# its title down to the row above one of these titles, or to the sheet's +# end. In all three EFOs the alcohol receipts forecasts follow the alcohol +# clearances, and no table follows the tobacco clearances. +CLEARANCE_TABLE_ENDS = { + TOBACCO_CLEARANCES: frozenset(), + ALCOHOL_CLEARANCES: frozenset({"alcohol receipts forecasts"}), +} + +# The product groups the OBR forecasts. The OBR includes other fermented +# products in Table 4.12's wine line and spirit-based ready-to-drinks in its +# spirits line (the table's footnotes 1 and 2). +CLEARANCE_SERIES_SPECS = ( + ClearanceSeriesSpec( + path=("per_capita", "tobacco_clearances", "cigarettes"), + title=TOBACCO_CLEARANCES, + row="cigarette clearances (million sticks)", + unit_cell=TOBACCO_STICKS, + ), + ClearanceSeriesSpec( + path=("per_capita", "tobacco_clearances", "hand_rolling_tobacco"), + title=TOBACCO_CLEARANCES, + row="hand rolled tobacco clearances", + unit_cell=TOBACCO_KILOGRAMS, + part_of_total=True, + ), + ClearanceSeriesSpec( + path=("per_capita", "tobacco_clearances", "other_tobacco_products"), + title=TOBACCO_CLEARANCES, + row="other tobacco products", + unit_cell=TOBACCO_KILOGRAMS, + part_of_total=True, + ), + ClearanceSeriesSpec( + path=("per_capita", "alcohol_clearances", "beer_and_cider"), + title=ALCOHOL_CLEARANCES, + row="beer and cider", + unit_cell=ALCOHOL_HECTOLITRES, + ), + ClearanceSeriesSpec( + path=("per_capita", "alcohol_clearances", "wine"), + title=ALCOHOL_CLEARANCES, + row="wine", + unit_cell=ALCOHOL_HECTOLITRES, + ), + ClearanceSeriesSpec( + path=("per_capita", "alcohol_clearances", "spirits"), + title=ALCOHOL_CLEARANCES, + row="spirits", + unit_cell=ALCOHOL_HECTOLITRES, + ), +) + GAMBLING_RATES = "gov.hmrc.gambling_duty" MACHINE_GAMES = f"{GAMBLING_RATES}.machine_games" @@ -865,6 +1038,8 @@ def machine_games_spec(higher_stake_share: float) -> ReceiptsSeriesSpec: (RateComponent(f"{GAMBLING_RATES}.pool_betting.rate"),), ((2024, 8_125), (2025, 3_970)), ), + # The alcohol and tobacco quantity inputs (#2596). + *CLEARANCE_SERIES_SPECS, ) @@ -952,8 +1127,9 @@ def growth_ratio( @dataclass(frozen=True) class ReceiptsLine: - """A receipts line as one EFO publishes it: £bn by the calendar year in - which each fiscal year starts, and which of those years are outturn.""" + """Hold a receipts line as the OBR publishes it in one EFO: £bn, or a + clearance line's own unit, by the calendar year in which each fiscal year + starts, and which of those years the OBR marks as outturn.""" table: str label: str @@ -965,17 +1141,37 @@ class ReceiptsLine: class ReceiptsGrowth: """Growth in a receipts line, by year, rounded to six decimals. - The first year is outturn growth: this EFO's first (outturn) year over - the year before in the previous EFO, where it is outturn. Each later - year is the ratio of this EFO's consecutive levels. A line with its own - outturn starts at that outturn's second year and has no previous line; - one with rates grows its base instead (see ``ReceiptsSeriesSpec``). + The first year holds outturn growth: this EFO's first (outturn) year + over the year before in the previous EFO, which marks that year as + outturn. Each later year holds the ratio of this EFO's consecutive + levels. A line with its own outturn starts at that outturn's second year + and has no previous line; one with rates grows its base instead (see + ``ReceiptsSeriesSpec``). A clearance line holds the clearances' growth, + which ``render_receipts_growth_yaml`` writes per head + (``per_head_growth``). + + ``ratios`` holds one plus each year's growth before rounding, as + ``compute_receipts_growth`` finds it, so ``per_head_growth`` rounds the + per-head series once. Each ratio must round to its year's value. """ values: dict[int, float] line: ReceiptsLine previous_line: ReceiptsLine | None spec: ReceiptsSeriesSpec | None = None + ratios: dict[int, float] | None = None + + def __post_init__(self): + if self.ratios is None: + return + if sorted(self.ratios) != sorted(self.values) or any( + round(ratio - 1, 6) + 0.0 != self.values[year] + for year, ratio in self.ratios.items() + ): + raise ValueError( + f"{self.line.label}: the growth before rounding {self.ratios} " + f"does not round to the growth {self.values}" + ) def fiscal_year_label(year: int) -> str: @@ -1085,13 +1281,156 @@ def validate_receipts_unit( ) +# A number as a worksheet stores it, such as "1272.4562775587838" or "1E-3". +NUMBER_RE = re.compile(r"[-+]?(?:\d+\.?\d*|\.\d+)(?:e[-+]?\d+)?") + + +def validate_clearance_table( + rows: list[dict[str, str | None]], + number: str, + spec: ClearanceSeriesSpec, + where: str, +) -> None: + """Refuse a clearance table that holds text no one has read. + + The importer checks every cell of the table numbered ``number`` and + titled ``spec.title`` on a sheet's ``rows``, in every column, from the + title's row down to the row above a title ``CLEARANCE_TABLE_ENDS`` + lists, or to the sheet's end. It skips the title, numbers and fiscal + years, and needs every other text in ``CLEARANCE_TABLE_TEXTS`` and the + line's ``unit_cell`` among them. It interprets no words, so a note that + states another unit, a line for another product or a unit column stops + the run until someone reads the text and lists it. + + The importer also refuses a sheet that repeats the table's number and + title, and names the rows (``SheetRow.number``, or a row's place in + ``rows`` counted from 1 for a row built as a plain dict). It would check + the text under one heading and could read the line under another. + """ + starts = [ + index + for index, row in enumerate(rows) + if row_title(row) == (number, spec.title) + ] + if len(starts) > 1: + numbers = [str(getattr(rows[index], "number", index + 1)) for index in starts] + raise ValueError( + f"{where} repeats its heading, in rows " + f"{', '.join(numbers[:-1])} and {numbers[-1]}. The importer would " + "check the text under one and could read a line under another, so " + "keep one heading." + ) + start = starts[0] + block = [rows[start]] + for row in rows[start + 1 :]: + title = row_title(row) + if title is not None and title[1] in CLEARANCE_TABLE_ENDS[spec.title]: + break + block.append(row) + read, unread = set(), [] + for index, row in enumerate(block): + years = _fiscal_year_columns(row) + cells = [ + (column, value) + for column, value in sorted( + row.items(), key=lambda item: column_index(item[0]) + ) + if receipts_label(value) + ] + # Skip the title, the first text on the title's row. + for column, value in cells[1:] if index == 0 else cells: + text = receipts_label(value) + if column in years or NUMBER_RE.fullmatch(text): + continue + read.add(text) + if text not in CLEARANCE_TABLE_TEXTS[spec.title]: + unread.append(f"{value!r} in column {column}") + if unread: + raise ValueError( + f"{where} holds text the importer has not read: " + + "; ".join(unread) + + ". Read each one. If it states another unit, convert the " + "table's figures to the units the read texts state. Then add the " + "text, as receipts_label gives it, to " + f'CLEARANCE_TABLE_TEXTS["{spec.title}"], or the title of a table ' + "that follows the clearances to CLEARANCE_TABLE_ENDS." + ) + if spec.unit_cell not in read: + raise ValueError( + f"{where} lacks {spec.unit_cell!r}, the cell in which the OBR " + f"states the unit of {'.'.join(spec.path)}" + ) + + +def validate_clearance_part( + table_rows: list[dict[str, str | None]], + line_index: int, + spec: ClearanceSeriesSpec, + where: str, +) -> None: + """Refuse a part of a total that does not sit under the total. + + The OBR states the unit of hand-rolled and other tobacco only in the + label of the total they break down, "Non-cigarette clearances ('000s + kg)", on the row directly above an "of which:" row. Going up from the + line in ``table_rows`` (the rows below the table's title), past the + total's other parts, the importer needs an "of which:" row with no other + cell, then a row that holds ``spec.unit_cell`` as its first text and one + or more numbers after it, and nothing else. A table of listed texts + could otherwise put the parts under another line, such as "Cigarette + clearances (million sticks)", and the importer would read them in the + wrong unit. + """ + + def texts(row: dict[str, str | None]) -> list[str]: + return [ + receipts_label(value) + for _, value in sorted(row.items(), key=lambda item: column_index(item[0])) + if receipts_label(value) + ] + + parts = { + other.row + for other in CLEARANCE_SERIES_SPECS + if other.part_of_total + and (other.title, other.unit_cell) == (spec.title, spec.unit_cell) + } + above = [texts(row) for row in table_rows[:line_index]] + while above and above[-1][:1] and above[-1][0] in parts: + above.pop() + # The two rows above the line and its fellow parts, empty past the top. + total, of_which = ([[], []] + above)[-2:] + if ( + of_which != ["of which:"] + or total[:1] != [spec.unit_cell] + or not total[1:] + or not all(NUMBER_RE.fullmatch(text) for text in total[1:]) + ): + raise ValueError( + f"{where} lacks the layout that gives the {'.'.join(spec.path)} line " + 'its unit: an "of which:" row with no other cell directly above the ' + "line and the total's other parts, and directly above that row the " + f"non-cigarette total's row, which holds its label ({spec.unit_cell!r}, " + "as receipts_label gives it), its figures and nothing else. The line " + "takes its unit from that total's label, so the importer refuses any " + "other layout." + ) + + def find_receipts_line( sheets: dict[str, list[dict[str, str | None]]], spec: ReceiptsSeriesSpec ) -> ReceiptsLine: - """The one row labelled ``spec.row`` in a table titled ``spec.title``. - - Only that table's rows are read: its cash unit, year headers and its - "Outturn" and "Forecast" markers, which may sit on one row or several. + """Find the row labelled ``spec.row`` in the table titled + ``spec.title``. + + The importer reads only that table's rows: its cash unit, its year + headers and its "Outturn" and "Forecast" markers, which may sit on one + row or several. Before it reads a clearance line's values, it checks the + line's table, from its title through its end, against the texts someone + has read (``validate_clearance_table``). It then refuses any text beside + the years in the fiscal-year row over the line, and a part of a total + that does not sit under the total (``validate_clearance_part``): a + table of listed texts can still put a unit over the wrong lines. """ matches = [] for sheet, rows in sheets.items(): @@ -1100,12 +1439,14 @@ def find_receipts_line( continue where = f"Table {table} (sheet {sheet})" years: dict[str, int] = {} + year_row: dict[str, str | None] = {} markers: dict[str, int] = {} header_rows = [] - for row in table_rows: + for index, row in enumerate(table_rows): row_years = _fiscal_year_columns(row) if row_years: years = row_years + year_row = row continue if not years: header_rows.append(row) @@ -1126,16 +1467,25 @@ def find_receipts_line( continue first_year_column = min(column_index(column) for column in years) labels = [ - value + (column, value) for column, value in sorted( row.items(), key=lambda item: column_index(item[0]) ) if column_index(column) < first_year_column and normalise_label(value) ] - if labels and receipts_label(labels[0]) == spec.row: + if labels and receipts_label(labels[0][1]) == spec.row: matches.append( - (table, sheet, labels[0], row, years, markers, header_rows) + ( + table, + sheet, + labels[0], + row, + years, + markers, + header_rows, + (year_row, table_rows, index), + ) ) name = ".".join(spec.path) @@ -1145,15 +1495,57 @@ def find_receipts_line( f'Expected one "{spec.row}" row in the "{spec.title}" table for ' f"{name}; found {found}" ) - table, sheet, label, row, years, markers, header_rows = matches[0] + ( + table, + sheet, + (label_column, label), + row, + years, + markers, + header_rows, + (year_row, table_rows, line_index), + ) = matches[0] where = f"Table {table} (sheet {sheet})" + if isinstance(spec, ClearanceSeriesSpec): + validate_clearance_table(sheets[sheet], table, spec, where) repeated = sorted({y for y in years.values() if list(years.values()).count(y) > 1}) if repeated: raise ValueError( f"{where} heads more than one column " + ", ".join(fiscal_year_label(y) for y in repeated) ) - validate_receipts_unit(header_rows, where) + # The importer reads a line's label and a value under each year and + # refuses anything else on the row: a unit or note beside the label would + # describe the values, which the importer would read in the table's unit. + beside = { + column: value + for column, value in row.items() + if normalise_label(value) and column != label_column and column not in years + } + if beside: + raise ValueError( + f"{where} holds {beside} beside the {name} row's label and values; " + "a unit or note on the row needs deliberate importer support" + ) + if isinstance(spec, ClearanceSeriesSpec): + # The OBR states each clearance unit in a label or a note. A text + # beside the years, such as a unit column's heading or a line's label, + # could head the values below it with another unit. + heading = { + column: value + for column, value in year_row.items() + if normalise_label(value) and column not in years + } + if heading: + raise ValueError( + f"{where} holds {heading} in the fiscal-year row over the {name} " + "line. A text beside the years could give the line another " + "unit, so the importer refuses any text there." + ) + if spec.part_of_total: + validate_clearance_part(table_rows, line_index, spec, where) + else: + validate_receipts_unit(header_rows, where) levels = {} for column, year in years.items(): text = row.get(column) @@ -1244,12 +1636,10 @@ def compute_receipts_growth( f"{name}: {fiscal_year_label(year)} receipts of {level} are " "not positive, so there is no growth to or from them" ) - values = { - year: round(growth_ratio(spec, levels, year) - 1, 6) + 0.0 - for year in sorted(levels)[1:] - } + ratios = {year: growth_ratio(spec, levels, year) for year in sorted(levels)[1:]} + values = {year: round(ratio - 1, 6) + 0.0 for year, ratio in ratios.items()} check_index(name, values) - result[spec.path] = ReceiptsGrowth(values, line, previous, spec) + result[spec.path] = ReceiptsGrowth(values, line, previous, spec, ratios) return result @@ -1293,36 +1683,27 @@ def growth_in_effect(series: dict[int, float], year: int) -> float: def per_head_growth( growth: ReceiptsGrowth, population: dict[int, float] ) -> ReceiptsGrowth: - """A series' growth per head, to six decimals: each year's growth before - rounding (``growth_ratio``), over one plus that year's ``ons.population`` - growth, minus one. + """Return a series' growth per head, to six decimals: one plus each + year's growth, over one plus that year's ``ons.population`` growth, + minus one. + It divides the series' own growth, before rounding where + ``growth.ratios`` holds it: a duty's base for a line with rates, whether + the importer links the line to the previous EFO or to its own outturn. Dataset projection grows each household's input by this and its weight by the population growth, so the weighted input grows by their product, - the line's growth: a duty's base, for a line with rates. + the series' growth. """ - spec = growth.spec - first = min(growth.line.levels) - if growth.previous_line is None: - earlier = { - year: value / 1e6 - for year, value in spec.outturn.receipts_thousands - if year < first - } - else: - earlier = {first - 1: growth.previous_line.levels[first - 1]} - levels = {**earlier, **growth.line.levels} - values = { - year: round( - growth_ratio(spec, levels, year) / (1 + growth_in_effect(population, year)) - - 1, - 6, - ) - + 0.0 - for year in growth.values + ratios = growth.ratios + if ratios is None: + ratios = {year: 1 + value for year, value in growth.values.items()} + per_head = { + year: ratio / (1 + growth_in_effect(population, year)) + for year, ratio in ratios.items() } + values = {year: round(ratio - 1, 6) + 0.0 for year, ratio in per_head.items()} check_index(".".join(growth.spec.path), values) - return replace(growth, values=values) + return replace(growth, values=values, ratios=per_head) def format_growth(value: float) -> str: @@ -1471,9 +1852,11 @@ def render_receipts_values( def receipts_basis(spec: ReceiptsSeriesSpec | None) -> str: - """How a line's growth is measured, for its comments: nothing for - receipts, the rates taken out for a base, and population for a base per - head.""" + """Return what the importer takes out of a line's growth, for its + comments: nothing for receipts, the rates for a base, and population for + clearances and for a base per head.""" + if isinstance(spec, ClearanceSeriesSpec): + return "; per head: clearance growth over one plus ons.population growth" if spec is None or not spec.rates: return "" if spec.per_head: @@ -1519,8 +1902,10 @@ def receipts_tail(content: str, growth: ReceiptsGrowth) -> tuple[dict[int, float comes from. A base under ``per_capita`` divides that growth by one plus ``ons.population`` growth (``per_head_tail``).""" last = max(growth.values) - path = average_earnings_path(content, last) spec = growth.spec + if isinstance(spec, ClearanceSeriesSpec): + return clearance_tail(content, last) + path = average_earnings_path(content, last) if spec is None or not spec.rates: return path, "the average_earnings long-run path above" # Rates hold their last value after their last change, as the path does. @@ -1555,6 +1940,25 @@ def receipts_tail(content: str, growth: ReceiptsGrowth) -> tuple[dict[int, float return tail, comment +def clearance_tail(content: str, last: int) -> tuple[dict[int, float], str]: + """Return a clearance series after the OBR's forecast, and its + comment. The importer holds the clearances flat, so per head they grow + by 1 / (1 + population) - 1, to six decimals, in each year to + ``ons.population``'s last. It writes a year only where the value + changes.""" + population = population_growth(content) + yearly = { + year: round(1 / (1 + growth_in_effect(population, year)) - 1, 6) + 0.0 + for year in range(last + 1, max(max(population), last + 1) + 1) + } + tail = { + year: value + for year, value in yearly.items() + if year == last + 1 or value != yearly[year - 1] + } + return tail, "clearances held flat: 1 / (1 + ons.population growth) - 1" + + def receipts_references( growth: ReceiptsGrowth, release: tuple[str, int], @@ -1614,12 +2018,13 @@ def render_receipts_growth_yaml( """Rewrite each receipts series' values and references in a yoy_growth.yaml. - Years before this EFO's first year are kept, so a series' index keeps - its base year. Years after the OBR's receipts forecast follow - average_earnings as ``content`` holds it. A series under ``per_capita`` - is written per head of ``ons.population`` as ``content`` holds it, in - every year to the last that population or the tail states. The result is - checked to differ from ``content`` only in those values and references. + It keeps the years before this EFO's first year, so a series' index + keeps its base year, and follows average_earnings as ``content`` holds + it after the OBR's receipts forecast. It writes a series under + ``per_capita`` per head of ``ons.population`` as ``content`` holds it, + in every year to the last that population or the tail states, and holds + a clearance series' clearances flat after the forecast. It checks that + the result differs from ``content`` only in those values and references. """ before = yaml.safe_load(content) expected = copy.deepcopy(before) @@ -2819,8 +3224,9 @@ def build_arg_parser() -> argparse.ArgumentParser: help=( "Do not update the receipts growth series (stamp_duty_land_tax, " "business_rates) that project corporate SDLT and business rates, " - "or the gambling_duty_bases series that project the gambling " - "duty inputs" + "the gambling_duty_bases series that project the gambling duty " + "inputs, or the per_capita clearance series that project the " + "alcohol and tobacco quantity inputs" ), ) parser.add_argument( diff --git a/policyengine_uk/variables/input/consumption/alcohol/beer_litres.py b/policyengine_uk/variables/input/consumption/alcohol/beer_litres.py index f1f41ec55c..0b2957ed15 100644 --- a/policyengine_uk/variables/input/consumption/alcohol/beer_litres.py +++ b/policyengine_uk/variables/input/consumption/alcohol/beer_litres.py @@ -7,3 +7,4 @@ class beer_litres(Variable): definition_period = YEAR label = "Annual litres of beer" unit = "litre" + uprating = "gov.economic_assumptions.indices.obr.per_capita.alcohol_clearances.beer_and_cider" diff --git a/policyengine_uk/variables/input/consumption/alcohol/cider_litres.py b/policyengine_uk/variables/input/consumption/alcohol/cider_litres.py index c977131f07..dbd8b0e28f 100644 --- a/policyengine_uk/variables/input/consumption/alcohol/cider_litres.py +++ b/policyengine_uk/variables/input/consumption/alcohol/cider_litres.py @@ -7,3 +7,4 @@ class cider_litres(Variable): definition_period = YEAR label = "Annual litres of cider" unit = "litre" + uprating = "gov.economic_assumptions.indices.obr.per_capita.alcohol_clearances.beer_and_cider" diff --git a/policyengine_uk/variables/input/consumption/alcohol/other_fermented_litres.py b/policyengine_uk/variables/input/consumption/alcohol/other_fermented_litres.py index 4f6e8fc9f6..14f8058254 100644 --- a/policyengine_uk/variables/input/consumption/alcohol/other_fermented_litres.py +++ b/policyengine_uk/variables/input/consumption/alcohol/other_fermented_litres.py @@ -7,3 +7,4 @@ class other_fermented_litres(Variable): definition_period = YEAR label = "Annual litres of other fermented" unit = "litre" + uprating = "gov.economic_assumptions.indices.obr.per_capita.alcohol_clearances.wine" diff --git a/policyengine_uk/variables/input/consumption/alcohol/sparkling_cider_litres.py b/policyengine_uk/variables/input/consumption/alcohol/sparkling_cider_litres.py index 4d634f136b..d7d0b38b67 100644 --- a/policyengine_uk/variables/input/consumption/alcohol/sparkling_cider_litres.py +++ b/policyengine_uk/variables/input/consumption/alcohol/sparkling_cider_litres.py @@ -7,3 +7,4 @@ class sparkling_cider_litres(Variable): definition_period = YEAR label = "Annual litres of sparkling cider" unit = "litre" + uprating = "gov.economic_assumptions.indices.obr.per_capita.alcohol_clearances.beer_and_cider" diff --git a/policyengine_uk/variables/input/consumption/alcohol/spirits_litres.py b/policyengine_uk/variables/input/consumption/alcohol/spirits_litres.py index 707e4b55dc..7509803e42 100644 --- a/policyengine_uk/variables/input/consumption/alcohol/spirits_litres.py +++ b/policyengine_uk/variables/input/consumption/alcohol/spirits_litres.py @@ -7,3 +7,6 @@ class spirits_litres(Variable): definition_period = YEAR label = "Annual litres of spirits" unit = "litre" + uprating = ( + "gov.economic_assumptions.indices.obr.per_capita.alcohol_clearances.spirits" + ) diff --git a/policyengine_uk/variables/input/consumption/alcohol/wine_litres.py b/policyengine_uk/variables/input/consumption/alcohol/wine_litres.py index f47ee347c2..8afabf63fb 100644 --- a/policyengine_uk/variables/input/consumption/alcohol/wine_litres.py +++ b/policyengine_uk/variables/input/consumption/alcohol/wine_litres.py @@ -7,3 +7,4 @@ class wine_litres(Variable): definition_period = YEAR label = "Annual litres of wine" unit = "litre" + uprating = "gov.economic_assumptions.indices.obr.per_capita.alcohol_clearances.wine" diff --git a/policyengine_uk/variables/input/consumption/tobacco/cigarettes_per_week.py b/policyengine_uk/variables/input/consumption/tobacco/cigarettes_per_week.py index 1e5e12513d..680bf799cf 100644 --- a/policyengine_uk/variables/input/consumption/tobacco/cigarettes_per_week.py +++ b/policyengine_uk/variables/input/consumption/tobacco/cigarettes_per_week.py @@ -7,4 +7,7 @@ class cigarettes_per_week(Variable): definition_period = YEAR label = "Cigarettes bought per week" unit = "cigarette" + uprating = ( + "gov.economic_assumptions.indices.obr.per_capita.tobacco_clearances.cigarettes" + ) quantity_type = STOCK diff --git a/policyengine_uk/variables/input/consumption/tobacco/cigars_grams_per_week.py b/policyengine_uk/variables/input/consumption/tobacco/cigars_grams_per_week.py index 4fbe27611d..2cf1b98fb8 100644 --- a/policyengine_uk/variables/input/consumption/tobacco/cigars_grams_per_week.py +++ b/policyengine_uk/variables/input/consumption/tobacco/cigars_grams_per_week.py @@ -7,4 +7,5 @@ class cigars_grams_per_week(Variable): definition_period = YEAR label = "Grams of cigars bought per week" unit = "gram" + uprating = "gov.economic_assumptions.indices.obr.per_capita.tobacco_clearances.other_tobacco_products" quantity_type = STOCK diff --git a/policyengine_uk/variables/input/consumption/tobacco/hand_rolling_tobacco_grams_per_week.py b/policyengine_uk/variables/input/consumption/tobacco/hand_rolling_tobacco_grams_per_week.py index 05e3678f02..56829d8abb 100644 --- a/policyengine_uk/variables/input/consumption/tobacco/hand_rolling_tobacco_grams_per_week.py +++ b/policyengine_uk/variables/input/consumption/tobacco/hand_rolling_tobacco_grams_per_week.py @@ -7,4 +7,5 @@ class hand_rolling_tobacco_grams_per_week(Variable): definition_period = YEAR label = "Grams of hand rolling tobacco bought per week" unit = "gram" + uprating = "gov.economic_assumptions.indices.obr.per_capita.tobacco_clearances.hand_rolling_tobacco" quantity_type = STOCK diff --git a/policyengine_uk/variables/input/consumption/tobacco/heated_tobacco_grams_per_week.py b/policyengine_uk/variables/input/consumption/tobacco/heated_tobacco_grams_per_week.py index e4415f473a..33ec50223a 100644 --- a/policyengine_uk/variables/input/consumption/tobacco/heated_tobacco_grams_per_week.py +++ b/policyengine_uk/variables/input/consumption/tobacco/heated_tobacco_grams_per_week.py @@ -7,4 +7,5 @@ class heated_tobacco_grams_per_week(Variable): definition_period = YEAR label = "Grams of heated tobacco bought per week" unit = "gram" + uprating = "gov.economic_assumptions.indices.obr.per_capita.tobacco_clearances.other_tobacco_products" quantity_type = STOCK diff --git a/policyengine_uk/variables/input/consumption/tobacco/other_tobacco_grams_per_week.py b/policyengine_uk/variables/input/consumption/tobacco/other_tobacco_grams_per_week.py index 213b11742e..8824f55835 100644 --- a/policyengine_uk/variables/input/consumption/tobacco/other_tobacco_grams_per_week.py +++ b/policyengine_uk/variables/input/consumption/tobacco/other_tobacco_grams_per_week.py @@ -7,4 +7,5 @@ class other_tobacco_grams_per_week(Variable): definition_period = YEAR label = "Grams of other tobacco bought per week" unit = "gram" + uprating = "gov.economic_assumptions.indices.obr.per_capita.tobacco_clearances.other_tobacco_products" quantity_type = STOCK