Skip to content

UK land value inputs for chronicle#408: MHCLG land values, VOA's 2026 stock, EPC D3, council tax by authority, the Market Value Survey, land by sector and HMRC stamp tables - #410

Merged
juaristi22 merged 16 commits into
bundle-stream-consumer-rowsfrom
uk-land-value-inputs-408
Nov 5, 2026
Merged

juaristi22 merged 16 commits into
bundle-stream-consumer-rowsfrom
uk-land-value-inputs-408

Conversation

@juaristi22

Copy link
Copy Markdown
Collaborator

This PR carries eight of the nine items #408 asks for: about 326,500 new facts and three raw-only registrations. Item 6, UCL's house price per square metre file, needs a way to store files over 300 MiB, and #409 asks for a ruling on that. It will be its own PR.

Each item is its own commit (items 2 and 8 share one, because they share a generator section and test module). The last commit adds the docs. A follow-up commit will re-pin the bundle test's counts (see "Bundle and CI").

  • scripts/build_uk_408_packages.py writes every new package, and the record sets added to the pinned Welsh, SLGFS and balance sheet packages, from the pinned bytes. --check compares instead of writing.
  • HMRC's tables are written by scripts/build_uk_1095_port_packages.py, which already writes that package.
  • Field contracts and caveats are in docs/data-sources/uk-land-value-inputs-408.md.

What it adds

Item 1: MHCLG land value estimates for policy appraisal 2023. mhclg-land-value-estimates-2023 holds 10,101 facts from all eight data sheets (valuation date 1 October 2023). The workbook is the copy PolicyEngine/uk-lvt-paper#14 pins, sha256 ca626b7e…8705.

  • Provenance: model_output. The guidelines say the values "do not represent the market value of land".
  • Residential: for 295 councils, the site density and the value in GBP per hectare at five percentiles of new-build house price, under each density.
    • Residential is the estimate before MHCLG's floor. Its - cells (negative residuals) emit no fact.
    • Residential (adj for floor) applies the floor.
  • Non-residential: by built-up area (ONS BUA 2022 codes), London location, region and car park location, as printed.

Item 3: VOA Council Tax stock of properties 2026. The summary workbook is the copy PolicyEngine/uk-lvt-paper#14 pins, sha256 9e9be694…29d9.

  • voa-council-tax-bands-2026 (8,566 facts) has CTSOP1.0 and 2.0 at every geography printed, including 318 billing authorities and 575 constituencies.
  • voa-council-tax-property-types-2026 (120,493 facts) and voa-council-tax-build-periods-2026 (93,529 facts) have CTSOP3.0 and 4.0.
  • Raw only: the three small-area zips (CTSOP1.1, 3.1 and 4.1) are kept in R2 only and git-ignored.

Item 7: floor area and dwelling density.

  • mhclg-epc-d3-domestic-2026-q3 (75,051 facts) has EPC D3's lodgements, floor area and habitable rooms. That's every quarter from 2008 Q4 to 2026 Q3 for England and Wales, each nation, region and local authority, plus calendar years for the nations. D3 counts certificates lodged, not dwellings.
  • mhclg-live-table-126-dwelling-density-2026 (5,406 facts) has dwellings per hectare by authority, 2011 to 2025.

Item 4: council tax collected by billing authority.

  • England: mhclg-council-tax-collection-by-authority-2025-26 (Table 6a, 1,208 facts) and mhclg-council-tax-qrc4-2025-26 (Table 9a, QRC4, 5,757 facts).
  • Wales and Scotland: the 22 Welsh authority rows of Table 2 in both pinned Welsh packages, and the 32 SLGFS council rows of Tables 2.8 and 2.11 (404 facts).
  • Not ported:
    • Cells the publisher defines as arithmetic over other cells.
    • Rows that restate ported facts (Table 6a's England row, QRC4 line 1). These are asserted equal in the tests.

Items 2 and 8: the Market Value Survey 2024, and land by sector.

  • Market Value Survey: mhclg-market-value-survey-2024-chapter-{1,3,4} (4,458 facts) has the annex tables the issue lists.
    • Values are in GBP thousands as printed.
    • The period is the valuation date.
    • Sample sizes are their own measure.
    • Provenance is survey_aggregate.
  • Balance sheets: both packages now port land for every sector their workbooks print: E43E, E43J, E43K, E43Z, E447, E44C, E44D, E44O and E44Q (279 facts).
  • 2022 series: ons-land-underlying-buildings-2022 (108 facts) is Figure 1 of ONS's 2022 article. It is a pre-revision vintage, with the total as a fourth series.

Item 5: HMRC stamp tables. hmrc-stamp-tax-statistics-2024-25 gains Tables 7, 3a, 5a, 9a, 9d and 2e for both years, going from 352 to 1,487 facts.

  • Its parser is now ods_numeric_text_used_range, because Table 5a's 2023-24 sheet stores some numbers as text. The 352 existing facts keep their keys and values.

Item 9: Tax Policy Associates' model. tpa-lvt-model-2026 registers config.py, cells.json and LICENSE at commit fef94f9 (MIT), raw only, with no facts. The files are kept in R2 only, because config.py is Python source.

Parity checks (the issue's, and some more)

Item 1

  • Both residential sheets have 295 authority rows.
  • No printed value falls as the percentile rises.
  • On the floor-adjusted sheet, Cumberland's median and 60th percentile values are equal (GBP 460,000 per hectare) under all three densities: the only values that don't strictly rise.
  • Every floor-adjusted value is at least the estimate before the floor.

Item 3

  • Bands sum to "All properties" within rounding, with the largest gap 30.
  • CTSOP3.0's and 4.0's "All" columns equal CTSOP1.0 in all 3,255 rows.
  • CTSOP1.0's England and Wales rows exceed CTSOP2.0's by 8,260 and 1,380, which together make 9,640, against CTSOP2.0's 9,630 properties whose postcodes weren't matched to a constituency.

Item 7

  • Lodgements and habitable rooms add up exactly from authorities to regions to nations to England and Wales.
  • Floor area misses by at most 11 m², and only in quarters up to 2012 Q1.

Item 4

  • Table 6a's England row equals Table 5: GBP 42,014,522 thousand and 44,981,384 thousand collectable, and 40,281,988 thousand and 43,002,543 thousand collected.
  • Table 9a's line 1 equals Table 6a's 2025-26 figure for every row.
  • Authorities sum exactly to England, to Total Wales and to Scotland.
  • Every QRC4 formula holds for all 303 rows.

Item 8

  • Every sector identity holds exactly in every year (E44D = E44N + E44O, E43Z = E447 + E44C, E44R = E43B + E43K + E43Z + E44D, …).
  • The whole economy less households and non-financial corporations was GBP 454,887m in 2024.
  • The 2022 series' components miss its total by at most GBP 0.1tn.

Item 5 (2024-25; 2023-24 agrees too)

  • Table 3a's receipts (GBP 10,380m residential and 3,505m non-residential) equal Tables 8a and 8b.
  • Table 5a's all-bands receipts equal 3a's residential receipts.
  • Table 7's 925,140 is within rounding of 8a's 925,100.
  • Table 9a's 155,400 claims equal 9d's total.
  • Table 2e's quarters from 2024 Q2 to 2025 Q1 sum to 155,368.45.

Item 2: structural checks pass. Sample sizes are positive integers, the A and B versions cover the same categories, complete percentage columns sum to 100, and every fact equals its cell in an independent parse.

Where the sources differ from the issue

  • Barnsley and Sheffield's codes.
    • E08000016 and E08000019 were current at the land value estimates' valuation date. ONS replaced them with E08000038 and E08000039 in 2025, not before April 2023 as the issue says.
    • Table 9a (2025-26) prints the new codes while Table 6a, in the same release, prints the old ones.
    • EPC D3 switches in 2025 Q4.
    • Every package keeps the codes as printed.
  • Live Table 126 is the 21 May 2026 edition. The page's 25 June update covered other tables.
  • HMRC Table 5a's price bands include corporate bodies paying the flat rate, which Table 8a reports on its own row. Above £500k they exceed 8a's by exactly that row's £405m of property value in 2024-25, so 5a's facts say so in a filter.
  • HMRC Table 7 prints counts below 50, although its note says [X] marks fewer than 50.
  • HMRC Table 2e prints fractional recent claims. Its note dates the series from 2017 Q2, but the data start in 2017 Q4.
  • HMRC's FTBR share of residential transactions (Tables 9a and 2e) is printed as percent text that divides published counts, so it is not ported, as with Table 6b.
  • The Market Value Survey's AT4.1 prints its 2024 all-dwellings base with two digits transposed against AT1.1A's. That cell is not ported, and a test records it.
  • VOA's 2026 CTSOP1.1 zip names its member CTSOP_1_1/CTSOP1_1_2026_03_31.csv (2025: CTSOP1.1/…). Microcosm's band-mix builder reads it by name.
  • QRC4 line 7. The QRC4 form describes it as receipts of 2025-26 council taxes; Table 9a's heading says 2026-27. The package follows the heading.

Choices a reviewer may want to check

  • Entities and roles. Non-residential land values are entity: firm, with the site type as entity_role; residential ones are dwelling. Land values are aggregation: mean; the guidelines call them "reasonable averages".
  • Non-standard geographies.
    • Built-up areas, London locations and car park postcode districts use statistical_scope.
    • Inner and Outer London are typed as regions, as DESNZ's packages type them.
  • Restated rows. These are asserted equal, not ported twice, as UK facts for chronicle#395: Northern Ireland domestic rates, Welsh bus journeys, the Childcare Offer for Wales, VED by payer and CPI gas weights #396 did, so the PR adds no semantic duplicates.
  • Wording normalised to one filter value. The Market Value Survey's tenure and value-band wording, where tables word the same category differently. Each table's printed text stays in its labels and guards.
  • Generated YAML size. The three largest packages are 19.7 MB (VOA property types), 12.8 MB (EPC D3) and 12.2 MB (VOA build periods). Each is under the largest existing package, 27 MB.

Bundle and CI

  • Not re-pinned yet. The bundle test's pinned counts are still main's, so tests/test_chronicle_bundle.py fails on this head.
    • Re-pinning needs a whole-bundle build, which peaks near 15 GB. It waits for other heavy jobs on the machine to finish.
    • A follow-up commit will re-pin the counts and give the build's local peak memory.
  • Memory risk in CI. CI's bundle job notes a peak near 15.6 GB on a 16 GB runner. This PR adds about 326,500 facts, about a fifth more than the bundle's 1,662,950, so that job may run short of memory. The follow-up will say what the local build shows.

Testing

Run before opening:

  • The PR's own test files all pass:
    • tests/test_source_uk_408_{facts,voa,epc,council_tax,mvs_land,stamp}.py;
    • tests/test_chronicle_uk_national_2025.py, tests/test_chronicle_council_tax_nation_totals.py and tests/test_issue_357_microcosm_1095_ports.py.
  • ruff check chronicle policyengine_chronicle db scripts tests is clean.
  • chronicle verify-raw passes for every R2-only registration.
  • At every commit, scripts/build_uk_408_packages.py --check rebuilds that commit's packages byte for byte.

Not yet run locally:

  • The rest of the CI surface is running now, one file per process.
  • tests/test_chronicle_bundle.py waits for the re-pin above.

Refs #408. Item 6 waits on #409.

🤖 Generated with Claude Code

juaristi22 and others added 8 commits November 4, 2026 14:09
chronicle#408 item 1: all eight data sheets of MHCLG's 2023 workbook (valuation date
1 October 2023), 10,101 facts, and the generator scripts/build_uk_408_packages.py that writes
the issue's packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 3: tables CTSOP1.0 to 4.0 of the 2026 summary workbook at every
geography they print (222,588 facts in three packages), and the three small-area zips
(CTSOP1.1, 3.1 and 4.1) registered raw-only, in R2 and git-ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 7: domestic certificates lodged, total floor area and habitable rooms
by quarter for England and Wales, each nation, region and local authority (75,051
facts), and dwelling stock density by local authority, 2011 to 2025 (5,406 facts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 4: MHCLG's Tables 6a and 9a for every English billing authority, and
the per-authority rows of the pinned Welsh Table 2 (both years) and SLGFS Tables 2.8 and
2.11. Rows that restate ported facts (Table 6a's England row, QRC4 line 1) are asserted
equal, not ported again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 items 8 and 2, which share a generator section and test module: land
(AN.211) for every sector in both pinned balance sheet workbooks, ONS's 2022 series for
land underlying dwellings, and the Market Value Survey 2024 annex tables the issue
lists. Chapter 4 rows that restate chapter 1 are asserted equal, not ported again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 5: more tables from the pinned Annual Stamp Tax Statistics 2024-25
workbook, both financial years, in hmrc-stamp-tax-statistics-2024-25 (352 to 1,487
facts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 9: preprocessing/config.py, webapp/cells.json and LICENSE of
DanNeidle/lvt_model_2026 at commit fef94f9 (MIT), as a raw-only registration with no
facts, kept in R2 only and git-ignored (config.py is third-party Python source).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field contracts and caveats of every item in
docs/data-sources/uk-land-value-inputs-408.md, a chronicle#408 section in the PE UK source
checklist (whose Welsh, Scottish and balance sheet rows now count the added rows), and the
changelog fragment. Item 6 waits on chronicle#409.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in the register (#408)

CI's surface job failed one test on 493fe89: the register's static check
(tests/test_dimension_value_register.py) found 396 values that two or more
of this PR's packages emit but chronicle/dimension_values/canonical_labels.json
does not list. Checking the packages' consumer rows with
chronicle.dimension_labels.dimension_label_issues also found values with no
label, which build-bundle refuses ("Every package must label every dimension
and value it emits"):

- Tables 6a and 9a (QRC4): the six billing authority classes;
- the land value estimates: site_density_scenario,
  new_build_price_percentile, agricultural_land_grade,
  office_accommodation_type and retail_location_type;
- the Market Value Survey's chapters 1, 3 and 4: the tenure columns, house
  or flat, occupancy, length of residence (chapter 3), and home value and
  value bands (chapter 4);
- the two VOA crossed packages: Wales' band I, which the register now lists.

scripts/build_uk_408_packages.py now declares those labels from the printed
text:
- each class takes its subtotal row's wording;
- item 1's values take the sheets' headings, and the percentiles, which the
  sheet heads with bare numbers, are labelled "25th percentile" to "75th
  percentile", with "50th percentile (median)";
- a Market Value Survey value takes the row's wording where the package
  prints it as a row, otherwise the column header as printed, otherwise the
  title of the block that sets it.
A value the register labels another way is left to the register, which a
package may not contradict.

The register gains the 396 values by 735e617's method: only the 14 packages
that emit them were built, at 2023. 384 have one wording and take it. Twelve
are worded two ways and take a label in CHOSEN_LABELS:
- "Yes" and "No" for ehs_dependent_children and ehs_household_disability
  and their mhclg. groupby twins. That is the wording most packages use;
  chapter 1's sample-size rows print "Dependent Children - Yes" and the like.
- gbp_1m_to_2m and gbp_2m_or_more, under mvs_market_value_band and its twin.
  Chapter 1 words them "£1,000,000 to £2,000,000" and "£2,000,000 or more",
  chapter 4 "£1m-£2m" and "£2m or more". With one package each, the rule
  does not decide, so chapter 1's wording is proposed for the maintainer's
  review.
The register now holds 8,553 values in 183 dimensions, up from 8,157 in 160.
It still records dece5e9 as its build commit, since only these packages
were rebuilt.

Checked:
- The static check reports no problem.
- All 18 packages this PR adds or changes build with no label issue.
- Seven of the 14 build byte-identical consumer rows before and after, so
  the register relabels nothing there: VOA bands 2025 and 2026, the Welsh
  dwellings package and both collection packages, the 2025 balance sheet and
  the 2022 series.
- The two VOA crossed packages change only by band I's new label.
- No package gains or loses a fact.
- The council tax and Market Value Survey test files pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi

Copy link
Copy Markdown

Review of 03f9689 (I started at 493fe89, and 03f9689 landed while I was reviewing). I checked each manifest's pin against the live files, did an independent cell parse of each source workbook (not the PR's parser) against the GOV.UK and ONS headline text, and read the package headers, units, periods and geography codes, the library diff, and the CI logs. I did not run anything locally: the PR's own tests ran in CI's surface job and passed there.

Must

  1. tests/test_chronicle_bundle.py (CI bundle job, run at 493fe89): this goes further than the count re-pin the body describes. The job never got as far as the count assertion. The runner "received a shutdown signal" 2h33m into the bundle test step, well inside the job's 300-minute timeout-minutes. On main at 89178aa the same job passes in about 2h20m end to end. That pattern looks like the runner running out of memory, the risk the body raises, but the log can't prove it. Before this leaves draft, the follow-up needs the local peak, and the bundle job has to finish on the 16 GB runner. If the peak is over 16 GB, a re-pin alone won't fix it: the job needs a larger runner, or the bundle needs splitting.

Should

  1. packages/mhclg/live_table_126_dwelling_density_2026/source_package.yaml: LT_126 marks 2022 to 2025 as [p] provisional and 2024 as [r] revised. The package keeps those markers only in the expected column header and the evidence notes, so the facts carry nothing a consumer can filter on. Example: Westminster's 2025 density (62.15 dwellings per hectare, England 1.98) looks the same as a final 2011 value. Main already has a way to do this: shared_filters: publication_status: preliminary in ons-national-balance-sheet-land-preliminary-2026, and publication_status: provisional in welshgov-dwelling-stock-by-tenure-2025. Each year is its own record set here, so a publication_status: provisional shared filter on the 2022 to 2025 sets is a small change in the generator.
  2. Barnsley and Sheffield codes. Keeping the printed codes is right. But the mapping UK land value inputs: MHCLG land value estimates 2023, the Market Value Survey 2024, VOA's 2026 stock of properties, council tax by billing authority, more HMRC stamp tax tables, and UCL's Price Paid–EPC file #408 asked for is recorded only in YAML comments and the docs, and this PR adds packages from the same release that disagree. For example, mhclg-council-tax-collection-by-authority-2025-26 (Table 6a) prints Barnsley as E08000016, with GBP 157,820 thousand collectable in 2025-26. Table 9a, VOA 2026 and main's mhclg-council-tax-levels-england-2026-27 all use E08000038. A consumer joining Table 6a to the levels or the VOA stock on geography_id silently drops both authorities. Within Tables 6a and 9a, the shared mhclg.billing_authority dimension also ends up with two values for each of them. For context, a gap that is already on main: chronicle/geography/uk_canonical_names.json has E08000016 and E08000019 but not E08000038 or E08000039, because scripts/build_uk_geography_register.py reads the December 2024 ONS lookups. To keep the mapping in the publisher's terms, the cleanest fix is to pin ONS's own record of the change (a December 2025 LAD lookup or the Code History Database) and rerun the register script. That would also add the names for the new codes. If that's out of scope here, one table in docs/data-sources/uk-land-value-inputs-408.md listing which code each package carries would at least make the split visible in one place.

Checked and fine

  • The register: at 493fe89 the surface job's one failure (1 failed, 5,767 passed) was test_the_static_check_passes_on_the_repository, with 397 shared values unlisted. 03f9689 adds them, along with the missing package labels. Twelve CHOSEN_LABELS entries are proposed for the maintainer. CI is rerunning on 03f9689.
  • Pins: each committed workbook's sha256 equals its manifest. The live GOV.UK URLs for the land value estimates, Table 6, Live Table 126 and the VOA summary tables, and TPA's LICENSE at fef94f9, return the same bytes today. Table 6 and 9 URLs are the 31 July revision listed on the collection rates page. HMRC stays on the 11 March 2026 _correction file. The CTSOP 1.1/3.1/4.1 URLs and sizes match the VOA page's attachments. For licences, the manifests state OGL v3.0 from the GOV.UK/ONS page footers, and MIT from TPA's LICENSE ("Copyright (c) 2026 Tax Policy Associates Limited").
  • VOA CTSOP1.0, as at 31 March 2026: England and Wales 27,438,780, Wales 1,493,990, England band A 6,123,220 of 25,944,800. These match the commentary's 27.4 million, Wales 5.4% of stock, and England band A 6.1 million (23.6%). Band D for England is 4,075,770. Barnsley is under E08000038.
  • MHCLG land value estimates, valuation date 1 October 2023: Westminster low density at the 25th percentile is GBP 41,660,000 per hectare. Cumberland's floor-adjusted median and 60th percentile are both GBP 460,000 at every density. The pre-floor sheet prints - (for example at Blackpool's and Burnley's 25th percentile), and the package header says those cells emit no fact. The floor-adjusted values are at or above the pre-floor ones in the rows I sampled (Stockport, Liverpool, Hyndburn).
  • MHCLG Table 6a, England: GBP 42,014,522 thousand and 44,981,384 thousand collectable, and 40,281,988 thousand and 43,002,543 thousand collected (95.6% in 2025-26). QRC4 England line 8 equals lines 3 + 6 + 7 (43,877,960), so leaving the formula lines out loses nothing. That identity also treats line 7 as next year's receipts, so following the 2026-27 column heading over the form text looks right.
  • HMRC 2024-25 (England and Northern Ireland): Table 3a's GBP 10,380m residential, 3,505m non-residential, 13,885m total and 1,049,600 residential transactions match the commentary. Table 9a/9d's 155,400 First Time Buyers' Relief transactions match too. Units: GBP million with value_scale: 1e6, except average relief, which is in GBP.
  • ONS balance sheet 2025: 2024 total land is GBP 7,117,771m, against the bulletin's "£7.1 trillion". Households and NPISH come to 4,609,744m (bulletin "£4.6 trillion"). The whole economy less households and non-financial corporations is 454,887m, as the body says. In the preliminary 2026 package, the sectors add up to the total.
  • Market Value Survey, valuation 1 July 2024: AT1.1a's all-dwellings mean is GBP 347.7 thousand and the median 275, against the main report's "£348,000 and ... £275,000". The owner occupier (398/320) and private renter (289/235) means and medians match too. Values are GBP thousands (value_scale: 1000).
  • EPC D3: England's 2025 quarters add up to the calendar-year row (1,651,150 lodgements), and the 2026 Q1 to Q3 record sets are present.
  • Periods: VOA counts are as at 31 March 2026 (month 2026-03), the land value estimates October 2023, collection data fiscal_year, and QRC4 lines 9 to 12 quarters. The ONS 2022 series is GBP trillion to one decimal place (value_scale: 1e12). Its "may not add up" note is kept, and nothing is derived from it.
  • The library diff only adds SOURCE_PACKAGE_ALIASES and UK_BUNDLE_SOURCES entries. The test edits keep the old counts as subsets, and the HMRC parser change uses main's existing ods_numeric_text_used_range. Raw-only registrations (the CTSOP zips and TPA's files) are git-ignored, with R2 keys. Tests: CI's surface job ran the six test_source_uk_408_* files, and they pass.

CI: at 493fe89, Chronicle surface failed on the register test (fixed in 03f9689) and Chronicle bundle failed when the runner shut down mid-test (item 1). Chronicle checks failed because it requires both (needs: [bundle, surface]). At 03f9689, bundle and surface are still running.

Verdict: the sources, pins and spot-checked figures hold up. The register is fixed. The bundle job still has to pass on the CI runner, and there are two smaller points on the provisional flags and the Barnsley/Sheffield codes, before it leaves draft.

🤖 Generated with Claude Code

juaristi22 and others added 2 commits November 4, 2026 18:51
… test

test_the_chosen_labels_are_the_registers_labels pins how many labels
scripts/build_dimension_value_register.py chooses. 03f9689 adds twelve for
the Market Value Survey 2024 packages: eight by the rule, four value bands
for the maintainer. That brings the count from 47 to 59, which is the
surface job's one failure at 03f9689 (1 failed, 5,770 passed). The test's
other check, that each chosen label is the register's label, holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…heffield's codes (#408)

Two points from the review of 03f9689:

- Live Table 126 marks 2022 to 2025 '[p]' provisional. Those years'
  record sets now carry shared_filters publication_status: provisional, as
  welshgov-dwelling-stock-by-tenure-2025 does, so a consumer can filter on
  it. The register already labels the value "Provisional". '[r]' (2024)
  stays in the measure header and notes. The fact count is unchanged.
- docs/data-sources/uk-land-value-inputs-408.md gains a table of the codes
  each package carries for Barnsley and Sheffield, E08000016 and E08000019
  or E08000038 and E08000039, including main's council tax levels package.
  The doc says that a join on geography_id drops both authorities unless
  the consumer maps one pair to the other.

Checked: scripts/build_uk_408_packages.py regenerates the package, and
tests/test_source_uk_408_epc.py and tests/test_dimension_value_register.py
pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks, Vahid. Taking the three points in order:

1. Bundle job (Must). Agreed: a re-pin alone won't fix it.

  • Main's own bundle step peaked at 15.4 GB on the 16 GB runner in today's run on main. On UK occupation facts: APS employment by SOC 2020 major and sub-major group, and ASHE earnings by two-digit occupation (PolicyEngine/microcosm#1490) #407's branch it peaked at 15.7 GB.
  • This PR adds about 326,500 facts, about a fifth more, so at the same rate it would need roughly 18 to 19 GB.
  • I'll measure the local peak next. The choice between a larger runner and cutting or splitting the bundle's peak memory comes before this leaves draft.
  • CI's rerun on 03f9689 failed the same way, so that's the second data point. In both runs the runner was shut down during test_build_bundle_writes_merged_consumer_contract, the whole-bundle build, after 2h33m and 3h04m. On main that test takes 2h14m. Neither run uploaded the bundle reports.

2. Live Table 126's provisional years (Should). Done in 160b4f9.

  • The 2022 to 2025 record sets now carry shared_filters: publication_status: provisional, as welshgov-dwelling-stock-by-tenure-2025 does. The register's label for that value is "Provisional".
  • [r] (2024) stays in the measure header and notes, with no filter of its own.
  • The fact count is unchanged.

3. Barnsley's and Sheffield's codes (Should). Done in 160b4f9.

  • docs/data-sources/uk-land-value-inputs-408.md now has a table that lists which pair of codes each package carries. Main's council tax levels package is included.
  • The doc says plainly that a join on geography_id drops both authorities unless the consumer maps one pair to the other.
  • It also notes that the mhclg.billing_authority values differ between Tables 6a and 9a.
  • Pinning ONS's own record of the change and rerunning scripts/build_uk_geography_register.py would also give the new codes their names. ONS's record could be the December 2025 LAD lookup or the Code History Database. That change touches the shared geography register and every package that names these areas, so I'd make it its own PR rather than fold it in here.

…anch

Both add packages to the same shared files. Keep both checklist sections,
#407's first as main has it, and count the 60 chosen labels: the 47 from
2026-10-26, #407's soc2020_major_group=3 and #408's twelve. The register
merged cleanly and its static check passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#414 writes and counts each package's rows as it reads them, so the
bundle no longer holds every row at once. The 16 GB CI runner could not
fit this branch's bundle without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 changed the base branch from main to bundle-stream-consumer-rows November 4, 2026 23:16
The bundle job now fits on the runner (3,578,392 KiB peak with #414's
streaming build) and failed only on the pinned counts. The new pins come
from that run's uploaded reports: 326,495 facts, 12 packages and source
tables, 275 geography keys and 46 period keys. Each change is attributed
per package from the packages' own facts reports, and the attributions
sum to the bundle's changes. Semantic duplicate keys (2,324) and warnings
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi

Copy link
Copy Markdown

Round 2 at 7bf09a5. I read the reply and the five commits since 03f9689 (04607e1, 160b4f9, the main merge and the restack, 7bf09a5), diffed the PR against its new base bundle-stream-consumer-rows (#414's own changes are not re-reviewed here), read the bundle job log at head, and checked the twelve proposed CHOSEN_LABELS against the three Market Value Survey workbooks. Nothing was run locally.

Round 1 items

  1. Bundle job (Must): fixed. On Write and count bundle rows per package instead of holding them all #414's streaming build the job finishes at 7bf09a5. test_build_bundle_writes_merged_consumer_contract passes in 11,348s, and the step's maximum resident set size is 3,581,556 kB, against the 15.4 GB you measured on main before the streaming build, so the 16 GB runner has plenty of room. The re-pinned counts all reconcile. The per-source deltas (hmrc +1,135, mhclg +101,981, voa +222,588, ons +387, scotgov +96, welshgov +308) add up to 326,495, and each one equals the sum of its packages' counts in the body. 1,735,989 + 326,495 = 2,062,484. There are 458 + 12 = 470 packages and 400 + 12 = 412 source tables. The geography split (109 + 109 + 25 + 25 + 3 + 1 + 3) comes to 275. The period split (45 EPC D3 quarters from 2008-Q4 to 2019-Q4, plus month 2015-10) comes to 46. The fiscal-year increments (513 + 1,368 + 5,303) equal 4,545 + 1,208 + 1,027 + 308 + 96.
  2. Live Table 126 provisional years (Should): fixed in 160b4f9. Exactly the four record sets for 2022 to 2025 carry shared_filters: publication_status: provisional and the matching constraint, and 2011 to 2021 carry neither. The package's dimension_labels and the generator agree.
  3. Barnsley and Sheffield codes (Should): partly, and that's fine for this PR. The new table in docs/data-sources/uk-land-value-inputs-408.md matches the packages. I grepped six of them: Table 6a, the land value estimates and Live Table 126 carry E08000016 in their facts, Table 9a and VOA bands carry E08000038, and EPC D3 carries E08000016 to 2025-Q4 and E08000038 from 2025-Q4, with both in 2025-Q4. The other code appears only in header comments. Doing the register change in its own PR makes sense. Nothing in the repo tracks it yet, though: a search for E08000038 or Barnsley finds only UK land value inputs: MHCLG land value estimates 2023, the Market Value Survey 2024, VOA's 2026 stock of properties, council tax by billing authority, more HMRC stamp tax tables, and UCL's Price Paid–EPC file #408. Could you open an issue for pinning the ONS December 2025 lookup or the Code History Database, and link it from that doc section, so it outlives this PR?

New findings

Nit

  1. scripts/build_dimension_value_register.py, the comment above the mvs_market_value_band entries: it says Chapter 4 prints "£1m-£2m" and "£2m or more". That is right for AT4.1 and AT4.2. But Chapter 4 also prints "£2m plus" as the column header in AT4.3 and AT4.4, and the package files it under gbp_2m_or_more too (its own header comment says so). Adding "and '£2m plus'" would give the maintainer the full set of printed wordings.

Checked and fine

  • The twelve CHOSEN_LABELS are factually accurate. In Chapter 1, the value rows print "Yes" and "No" (six each), and the sample-size rows print "Dependent Children - Yes" and "Disability or long term illness - Yes". Chapter 4's value rows print "Yes" and "No". So "Yes" and "No" are the wording most packages use, under the harness rule. Chapter 1 prints "£1,000,000 to £2,000,000" and "£2,000,000 or more", alongside "£700,000 to £1,000,000". Chapter 4 prints "£1m-£2m" and "£2m or more". That is one MHCLG package each, so the rule's statistics-office tie-break doesn't decide it, and leaving it to the maintainer is the right call. Chapter 3 prints Chapter 1's wording but ports no value-band filter. None of the four dimensions was in the register on the base, so this PR adds every wording. The register test's count of 60 (47 + 1 + 12) holds.
  • The library diff against the new base is still only the UK_BUNDLE_SOURCES and SOURCE_PACKAGE_ALIASES entries for the twelve packages, and none of Write and count bundle rows per package instead of holding them all #414's streaming code.

CI at 7bf09a5: Chronicle bundle passed (3h16m), surface passed (1h49m), and checks passed. The bundle test now takes 3h09m. For comparison, #414's whole bundle job took 2h55m, and the test took about 2h14m on main before streaming. That is inside the 300-minute limit, but the workflow's own comment records runner speed varying 1.6x, so it's worth watching.

Verdict: all three round-1 points are dealt with (the geography register fix is deferred to its own PR, which needs an issue). The counts reconcile, and the label proposals match the sources. Only the nit is left from my side.

🤖 Generated with Claude Code

juaristi22 and others added 2 commits November 5, 2026 11:51
Review nit on #410: Chapter 4 of the Market Value Survey 2024 prints
"£2m or more" in AT4.1 and AT4.2 but "£2m plus" as the column header in
AT4.3 and AT4.4, and the package files both under gbp_2m_or_more. The
comment above the proposed labels now names every printed wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review on #410, round 2, item 3: Chronicle keeps the codes each table
prints, and the consumer maps one vintage to the other. Microcosm's
local-authority crosswalk already lists E08000038 and E08000039 as aliases
of E08000016 and E08000019 (PolicyEngine/microcosm#931, item (e)), so the
doc section now says so instead of pointing at a Chronicle register change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks, Vahid.

@vahid-ahmadi

Copy link
Copy Markdown

Round 3 at d2737e6. I read the reply and the two commits since 7bf09a5 (fcbe61d, d2737e6), diffed them against round 2 (the base bundle-stream-consumer-rows has not moved, so #410's own diff is just these two files), and checked the Microcosm crosswalk the doc now links to. Nothing was run locally.

Round 2 items

  1. Barnsley and Sheffield codes: fixed, by a different route. Keeping printed codes in Chronicle and mapping them in the consumer is reasonable, and the mapping exists. Microcosm's tools/generate_uk_local_area_crosswalk.py has LOCAL_AUTHORITY_CODE_ALIASES listing E08000038 and E08000039 as lad_2025 aliases of E08000016 and E08000019, and the generated local_area_crosswalk.json carries them. Item (e) of microcosm#931 is ticked. The new paragraph in docs/data-sources/uk-land-value-inputs-408.md says this and links there, and the section already records that the name register lacks the new codes. No Chronicle issue needed.
  2. Nit, "£2m plus": fixed in fcbe61d. scripts/build_dimension_value_register.py:157-161 now names "£1m-£2m" and "£2m or more" (AT4.1, AT4.2) and "£2m plus" (AT4.3, AT4.4).
  • Bundle test time: open as a watch item, and your plan (an issue if a run gets within half an hour of the 300-minute limit) covers it.

Checked and fine

  • Both commits are a comment and a docs paragraph. No code, package or count changes.
  • A small point outside this PR: the doc says ONS recoded the two authorities "after a boundary change", and your reply says the same. The comment above Microcosm's LOCAL_AUTHORITY_CODE_ALIASES says "Nothing else about the two authorities changed (microcosm#929)". If the boundary did move, that Microcosm comment may be worth a word next time you are in that file. It doesn't affect anything here.
  • Thanks for the GiB correction on main's peak. It doesn't change round 2's conclusion.

CI at d2737e6: Chronicle bundle and Chronicle surface are still running (started 12:15 UTC; the bundle took 3h16m at 7bf09a5). I'll treat a pass there as the last check.

Verdict: nothing left from my side beyond the timing watch item and CI finishing.

🤖 Generated with Claude Code

@juaristi22
juaristi22 marked this pull request as ready for review November 5, 2026 14:42
@juaristi22
juaristi22 merged commit 41c5b37 into bundle-stream-consumer-rows Nov 5, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants