Repository navigation
UK land value inputs for chronicle#408: MHCLG land values, VOA's 2026 stock, EPC D3, council tax by authority, the Market Value Survey, land by sector and HMRC stamp tables - #410
Conversation
chronicle#408 item 1: all eight data sheets of MHCLG's 2023 workbook (valuation date 1 October 2023), 10,101 facts, and the generator scripts/build_uk_408_packages.py that writes the issue's packages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 3: tables CTSOP1.0 to 4.0 of the 2026 summary workbook at every geography they print (222,588 facts in three packages), and the three small-area zips (CTSOP1.1, 3.1 and 4.1) registered raw-only, in R2 and git-ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 7: domestic certificates lodged, total floor area and habitable rooms by quarter for England and Wales, each nation, region and local authority (75,051 facts), and dwelling stock density by local authority, 2011 to 2025 (5,406 facts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 4: MHCLG's Tables 6a and 9a for every English billing authority, and the per-authority rows of the pinned Welsh Table 2 (both years) and SLGFS Tables 2.8 and 2.11. Rows that restate ported facts (Table 6a's England row, QRC4 line 1) are asserted equal, not ported again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 items 8 and 2, which share a generator section and test module: land (AN.211) for every sector in both pinned balance sheet workbooks, ONS's 2022 series for land underlying dwellings, and the Market Value Survey 2024 annex tables the issue lists. Chapter 4 rows that restate chapter 1 are asserted equal, not ported again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 5: more tables from the pinned Annual Stamp Tax Statistics 2024-25 workbook, both financial years, in hmrc-stamp-tax-statistics-2024-25 (352 to 1,487 facts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronicle#408 item 9: preprocessing/config.py, webapp/cells.json and LICENSE of DanNeidle/lvt_model_2026 at commit fef94f9 (MIT), as a raw-only registration with no facts, kept in R2 only and git-ignored (config.py is third-party Python source). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field contracts and caveats of every item in docs/data-sources/uk-land-value-inputs-408.md, a chronicle#408 section in the PE UK source checklist (whose Welsh, Scottish and balance sheet rows now count the added rows), and the changelog fragment. Item 6 waits on chronicle#409. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in the register (#408) CI's surface job failed one test on 493fe89: the register's static check (tests/test_dimension_value_register.py) found 396 values that two or more of this PR's packages emit but chronicle/dimension_values/canonical_labels.json does not list. Checking the packages' consumer rows with chronicle.dimension_labels.dimension_label_issues also found values with no label, which build-bundle refuses ("Every package must label every dimension and value it emits"): - Tables 6a and 9a (QRC4): the six billing authority classes; - the land value estimates: site_density_scenario, new_build_price_percentile, agricultural_land_grade, office_accommodation_type and retail_location_type; - the Market Value Survey's chapters 1, 3 and 4: the tenure columns, house or flat, occupancy, length of residence (chapter 3), and home value and value bands (chapter 4); - the two VOA crossed packages: Wales' band I, which the register now lists. scripts/build_uk_408_packages.py now declares those labels from the printed text: - each class takes its subtotal row's wording; - item 1's values take the sheets' headings, and the percentiles, which the sheet heads with bare numbers, are labelled "25th percentile" to "75th percentile", with "50th percentile (median)"; - a Market Value Survey value takes the row's wording where the package prints it as a row, otherwise the column header as printed, otherwise the title of the block that sets it. A value the register labels another way is left to the register, which a package may not contradict. The register gains the 396 values by 735e617's method: only the 14 packages that emit them were built, at 2023. 384 have one wording and take it. Twelve are worded two ways and take a label in CHOSEN_LABELS: - "Yes" and "No" for ehs_dependent_children and ehs_household_disability and their mhclg. groupby twins. That is the wording most packages use; chapter 1's sample-size rows print "Dependent Children - Yes" and the like. - gbp_1m_to_2m and gbp_2m_or_more, under mvs_market_value_band and its twin. Chapter 1 words them "£1,000,000 to £2,000,000" and "£2,000,000 or more", chapter 4 "£1m-£2m" and "£2m or more". With one package each, the rule does not decide, so chapter 1's wording is proposed for the maintainer's review. The register now holds 8,553 values in 183 dimensions, up from 8,157 in 160. It still records dece5e9 as its build commit, since only these packages were rebuilt. Checked: - The static check reports no problem. - All 18 packages this PR adds or changes build with no label issue. - Seven of the 14 build byte-identical consumer rows before and after, so the register relabels nothing there: VOA bands 2025 and 2026, the Welsh dwellings package and both collection packages, the 2025 balance sheet and the 2022 series. - The two VOA crossed packages change only by band I's new label. - No package gains or loses a fact. - The council tax and Market Value Survey test files pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review of 03f9689 (I started at 493fe89, and 03f9689 landed while I was reviewing). I checked each manifest's pin against the live files, did an independent cell parse of each source workbook (not the PR's parser) against the GOV.UK and ONS headline text, and read the package headers, units, periods and geography codes, the library diff, and the CI logs. I did not run anything locally: the PR's own tests ran in CI's surface job and passed there. Must
Should
Checked and fine
CI: at 493fe89, Chronicle surface failed on the register test (fixed in 03f9689) and Chronicle bundle failed when the runner shut down mid-test (item 1). Chronicle checks failed because it requires both ( Verdict: the sources, pins and spot-checked figures hold up. The register is fixed. The bundle job still has to pass on the CI runner, and there are two smaller points on the provisional flags and the Barnsley/Sheffield codes, before it leaves draft. 🤖 Generated with Claude Code |
… test test_the_chosen_labels_are_the_registers_labels pins how many labels scripts/build_dimension_value_register.py chooses. 03f9689 adds twelve for the Market Value Survey 2024 packages: eight by the rule, four value bands for the maintainer. That brings the count from 47 to 59, which is the surface job's one failure at 03f9689 (1 failed, 5,770 passed). The test's other check, that each chosen label is the register's label, holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…heffield's codes (#408) Two points from the review of 03f9689: - Live Table 126 marks 2022 to 2025 '[p]' provisional. Those years' record sets now carry shared_filters publication_status: provisional, as welshgov-dwelling-stock-by-tenure-2025 does, so a consumer can filter on it. The register already labels the value "Provisional". '[r]' (2024) stays in the measure header and notes. The fact count is unchanged. - docs/data-sources/uk-land-value-inputs-408.md gains a table of the codes each package carries for Barnsley and Sheffield, E08000016 and E08000019 or E08000038 and E08000039, including main's council tax levels package. The doc says that a join on geography_id drops both authorities unless the consumer maps one pair to the other. Checked: scripts/build_uk_408_packages.py regenerates the package, and tests/test_source_uk_408_epc.py and tests/test_dimension_value_register.py pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, Vahid. Taking the three points in order: 1. Bundle job (Must). Agreed: a re-pin alone won't fix it.
2. Live Table 126's provisional years (Should). Done in 160b4f9.
3. Barnsley's and Sheffield's codes (Should). Done in 160b4f9.
|
…anch Both add packages to the same shared files. Keep both checklist sections, #407's first as main has it, and count the 60 chosen labels: the 47 from 2026-10-26, #407's soc2020_major_group=3 and #408's twelve. The register merged cleanly and its static check passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#414 writes and counts each package's rows as it reads them, so the bundle no longer holds every row at once. The 16 GB CI runner could not fit this branch's bundle without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bundle job now fits on the runner (3,578,392 KiB peak with #414's streaming build) and failed only on the pinned counts. The new pins come from that run's uploaded reports: 326,495 facts, 12 packages and source tables, 275 geography keys and 46 period keys. Each change is attributed per package from the packages' own facts reports, and the attributions sum to the bundle's changes. Semantic duplicate keys (2,324) and warnings are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Round 2 at 7bf09a5. I read the reply and the five commits since 03f9689 (04607e1, 160b4f9, the main merge and the restack, 7bf09a5), diffed the PR against its new base Round 1 items
New findings Nit
Checked and fine
CI at 7bf09a5: Chronicle bundle passed (3h16m), surface passed (1h49m), and checks passed. The bundle test now takes 3h09m. For comparison, #414's whole bundle job took 2h55m, and the test took about 2h14m on main before streaming. That is inside the 300-minute limit, but the workflow's own comment records runner speed varying 1.6x, so it's worth watching. Verdict: all three round-1 points are dealt with (the geography register fix is deferred to its own PR, which needs an issue). The counts reconcile, and the label proposals match the sources. Only the nit is left from my side. 🤖 Generated with Claude Code |
Review nit on #410: Chapter 4 of the Market Value Survey 2024 prints "£2m or more" in AT4.1 and AT4.2 but "£2m plus" as the column header in AT4.3 and AT4.4, and the package files both under gbp_2m_or_more. The comment above the proposed labels now names every printed wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review on #410, round 2, item 3: Chronicle keeps the codes each table prints, and the consumer maps one vintage to the other. Microcosm's local-authority crosswalk already lists E08000038 and E08000039 as aliases of E08000016 and E08000019 (PolicyEngine/microcosm#931, item (e)), so the doc section now says so instead of pointing at a Chronicle register change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, Vahid.
|
|
Round 3 at d2737e6. I read the reply and the two commits since 7bf09a5 (fcbe61d, d2737e6), diffed them against round 2 (the base Round 2 items
Checked and fine
CI at d2737e6: Chronicle bundle and Chronicle surface are still running (started 12:15 UTC; the bundle took 3h16m at 7bf09a5). I'll treat a pass there as the last check. Verdict: nothing left from my side beyond the timing watch item and CI finishing. 🤖 Generated with Claude Code |
This PR carries eight of the nine items #408 asks for: about 326,500 new facts and three raw-only registrations. Item 6, UCL's house price per square metre file, needs a way to store files over 300 MiB, and #409 asks for a ruling on that. It will be its own PR.
Each item is its own commit (items 2 and 8 share one, because they share a generator section and test module). The last commit adds the docs. A follow-up commit will re-pin the bundle test's counts (see "Bundle and CI").
scripts/build_uk_408_packages.pywrites every new package, and the record sets added to the pinned Welsh, SLGFS and balance sheet packages, from the pinned bytes.--checkcompares instead of writing.scripts/build_uk_1095_port_packages.py, which already writes that package.docs/data-sources/uk-land-value-inputs-408.md.What it adds
Item 1: MHCLG land value estimates for policy appraisal 2023.
mhclg-land-value-estimates-2023holds 10,101 facts from all eight data sheets (valuation date 1 October 2023). The workbook is the copy PolicyEngine/uk-lvt-paper#14 pins, sha256ca626b7e…8705.model_output. The guidelines say the values "do not represent the market value of land".Residentialis the estimate before MHCLG's floor. Its-cells (negative residuals) emit no fact.Residential (adj for floor)applies the floor.Item 3: VOA Council Tax stock of properties 2026. The summary workbook is the copy PolicyEngine/uk-lvt-paper#14 pins, sha256
9e9be694…29d9.voa-council-tax-bands-2026(8,566 facts) has CTSOP1.0 and 2.0 at every geography printed, including 318 billing authorities and 575 constituencies.voa-council-tax-property-types-2026(120,493 facts) andvoa-council-tax-build-periods-2026(93,529 facts) have CTSOP3.0 and 4.0.Item 7: floor area and dwelling density.
mhclg-epc-d3-domestic-2026-q3(75,051 facts) has EPC D3's lodgements, floor area and habitable rooms. That's every quarter from 2008 Q4 to 2026 Q3 for England and Wales, each nation, region and local authority, plus calendar years for the nations. D3 counts certificates lodged, not dwellings.mhclg-live-table-126-dwelling-density-2026(5,406 facts) has dwellings per hectare by authority, 2011 to 2025.Item 4: council tax collected by billing authority.
mhclg-council-tax-collection-by-authority-2025-26(Table 6a, 1,208 facts) andmhclg-council-tax-qrc4-2025-26(Table 9a, QRC4, 5,757 facts).Items 2 and 8: the Market Value Survey 2024, and land by sector.
mhclg-market-value-survey-2024-chapter-{1,3,4}(4,458 facts) has the annex tables the issue lists.survey_aggregate.ons-land-underlying-buildings-2022(108 facts) is Figure 1 of ONS's 2022 article. It is a pre-revision vintage, with the total as a fourth series.Item 5: HMRC stamp tables.
hmrc-stamp-tax-statistics-2024-25gains Tables 7, 3a, 5a, 9a, 9d and 2e for both years, going from 352 to 1,487 facts.ods_numeric_text_used_range, because Table 5a's 2023-24 sheet stores some numbers as text. The 352 existing facts keep their keys and values.Item 9: Tax Policy Associates' model.
tpa-lvt-model-2026registersconfig.py,cells.jsonandLICENSEat commit fef94f9 (MIT), raw only, with no facts. The files are kept in R2 only, becauseconfig.pyis Python source.Parity checks (the issue's, and some more)
Item 1
Item 3
Item 7
Item 4
Item 8
Item 5 (2024-25; 2023-24 agrees too)
Item 2: structural checks pass. Sample sizes are positive integers, the A and B versions cover the same categories, complete percentage columns sum to 100, and every fact equals its cell in an independent parse.
Where the sources differ from the issue
[X]marks fewer than 50.CTSOP_1_1/CTSOP1_1_2026_03_31.csv(2025:CTSOP1.1/…). Microcosm's band-mix builder reads it by name.Choices a reviewer may want to check
entity: firm, with the site type asentity_role; residential ones aredwelling. Land values areaggregation: mean; the guidelines call them "reasonable averages".statistical_scope.Bundle and CI
tests/test_chronicle_bundle.pyfails on this head.Testing
Run before opening:
tests/test_source_uk_408_{facts,voa,epc,council_tax,mvs_land,stamp}.py;tests/test_chronicle_uk_national_2025.py,tests/test_chronicle_council_tax_nation_totals.pyandtests/test_issue_357_microcosm_1095_ports.py.ruff check chronicle policyengine_chronicle db scripts testsis clean.chronicle verify-rawpasses for every R2-only registration.scripts/build_uk_408_packages.py --checkrebuilds that commit's packages byte for byte.Not yet run locally:
tests/test_chronicle_bundle.pywaits for the re-pin above.Refs #408. Item 6 waits on #409.
🤖 Generated with Claude Code