[anth-volk]

Fixes #968

Merge requirement

Don't merge this until #976 is merged and deployed. #976 makes the contract and the deployed gateway accept constituency_withheld_codes and local_authority_withheld_codes, while no executor sends them. This PR makes the executor send them.

The deployment workflow redeploys the executor before the stable gateway (deploy_gateway needs deploy_executor). If this PR deployed first, the old gateway, whose contract forbids unknown keys, would report every breakdown that completed in between as failed. With #976 deployed first, the gateway already accepts the fields.

After #976 merges, main gets merged into this branch. Its diff then shows only the executor commits: the contract commit 762f2c6f is the first commit of both PRs.

What changes

Each level of a UK local-area breakdown result (kind: local_areas) now lists the ONS codes of the areas disclosure control withheld, beside the published records. A client can then tell an area that the run estimated and withheld from one it never estimated, such as an area with no households in the local-area dataset.

The executor fills the fields that #976 adds to LocalAreaBreakdownOutput:

Field	Meaning
constituency_withheld_codes	The ONS codes, sorted, of the constituencies the run estimated but disclosure control did not publish.
local_authority_withheld_codes	The same for local authorities.

• An area the run never estimated is in neither the records nor the list.
• [] means that no area of that level was withheld.
• Executors deployed for older .py versions keep leaving both keys out.

Where the codes come from

The executor computes them as the estimated areas minus the published ones, in UKGeographyDisclosureOutput.withheld_codes:

• Estimated areas: for each level, the area codes that have household weight. compute_uk_geography_disclosure_output already computes them (_uk_area_household_counts), to withhold the unpublished share cells and to take each area's population. This is the set of areas .py's release estimates: AreaWeights.from_labels normalises the codes with normalise_area_codes, treats blank codes as unassigned and keeps only the areas with positive weight.
• Published areas: the areas _published_area_changes turns into records, which are the breakdown's records.

I didn't read the withheld cells from UKGeographyDisclosure.release, because they don't match what the result leaves out:

• An area is published only when both its average and its relative change are public. An area whose relative change alone is withheld, for example because its total income is not positive, still has a public average change in the release, yet the result leaves it out.
• When withholding cells cannot resolve the disclosure findings, the executor publishes no area at all (_resolved_public_table returns None). The release has not marked those cells withheld, and gives no public table to read them from.
• The release keeps its estimates and the cells withheld by hand in private attributes (Release._estimates, Release._manual).

The difference is right in each of these cases. The tests check that the published and withheld areas never overlap and together equal the estimated areas.

Commits

1. 762f2c6f Add optional withheld-area code lists to the local-area breakdown result (contract). This is also Accept optional withheld-area code lists on UK local-area breakdown results #976's first commit, so it leaves this PR's diff once main is merged in. Seen failing first: the contract tests and the entrypoint's OpenAPI test.
2. bfb7e81f Record which estimated UK areas the disclosure output did not publish (executor). UKGeographyDisclosureOutput keeps each level's estimated codes and gains withheld_codes(level). Seen failing first, with AttributeError: 'UKGeographyDisclosureOutput' object has no attribute 'withheld_codes': for both levels, the published and withheld areas are disjoint and together equal the estimated ones; the thin, undefined and complementary seats are listed; a seat whose households all have zero weight is in neither; and every estimated area is withheld when the findings cannot be resolved.
3. 3a71ed3b Report each level's withheld UK areas in the local-area breakdown result (executor). LocalAreaBreakdownBuilder.build fills both fields from the disclosure output, and refuses to build without one. Seen failing first: the serialized breakdown held None instead of ['E14000529', 'E14000531'] and [], and a missing disclosure output raised nothing.
4. d482efea Check the withheld-area lists in the beta local-area breakdown test (apis-integ). assert_uk_local_area_breakdown requires each level's list, sorted, without repeats and without a published area. Seen failing first: the four new rejection cases (each list missing, an unsorted list, a published code in the list) raised nothing.
5. 18d7fc96 Describe the withheld-area code lists in the changelog. Documentation only.
6. 022deac2 Limit the changelog entry to the executor change. Accept optional withheld-area code lists on UK local-area breakdown results #976 has its own entry for the contract and gateway change. This entry sits beside the local-area breakdown entry, so merging main doesn't touch the same text. A trial merge of Accept optional withheld-area code lists on UK local-area breakdown results #976's branch into this one merged cleanly.

The stricter beta check

• test_calculate_uk_local_area_breakdown now fails when a breakdown carries no withheld-code lists. The deployment runs it against the executor it has just deployed, at the pinned UK version, so after this PR deploys the lists are present.
• Stage 12-only deployments: stage12-deploy.yml (deploy_existing_stack: false) runs the integration tests without redeploying the executor. If one ran after this PR merges but before a full deployment of it had succeeded, the beta check would run against the previous executor and fail.

Deployment

• Executors for older .py versions never change. This reaches the 6.10.0 executor only if 6.10.0 is still the pinned version when it deploys.
• Like the rest of this project, it waits to merge until after the UK Budget on 28 October.
• No workflow, Docker image or environment variable changes.

Overlap with #919

#919 also edits macro_output.py (the area records' x and y). After main is merged in, this PR no longer changes that file, so the overlap is #976's.

Checks run

• Focused tests:
  • the contract's test_local_area_breakdown.py and test_gateway_models.py;
  • the entrypoint's test_openapi.py;
  • the gateway's test_openapi_golden.py and test_local_area_breakdown.py;
  • the executor's test_simulation_output_geographic.py, test_simulation_output_uk_local_areas.py, test_simulation_output_impacts.py, test_simulation_api_contracts.py, test_microdata_weight_boundaries.py and test_runtime_evaluated_annotations.py;
  • apis-integ's test_calculate.py without the beta_only tests.
• On a trial merge of Accept optional withheld-area code lists on UK local-area breakdown results #976's branch into this one, the executor's local-area breakdown tests and the contract's breakdown tests pass.
• make check passes: ruff format and lint, and Pyright in every package, with no errors.
• python scripts/run_quality_guards.py with GITHUB_BASE_REF=main passes every guard. simulation_output_geographic.py is now 795 lines, under the 800-line limit.
• The beta test itself runs only against a deployed beta, so it hasn't run.

The work was done red-green: each behaviour change's tests were written first and seen failing for the expected reason, then the code was written, and both were committed together.

🤖 Generated with Claude Code