Skip to content

Document UK breakdown results' withheld area codes, and stop serving breakdowns cached without them - #4104

Draft
anth-volk wants to merge 3 commits into
masterfrom
feat/uk-breakdown-withheld-codes
Draft

anth-volk wants to merge 3 commits into
masterfrom
feat/uk-breakdown-withheld-codes

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4091

Summary

The simulation API is adding two fields to a completed UK breakdown=local_areas result (kind: local_areas) in PolicyEngine/policyengine-sim-api#968: constituency_withheld_codes and local_authority_withheld_codes. Each is a sorted list of the ONS codes of that level's areas that the run estimated but disclosure control didn't publish. [] means none were withheld; null or an absent field means the result doesn't report them, as in older results.

  • Pass-through, no code change. The Public API already stores and returns a breakdown result unchanged once its kind is local_areas (_breakdown_result_kind_error in services/economy_impact_execution.py); the service only adds policyengine_bundle and resolved_app_name, as it does to every economy result. The route-level storage test now runs for a result with the codes, one with null codes and one without them, and the simulation gateway contract test polls a result with the codes over HTTP.
  • Breakdown results cached without the codes are recomputed. A simulation API deploy that keeps the same .py version doesn't change the cache identity, so breakdowns cached before the change would be served for up to 30 days without the fields. _build_options_hash now adds breakdown_schema=2 (LOCAL_AREA_BREAKDOWN_RESULT_SCHEMA_VERSION) to a request with a breakdown, after the version fields and before runtime_app_name, as it adds spm_provenance_schema for an SPM selection. The lookup matches the exact hash or hashes that start with it, and an earlier breakdown hash doesn't start with the new one, so the next breakdown request submits a new job. Requests without a breakdown get no new field: their hashes are byte-for-byte the same and their cache entries are still read. RUNTIME_CACHE_SCHEMA_VERSIONS["economy_impact"] is not raised, since that would recompute every cached economy result.
  • Documentation. The annual economy route's OpenAPI description declares both fields as nullable string arrays, with what [], null and absence mean, and says a breakdown cached before the fields existed is computed again. docs/canonical-spm.md describes the same, including the breakdown_schema=2 hash field.

For a client, an existing breakdown report shows computing once more after this deploys, then completes with the new fields.

Merge order

  • Merge after the simulation API's executor change, Fill the withheld-area code lists in UK local-area breakdown results policyengine-sim-api#975, is deployed to the executor for the .py version the Public API pins. The simulation API's contract and gateway change, which only accepts the fields, is not enough: until the executor fills them, breakdowns recomputed after this change come back without the codes and stay cached under the new hash for 30 days. Executors for older .py versions never change, so the same applies if Update PolicyEngine UK to 0.62.1 #975 reaches only a different .py version's executor.
  • Like the rest of this work, merge after the UK Budget on 28 October 2026.

Testing

The work was done red-green: each behaviour change's tests were written first and seen failing before the code, and committed together with it.

  • Test that breakdown results keep their withheld area codes: characterization tests of existing behaviour, so they passed when written. To check they detect a regression, _with_policyengine_bundle was temporarily changed to drop keys ending in _withheld_codes; the withheld-codes and null-withheld-codes cases then failed, and the case without the fields passed. The change was reverted.
  • Stop serving breakdown results cached without withheld area codes: before the code change, the pinned breakdown-hash tests failed (no breakdown_schema=2), and test_a_breakdown_cached_before_the_withheld_codes_is_recomputed_but_not_national failed because the earlier breakdown result was returned with status ok instead of a new job being submitted. test_requests_without_a_breakdown_keep_their_stored_hashes pins, as written-out strings, the stored hashes of UK national, Scotland and Northern Ireland requests and US national and state requests, and passed before and after.
  • Describe breakdown results' withheld area codes in the OpenAPI document: test_annual_economy_result_declares_the_breakdown_withheld_area_codes failed until the description was added.

Run locally on Python 3.12 (focused, not the full suite):

  • FLASK_DEBUG=1 uv run --frozen pytest on tests/unit/services/test_economy_service_request_plumbing.py, test_economy_service_simulation_options.py, test_economy_service_stored_impacts.py, test_economy_service_annual_cache_identity.py, test_economy_service_annual_simulation_requests.py, test_worker_spm.py, test_economy_impact_types.py, tests/unit/routes/test_economy_spm_query.py, tests/unit/test_openapi_spm.py, tests/unit/test_openapi_report_routes.py, tests/unit/test_constants.py and tests/contract/test_simulation_gateway_contract.py: all pass.
  • make format, ruff check on the changed Python files, uv run --frozen --extra dev mypy and python scripts/run_quality_guards.py: pass.

No inline ruff suppressions are added.

🤖 Generated with Claude Code

anth-volk and others added 3 commits October 28, 2026 15:37
The simulation API is adding, to each level of a completed
`breakdown=local_areas` result, the sorted ONS codes of the areas the run
estimated but disclosure control withheld:
`constituency_withheld_codes` and `local_authority_withheld_codes`
(PolicyEngine/policyengine-sim-api#968). `[]` means none were withheld;
null or an absent field means the result doesn't report them.

The Public API stores and returns a breakdown result unchanged once its
`kind` is `local_areas`, so the fields need no code here. These tests
record that:

- The route-level test that submits a breakdown, polls it to completion
  and then reads it back from storage now runs for a result with the
  codes (one level listing two, the other `[]`), one with null for both,
  and one without the fields. Each answer carries the simulation API's
  fields unchanged, plus only `policyengine_bundle` and
  `resolved_app_name`.
- The simulation gateway contract test also polls a result with the
  codes over HTTP, and the client returns it unchanged.

They describe existing behaviour and passed when written. To check that
they detect a regression, `_with_policyengine_bundle` was changed
temporarily to drop keys ending in `_withheld_codes`: the withheld-codes
and null-withheld-codes cases failed on the unchanged-fields assertion,
and the case without the fields passed. The change was reverted.

Refs #4091

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Breakdown results cached before the simulation API listed each level's
withheld area codes (PolicyEngine/policyengine-sim-api#968) would be
served for up to 30 days without them. A simulation API deploy that keeps
the same .py version doesn't change any part of the cache identity, and
raising `RUNTIME_CACHE_SCHEMA_VERSIONS["economy_impact"]` would recompute
every cached economy result, not just breakdowns.

`_build_options_hash` now adds `breakdown_schema=2` to the hash of a
request with a breakdown, after the version fields and before
`runtime_app_name`, as it adds `spm_provenance_schema` for an SPM
selection. The number is `LOCAL_AREA_BREAKDOWN_RESULT_SCHEMA_VERSION` in
`economy_impact_types`, whose comment says when to raise it. The cache
lookup matches the exact hash or hashes that start with it, and an
earlier breakdown hash doesn't start with the new one, so an earlier
breakdown result is never selected and the next request submits a new
job. Requests without a breakdown get no new field, so their hashes are
byte-for-byte the same and their cache entries are still read.

Tests, written first:

- `test_options_hash_marks_the_breakdown_result_schema` pins the hash of
  a breakdown with a resolved worker application.
- The pinned UK breakdown hash used by the submission, lookup-matching
  and Northern Ireland tests now ends in `&breakdown_schema=2`.
- `test_a_breakdown_cached_before_the_withheld_codes_is_recomputed_but_not_national`
  stores a completed breakdown under its earlier hash and a completed
  national result under its hash. The national request is answered from
  its entry; the breakdown request submits a new job and stores it under
  the new hash.
- `test_requests_without_a_breakdown_keep_their_stored_hashes` pins, as
  written-out strings, the stored hashes of UK national, Scotland and
  Northern Ireland requests and of US national and state requests.

Before the change, the first three groups failed: the hashes had no
`breakdown_schema=2`, and the earlier breakdown result was returned with
status ok instead of a new job being submitted. The unchanged-hash cases
passed, as they should before and after.

Refs #4091

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The annual economy route's OpenAPI description now declares
`constituency_withheld_codes` and `local_authority_withheld_codes` in its
completed result, as nullable arrays of strings marked
"breakdown=local_areas only". Each description says the list holds the
sorted ONS codes of the areas the run estimated but disclosure control
withheld, which the level's impact list leaves out; that `[]` means none
were withheld; that null or an absent field means the result doesn't
report them, as in results computed before the simulation API reported
them; and that an area the run never estimated isn't listed. The 200
description names the two fields among a breakdown result's fields and
says that a breakdown cached before the simulation API reported them is
not returned, and the next request computes it again.

`docs/canonical-spm.md` describes the same fields, says the Public API
stores and returns them unchanged, and explains the `breakdown_schema=2`
field in a breakdown's options hash.

`test_annual_economy_result_declares_the_breakdown_withheld_area_codes`
was written first and failed because the 200 description didn't name
`constituency_withheld_codes`; it passes with the new description.

Refs #4091

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.84%. Comparing base (c122e76) to head (252fd94).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #4104       +/-   ##
===========================================
+ Coverage   50.82%   87.84%   +37.01%     
===========================================
  Files         254      270       +16     
  Lines       14635    16061     +1426     
  Branches     2339     2627      +288     
===========================================
+ Hits         7438    14108     +6670     
+ Misses       6570     1284     -5286     
- Partials      627      669       +42     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document UK breakdown results' withheld area codes, and stop serving breakdowns cached without them

1 participant