Repository navigation
Document UK breakdown results' withheld area codes, and stop serving breakdowns cached without them - #4104
Draft
anth-volk wants to merge 3 commits into
Draft
Document UK breakdown results' withheld area codes, and stop serving breakdowns cached without them#4104anth-volk wants to merge 3 commits into
anth-volk wants to merge 3 commits into
Conversation
The simulation API is adding, to each level of a completed `breakdown=local_areas` result, the sorted ONS codes of the areas the run estimated but disclosure control withheld: `constituency_withheld_codes` and `local_authority_withheld_codes` (PolicyEngine/policyengine-sim-api#968). `[]` means none were withheld; null or an absent field means the result doesn't report them. The Public API stores and returns a breakdown result unchanged once its `kind` is `local_areas`, so the fields need no code here. These tests record that: - The route-level test that submits a breakdown, polls it to completion and then reads it back from storage now runs for a result with the codes (one level listing two, the other `[]`), one with null for both, and one without the fields. Each answer carries the simulation API's fields unchanged, plus only `policyengine_bundle` and `resolved_app_name`. - The simulation gateway contract test also polls a result with the codes over HTTP, and the client returns it unchanged. They describe existing behaviour and passed when written. To check that they detect a regression, `_with_policyengine_bundle` was changed temporarily to drop keys ending in `_withheld_codes`: the withheld-codes and null-withheld-codes cases failed on the unchanged-fields assertion, and the case without the fields passed. The change was reverted. Refs #4091 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Breakdown results cached before the simulation API listed each level's withheld area codes (PolicyEngine/policyengine-sim-api#968) would be served for up to 30 days without them. A simulation API deploy that keeps the same .py version doesn't change any part of the cache identity, and raising `RUNTIME_CACHE_SCHEMA_VERSIONS["economy_impact"]` would recompute every cached economy result, not just breakdowns. `_build_options_hash` now adds `breakdown_schema=2` to the hash of a request with a breakdown, after the version fields and before `runtime_app_name`, as it adds `spm_provenance_schema` for an SPM selection. The number is `LOCAL_AREA_BREAKDOWN_RESULT_SCHEMA_VERSION` in `economy_impact_types`, whose comment says when to raise it. The cache lookup matches the exact hash or hashes that start with it, and an earlier breakdown hash doesn't start with the new one, so an earlier breakdown result is never selected and the next request submits a new job. Requests without a breakdown get no new field, so their hashes are byte-for-byte the same and their cache entries are still read. Tests, written first: - `test_options_hash_marks_the_breakdown_result_schema` pins the hash of a breakdown with a resolved worker application. - The pinned UK breakdown hash used by the submission, lookup-matching and Northern Ireland tests now ends in `&breakdown_schema=2`. - `test_a_breakdown_cached_before_the_withheld_codes_is_recomputed_but_not_national` stores a completed breakdown under its earlier hash and a completed national result under its hash. The national request is answered from its entry; the breakdown request submits a new job and stores it under the new hash. - `test_requests_without_a_breakdown_keep_their_stored_hashes` pins, as written-out strings, the stored hashes of UK national, Scotland and Northern Ireland requests and of US national and state requests. Before the change, the first three groups failed: the hashes had no `breakdown_schema=2`, and the earlier breakdown result was returned with status ok instead of a new job being submitted. The unchanged-hash cases passed, as they should before and after. Refs #4091 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The annual economy route's OpenAPI description now declares `constituency_withheld_codes` and `local_authority_withheld_codes` in its completed result, as nullable arrays of strings marked "breakdown=local_areas only". Each description says the list holds the sorted ONS codes of the areas the run estimated but disclosure control withheld, which the level's impact list leaves out; that `[]` means none were withheld; that null or an absent field means the result doesn't report them, as in results computed before the simulation API reported them; and that an area the run never estimated isn't listed. The 200 description names the two fields among a breakdown result's fields and says that a breakdown cached before the simulation API reported them is not returned, and the next request computes it again. `docs/canonical-spm.md` describes the same fields, says the Public API stores and returns them unchanged, and explains the `breakdown_schema=2` field in a breakdown's options hash. `test_annual_economy_result_declares_the_breakdown_withheld_area_codes` was written first and failed because the 200 description didn't name `constituency_withheld_codes`; it passes with the new description. Refs #4091 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4104 +/- ##
===========================================
+ Coverage 50.82% 87.84% +37.01%
===========================================
Files 254 270 +16
Lines 14635 16061 +1426
Branches 2339 2627 +288
===========================================
+ Hits 7438 14108 +6670
+ Misses 6570 1284 -5286
- Partials 627 669 +42 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4091
Summary
The simulation API is adding two fields to a completed UK
breakdown=local_areasresult (kind: local_areas) in PolicyEngine/policyengine-sim-api#968:constituency_withheld_codesandlocal_authority_withheld_codes. Each is a sorted list of the ONS codes of that level's areas that the run estimated but disclosure control didn't publish.[]means none were withheld;nullor an absent field means the result doesn't report them, as in older results.kindislocal_areas(_breakdown_result_kind_errorinservices/economy_impact_execution.py); the service only addspolicyengine_bundleandresolved_app_name, as it does to every economy result. The route-level storage test now runs for a result with the codes, one withnullcodes and one without them, and the simulation gateway contract test polls a result with the codes over HTTP._build_options_hashnow addsbreakdown_schema=2(LOCAL_AREA_BREAKDOWN_RESULT_SCHEMA_VERSION) to a request with a breakdown, after the version fields and beforeruntime_app_name, as it addsspm_provenance_schemafor an SPM selection. The lookup matches the exact hash or hashes that start with it, and an earlier breakdown hash doesn't start with the new one, so the next breakdown request submits a new job. Requests without a breakdown get no new field: their hashes are byte-for-byte the same and their cache entries are still read.RUNTIME_CACHE_SCHEMA_VERSIONS["economy_impact"]is not raised, since that would recompute every cached economy result.[],nulland absence mean, and says a breakdown cached before the fields existed is computed again.docs/canonical-spm.mddescribes the same, including thebreakdown_schema=2hash field.For a client, an existing breakdown report shows
computingonce more after this deploys, then completes with the new fields.Merge order
Testing
The work was done red-green: each behaviour change's tests were written first and seen failing before the code, and committed together with it.
Test that breakdown results keep their withheld area codes: characterization tests of existing behaviour, so they passed when written. To check they detect a regression,_with_policyengine_bundlewas temporarily changed to drop keys ending in_withheld_codes; the withheld-codes and null-withheld-codes cases then failed, and the case without the fields passed. The change was reverted.Stop serving breakdown results cached without withheld area codes: before the code change, the pinned breakdown-hash tests failed (nobreakdown_schema=2), andtest_a_breakdown_cached_before_the_withheld_codes_is_recomputed_but_not_nationalfailed because the earlier breakdown result was returned with statusokinstead of a new job being submitted.test_requests_without_a_breakdown_keep_their_stored_hashespins, as written-out strings, the stored hashes of UK national, Scotland and Northern Ireland requests and US national and state requests, and passed before and after.Describe breakdown results' withheld area codes in the OpenAPI document:test_annual_economy_result_declares_the_breakdown_withheld_area_codesfailed until the description was added.Run locally on Python 3.12 (focused, not the full suite):
FLASK_DEBUG=1 uv run --frozen pytestontests/unit/services/test_economy_service_request_plumbing.py,test_economy_service_simulation_options.py,test_economy_service_stored_impacts.py,test_economy_service_annual_cache_identity.py,test_economy_service_annual_simulation_requests.py,test_worker_spm.py,test_economy_impact_types.py,tests/unit/routes/test_economy_spm_query.py,tests/unit/test_openapi_spm.py,tests/unit/test_openapi_report_routes.py,tests/unit/test_constants.pyandtests/contract/test_simulation_gateway_contract.py: all pass.make format,ruff checkon the changed Python files,uv run --frozen --extra dev mypyandpython scripts/run_quality_guards.py: pass.No inline ruff suppressions are added.
🤖 Generated with Claude Code