[anth-volk]

Fixes #786

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

Stage 12 rejected every aggregate report without an SPM calculation, which includes every UK report. Between 6 and 13 October 2026 this caused 42 of the 153 production Stage 12 failures. All 42 were UK nationwide reports whose baseline and reform simulations had already succeeded.

• Cause: SingleYearMacroOutput.spm_provenance is optional and defaults to None, so a report with no SPM calculation serializes "spm_provenance": null. AggregateReportArtifactPayload.validate_spm_provenance checked only that the key was present. It then validated null as an SPMComparisonProvenance object, which failed inside Stage12ArtifactStore.write_aggregate. The coordinator recorded the failure as the generic Stage 12 report coordination failed.
• Fix: the validator now validates spm_provenance only when a value is supplied. An absent or null field is accepted. Every non-null value is still validated, so malformed or mismatched provenance records are still rejected.
• Why the tests missed it: the coordinator tests' artifact double, FakeArtifacts.write_aggregate, saved any payload without validating it. It now writes through the real Stage12ArtifactStore, so the coordinator tests enforce the same aggregate contract as production.
• Deployment: the change is in the contract library used by the Stage 12 coordinator. It takes effect with a stage12-deploy.yml run and needs no change to the selected worker version. The other changes are test-only.

Testing

• Rebased on main after Release Stage 12 database sessions and keep original failures visible #789 and Add a rules registry that makes every enforced and unenforced rule visible #752. The three coordinator tests Release Stage 12 database sessions and keep original failures visible #789 added now use _aggregate_payload too, because one of them aggregates successfully and the real store rejects {"result": "complete"}.
• New contract tests in libs/policyengine-simulation-contract/tests/test_stage12_execution.py:
  • An absent spm_provenance and a null one are both accepted.
  • Present but invalid values ({}, a string, a list, 0) are rejected.
• New executor test in projects/policyengine-simulation-executor/tests/test_stage12_runtime.py:
  • It builds a UK report aggregate with build_aggregate_report, using the real SingleYearMacroOutput serialization.
  • It writes that aggregate with FakeArtifacts.write_aggregate, which now calls the real Stage12ArtifactStore.write_aggregate over an in-memory object store.
  • It checks that the stored artifact keeps spm_provenance: null and matches the returned digest.
• The coordinator test double now validates aggregates:
  • FakeArtifacts.write_aggregate delegates to a real Stage12ArtifactStore, so it runs the same AggregateReportArtifactPayload validation and encoding as production. The stored aggregate path now follows the production layout under the run's artifact prefix, and one coordinator test checks that path.
  • With validation on, 10 coordinator tests failed because their stand-in aggregators returned payloads that never met the contract, such as {"result": "complete"}. They now return a valid payload from a _aggregate_payload helper, with the same result as before.
  • The stand-in aggregators that must not run also return a valid payload. If one ran, its aggregate would be saved and the existing aggregate_writes == [] checks would fail, rather than a validation error hiding the run.
• Shared test support, following docs/engineering/skills/testing.md:
  • fixtures/fake_object_store.py holds InMemoryObjectStore. It replaces the two identical copies in test_stage12_artifacts.py and test_stage12_runtime.py.
  • fixtures/macro_output.py, renamed from fixtures/test_simulation_api_contracts.py, holds the shared macro result and its key sets. The Stage 12 runtime, output builder and standalone app tests no longer import the API contract tests' fixture module.
• The new tests reproduce the defect. With the previous validator, the null-provenance contract case and the UK aggregate test both fail with ValidationError. In a one-off check with the previous validator, a coordinator run whose aggregate carries spm_provenance: null also failed through the validating double, recording ValidationError. The absent-field and invalid-value cases pass both before and after, as guards should.
• Suites, on Python 3.13 with uv run --frozen:
  • libs/policyengine-simulation-contract: 153 passed.
  • projects/policyengine-simulation-executor, all of tests/: 1058 passed, 22 skipped, 3 deselected (the integration marker).
  • Of those, tests/test_stage12_runtime.py and tests/test_stage12_artifacts.py: 54 passed.
• Formatting and lint: with CI's pinned ruff 0.15.22, ruff format --check, ruff check and git diff --check pass on every changed file.
• Replay of a failed report: the coordinator's aggregate for one failed production UK report was rebuilt from its saved simulation outputs. It failed validation before this change and validates after it.

🤖 Generated with Claude Code