Certify Microcosm builds in their own release folders, and tagged builds only where a repository has no releases.json

changelog.d/682.added.md

@@ -0,0 +1 @@
+Certify Microcosm builds published side by side in their own release folders (`releases/<release>/<build>/`, with no Hugging Face tag): a release manifest that names its `release` and `build` as strings is certified from that folder, its own files are pinned inside it at the certified revision, and its `build` names the certificate, so two builds of one release never match. Certification refuses such a build unless the repository's `releases.json` at `main` lists it, has not withdrawn it, and is a record the release writer's parser accepts, and refuses any of its own files that is not a bare file name directly inside its folder. Such a manifest's build details (`built_at`, `built_with_model_package`, `built_with_core_package`) are read from its top level. At import, a remote or local manifest that names another release or build than the bundle's pinned folder is refused. Manifests published before this layout are read as before, and certified only from a repository with no `releases.json` at `main`.

docs/engineering/skills/data-certification.md

@@ -34,6 +34,63 @@ uses the cut tag as the build id (`data_releases.<country>.version`, the
 certification's `data_build_id` and the changelog fragment name), so two cuts
 of one line never look identical.
 
+### Builds in their own release folders
+
+Once Microcosm publishes each build in its own folder,
+`releases/<release>/<build>/`, a build is never changed after publishing and
+gets no Hugging Face tag. The repository's root `releases.json` records which
+build of each release is current. That build's `release_manifest.json` names
+its `release` and `build` as strings, for example
+`microcosm-uk-2024-25-national` and
+`uk-frs-calibration-attempt-20261019T083323Z-6cba5d71`. Certify the build from
+its own folder, at the commit that published it:
+
+```bash
+python scripts/bundle.py certify-data --country uk --data-producer populace \
+  --manifest-uri "hf://dataset/policyengine/populace-uk-private@<commit>/releases/<release>/<build>/release_manifest.json"
+```
+
+- The URI must be the build's own folder: a manifest that names another
+  release or build than the folder it was read from is refused.
+- The repository's `releases.json` at `main` must list the build and must not
+  mark it withdrawn, or certification refuses it. A local-area release passed
+  with `--regional-manifest-uri` is checked the same way. Which build is
+  `current` doesn't matter: a certificate pins one build.
+- The whole record must also be one the release writer's parser
+  (microcosm-emitter's `ReleasesRecord.parse`) accepts, or certification
+  refuses it. policyengine.py repeats the writer's rules: exactly the keys the writer
+  writes, at every level; `schema_version` 1; safe release and build names and
+  valid roles; a non-empty `published_at` and a boolean `withdrawn` on every
+  build; a `default` that is null or a listed release; a `current` that is null
+  or a listed build that isn't withdrawn; and at most 4 MiB.
+- The build's own files are pinned at `releases/<release>/<build>/<file>` and
+  at the URI's revision, whatever revision their manifest entries name. Each
+  one's `path` must be a bare file name, alone or after the folder, as the
+  release writer publishes it; a nested path, a `.` or `..` segment or a
+  backslash is refused. Artifacts from other repositories keep their own pins.
+- The build's details are at the manifest's top level: `built_at`,
+  `built_with_model_package` and `built_with_core_package`. They are read as
+  the build's own, as an older manifest's `build` object holds them, so the
+  `built_with_model_package` basis, the data build fingerprint and the TRO's
+  start time apply as before. A dump writes them back at the top level.
+- The certificate names the build by `build`
+  (`data_releases.<country>.version`, the certification's `data_build_id` and
+  the changelog fragment name), so two builds of one release never match.
+- At import, a remote manifest that names another release or build than the
+  bundle's pinned folder is refused, even when the bundle carries a
+  certificate. Import doesn't read `releases.json`: a build withdrawn after it
+  was certified keeps loading in every installed .py release whose bundle pins
+  it. Certifying again changes only the .py releases published afterwards.
+
+Manifests published before this layout, whose `build` is an object, are
+certified as described above, but only from a repository that has no
+`releases.json` at `main`. A repository is read in one layout: once it has
+`releases.json`, a tagged manifest (primary or `--regional-manifest-uri`) is
+refused, and the build is certified from its own folder instead. Only Hugging
+Face's 404 for `releases.json` means the repository has none; a 404 whose
+error code is `RepoNotFound` or `RevisionNotFound`, or any other failure,
+refuses certification rather than being taken for a tagged repository.
+
 ### UK constituency and local-authority regions
 
 UK constituency and local-authority regions row-filter the dataset a regional

src/policyengine/provenance/certification.py

@@ -29,6 +29,13 @@
 Pin the cut tag (``@<cut-tag>``), never a branch: a release line's directory
 (``releases/<release-id>/``) is rewritten by each cut.
 
+A build published in its own folder (``releases/<release>/<build>/``, named
+by its manifest's ``release`` and ``build`` strings) has no tag. Certify it
+from that folder at the commit that published it; its own files are pinned
+inside the folder at that revision, and its ``build`` names the certificate.
+The repository's ``releases.json`` at ``main`` must list the build and not
+have withdrawn it, and must be a record the release writer accepts.
+
 ``scripts/bundle.py certify-data`` is the operator-facing wrapper. Network
 access is required (HF manifest fetch + PyPI wheel metadata). Countries whose
 data release predates release manifests need a dedicated data-producer strategy
@@ -57,12 +64,17 @@
 from policyengine.provenance.bundle_validation import shadowed_dataset_overlays
 from policyengine.provenance.manifest import (
     HF_REQUEST_TIMEOUT_SECONDS,
+    RELEASES_RECORD_NAME,
+    RELEASES_RECORD_REVISION,
     DataReleaseArtifact,
     DataReleaseManifest,
     RegionalInputSurfaceEvidence,
     RegionalReleaseKind,
+    ReleaseBuildFolderError,
     _specifier_matches,
+    check_build_listed,
     fetch_pypi_wheel_metadata,
+    read_build_folder,
     release_build_id,
 )
 from policyengine.provenance.manifest import (
@@ -246,12 +258,112 @@ def release_manifest_dir(parts: dict) -> str:
     return parts["path"].rsplit("/", 1)[0]
 
 
+def https_releases_record_url(parts: dict) -> str:
+    """The repository's ``releases.json`` at ``main``, its current record."""
+    prefix = "datasets/" if parts["repo_type"] == "dataset" else ""
+    return (
+        f"https://huggingface.co/{prefix}{parts['repo_id']}/resolve/"
+        f"{RELEASES_RECORD_REVISION}/{RELEASES_RECORD_NAME}"
+    )
+
+
+#: Hugging Face's error codes for a 404 that is not about the file: the
+#: repository or the revision is missing (or unreadable with this token).
+_MISSING_REPOSITORY_OR_REVISION = frozenset({"RepoNotFound", "RevisionNotFound"})
+
+
+def _releases_record_at_main(parts: dict, headers: dict) -> Optional[bytes]:
+    """The repository's ``releases.json`` at ``main``, or ``None`` when it has
+    none.
+
+    Only Hugging Face's 404 for the file means "none", unless its error code
+    (``X-Error-Code``) says the repository or the revision is missing. Any
+    other answer is a ``CertificationError``, so certification never takes a
+    failed read for a repository without the record.
+    """
+    url = https_releases_record_url(parts)
+    response = requests.get(url, headers=headers, timeout=HF_REQUEST_TIMEOUT_SECONDS)
+    error_code = (getattr(response, "headers", None) or {}).get("X-Error-Code")
+    if response.status_code == 404 and error_code not in (
+        _MISSING_REPOSITORY_OR_REVISION
+    ):
+        return None
+    if response.status_code != 200:
+        detail = (
+            f"{response.status_code}, {error_code}"
+            if error_code
+            else (f"{response.status_code}")
+        )
+        raise CertificationError(
+            f"Could not fetch {RELEASES_RECORD_NAME} ({detail}) from {url}; "
+            "certification reads it to tell which layout the repository "
+            "publishes."
+        )
+    return response.content
+
+
+def _require_listed_build(
+    manifest: DataReleaseManifest, parts: dict, headers: dict
+) -> None:
+    """Refuse a build in its own folder that ``releases.json`` at ``main``
+    does not list, or has withdrawn.
+
+    The record is read at ``main``, not at the URI's revision: a build is
+    withdrawn after the commit that published it.
+    """
+    release, build = manifest.release, release_build_id(manifest.build)
+    if release is None or build is None:
+        return
+    record = _releases_record_at_main(parts, headers)
+    if record is None:
+        raise CertificationError(
+            f"{parts['repo_id']} has no {RELEASES_RECORD_NAME} at "
+            f"{RELEASES_RECORD_REVISION}; a build in its own release folder is "
+            "certified only while that record lists it."
+        )
+    try:
+        check_build_listed(record, release=release, build=build)
+    except ReleaseBuildFolderError as exc:
+        raise CertificationError(str(exc)) from exc
+
+
+def require_tagged_layout(
+    manifest: DataReleaseManifest, parts: dict, token: Optional[str] = None
+) -> None:
+    """Refuse a manifest in the tagged layout from a repository that has
+    ``releases.json`` at ``main``.
+
+    A repository is read in one layout only. Once it records its builds in
+    ``releases.json``, only builds in their own release folders are
+    certified, so a stray tag or pointer file has no effect; where it has no
+    ``releases.json``, its tagged builds are certified as before. A build in
+    its own folder is checked by ``fetch_release_manifest`` instead.
+    """
+    if manifest.build_folder is not None:
+        return
+    headers = {"Authorization": f"Bearer {token}"} if token else {}
+    if _releases_record_at_main(parts, headers) is not None:
+        raise CertificationError(
+            f"{parts['repo_id']} has {RELEASES_RECORD_NAME} at "
+            f"{RELEASES_RECORD_REVISION}, so only builds in their own release "
+            "folders are certified; certify "
+            "releases/<release>/<build>/release_manifest.json instead of "
+            f"{parts['path']}."
+        )
+
+
 def fetch_release_manifest(
     manifest_uri: str,
     token: Optional[str] = None,
 ) -> tuple[DataReleaseManifest, str, dict]:
     """Fetch and parse the data release manifest; returns
-    (manifest, sha256 of the fetched bytes, parsed URI parts)."""
+    (manifest, sha256 of the fetched bytes, parsed URI parts).
+
+    A build published in its own release folder must be read from that
+    folder, and its own files are pinned inside it at the URI's revision
+    (``read_build_folder``). The repository's ``releases.json`` at ``main``
+    must list the build and not have withdrawn it, and must be a record the
+    release writer accepts (``check_build_listed``)."""
     parts = parse_manifest_uri(manifest_uri)
     url = https_manifest_url(parts)
     headers = {"Authorization": f"Bearer {token}"} if token else {}
@@ -264,6 +376,17 @@ def fetch_release_manifest(
     sha256 = hashlib.sha256(response.content).hexdigest()
     payload = json.loads(response.content)
     manifest = DataReleaseManifest.model_validate(payload)
+    try:
+        manifest = read_build_folder(
+            manifest,
+            repo_id=parts["repo_id"],
+            manifest_path=parts["path"],
+            revision=parts["revision"],
+        )
+    except ReleaseBuildFolderError as exc:
+        raise CertificationError(str(exc)) from exc
+    if manifest.build_folder is not None:
+        _require_listed_build(manifest, parts, headers)
     return manifest, sha256, parts
 
 
@@ -1461,6 +1584,7 @@ def certify(
         manifest, manifest_sha256, uri_parts = fetch_release_manifest(
             manifest_uri, token=token
         )
+        require_tagged_layout(manifest, uri_parts, token)
         primary_manifest = manifest
         regional_manifest = None
         regional_manifest_sha256 = None
@@ -1484,6 +1608,7 @@ def certify(
             regional_manifest, regional_manifest_sha256, regional_uri_parts = (
                 fetch_release_manifest(regional_manifest_uri, token=token)
             )
+            require_tagged_layout(regional_manifest, regional_uri_parts, token)
             _regional_build_model_version(regional_manifest, model_package)
             # The regional release runs on the same model and core, so it must
             # cover the pinned versions by its own build or claims. A qualified