test(mutationlab): run the lab on a cluster of its own

.github/workflows/mutation-lab.yml

@@ -0,0 +1,78 @@
+# Mutation-capture lab: creates the lab's own k3d cluster, runs every scenario, and fails on any
+# difference from the committed corpus under test/mutationlab/corpus/
+# (docs/design/mutation-capture-lab-own-cluster.md).
+#
+# Nightly and on demand rather than per PR. Its job is to notice when something stops the lab
+# from working, or makes it record something else, before the next Kubernetes upgrade depends on
+# it; a break shows up within a day of the change that caused it. Nothing here publishes, and the
+# run needs no secrets beyond a read token to pull the CI container.
+name: Mutation lab
+
+on:
+  schedule:
+    - cron: "17 4 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: mutation-lab-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  # The CI base container main last published (ci.yml moves :latest only from main): the same
+  # Go, kubectl, k3d, and task the e2e legs run with.
+  CI_CONTAINER: ghcr.io/configbutler/gitops-reverser-ci:latest
+  CI_WORKDIR: /workspaces/${{ github.event.repository.name }}
+
+jobs:
+  capture:
+    name: Capture and compare to the corpus
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    permissions:
+      contents: read
+      packages: read
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Login to Docker registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
+
+      - name: Run lab-capture in the CI container
+        # One container for the whole run: the lab's kube context and its records-API forward
+        # live only as long as it does. Host networking reaches the k3d API port and the forward.
+        run: |
+          docker run --rm \
+            --network host \
+            -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \
+            -v /var/run/docker.sock:/var/run/docker.sock \
+            -w "${{ env.CI_WORKDIR }}" \
+            -e HOST_PROJECT_PATH=${{ github.workspace }} \
+            ${{ env.CI_CONTAINER }} \
+            bash -c "
+              git config --global --add safe.directory ${{ env.CI_WORKDIR }}
+              task lab-capture
+            "
+
+      - name: Show the lab on failure
+        if: failure()
+        run: |
+          docker run --rm \
+            --network host \
+            -v "${GITHUB_WORKSPACE}:${{ env.CI_WORKDIR }}" \
+            -v /var/run/docker.sock:/var/run/docker.sock \
+            -w "${{ env.CI_WORKDIR }}" \
+            ${{ env.CI_CONTAINER }} \
+            bash -c "
+              k3d kubeconfig merge gitops-reverser-capture-lab --kubeconfig-switch-context=false \
+                --kubeconfig-merge-default >/dev/null 2>&1 || exit 0
+              kubectl --context k3d-gitops-reverser-capture-lab get pods -A -o wide || true
+              kubectl --context k3d-gitops-reverser-capture-lab -n mutation-capture-lab \
+                logs deployment/mutation-capture-lab --tail=200 || true
+            "

Taskfile.yml

@@ -24,6 +24,10 @@ includes:
     taskfile: ./test/e2e/Taskfile-e2e.yml
     dir: .
     flatten: true
+  lab:
+    taskfile: ./test/mutationlab/Taskfile-lab.yml
+    dir: .
+    flatten: true
 
 tasks:
   default:

cmd/mutation-capture-lab/main.go

@@ -6,9 +6,11 @@
 // deliberately NOT a second GitOps Reverser; see
 // docs/spec/mutation-capture-lab-design.md.
 //
-// It deliberately serves the SAME webhook URLs as the product
-// (/validate-all, /audit-webhook) so a lab deployment can swap the product image
-// without reconfiguring the cluster's admission/audit wiring.
+// It runs on a cluster of its own (test/mutationlab/Taskfile-lab.yml), which
+// delivers to it the way any cluster delivers: the API server's audit webhook
+// backend posts to /audit-webhook over mutual TLS, a ValidatingWebhookConfiguration
+// sends admission reviews to /validate-all, and the scenario CRD's conversion
+// webhook calls /convert.
 package main
 
 import (
@@ -121,12 +123,6 @@ func buildServers(
 
 	auditMux := http.NewServeMux()
 	auditMux.Handle("/audit-webhook", audit)
-	// Audit routes are NAMED in the e2e cluster: its bootstrap kubeconfig posts to
-	// /audit-webhook/<source-name> (the chart's local KubeProvider), because the bare
-	// path is the product's shared annotation-routed endpoint. The lab records what
-	// arrives regardless of which source it was addressed to, so it takes the whole
-	// subtree — an exact-path-only mux 404s every event the cluster sends.
-	auditMux.Handle("/audit-webhook/", audit)
 
 	return []server{
 		{name: "admission", certDir: cfg.admissionCert, srv: &http.Server{

docs/ci-overview.md

@@ -20,6 +20,7 @@ so local and CI results cannot drift apart.
 | Release & publish | [release.yml](../.github/workflows/release.yml) tail jobs | after trusted validation is green | `GITHUB_TOKEN` + OIDC | packages, releases, attestations | Version (release-please), **retag** the CI-built multi-arch digests to semver + `latest` (zero rebuilds), publish chart, sign, attest |
 | Hygiene | [scorecard.yml](../.github/workflows/scorecard.yml) | weekly + `main` | none | security-events | OpenSSF Scorecard supply-chain checks |
 | Hygiene | [pr-title.yml](../.github/workflows/pr-title.yml) | `pull_request` (including `edited`) | none | none | The PR title is a Conventional Commit, since it becomes the squash-merge subject release-please classifies |
+| Hygiene | [mutation-lab.yml](../.github/workflows/mutation-lab.yml) | nightly + manual | `GITHUB_TOKEN` (read) | none | The mutation-capture lab on its own cluster, compared to its committed corpus ([design](design/mutation-capture-lab-own-cluster.md)) |
 
 Three properties are worth calling out:
 
@@ -164,5 +165,5 @@ GitHub's OIDC identity and logged in the public Rekor transparency log.
 | Tool versions | [.devcontainer/Dockerfile](../.devcontainer/Dockerfile) (single source for devcontainer *and* CI) |
 | Validation pipeline | [.github/workflows/ci.yml](../.github/workflows/ci.yml) |
 | Release pipeline | [.github/workflows/release.yml](../.github/workflows/release.yml), [.github/workflows/release-pr.yml](../.github/workflows/release-pr.yml) |
-| Hygiene | [.github/workflows/scorecard.yml](../.github/workflows/scorecard.yml), [.github/dependabot.yml](../.github/dependabot.yml) |
+| Hygiene | [.github/workflows/scorecard.yml](../.github/workflows/scorecard.yml), [.github/workflows/mutation-lab.yml](../.github/workflows/mutation-lab.yml), [.github/dependabot.yml](../.github/dependabot.yml) |
 | Release process details | [.github/RELEASES.md](../.github/RELEASES.md) |

docs/design/mutation-capture-lab-own-cluster.md

@@ -1,6 +1,6 @@
 # Mutation-capture lab on its own cluster
 
-Status: proposed, 2026-10-27. Not built.
+Status: built, 2026-10-27. [As built](#as-built) lists where the build departs from this design.
 
 The [mutation-capture lab](../spec/mutation-capture-lab-design.md) records what Kubernetes emits
 through watches, audit, admission, CRD conversion, and an aggregated API, and commits it as a
@@ -15,7 +15,7 @@ the requirements that keep the capture faithful and the lab working.
 ## What broke, and why it could
 
 The lab never owned its wiring. `task lab-e2e` swapped the lab image into the product's controller
-Deployment on the e2e cluster ([`swap-image.sh`](../../test/mutationlab/swap-image.sh)) and borrowed
+Deployment on the e2e cluster (`swap-image.sh`, since removed) and borrowed
 everything around it:
 
 | The lab needs | It came from | Since `fd6f6933` |
@@ -181,7 +181,7 @@ context rather than the current one. Otherwise bootstrapping the lab would move
 | the lab binary, recorders, normalizer, corpus, and driver | unchanged, apart from the client-CA flag, the context, and the readiness check |
 | the content-hashed lab image and its build | unchanged |
 | `lab-port-forward.sh` | with the lab's namespace and selector |
-| the audit kubeconfig template | from `generate-audit-webhook-kubeconfig.sh`, fed from files instead of Secrets |
+| the audit kubeconfig | written by the certificate command, not by `generate-audit-webhook-kubeconfig.sh` (see [As built](#as-built)) |
 
 Not reused: `inject-webhook-tls.sh`, whose node restart the lab avoids, and the Flux stack.
 
@@ -230,10 +230,28 @@ Each one names the failure it prevents.
 7. The scheduled CI job, the static self-containment check, and the lab's README and
    [spec](../spec/mutation-capture-lab-design.md#isolated-test-setup) rewritten to match.
 
-## Open questions
-
-- **CI cadence.** Nightly plus manual dispatch is the proposal. A per-PR job would catch a break
-  in the PR that causes it, for a few runner minutes per PR.
-- **Memory beside e2e.** One k3s node with the lab and the sample API server should stay near
-  1 GiB, but that is an estimate until measured. If it is more, `lab-capture` can take the e2e lock
-  as well, so the two run in turn.
+## As built
+
+- **Two Services, not one.** `mutation-capture-lab` is a ClusterIP Service for admission,
+  conversion (443 to 9443), and the records API (8080); `mutation-capture-lab-audit` is the NodePort
+  30444 to 9444. A NodePort Service gives every port it lists a node port.
+- **The certificate command writes the audit kubeconfig.** `generate-audit-webhook-kubeconfig.sh`
+  defaults to the product's names and reads Secrets; requirement 5 keeps the lab off it.
+- **The webhook keeps the name `validate-all.configbutler.ai`.** The API server keys the admission
+  recorder's audit annotation by it (`validate-all.configbutler.ai/scenario`) and quotes it in row
+  12's rejection, so the name is part of the corpus, like the match rules of requirement 3.
+- **The checks are tests in the driver.** `TestSourcesLive` is the five-source check:
+  `lab-bootstrap` runs it, and `lab-capture` skips it. `TestClusterProvenance` writes `CLUSTER.md`
+  on an update and fails a capture when the cluster serves another version. The static check is
+  [`selfcontained_test.go`](../../test/mutationlab/selfcontained_test.go), in `task test`.
+- **The capture on v1.36.4 differs in one field.** 19 of 21 scenarios reproduced the corpus byte for
+  byte. The other two, rows 8a and 19, gained `authenticationMetadata.impersonationConstraint:
+  impersonate:user-info` on each of their 15 impersonated audit events, and on nothing else. A plain
+  k3s v1.36.1 and v1.36.4 cluster with an audit log backend and none of the lab's wiring records the
+  same field for the same request, so it comes from the API server (constrained impersonation, beta
+  in 1.36), not from the new delivery path. Why the committed rows, captured on the e2e cluster,
+  lack it is not known.
+- **Memory.** The lab cluster's node used about 600 MiB with the lab and the sample API server
+  running, so `lab-capture` does not take the e2e lock.
+- **CI.** [`mutation-lab.yml`](../../.github/workflows/mutation-lab.yml) runs `lab-capture` nightly
+  and on manual dispatch.

docs/proud.md

@@ -96,7 +96,8 @@ be **SSH-signed**. It even captures resources served by an **aggregated API serv
 - A **self-ratcheting coverage gate** (`cover-check`) that only ever moves the floor up, with
   merged unit + e2e coverage reported to Codecov.
 - **Three install modes** — config-dir, Helm, and plain manifests — all exercised by e2e.
-- A **mutation-capture lab** (`lab-e2e`) for stress-testing capture correctness.
+- A **mutation-capture lab** (`lab-capture`), on a cluster of its own, that records what Kubernetes
+  emits and diffs it against a committed corpus.
 - `actionlint` on the CI workflows, Conventional Commits driving `release-please`.
 - A genuinely deep [docs/architecture.md](architecture.md) that answers the "but how does it
   actually…" questions instead of hand-waving.

docs/spec/mutation-capture-lab-design.md

@@ -8,14 +8,15 @@ webhooks** — at every interesting moment — and commits those structures as a
 **corpus**. It is not a second implementation of GitOps Reverser; it is reference material and a
 regression harness.
 
-The corpus is captured against **k8s v1.36.4+k3s1** (recorded in
+The corpus is captured against **k8s v1.37.1+k3s1** (recorded in
 `test/mutationlab/corpus/CLUSTER.md`). **All seventeen** catalogued scenarios are captured today (see
 the [Difficult Cases Catalog](#difficult-cases-catalog)), plus the rows that extend it, which
 `test/mutationlab/README.md` indexes.
 
-The lab reuses the product's webhook URLs (`/validate-all`, `/audit-webhook`) on the same ports and
-TLS mounts, so capturing is just an **image swap** on an already-prepared e2e cluster — no new audit
-policy, webhook config, or certificates. The binary lives under `cmd/mutation-capture-lab/`, the
+The lab runs on a k3d cluster of its own, which delivers audit, admission, conversion, and the
+aggregated API to it through their real mechanisms
+([design](../design/mutation-capture-lab-own-cluster.md)). The binary lives under
+`cmd/mutation-capture-lab/`, the
 recorders/normalizer/store/
 record model under `internal/mutationlab/`, and the live-cluster driver plus golden corpus under
 `test/mutationlab/` (driver behind the `mutationlab_e2e` build tag, so the normal lane never needs a
@@ -386,13 +387,14 @@ sometimes itself the behavior under test.
 
 ### Golden workflow
 
-- **Default run:** capture → normalize → compare against the committed file; a mismatch fails the test
-  and prints a unified diff.
-- **Update run:** `MUTATIONLAB_UPDATE=1 task lab-e2e` (or `task lab-corpus-update`) rewrites the corpus
-  from the live capture.
-- **Validating a new Kubernetes version:** bump the k3d image, run the update, and review the `corpus/`
-  diff — it *is* the behavioral changelog for the upgrade, scoped to exactly the behaviors GitOps
-  Reverser depends on, with volatile fields normalized away. An empty diff is a positive result.
+- **Default run:** `task lab-capture`: capture → normalize → compare against the committed file; a
+  mismatch fails the test and prints a unified diff.
+- **Update run:** `task lab-corpus-update` rewrites the corpus from the live capture, and the k3s
+  image, server version, and date in `CLUSTER.md`.
+- **Validating a new Kubernetes version:** recreate the lab cluster with
+  `task lab-bootstrap LAB_K3S_IMAGE=<image>`, run the update, and review the `corpus/` diff — it
+  *is* the behavioral changelog for the upgrade, scoped to exactly the behaviors GitOps Reverser
+  depends on, with volatile fields normalized away. An empty diff is a positive result.
 
 ## Record model
 
@@ -438,13 +440,17 @@ not) after the request — never guessed from the payload alone.
 The lab gets its own everything and reuses only the integration *shape* of the real install:
 
 - **Separate binary** (`cmd/mutation-capture-lab/main.go`), built as its own image.
-- **Swap the image, reuse the wiring.** Rather than authoring its own audit policy, webhook configs,
-  and certificates, the lab serves the **same** webhook URLs as the product on the same ports and TLS
-  mounts, so `task lab-e2e` integrates by swapping the controller image on the already-prepared e2e
-  cluster (`test/mutationlab/swap-image.sh`). The cluster is left running the lab image; restore with
-  `task clean-cluster && task test-e2e`.
-- **Separate, opt-in, serial task targets** (`task lab-e2e`, `task lab-corpus-update`) — not invoked by
-  `task test-e2e` or the default CI lane; the driver is behind the `mutationlab_e2e` build tag.
+- **Its own cluster.** The lab runs on a k3d cluster of its own, `gitops-reverser-capture-lab`,
+  with its own certificates, `ValidatingWebhookConfiguration`, Service, and RBAC
+  (`test/mutationlab/deploy/`), and nothing of the product. The API server's audit webhook backend,
+  the admission registration, the Widget CRD's conversion webhook, and the `wardle` aggregated API
+  deliver to it as they would on any cluster; it shares only the cluster script, the audit policy,
+  and the sample API server's manifests with e2e. See
+  [the mutation-capture lab on its own cluster](../design/mutation-capture-lab-own-cluster.md).
+- **Separate, opt-in, serial tasks** (`task lab-bootstrap`, `task lab-capture`,
+  `task lab-corpus-update`) in `test/mutationlab/Taskfile-lab.yml`, not invoked by `task test-e2e`
+  or the per-PR CI lanes; a scheduled workflow runs `lab-capture`. The driver is behind the
+  `mutationlab_e2e` build tag.
 
 Because audit delivery is asynchronous and batched, a late event from one scenario can land after the
 next has started. The lab **isolates** scenarios rather than trusting a clean slate:
@@ -502,7 +508,7 @@ so each row remains a deliberate stop-and-read point.
 Write Git commits · reuse the GitOps Reverser controller runtime · implement the full
 `WatchRule`/`GitTarget` model · solve HA/retention/multi-cluster · prove every resource behaves
 identically · build a **mutating** admission recorder (cited, not re-proven) · re-prove **Cite**-half
-contracts · hand-author corpus files · wire the lab into `task test-e2e` or the default CI lane · add
+contracts · hand-author corpus files · wire the lab into `task test-e2e` or the per-PR CI lanes · add
 lab-only behavior to the Helm chart.
 
 ## Related

docs/tasks-overview.md

@@ -311,9 +311,13 @@ flowchart LR
 
   FULL --> ALLURE["allure-e2e-results/report/open"]
   FULL --> COV["e2e-coverage-collect<br/>(when E2E_COVERAGE=1)"]
-  PREP --> LAB["lab-e2e / lab-corpus-update<br/>(serial opt-in)"]
 ```
 
+The mutation-capture lab is not in this graph: it runs on a cluster of its own, through
+`task lab-bootstrap`, `lab-capture`, and `lab-corpus-update`
+([`test/mutationlab/Taskfile-lab.yml`](../test/mutationlab/Taskfile-lab.yml)), and no task here
+depends on it or it on them.
+
 ## Why the DAG pays off: only what changed re-runs
 
 Because each task declares its real inputs and outputs, Task skips any step already up to