Skip to content

test(mutationlab): run the lab on a cluster of its own - #458

Merged
sunib merged 3 commits into
feat/gitops-api-asks-converge-root-cafrom
feat/mutation-capture-lab-own-cluster
Oct 27, 2026
Merged

sunib merged 3 commits into
feat/gitops-api-asks-converge-root-cafrom
feat/mutation-capture-lab-own-cluster

Conversation

@sunib

@sunib sunib commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #456.

Implements docs/design/mutation-capture-lab-own-cluster.md.

Why

The mutation-capture lab swapped its image into the product's controller Deployment on the e2e cluster and borrowed its admission registration, serving certificate, and Service. When #456 removed the admission webhook (fd6f693), the lab lost its admission source and the CA its CRD conversion webhook used. Nothing noticed, because nothing ran the lab.

What

The lab now runs on its own k3d cluster, gitops-reverser-capture-lab, with nothing of the product on it. Each source arrives through its real mechanism:

Source How it reaches the lab
audit the API server's audit webhook backend → NodePort 30444, mutual TLS (the lab now requires the client certificate)
admission its own ValidatingWebhookConfiguration, with the match rules and the name the corpus was captured under
conversion the Widget CRD's webhook → the lab's Service, CA read from the stamps
aggregated API the same wardle manifests e2e uses, through a new kustomization.yaml
watch the lab's own ServiceAccount with read-only RBAC
  • test/mutationlab/tools/lab-certs issues the CA, the serving certificate, and the API server's client certificate before the cluster exists. The audit kubeconfig is final at startup, so the node never restarts and one node is enough.
  • test/mutationlab/Taskfile-lab.yml provides lab-bootstrap, lab-capture, lab-corpus-update, lab-status, and lab-cleanup.
  • Coexistence with e2e: the lab has its own stamps, lab.lock, and port 18081. start-cluster.sh gains K3D_SWITCH_CONTEXT=false, so creating the lab cluster never moves the current context.
  • Requirement checks in the driver: TestSourcesLive (run by lab-bootstrap) proves all five sources with probe writes. TestClusterProvenance fails a capture when the cluster serves another version than CLUSTER.md records, and writes those rows on an update.
  • Self-containment: test/mutationlab/selfcontained_test.go (in task test) fails on a lab task that calls a non-lab task, a product resource name, or an unshared path.
  • Scheduled run: .github/workflows/mutation-lab.yml runs lab-capture nightly and on dispatch.
  • Removed: lab-e2e, _lab-deployed, swap-image.sh, hack/e2e/lab-port-forward.sh.

Does it still capture the same thing?

The corpus was recaptured on the same k3s v1.36.4, in its own commit:

  • 19 of 21 scenarios reproduce byte for byte.
  • In rows 8a and 19, the audit events of impersonated actors (15 files) each gain authenticationMetadata.impersonationConstraint: impersonate:user-info. The ServiceAccount-token patch in row 8a does not.
  • That field comes from the API server, not from the new wiring. Plain k3s v1.36.1 and v1.36.4 clusters with an audit log backend, and none of the lab's wiring, record it for the same request.
  • I don't know why the committed rows lacked it. The fact-index tests that replay them pass unchanged.

The webhook rename taught one thing: the API server keys the recorder's audit annotation (validate-all.configbutler.ai/scenario) by the webhook's name. The name is therefore part of the corpus and stays as it was.

Then on v1.37.1

A third commit moves the pin to v1.37.1, the e2e cluster's version, so its diff is the Kubernetes change alone. Each change reproduced in a second capture:

  • deletecollection: the response's list resourceVersion is now the last deleted object's own, so the watch DELETED placeholders shift by one.
  • graceful pod delete: PodReadyToStartContainers gains reason: PodSandboxNotReady and a message.
  • owner-ref cascade: the garbage collector's userAgent now names v1.37.1.

Validation

  • task lint and task test pass.
  • task lab-capture passes on v1.36.4 (25 tests), and twice on v1.37.1. It ran beside the running e2e cluster, which kept the current context.
  • The e2e cluster's kustomize render of test/e2e/setup/manifests is byte-identical.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added a standalone mutation-capture lab that runs on its own Kubernetes cluster and records audit, admission, conversion, watch, and aggregated API activity for comparison with a committed reference.
    • Added workflows to set up the lab, capture or update records, check status, and clean up.
    • The lab can run nightly or be started manually, and supports validation against other Kubernetes versions.
  • Documentation
    • Updated lab and CI guides with setup, capture, and version-validation instructions.
    • Clarified that lab runs are separate from the standard end-to-end test flow.

@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f400072d-5570-41a9-8fb9-7ca930144c7e
Commits

Reviewing files that changed from the base of the PR and between 7431ea3 and c7707d5.

Files selected for processing: 58
  • .github/workflows/mutation-lab.yml
  • Taskfile.yml
  • cmd/mutation-capture-lab/main.go
  • docs/ci-overview.md
  • docs/design/mutation-capture-lab-own-cluster.md
  • docs/proud.md
  • docs/spec/mutation-capture-lab-design.md
  • docs/tasks-overview.md
  • hack/e2e/lab-port-forward.sh
  • internal/mutationlab/labserver/api.go
  • test/e2e/Taskfile-e2e.yml
  • test/e2e/cluster/README.md
  • test/e2e/cluster/start-cluster.sh
  • test/e2e/setup/manifests/kustomization.yaml
  • test/e2e/setup/manifests/sample-apiserver/kustomization.yaml
  • test/mutationlab/Dockerfile
  • test/mutationlab/README.md
  • test/mutationlab/Taskfile-lab.yml
  • test/mutationlab/corpus/CLUSTER.md
  • test/mutationlab/corpus/configmap/deletecollection/audit.deletecollection.yaml
  • test/mutationlab/corpus/configmap/deletecollection/watch.deleted.cm-a.yaml
  • test/mutationlab/corpus/configmap/deletecollection/watch.deleted.cm-b.yaml
  • test/mutationlab/corpus/configmap/deletecollection/watch.deleted.cm-c.yaml
  • test/mutationlab/corpus/configmap/deletion-intent-actor/audit.delete.yaml
  • test/mutationlab/corpus/configmap/owner-ref-cascade/audit.delete.cm-child.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-exit-deleted.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-gone.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-held.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-member.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-outsider.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.delete.cm-exit-deleted.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.delete.cm-gone.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.delete.cm-held.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.patch.1.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.patch.3.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.patch.4.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.update.1.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.update.2.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.update.3.yaml
  • test/mutationlab/corpus/pod/graceful-delete/watch.deleted.yaml
  • test/mutationlab/deploy.sh
  • test/mutationlab/deploy/deployment.yaml
  • test/mutationlab/deploy/kustomization.yaml
  • test/mutationlab/deploy/namespace.yaml
  • test/mutationlab/deploy/rbac.yaml
  • test/mutationlab/deploy/service.yaml
  • test/mutationlab/deploy/validating-webhook.yaml
  • test/mutationlab/e2e/cluster_provenance_test.go
  • test/mutationlab/e2e/crd_conversion_test.go
  • test/mutationlab/e2e/doc.go
  • test/mutationlab/e2e/harness_test.go
  • test/mutationlab/e2e/sources_live_test.go
  • test/mutationlab/e2e/workload_scenarios_test.go
  • test/mutationlab/port-forward.sh
  • test/mutationlab/selfcontained_test.go
  • test/mutationlab/swap-image.sh
  • test/mutationlab/tools/lab-certs/main.go
  • test/mutationlab/tools/lab-certs/main_test.go

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: dfafcfa2-d0a3-48cd-9c15-ded83e087695
Commits

Reviewing files that changed from the base of the PR and between 1576bd9 and 7431ea3.

Files selected for processing: 58
  • .github/workflows/mutation-lab.yml
  • Taskfile.yml
  • cmd/mutation-capture-lab/main.go
  • docs/ci-overview.md
  • docs/design/mutation-capture-lab-own-cluster.md
  • docs/proud.md
  • docs/spec/mutation-capture-lab-design.md
  • docs/tasks-overview.md
  • hack/e2e/lab-port-forward.sh
  • internal/mutationlab/labserver/api.go
  • test/e2e/Taskfile-e2e.yml
  • test/e2e/cluster/README.md
  • test/e2e/cluster/start-cluster.sh
  • test/e2e/setup/manifests/kustomization.yaml
  • test/e2e/setup/manifests/sample-apiserver/kustomization.yaml
  • test/mutationlab/Dockerfile
  • test/mutationlab/README.md
  • test/mutationlab/Taskfile-lab.yml
  • test/mutationlab/corpus/CLUSTER.md
  • test/mutationlab/corpus/configmap/deletecollection/audit.deletecollection.yaml
  • test/mutationlab/corpus/configmap/deletecollection/watch.deleted.cm-a.yaml
  • test/mutationlab/corpus/configmap/deletecollection/watch.deleted.cm-b.yaml
  • test/mutationlab/corpus/configmap/deletecollection/watch.deleted.cm-c.yaml
  • test/mutationlab/corpus/configmap/deletion-intent-actor/audit.delete.yaml
  • test/mutationlab/corpus/configmap/owner-ref-cascade/audit.delete.cm-child.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-exit-deleted.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-gone.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-held.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-member.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.create.cm-outsider.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.delete.cm-exit-deleted.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.delete.cm-gone.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.delete.cm-held.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.patch.1.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.patch.3.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.patch.4.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.update.1.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.update.2.yaml
  • test/mutationlab/corpus/configmap/selector-membership/audit.update.3.yaml
  • test/mutationlab/corpus/pod/graceful-delete/watch.deleted.yaml
  • test/mutationlab/deploy.sh
  • test/mutationlab/deploy/deployment.yaml
  • test/mutationlab/deploy/kustomization.yaml
  • test/mutationlab/deploy/namespace.yaml
  • test/mutationlab/deploy/rbac.yaml
  • test/mutationlab/deploy/service.yaml
  • test/mutationlab/deploy/validating-webhook.yaml
  • test/mutationlab/e2e/cluster_provenance_test.go
  • test/mutationlab/e2e/crd_conversion_test.go
  • test/mutationlab/e2e/doc.go
  • test/mutationlab/e2e/harness_test.go
  • test/mutationlab/e2e/sources_live_test.go
  • test/mutationlab/e2e/workload_scenarios_test.go
  • test/mutationlab/port-forward.sh
  • test/mutationlab/selfcontained_test.go
  • test/mutationlab/swap-image.sh
  • test/mutationlab/tools/lab-certs/main.go
  • test/mutationlab/tools/lab-certs/main_test.go
Files with no reviewable changes: 3
  • test/mutationlab/swap-image.sh
  • hack/e2e/lab-port-forward.sh
  • test/e2e/Taskfile-e2e.yml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

The mutation-capture lab now runs on a dedicated k3d cluster with separate deployment, certificates, and capture tasks. E2E checks verify its traffic sources, and capture updates a versioned corpus. A nightly and manually triggered workflow runs capture.

Changes: Mutation Capture Lab

Layer / File(s) Summary
Dedicated cluster and task lifecycle
Taskfile.yml, test/e2e/Taskfile-e2e.yml, test/e2e/cluster/*, test/e2e/setup/manifests/*, test/mutationlab/Taskfile-lab.yml, test/mutationlab/README.md, test/mutationlab/Dockerfile, test/mutationlab/swap-image.sh, hack/e2e/lab-port-forward.sh, docs/spec/*, docs/design/mutation-capture-lab-own-cluster.md, docs/tasks-overview.md, docs/proud.md
The lab has separate bootstrap, capture, corpus-update, status, and cleanup tasks. Cluster creation can preserve the current kubeconfig context. The prior e2e image-swap tasks and lab port-forward script are removed.
Lab deployment and capture sources
test/mutationlab/deploy.sh, test/mutationlab/deploy/*, test/mutationlab/tools/lab-certs/*, test/mutationlab/port-forward.sh, test/mutationlab/e2e/*, cmd/mutation-capture-lab/main.go, internal/mutationlab/labserver/api.go
The lab deployment adds separate webhook/API and audit Services, a validating webhook, and read-only watch permissions. The certificate command creates serving and client certificates and the audit kubeconfig. E2E checks probe audit, admission, conversion, watch, and aggregated API capture paths.
Capture provenance and corpus updates
test/mutationlab/e2e/cluster_provenance_test.go, test/mutationlab/corpus/*, test/mutationlab/selfcontained_test.go, docs/design/mutation-capture-lab-own-cluster.md
The provenance test checks the cluster version against the image tag and can update the recorded image, version, and date. The corpus records impersonation metadata and Kubernetes output changes. Self-containment tests check task references and lab wiring.
Scheduled capture workflow
.github/workflows/mutation-lab.yml, docs/ci-overview.md, docs/design/mutation-capture-lab-own-cluster.md
The workflow runs capture nightly or by manual dispatch with read-only permissions. On failure, it attempts to collect pod and deployment logs. CI documentation lists the workflow.

Priority: Low

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant SourceTest as TestSourcesLive
  participant Cluster as Kubernetes cluster
  participant Lab as Mutation capture lab
  participant Records as Records API
  SourceTest->>Cluster: Create probes for capture sources
  Cluster->>Lab: Send capture traffic
  SourceTest->>Records: Poll for scenario records
  Records-->>SourceTest: Return captured records
Loading
Merge Risk: Minimal · previous review 7431e

No specific issue is established that would prevent merging the standalone lab change after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 43.90% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surv… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the primary change: moving the mutation lab to its own cluster.
Description check Passed The description is detailed, directly related to the changes, and includes the motivation, implementation scope, compatibility details, and validation results. It does not use the template headings or…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description is detailed, directly related to the changes, and includes the motivation, implementation scope, compatibility details, and validation results. It does not use the template headings or checkbox sections for change type and checklist, but the substantive information is mostly complete.

Full details: Docstring Coverage

Explanation

Docstring coverage is 43.90% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 41 functions across 14 files. (41 skipped: 41 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

sunib and others added 3 commits October 27, 2026 16:47
The lab swapped its image into the product's controller Deployment on
the e2e cluster and borrowed its admission registration, serving
certificate, and Service. fd6f693 removed the admission webhook, and
the lab lost its admission source and the conversion webhook's CA while
every check stayed green, because nothing ran the lab.

The lab now runs on its own k3d cluster, gitops-reverser-capture-lab,
with nothing of the product on it. Every source arrives through its
real mechanism: the API server's audit webhook backend posts to a
NodePort over mutual TLS (the lab now requires the client certificate),
a ValidatingWebhookConfiguration of its own sends admission reviews,
the Widget CRD's conversion webhook calls the lab's Service, the wardle
sample API server serves flunders, and the lab watches with read-only
RBAC. lab-certs issues the CA, serving certificate, and the API server's
client certificate before the cluster exists, so the audit kubeconfig
is final at startup and the node never restarts.

The registration keeps the match rules and the name the corpus was
captured under: the API server keys the recorder's audit annotation by
the webhook's name, so renaming it changes 13 corpus rows.

test/mutationlab/Taskfile-lab.yml carries lab-bootstrap, lab-capture,
lab-corpus-update, lab-status, and lab-cleanup. The cluster can run
beside e2e: its own stamps, lock, and port, and start-cluster.sh gains
K3D_SWITCH_CONTEXT=false so creating it never moves the current
context. lab-bootstrap ends with TestSourcesLive, which proves each of
the five sources with a probe write; TestClusterProvenance fails a
capture on another server version than CLUSTER.md records, and writes
those rows on an update. selfcontained_test.go keeps the lab off
product tasks, product resource names, and unshared paths, and
mutation-lab.yml runs lab-capture nightly.

lab-e2e, _lab-deployed, swap-image.sh, and hack/e2e/lab-port-forward.sh
are removed. The sample API server gains a kustomization.yaml that both
clusters apply; the e2e render is byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…36.4

Same k3s version as before (v1.36.4+k3s1); only the delivery path
changed. 19 of 21 scenarios reproduce byte for byte. The audit events
of the impersonated actors in rows 8a and 19 (15 files) each gain
authenticationMetadata.impersonationConstraint: impersonate:user-info;
the ServiceAccount-token patch in row 8a does not.

The field comes from the API server, not the new wiring: a plain k3s
v1.36.1 and v1.36.4 cluster with an audit log backend and none of the
lab's wiring records it for the same impersonated write. Why the
committed rows, captured on the e2e cluster, lacked it is not known.
The fact-index tests that replay these rows pass unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s version

Only the Kubernetes version changed since the previous commit, so this
diff is the 1.36.4 → 1.37.1 changelog for what the corpus records. Each
change reproduced in a second capture:

- deletecollection: the response's list resourceVersion is the last
  deleted object's own, where 1.36 returned a newer one, so the watch
  DELETED placeholders shift by one;
- graceful pod delete: PodReadyToStartContainers gains reason
  PodSandboxNotReady and a message;
- owner-ref cascade: the garbage collector's userAgent names v1.37.1.

The fact-index tests that replay the corpus pass unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sunib
sunib force-pushed the feat/mutation-capture-lab-own-cluster branch from 7431ea3 to c7707d5 Compare October 27, 2026 16:50
@sunib
sunib merged commit c99175b into main Oct 27, 2026
25 of 34 checks passed
@sunib
sunib deleted the feat/mutation-capture-lab-own-cluster branch October 27, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant