diff --git a/promptbook/prompts.json b/promptbook/prompts.json index eb171f9..39f11a0 100644 --- a/promptbook/prompts.json +++ b/promptbook/prompts.json @@ -779,6 +779,26 @@ ], "formatNotes": "All three. Point the tool at the model client or gateway code, the retry and fallback config, the application config with model aliases, the auth or caller-identity code, and the metrics setup. Copilot: llm-cost path instructions plus those files in agent mode (or save it as `.github/prompts/add-llm-token-budgets-and-cost-tracking.prompt.md`). Claude Code: the llm-cost-latency-budget-guardrails skill plus the files. Codex: AGENTS.md LLM cost and latency section, then the diff.", "body": "Task: Add token budgets, cost tracking, and latency measurement to the LLM calls in the service I specify. Inventory every model call, meter usage from provider responses, price it from a versioned table, cap each request, add caller and global budgets, fix retry accounting, and finish with a minimal diff, tests, and a cost report.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Service or module root: {{APP_PATH}}\n- Stack: {{STACK}} (Spring Boot + Spring AI | LangChain4j | plain HTTP client | Python SDK | gateway)\n- Providers and models: {{PROVIDERS}} (e.g. Azure OpenAI deployments, OpenAI-compatible endpoints, OpenRouter, Anthropic)\n- Callers to attribute: {{CALLERS}} (service, tenant, user, or feature; and where the identity comes from)\n- Price source: {{PRICES}} (list prices with date, or negotiated rates; currency)\n- Budgets: {{BUDGETS}} (e.g. 200k tokens per caller per day, $50 global per day, max 8 agent steps)\n- Latency SLO: {{SLO}} (e.g. p95 time to first token ≤ 1.5 s, p95 total ≤ 8 s)\n- Usage store and metrics: {{STORE}} (database table, append-only log, Micrometer/OpenTelemetry metrics backend)\n- Mode: {{MODE}} (report-only | enforce)\n\nRequirements:\n\n1. Inventory\n - Table of every model call in {{APP_PATH}}: file and method, route or feature, caller, model alias, max output tokens (or \"not set\"), streaming, tools or agent loop, retry and fallback policy\n - Flag calls with no max output tokens, streams that never read usage, agent loops with no step cap, and retries of non-retryable errors\n\n2. Metering\n - One usage hook (client interceptor, gateway filter, or callback), not per-call code\n - Read provider usage: input, output, and, where reported, cached-input and reasoning tokens; for OpenAI-compatible streams request the final usage chunk and count streams that end without it\n - Record: request ID, trace ID, route, caller ID (stable ID, never an email), requested alias, resolved provider and model version, attempt number, outcome, token counts, timings, cost, price table version\n - Never store prompts or completions in the usage store; redact them in logs\n\n3. Pricing\n - Versioned price table from {{PRICES}} keyed by provider, model, and token type, with effective date and currency\n - Cost computed at write time; unit tests for every token type and for a model missing from the table (record it, flag it, don't price it as zero)\n\n4. Request caps\n - Explicit max output tokens per route, input and retrieved-context caps, tool-result size cap, agent step and total-token cap per run\n - A clear error or trim policy when a cap is hit; a CI or unit check that fails when a route has no max output tokens\n\n5. Budgets\n - Per-caller per-window and global daily budgets from {{BUDGETS}}\n - Reserve an estimate before the call, reconcile with actual usage after it, release on failure; safe under concurrency\n - Blocked response: documented status (e.g. 429 with reason and Retry-After) and a metric; in report-only {{MODE}}, log and count would-be blocks without rejecting\n - Alert at a threshold below the global ceiling and on sudden jumps in tokens per request\n\n6. Retries and fallbacks\n - Retry only timeouts, 429, and 5xx, with capped exponential backoff and jitter, honouring Retry-After, with a total attempt and time limit\n - Every attempt's tokens count against the caller's budget; record which provider and model each attempt used\n - Cancel the upstream call when the client disconnects\n\n7. Latency\n - Histograms for total duration, time to first token, output tokens per second, gateway overhead, and retry wait, tagged by route, provider, model, and outcome\n - Labels stay low-cardinality: no user IDs, request IDs, or prompt text; if using OpenTelemetry GenAI conventions, pin the version\n - Report against {{SLO}}\n\n8. Report and tests\n - A query or endpoint that reports cost, tokens, p95 latency, and error rate per day, route, caller, and model from {{STORE}}\n - Tests: pricing, reserve-and-reconcile under concurrency, streaming with and without final usage, retry accounting, blocked response, missing-price handling\n\nConstraints:\n- Minimal diff. Don't change prompts, models, or routing unless a finding requires it; report findings instead.\n- Don't invent prices, token counts, or traffic numbers. Use placeholders and state assumptions.\n- Don't put prompts, completions, secrets, or personal data in the usage store, metrics, or test fixtures.\n- Put price table changes in their own commit with an effective date.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave dashboard building and provider migrations as follow-up notes.\n\nOutput format:\n1. Inventory table with flagged calls\n2. Usage record schema and where it is written\n3. Caps and budgets table: route or caller | limit | window | current behaviour | new behaviour\n4. Retry and fallback policy before and after\n5. Patch (usage hook, price table, caps, budgets, retry changes, metrics, report query, tests)\n6. Sample report output with placeholder numbers clearly marked, plus follow-ups\n\nStart from {{APP_PATH}}. Meter before you limit: get honest usage numbers first, then add caps and budgets.\n" + }, + { + "id": "scaffold-kustomize-aks-overlays", + "title": "Scaffold or Clean Up Kustomize Overlays for AKS", + "whenToUse": "Set up new Kustomize manifests or clean up existing ones for a service on AKS: map overlays, replace deprecated fields with proof the rendered output is unchanged, split into base, components, and overlays, fix labels, images, generators, and secrets, and add a CI gate that builds, validates, policy-checks, and diffs every overlay.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "kubernetes", + "kustomize", + "aks", + "gitops", + "ci", + "devsecops" + ], + "formatNotes": "All three. Point the tool at the manifests folder, the CI workflow that deploys it, and any per-environment values files. Copilot: kustomize path instructions plus those files in agent mode (or save it as `.github/prompts/scaffold-kustomize-aks-overlays.prompt.md`). Claude Code: the kustomize-aks-overlay-guardrails skill plus the files. Codex: AGENTS.md Kubernetes manifests section, then the diff.", + "body": "Task: Set up or clean up Kustomize manifests for the service I specify so each environment is a small overlay on one base, every overlay renders and validates in CI, and reviewers see the rendered diff. Finish with a minimal diff, the CI job, and a rendered-output comparison.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Manifests root: {{K8S_PATH}} (e.g. deploy/k8s)\n- Environments and clusters: {{ENVS}} (e.g. dev, stage, prod on AKS; namespace per env)\n- Kubernetes versions: {{K8S_VERSIONS}} (per cluster)\n- Image registry and tagging: {{IMAGES}} (e.g. ACR, digest pinned by CI)\n- Config and secrets: {{CONFIG}} (which values differ per env; where secrets live, e.g. Azure Key Vault)\n- Identity: {{IDENTITY}} (e.g. AKS workload identity client ID per env)\n- Optional pieces: {{COMPONENTS}} (e.g. monitoring annotations, PodDisruptionBudget, topology spread for prod)\n- CI system: {{CI}} (GitHub Actions | Azure Pipelines | Jenkins)\n- Tool versions: {{TOOLS}} (kustomize, kubeconform, kube-linter or policy tool)\n\nRequirements:\n\n1. Map the current state\n - Table: overlay | what it changes from base | cluster and namespace | deprecated fields | copied base resources | secrets in Git\n - Render every overlay as it is today with the pinned kustomize version and save the output for comparison\n\n2. Modernize\n - Replace `bases`, `commonLabels`, `patchesStrategicMerge`, `patchesJson6902`, and `vars` with `resources`, `labels`, `patches`, and `replacements` (`kustomize edit fix` first, then hand-check)\n - Rendered output must match the saved output, or list and justify every difference\n\n3. Structure\n - `base/` environment-neutral; `components/` for {{COMPONENTS}}; `overlays//` with only real differences\n - Strategic merge patches for values; JSON 6902 patches with explicit targets for list edits (`args`, `command`, env lists without a merge key) and removals\n - Small, named patch files; no overlay copies a full base resource\n\n4. Labels and selectors\n - Stable identity labels in base selectors; extra labels via `labels:` with `includeSelectors: false`\n - Flag any change that would alter an existing Deployment or StatefulSet selector and stop for confirmation instead of applying it\n\n5. Images and config\n - Images set with the `images:` field in each overlay; prod pinned to a digest or immutable tag; no `latest` in any rendered output\n - ConfigMaps from `configMapGenerator` with hash suffixes on; document any `disableNameSuffixHash`\n\n6. Secrets and identity\n - No secret values or committed secret files; use a SecretProviderClass for Azure Key Vault through the Secrets Store CSI driver with workload identity from {{IDENTITY}}\n - List any secret that was committed in history so it can be rotated\n\n7. CI gate in {{CI}}\n - For each overlay: build with pinned kustomize, fail on warnings; kubeconform strict against {{K8S_VERSIONS}} with CRD schemas; policy check with the pinned tool\n - On pull requests, render each overlay on the base and head branches and publish the diff as a comment or artifact\n - Pin every tool version and action\n\nConstraints:\n- Minimal diff. Don't change resource requests, replicas, probes, or app config values unless a finding requires it; report findings instead.\n- Don't apply anything to a cluster. `kubectl diff -k` against non-production is the most you suggest running.\n- Don't invent cluster names, versions, IDs, or digests; use placeholders and state assumptions.\n- Don't add Helm, and don't enable Helm inflation inside Kustomize.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n\nOutput format:\n1. Current-state table with flagged issues\n2. New directory tree\n3. Patch (kustomization files, patches, components, generators, SecretProviderClass, CI job)\n4. Rendered-output comparison per overlay: identical, or each difference with its reason\n5. Risk list (selector changes, secrets to rotate, version gaps) and follow-ups\n\nStart from {{K8S_PATH}}. Render before you refactor, and prove the rendered output is unchanged unless you meant to change it.\n" } ] } diff --git a/skillbook/skills.json b/skillbook/skills.json index 58a600d..8b3f179 100644 --- a/skillbook/skills.json +++ b/skillbook/skills.json @@ -777,6 +777,26 @@ ], "formatNotes": "All three. Copilot: path-scoped `.github/instructions/llm-cost.instructions.md` with applyTo for `**/llm/**`, `**/gateway/**`, `**/*Client*.java`, `**/*Provider*.java`, `**/*Budget*.java`, `**/pricing/**`, and `**/application*.yml`. Optional `.github/skills/llm-cost-latency-budget-guardrails/SKILL.md`. Claude Code: `.claude/skills/llm-cost-latency-budget-guardrails/SKILL.md` plus a CLAUDE.md fact that every model call sets max output tokens, records provider usage with the resolved model and caller, prices it from the versioned table, and counts retries against budgets. Codex: an \"LLM cost and latency\" section in root `AGENTS.md` or a nested one in the gateway module. Pair with spring-boot-http-client-guardrails for timeouts and connection pools, spring-boot-micrometer-prometheus-guardrails for metric naming and cardinality, llm-eval-ci-regression-gate and rag-retrieval-eval-guardrails for the quality side of a model change. This skill owns usage metering, pricing, caps, budgets, retry accounting, and LLM latency stages.", "body": "# Skill: LLM Token Budget, Cost, and Latency Guardrails\n\n## Goal\nEvery LLM call your service makes is metered, attributed, and bounded. You can say what a request, a feature, a tenant, and a day cost, where the time went (queueing, time to first token, generation, retries), and a runaway prompt, retry loop, or agent cannot burn through a month of budget overnight. Numbers come from the provider's usage fields and a versioned price table, not from guesses, and they live next to quality metrics so a cheaper model is never judged on cost alone.\n\n## When to apply\n- Adding the first LLM call to a service, or putting several features behind a shared gateway or router (Azure OpenAI, OpenAI-compatible providers, OpenRouter, Together, Modal, Anthropic, local models)\n- A bill or a rate-limit error arrived that nobody can explain by feature or caller\n- Adding streaming, tool calling, agents, retries, fallbacks, or long-context RAG, all of which multiply tokens quietly\n- Choosing between models or providers, or turning on prompt caching or batch processing\n- Setting per-tenant, per-user, or per-feature quotas, or a latency SLO for a chat or completion endpoint\n\n## Baseline rules\n1. **Record usage from the response, not from your own tokenizer.** Read the provider's usage block on every call: input and output tokens, plus cached-input and reasoning tokens when the provider reports them separately. For streaming, request the final usage chunk (for OpenAI-compatible APIs, `stream_options: {\"include_usage\": true}`) and handle streams that end without one. Estimate with a local tokenizer only for pre-flight budget checks, and label estimates as such.\n2. **Price with a versioned table.** Keep a price table in config or a data file keyed by provider, model, and token type (input, cached input, output, reasoning if billed separately), with an effective date and currency. Compute cost at write time and store the table version with the record. Never hard-code prices in business logic, and never mix list prices with your negotiated rates without saying which one a report uses.\n3. **Attribute every call.** Each usage record carries a request ID, trace ID, feature or route, caller identity (service, tenant, or user, as a stable ID, not an email), requested model alias, resolved provider and model version, attempt number, and outcome (ok, error class, refused, cancelled, budget-blocked). Without the resolved model, a fallback silently changes your cost and quality.\n4. **Bound each request before it is sent.** Set an explicit max output tokens on every call. Cap input size (characters or estimated tokens) per route, cap retrieved context and tool results before they are added to the prompt, and cap tool-call iterations and total tokens per agent run. Reject or trim with a clear error instead of letting the provider fail at the context limit.\n5. **Enforce budgets in layers.** Per request (max tokens), per run (agent step and token cap), per caller per window (tokens or cost per minute and per day), and a global daily spend ceiling with an alert well before it. Reserve an estimate before the call and reconcile with actual usage after it, so concurrent requests can't all pass a nearly-empty budget. Return a distinct, documented status (for example HTTP 429 with a reason and `Retry-After`) when a budget blocks.\n6. **Retries and fallbacks cost money too.** Retry only retryable errors (timeouts, 429, 5xx) with capped exponential backoff and jitter, honour `Retry-After`, and set a total attempt and total time limit per request. Count every attempt's tokens against the caller's budget. Don't retry client errors such as 400 or 401 on the next provider unless you have decided that on purpose and documented it. Cancel the upstream call when the client disconnects.\n7. **Measure latency in stages.** Record total duration, time to first token for streams, output tokens per second, gateway overhead (auth, redaction, routing), and retry or backoff wait, as histograms with p50, p95, and p99. Tag by route, provider, model, and outcome. A p95 that hides a 30 s retry chain is not a latency number.\n8. **Keep metric labels low-cardinality.** Model, provider, route, outcome, and token type are labels. Tenant can be a label only if the tenant count is small; user IDs, request IDs, and prompt text are never labels. Put per-request detail in the usage store or traces, not in metric dimensions. If you follow the OpenTelemetry GenAI semantic conventions (for example `gen_ai.client.token.usage` and `gen_ai.client.operation.duration`), pin the convention version, because it is still evolving.\n9. **Store usage durably and privately.** Write usage records to your own store (a table or an append-only log) that survives metric retention, so monthly reports and chargeback can be rebuilt. Store token counts, cost, timings, and IDs; do not store prompts or completions there by default, and redact them anywhere they are logged.\n10. **Report cost next to quality.** Model and prompt changes are judged on a fixed eval set with quality, cost per request, and p95 latency side by side. Prompt caching, smaller models, and context trimming count as wins only if the quality metrics hold.\n11. **Test the guards.** Unit-test cost calculation against the price table, budget reservation and reconciliation under concurrency, streaming with and without a final usage chunk, retry accounting, and the budget-blocked response. Add a CI check that fails when a route has no max output tokens set.\n\n## Workflow\n1. **Inventory.** List every place the code calls a model: client, route or feature, model alias, max tokens, streaming, tools, retries, fallbacks, and who the caller is.\n2. **Meter.** Add a single usage hook (client interceptor, gateway filter, or callback) that reads provider usage, resolves the model, computes cost from the price table, and writes a usage record plus metrics, per rules 1–3, 7, and 8.\n3. **Bound.** Add max output tokens, input and context caps, and agent step caps per route (rule 4). Make the defaults conservative and override per route in config.\n4. **Budget.** Add caller and global budgets with reserve-then-reconcile and a clear blocked response (rule 5). Start in report-only mode for a few days, compare against real traffic, then enforce.\n5. **Fix retries.** Bring retry and fallback policy in line with rule 6 and count every attempt.\n6. **Report.** Build the cost report from the usage store: per day, feature, caller, and model, with p95 latency and error rate next to it, and an alert on spend ceiling and on sudden token-per-request jumps.\n7. **Verify.** Run the tests in rule 11 and a short load test with a deliberately tiny budget to see the block, the alert, and the report all line up.\n\n## Tool-specific notes\n- **GitHub Copilot:** path-scoped `.github/instructions/llm-cost.instructions.md` with `applyTo` for `**/llm/**`, `**/gateway/**`, `**/*Client*.java`, `**/*Provider*.java`, `**/*Budget*.java`, `**/pricing/**`, and `**/application*.yml`. Put rules 1, 4, 5, and 8 at the top. Optional `.github/skills/llm-cost-latency-budget-guardrails/SKILL.md`. Ask agent mode for the inventory table before any change.\n- **Claude Code:** `.claude/skills/llm-cost-latency-budget-guardrails/SKILL.md` with the workflow above, plus a short `CLAUDE.md` fact: \"Every model call sets max output tokens, records provider usage with the resolved model and caller, prices it from the versioned price table, and counts retries against the caller's budget; prompts are never stored in the usage store.\"\n- **Codex:** an \"LLM cost and latency\" section in root `AGENTS.md`, or a nested `AGENTS.md` in the gateway or LLM module, with the commands that run the pricing and budget tests and the rule that price table changes go in their own commit with an effective date.\n\n## Anti-patterns\n- Counting tokens with a local tokenizer and calling it the bill\n- Streaming responses with no usage captured, so the busiest route reports zero cost\n- Prices hard-coded in a service class, or updated in place with no effective date\n- Usage records that show the alias `gpt-default` but not the model a fallback actually used\n- No max output tokens, so one prompt can generate until the context limit\n- An agent loop with no step or token cap\n- A daily budget checked after the call, so a burst of concurrent requests sails past it\n- Retrying a 400 on three providers and paying three times for the same bad request\n- User IDs or prompt text as metric labels\n- Prompts and completions written to the usage table \"for debugging\"\n- Switching to a cheaper model because the cost chart dropped, with no eval run\n\n## Done when\n- [ ] Every model call is in the inventory with route, caller, model alias, max tokens, streaming, tools, and retry policy\n- [ ] Usage records carry provider usage (including cached and reasoning tokens where reported), resolved model, caller, attempt, outcome, cost, and price table version\n- [ ] Streaming calls capture final usage, and missing usage is counted and alerted\n- [ ] Max output tokens, input and context caps, and agent step caps are set per route\n- [ ] Caller and global budgets use reserve-then-reconcile and return a documented blocked response\n- [ ] Retries and fallbacks are limited, honour `Retry-After`, and count against budgets\n- [ ] Latency is recorded in stages with p50, p95, and p99, using low-cardinality labels\n- [ ] A cost report per day, feature, caller, and model exists, with spend and token-spike alerts\n- [ ] Model and prompt changes are compared on quality, cost, and latency together\n- [ ] Pricing, budget, streaming, and retry-accounting tests pass in CI\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: llm-cost-latency-budget-guardrails\ndescription: Use when adding or reviewing LLM calls, a model gateway, or a router — record provider token usage (including streaming, cached, and reasoning tokens), price it from a versioned price table, attribute every call to route, caller, and resolved model, cap output, input, context, and agent steps, enforce per-caller and global budgets with reserve-then-reconcile, count retries and fallbacks against budgets, measure latency in stages with low-cardinality labels, keep a private usage store, and judge model changes on cost and quality together.\n---\n\nFollow LLM Token Budget, Cost, and Latency Guardrails: read token usage from every provider response, including the final usage chunk of a stream. Price calls from a versioned price table and store the table version with each record. Attribute each call to its route, caller, requested alias, resolved model, attempt, and outcome. Set max output tokens on every call and cap input, retrieved context, tool results, and agent steps. Enforce per-caller and global budgets by reserving before the call and reconciling after it, with a documented blocked response. Retry only retryable errors with capped backoff and count every attempt. Record total duration, time to first token, and retry wait as histograms with low-cardinality labels. Keep usage records without prompt text, and compare model or prompt changes on quality, cost, and latency together.\n```\n" + }, + { + "id": "kustomize-aks-overlay-guardrails", + "title": "Kustomize Overlay Guardrails for AKS", + "whenToUse": "Writing or reviewing Kustomize manifests for AKS or any Kubernetes cluster — a base, optional components, and a small overlay per environment; current fields only (resources, labels, patches, replacements) instead of the deprecated ones; strategic merge for values and JSON 6902 for list edits; stable selector labels; images pinned with the images field; ConfigMap generators that roll pods; no secret values in Git; pinned remote resources; and every overlay built, schema-validated, and policy-checked in CI with its rendered diff shown in the pull request.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "kubernetes", + "kustomize", + "aks", + "gitops", + "ci", + "devsecops" + ], + "formatNotes": "All three. Copilot: path-scoped `.github/instructions/kustomize.instructions.md` with applyTo for `**/kustomization.yaml`, `**/kustomization.yml`, `deploy/**/*.yaml`, `k8s/**/*.yaml`, and `.github/workflows/*k8s*.yml`. Optional `.github/skills/kustomize-aks-overlay-guardrails/SKILL.md`. Claude Code: `.claude/skills/kustomize-aks-overlay-guardrails/SKILL.md` plus a CLAUDE.md fact that every overlay must build without warnings, pass kubeconform and the policy check, and show its rendered diff in the PR. Codex: a \"Kubernetes manifests\" section in root `AGENTS.md` or a nested one in the deploy folder. Pair with aks-workload-hardening for pod security, k8s-probes-graceful-shutdown-guardrails and aks-hpa-requests-limits-guardrails for what goes inside the Deployment, aks-workload-identity-guardrails and azure-key-vault-app-integration for identity and secrets, and helm-chart-aks-guardrails if a chart is also in play. This skill owns manifest layout, Kustomize fields, patches, generators, and the render-validate-diff CI gate.", + "body": "# Skill: Kustomize Overlay Guardrails for AKS\n\n## Goal\nOne base and a small overlay per environment render the exact manifests that reach each AKS cluster. Every overlay builds cleanly, validates against the cluster's Kubernetes version, and shows its rendered diff in the pull request, so a reviewer sees what will change in the cluster rather than guessing from patch fragments. No secret values live in Git, images are pinned, and selector labels never change after the first deploy.\n\n## When to apply\n- Starting Kubernetes manifests for a new service, or replacing a Helm chart with plain manifests plus Kustomize\n- Adding a new environment (dev, test, stage, prod) or a new cluster or region\n- Reviewing a pull request that touches `kustomization.yaml`, a patch, a generator, or a component\n- Seeing deprecation warnings (`bases`, `commonLabels`, `patchesStrategicMerge`, `patchesJson6902`, `vars`) from `kustomize build`\n- An apply failed with \"field is immutable\" on a Deployment selector, or a ConfigMap change did not roll the pods\n\n## Baseline rules\n1. **Layout is base, components, overlays.** `base/` holds environment-neutral resources. `overlays//` holds only what differs: namespace, replicas, resources, image tag, config values. `components/` holds optional cross-cutting pieces (for example monitoring annotations, a PodDisruptionBudget, a high-availability topology spread) that some overlays opt into. An overlay never copies a whole resource from the base.\n2. **Use current fields only.** `resources` instead of `bases`, `labels` instead of `commonLabels`, `patches` instead of `patchesStrategicMerge` and `patchesJson6902`, `replacements` instead of `vars`. Run `kustomize edit fix` on old files and diff the rendered output before and after the migration.\n3. **Choose the patch type by intent.** Strategic merge patches for changing values; they read like the resource. JSON 6902 patches with an explicit `target` for list edits and removals, because strategic merge replaces lists without a merge key (`args`, `command`) wholesale. Keep each patch small and named for what it changes (`replicas.yaml`, `resources.yaml`).\n4. **Set selector labels once.** Put the stable identity labels (`app.kubernetes.io/name`, `app.kubernetes.io/instance`) in the base resources' selectors directly. Add common labels with `labels:` and `includeSelectors: false` unless you are creating the Deployments for the first time, because a Deployment's selector is immutable and changing it later forces delete-and-recreate.\n5. **Pin images in one place.** Set image names and tags or digests with the `images:` field in the overlay (or a CI step running `kustomize edit set image`), never by patching the container spec. Production uses a digest or an immutable tag from your registry; `latest` never appears in rendered output.\n6. **Let generators roll config.** Generate ConfigMaps with `configMapGenerator` and keep the default hash suffix, so a config change produces a new name and rolls the pods. Don't set `disableNameSuffixHash: true` unless something outside Kustomize references the name, and document why when you do.\n7. **No secret values in Git.** Don't use `secretGenerator` with literal values or committed files for real secrets. On AKS, mount secrets from Azure Key Vault through the Secrets Store CSI driver with workload identity, or use an operator you already run; the overlay only references the SecretProviderClass and the identity.\n8. **Pin remote resources.** Avoid remote bases. If you must use one, pin it to a tag or commit (`?ref=`) and review its rendered output like your own code. Don't enable Helm inflation inside Kustomize (`--enable-helm`) as a shortcut; render the chart separately if you truly need it.\n9. **Validate the rendered output in CI.** For every overlay: `kustomize build` with the same Kustomize version you deploy with (the one embedded in `kubectl` can lag the standalone binary), then schema validation against the target Kubernetes version (for example kubeconform in strict mode, with CRD schemas for your custom resources), then a policy check (for example kube-linter or your admission policies run locally). Fail the build on any error or deprecation warning.\n10. **Show the rendered diff in the pull request.** CI renders each overlay on the base branch and on the PR branch and posts or uploads the diff. Reviewers approve rendered changes, especially for prod.\n11. **Keep overlays boring.** No logic, no templating, no environment-specific copies of base resources. If two overlays need the same change, it belongs in the base or a component.\n\n## Workflow\n1. **Map.** List every overlay, what it changes from the base, the target cluster and namespace, and the Kubernetes version of each cluster.\n2. **Modernize.** Run `kustomize edit fix`, replace deprecated fields, and confirm the rendered output for each overlay is byte-identical, or explain each difference.\n3. **Restructure.** Move duplicated overlay content into the base or a component. Move image settings to `images:` and config to generators.\n4. **Remove secrets.** Replace any committed secret material with Key Vault references through the CSI driver and workload identity, and rotate anything that was ever committed.\n5. **Guard.** Add the CI job from rules 9 and 10 for every overlay, using pinned tool versions.\n6. **Verify.** Run `kubectl diff -k overlays/` against a non-production cluster, then deploy and watch the rollout.\n\n## Review checklist\n- [ ] Base, components, and overlays are separate; no overlay copies a full base resource\n- [ ] No deprecated fields; `kustomize build` prints no warnings\n- [ ] List edits and removals use JSON 6902 patches with explicit targets\n- [ ] Selector labels are stable; added labels use `includeSelectors: false`\n- [ ] Images are set with `images:`; prod uses a digest or immutable tag; no `latest`\n- [ ] ConfigMaps come from generators with hash suffixes left on\n- [ ] No secret values in Git; secrets come from Key Vault via the CSI driver and workload identity\n- [ ] Remote resources are absent or pinned to a ref\n- [ ] CI builds, schema-validates, and policy-checks every overlay with pinned tool versions\n- [ ] The PR shows the rendered diff for each changed overlay\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: kustomize-aks-overlay-guardrails\ndescription: Use when writing or reviewing Kustomize manifests for AKS or any Kubernetes cluster — base, components, and per-environment overlays; current fields only (resources, labels, patches, replacements); strategic merge for values and JSON 6902 for list edits; stable selector labels; images pinned with the images field; ConfigMap generators with hash suffixes; no secret values in Git; pinned remote resources; every overlay built, schema-validated, and policy-checked in CI with its rendered diff shown in the pull request.\n---\n\nFollow Kustomize Overlay Guardrails: keep environment-neutral resources in base, optional cross-cutting pieces in components, and only real differences in each overlay. Use resources, labels, patches, and replacements, never the deprecated fields. Use strategic merge patches for values and JSON 6902 patches with explicit targets for list edits and removals. Never change Deployment selector labels after the first deploy; add labels with includeSelectors false. Set images with the images field and pin prod to a digest or immutable tag. Generate ConfigMaps with hash suffixes so config changes roll pods. Keep secret values out of Git and mount them from Azure Key Vault through the CSI driver and workload identity. Build, validate, and policy-check every overlay in CI with pinned tool versions, and show the rendered diff in the pull request.\n```\n" } ] }