diff --git a/promptbook/prompts.json b/promptbook/prompts.json index 12c0d90..eb171f9 100644 --- a/promptbook/prompts.json +++ b/promptbook/prompts.json @@ -759,6 +759,26 @@ ], "formatNotes": "All three. Point the tool at the ingest and chunking code, the embedding and vector-store or search client, the retriever and re-ranker, the prompt template and the class that builds the chat request with citations, the eval folder and golden set, and the CI workflow. Copilot: rag-eval path instructions plus those files in agent mode (or save it as `.github/prompts/evaluate-rag-retrieval-quality.prompt.md`). Claude Code: the rag-retrieval-eval-guardrails skill plus the files. Codex: AGENTS.md RAG evals section, then the diff.", "body": "Task: Evaluate the retrieval and grounding quality of the RAG feature I specify. Build or audit a golden question set with labelled evidence, score the retriever on its own, score answers for faithfulness, citation correctness, and refusal when evidence is missing, calibrate any LLM judge against human labels, and finish with a minimal diff that runs these evals in CI with regression thresholds.\n\nContext placeholders (fill from the repo or ask only if missing):\n- RAG module root: {{APP_PATH}}\n- Stack: {{STACK}} (Spring AI | LangChain4j | LlamaIndex | LangChain | custom)\n- Retriever: {{RETRIEVER}} (vector store and distance metric | keyword search | hybrid; plus re-ranker or none)\n- Chunking and embedding today: {{CHUNKING}} (splitter, chunk size, overlap; embedding model and dimensions)\n- Top-k used in the prompt: {{TOP_K}}\n- Corpus snapshot: {{CORPUS}} (where it lives and how it is versioned)\n- Golden set: {{GOLDEN}} (e.g. evals/rag/golden.jsonl, or \"none yet\")\n- Change under test: {{CHANGE}} (e.g. chunk 512→1024, new embedding model, add re-ranker, new prompt, new model, or \"baseline only\")\n- Judge model: {{JUDGE}} (pinned version, or none)\n- Eval runner and CI: {{EVAL_RUNNER}} (pytest | JUnit | Ragas or similar library | custom harness; GitHub Actions | Jenkins)\n- Budgets: {{BUDGETS}} (e.g. p95 latency ≤ 2.5 s, cost per query ≤ $0.004)\n\nRequirements:\n\n1. Inventory\n - Document the pipeline in {{APP_PATH}}: loaders, splitter and chunk settings, embedding model, index and filters, hybrid weights, {{TOP_K}}, re-ranker, prompt template and version, model, and where citations and refusals are produced\n - List existing eval code, data, and metrics, and note anything unpinned (model aliases, corpus not versioned, judge unpinned)\n - Flag any golden or fixture data that contains real customer data or PHI\n\n2. Golden set\n - Each case: stable `id`, `question`, `relevant_ids` as stable document IDs plus span or required facts (graded 0–3 if possible), short reference answer or required facts, and tags (lookup, multi-hop, comparison, numeric, recent-change, no-answer, ambiguous, past-incident)\n - At least 10% `no-answer` cases whose answer is not in {{CORPUS}}\n - A frozen held-out slice that is not used for tuning\n - Map document-level labels onto current chunks at eval time so a chunking change doesn't break labels\n - If {{GOLDEN}} is missing, propose 30 starter cases from the corpus and mark them for human review; drop synthetic questions that copy chunk text nearly verbatim\n - Don't rewrite or delete existing cases; flag questionable ones in the report\n\n3. Retrieval metrics (retriever only, no LLM)\n - recall@k, MRR, and nDCG@k at {{TOP_K}} (and at 2×{{TOP_K}} to show headroom), context precision at k, and retrieved-context tokens, overall and per tag\n\n4. Generation metrics (end to end)\n - Claim-level faithfulness against retrieved context only: supported-claim share and the count of answers with any unsupported claim\n - Citation validity (every cited ID was in the prompt for that request, checked in code), citation precision, and citation recall\n - Correct-refusal rate on no-answer cases and on cases with forced irrelevant context; false-refusal rate on answerable cases\n - Answer correctness against the reference or required facts\n - Per-query retrieval, re-rank, and generation latency (p50/p95), prompt, completion, and context tokens, and cost\n\n5. Judge calibration (only if a judge is used)\n - Hand-label 100–200 answers with a written binary rubric per claim, split dev and held-out\n - Report judge precision, recall, and Cohen's kappa on held-out labels; pin judge model, prompt, temperature, and parser\n - Keep judge metrics warn-only until agreement is acceptable; the model under test never judges itself\n\n6. Compare {{CHANGE}}\n - Same golden set, same corpus snapshot, same {{TOP_K}} unless top-k is the change; full re-embed for embedding changes\n - Paired per-case comparison against the current configuration, per tag, with token, latency, and cost deltas\n - Stage-tag every failure: not retrieved, ranked below k, in context but ignored, unsupported claim, wrong citation, should have refused\n\n7. CI gate\n - Commit the accepted run as a baseline (separate commit, with the pins it came from)\n - Per-metric floor and tolerance: recall@k, MRR, faithfulness, citation validity, correct-refusal rate; budgets from {{BUDGETS}} for p95 latency and cost per query\n - Deterministic retrieval and citation-validity metrics block now; calibrated judge metrics block once step 5 passes\n - Path-filter the job on retrieval, ingest, prompt, model config, and eval code; write a results file with pins and a short Markdown summary; exit non-zero on regression\n\n8. Leakage checks\n - Confirm no golden question or reference answer appears in prompts, few-shot examples, fine-tuning data, or re-ranker or embedder training data; add a test that fails if one does\n\nConstraints:\n- Minimal diff. Don't change retrieval or prompt behaviour unless {{CHANGE}} asks for it; report findings instead.\n- Don't edit the golden set, labels, or baseline in the same commit as a retrieval, prompt, or model change.\n- Don't invent scores, labels, or corpus contents. Use placeholders and state assumptions.\n- Don't put real customer data, PHI, or secrets in fixtures, logs, or reports.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave index migrations and model swaps as follow-up notes unless they are {{CHANGE}}.\n\nOutput format:\n1. Pipeline summary with pins and unpinned items\n2. Golden set report: case counts per tag, no-answer share, held-out size, flagged cases\n3. Metrics table: metric | baseline | candidate | delta | per-tag notes (retrieval, generation, refusal, cost, latency)\n4. Judge calibration table (if used): metric | precision | recall | kappa | blocking yes/no\n5. Stage-tagged failure table: case id | stage | evidence | suggested fix\n6. Patch (eval harness, metrics, gate config, CI wiring, leakage test) plus follow-ups\n\nStart from {{APP_PATH}}. Score the retriever on its own before judging any answer.\n" + }, + { + "id": "add-llm-token-budgets-and-cost-tracking", + "title": "Add LLM Token Budgets & Cost Tracking", + "whenToUse": "Meter, price, and bound the LLM calls in a service or gateway: inventory every model call, read token usage from provider responses (including streams), price it from a versioned table, attribute it to route, caller, and resolved model, add max-token, context, and agent-step caps, per-caller and global budgets with reserve-then-reconcile and a report-only mode, retry accounting, staged latency histograms, a cost report, and tests.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "llm", + "cost", + "latency", + "budgets", + "observability", + "gateway" + ], + "formatNotes": "All three. Point the tool at the model client or gateway code, the retry and fallback config, the application config with model aliases, the auth or caller-identity code, and the metrics setup. Copilot: llm-cost path instructions plus those files in agent mode (or save it as `.github/prompts/add-llm-token-budgets-and-cost-tracking.prompt.md`). Claude Code: the llm-cost-latency-budget-guardrails skill plus the files. Codex: AGENTS.md LLM cost and latency section, then the diff.", + "body": "Task: Add token budgets, cost tracking, and latency measurement to the LLM calls in the service I specify. Inventory every model call, meter usage from provider responses, price it from a versioned table, cap each request, add caller and global budgets, fix retry accounting, and finish with a minimal diff, tests, and a cost report.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Service or module root: {{APP_PATH}}\n- Stack: {{STACK}} (Spring Boot + Spring AI | LangChain4j | plain HTTP client | Python SDK | gateway)\n- Providers and models: {{PROVIDERS}} (e.g. Azure OpenAI deployments, OpenAI-compatible endpoints, OpenRouter, Anthropic)\n- Callers to attribute: {{CALLERS}} (service, tenant, user, or feature; and where the identity comes from)\n- Price source: {{PRICES}} (list prices with date, or negotiated rates; currency)\n- Budgets: {{BUDGETS}} (e.g. 200k tokens per caller per day, $50 global per day, max 8 agent steps)\n- Latency SLO: {{SLO}} (e.g. p95 time to first token ≤ 1.5 s, p95 total ≤ 8 s)\n- Usage store and metrics: {{STORE}} (database table, append-only log, Micrometer/OpenTelemetry metrics backend)\n- Mode: {{MODE}} (report-only | enforce)\n\nRequirements:\n\n1. Inventory\n - Table of every model call in {{APP_PATH}}: file and method, route or feature, caller, model alias, max output tokens (or \"not set\"), streaming, tools or agent loop, retry and fallback policy\n - Flag calls with no max output tokens, streams that never read usage, agent loops with no step cap, and retries of non-retryable errors\n\n2. Metering\n - One usage hook (client interceptor, gateway filter, or callback), not per-call code\n - Read provider usage: input, output, and, where reported, cached-input and reasoning tokens; for OpenAI-compatible streams request the final usage chunk and count streams that end without it\n - Record: request ID, trace ID, route, caller ID (stable ID, never an email), requested alias, resolved provider and model version, attempt number, outcome, token counts, timings, cost, price table version\n - Never store prompts or completions in the usage store; redact them in logs\n\n3. Pricing\n - Versioned price table from {{PRICES}} keyed by provider, model, and token type, with effective date and currency\n - Cost computed at write time; unit tests for every token type and for a model missing from the table (record it, flag it, don't price it as zero)\n\n4. Request caps\n - Explicit max output tokens per route, input and retrieved-context caps, tool-result size cap, agent step and total-token cap per run\n - A clear error or trim policy when a cap is hit; a CI or unit check that fails when a route has no max output tokens\n\n5. Budgets\n - Per-caller per-window and global daily budgets from {{BUDGETS}}\n - Reserve an estimate before the call, reconcile with actual usage after it, release on failure; safe under concurrency\n - Blocked response: documented status (e.g. 429 with reason and Retry-After) and a metric; in report-only {{MODE}}, log and count would-be blocks without rejecting\n - Alert at a threshold below the global ceiling and on sudden jumps in tokens per request\n\n6. Retries and fallbacks\n - Retry only timeouts, 429, and 5xx, with capped exponential backoff and jitter, honouring Retry-After, with a total attempt and time limit\n - Every attempt's tokens count against the caller's budget; record which provider and model each attempt used\n - Cancel the upstream call when the client disconnects\n\n7. Latency\n - Histograms for total duration, time to first token, output tokens per second, gateway overhead, and retry wait, tagged by route, provider, model, and outcome\n - Labels stay low-cardinality: no user IDs, request IDs, or prompt text; if using OpenTelemetry GenAI conventions, pin the version\n - Report against {{SLO}}\n\n8. Report and tests\n - A query or endpoint that reports cost, tokens, p95 latency, and error rate per day, route, caller, and model from {{STORE}}\n - Tests: pricing, reserve-and-reconcile under concurrency, streaming with and without final usage, retry accounting, blocked response, missing-price handling\n\nConstraints:\n- Minimal diff. Don't change prompts, models, or routing unless a finding requires it; report findings instead.\n- Don't invent prices, token counts, or traffic numbers. Use placeholders and state assumptions.\n- Don't put prompts, completions, secrets, or personal data in the usage store, metrics, or test fixtures.\n- Put price table changes in their own commit with an effective date.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave dashboard building and provider migrations as follow-up notes.\n\nOutput format:\n1. Inventory table with flagged calls\n2. Usage record schema and where it is written\n3. Caps and budgets table: route or caller | limit | window | current behaviour | new behaviour\n4. Retry and fallback policy before and after\n5. Patch (usage hook, price table, caps, budgets, retry changes, metrics, report query, tests)\n6. Sample report output with placeholder numbers clearly marked, plus follow-ups\n\nStart from {{APP_PATH}}. Meter before you limit: get honest usage numbers first, then add caps and budgets.\n" } ] } diff --git a/skillbook/skills.json b/skillbook/skills.json index 91047ee..58a600d 100644 --- a/skillbook/skills.json +++ b/skillbook/skills.json @@ -757,6 +757,26 @@ ], "formatNotes": "All three. Copilot: path-scoped `.github/instructions/rag-eval.instructions.md` with applyTo for `**/rag/**`, `**/retrieval/**`, `**/*Retriever*.java`, `**/*Chunk*.java`, `**/*Embedding*.java`, `**/ingest/**`, `**/prompts/**`, and `evals/**`. Optional `.github/skills/rag-retrieval-eval-guardrails/SKILL.md`. Claude Code: `.claude/skills/rag-retrieval-eval-guardrails/SKILL.md` plus a CLAUDE.md fact that retrieval changes are judged on recall@k and MRR against the golden set, answers must be grounded with valid citations, and golden questions never go into prompts or training data. Codex: a \"RAG retrieval and grounding evals\" section in root `AGENTS.md` or a nested `evals/AGENTS.md`. Pair with llm-eval-ci-regression-gate for the CI gate mechanics (baseline, noise floor, workflow permissions), llm-app-prompt-injection-guardrails for access-filtered retrieval and injection red-teaming, and llm-eval-agent-workflow for coding-agent rubrics. This skill owns the golden evidence labels, retrieval metrics, grounding and citation checks, refusal cases, judge calibration, and leakage control.", "body": "# Skill: RAG Retrieval and Grounding Evaluation Guardrails\n\n## Goal\nKnow, with numbers, whether a RAG feature finds the right evidence and answers only from it. Retrieval is scored on its own against labelled relevant chunks, generation is scored on faithfulness, citation correctness, and refusal when the evidence is missing, and every chunking, embedding, index, re-ranker, prompt, or model change is compared on the same frozen question set before it ships. When an answer is wrong, the eval report tells you which stage broke: retrieval missed the chunk, the ranker buried it, or the model ignored it.\n\n## When to apply\n- Building or reviewing a RAG service (Spring AI, LangChain4j, LlamaIndex, LangChain, or hand-rolled) over a vector store, search index, or hybrid of both\n- Changing chunk size or overlap, the splitter, the embedding model or its dimensions, the index type, top-k, hybrid weights, filters, or the re-ranker\n- Answer quality complaints that nobody can pin on retrieval or on the model\n- Adding citations, source links, or \"I don't know\" behaviour to an assistant\n- Introducing an LLM judge for faithfulness or relevance and needing to know whether to trust it\n- Before a model, prompt, or knowledge-base migration, and after any re-index\n\n## Baseline rules\n1. **Build a golden question set with labelled evidence.** Each case has a stable `id`, the `question`, the IDs of the chunks or documents that answer it (`relevant_ids`, optionally graded 0–3), a short reference answer or list of required facts, and tags (`lookup`, `multi-hop`, `comparison`, `numeric`, `recent-change`, `no-answer`, `ambiguous`, `past-incident`). Start with 100–300 cases written or reviewed by people who know the content, mixed with real (redacted) user questions. No real customer data or PHI in fixtures.\n2. **Label documents, not just chunk offsets.** Chunk IDs change when you re-chunk. Store relevance against a stable document ID plus a span or a set of required facts, and map them onto the current chunks at eval time, so a chunking experiment doesn't invalidate the labels.\n3. **Score retrieval separately from generation.** Run the retriever alone and compute deterministic ranking metrics at the k the prompt actually uses: recall@k (did any or all relevant evidence arrive), MRR (how high the first hit lands), and nDCG@k when you have graded labels. Also report context precision at k (how much of the prompt is noise) and the hit rate per tag. These need no LLM and should be the first blocking numbers.\n4. **Compare retrieval changes like experiments.** Chunking, embedding, index, hybrid weighting, and re-ranker changes run against the same golden set and the same corpus snapshot, side by side with the current configuration, per tag, with token cost of the retrieved context. Re-embed the full corpus for an embedding change; never mix vectors from two models in one index. Report the paired difference per case, not just the means.\n5. **Faithfulness is claim-level.** Split each answer into atomic claims and check every claim against the retrieved context only, not against the reference or the model's own knowledge. Report the share of supported claims per answer and the count of answers with any unsupported claim. A correct answer drawn from parametric memory still fails faithfulness, because it will fail silently when the corpus changes.\n6. **Check citations mechanically first.** Every citation must point to a chunk that was in the prompt for that request (deterministic check), and the cited chunk must support the sentence it is attached to (judge check). Track citation precision (cited chunks that support the claim) and citation recall (claims that carry a valid citation). A link to a real document that doesn't say the thing is a defect, not a near-miss.\n7. **Refusal when evidence is missing is a metric, not a hope.** Keep `no-answer` cases whose answer is not in the corpus, plus cases where retrieval is forced to return irrelevant chunks. Measure correct-refusal rate on those and false-refusal rate on answerable cases; both matter. The app decides refusal from evidence (top score below threshold, no supporting chunk) as well as from the model's wording.\n8. **Calibrate any LLM judge before it gates.** Label 100–200 answers by hand with a written rubric (binary supported or not supported per claim, with examples of borderline cases), split into dev and held-out sets, and measure judge agreement (precision, recall, Cohen's kappa) on held-out labels. Pin the judge model, prompt, temperature, and parser. Keep a judge metric warn-only until agreement is acceptable, re-check it after any judge change, and never let the model under test judge itself.\n9. **Track cost and latency per query.** Record retrieval latency, re-ranker latency, generation latency, prompt and completion tokens, retrieved-context tokens, and cost per query, as p50/p95 per tag. A retrieval change that adds two points of recall and doubles context tokens is a trade-off to decide on, not a free win.\n10. **Gate regressions in CI with per-metric thresholds.** Commit a baseline of the accepted run and fail when recall@k, MRR, faithfulness, citation validity, or correct-refusal rate drops more than its tolerance or under its floor, or when p95 latency or cost per query exceeds its budget. Deterministic retrieval metrics block from day one; calibrated judge metrics block once rule 8 is met. Pin the corpus snapshot, embedding model, index settings, and model versions in the results file.\n11. **Keep the test set from leaking.** Never use golden questions or reference answers as few-shot examples, in prompts, in fine-tuning data, or as synthetic training for the embedder or re-ranker. Keep a held-out slice that nobody tunes against and look at it only at release time. When you generate synthetic questions from the corpus, have a person review them and drop near-copies of the chunk text, since they inflate recall.\n12. **Keep the corpus and the set in step.** Version the corpus snapshot used for evals. When documents are added, removed, or rewritten, re-validate the labels for affected cases in a reviewed change, and add new cases for content users actually ask about (sample from logs, redacted).\n\n## Workflow\n1. **Inventory.** Document the pipeline: loaders, splitter and chunk settings, embedding model and dimensions, index and distance metric, filters, hybrid weights, top-k, re-ranker, prompt template, model, and where citations and refusals are produced.\n2. **Golden set.** Build or audit the set per rules 1, 2, and 11: coverage of every tag, stable document-level labels, a `no-answer` slice of at least 10%, and a frozen held-out slice.\n3. **Retrieval baseline.** Run the retriever alone and record recall@k, MRR, nDCG@k, context precision, and context tokens per tag on the pinned corpus snapshot.\n4. **Generation baseline.** Run end to end and record claim-level faithfulness, citation validity, citation precision and recall, correct- and false-refusal rates, and answer correctness against the reference, plus per-query latency and cost.\n5. **Calibrate the judge.** Hand-label a sample, measure agreement on held-out labels, fix the rubric or judge, and mark which judge metrics may block.\n6. **Diagnose.** For every failing case, tag the stage: not retrieved, retrieved but ranked below k, in context but ignored, unsupported claim added, wrong citation, should have refused. Fix the stage with the most failures first.\n7. **Experiment and gate.** Change one thing at a time, compare paired per-case results against the baseline, and wire the accepted thresholds into CI (pair with llm-eval-ci-regression-gate for the mechanics).\n\n## Tool-specific notes\n- **GitHub Copilot:** path-scoped `.github/instructions/rag-eval.instructions.md` with `applyTo` for `**/rag/**`, `**/retrieval/**`, `**/*Retriever*.java`, `**/*Chunk*.java`, `**/*Embedding*.java`, `**/ingest/**`, `**/prompts/**`, and `evals/**`. Put rules 3, 5, 7, and 11 at the top. Optional `.github/skills/rag-retrieval-eval-guardrails/SKILL.md`. Ask agent mode for the stage-tagged failure table before any change.\n- **Claude Code:** `.claude/skills/rag-retrieval-eval-guardrails/SKILL.md` with the workflow above, plus a short `CLAUDE.md` fact: \"Retrieval changes are judged on recall@k and MRR against evals/golden; answers must be grounded in retrieved chunks with valid citations; never put golden questions into prompts or training data.\"\n- **Codex:** a \"RAG retrieval and grounding evals\" section in root `AGENTS.md`, or a nested `evals/AGENTS.md`, with the commands that run the retrieval-only and end-to-end evals and the rule not to edit the golden set, labels, or baseline in the same change as a retrieval or prompt change.\n\n## Anti-patterns\n- Judging a RAG change only by end-to-end answer scores, with no retrieval-only metrics\n- Labels keyed to chunk offsets that silently break on the next re-chunk\n- Comparing chunk sizes on different corpus snapshots or with different top-k\n- Re-indexing half the corpus with a new embedding model\n- Counting an answer as good because it matches the reference, even though the retrieved context never said it\n- Citations rendered from document titles the model invented, with no check that the chunk was in the prompt\n- No unanswerable questions in the set, so the assistant is never tested on saying \"I don't know\"\n- A 1–10 judge score with no human labels gating merges, or the model under test grading itself\n- Golden questions reused as few-shot examples or as synthetic training data for the re-ranker\n- Ignoring the extra context tokens and p95 latency that came with a better recall number\n\n## Done when\n- [ ] Pipeline settings (chunking, embedding, index, filters, top-k, re-ranker, prompt, model) are documented and pinned in eval results\n- [ ] A versioned golden set has document-level relevance labels, tags, a no-answer slice, and a frozen held-out slice, with no real customer data\n- [ ] Retrieval-only recall@k, MRR, nDCG@k, and context precision are reported per tag\n- [ ] Claim-level faithfulness, citation validity, citation precision and recall, and correct- and false-refusal rates are reported\n- [ ] Any LLM judge is calibrated against human labels on held-out data before it blocks\n- [ ] Per-query latency (p50/p95), tokens, and cost are tracked next to quality\n- [ ] CI fails on threshold breaches against a committed baseline\n- [ ] Golden data is kept out of prompts, few-shot examples, and training data\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: rag-retrieval-eval-guardrails\ndescription: Use when building, changing, or reviewing a RAG pipeline (chunking, embeddings, index, top-k, re-ranker, prompt, model) — golden question set with document-level relevance labels, retrieval-only recall@k, MRR, and nDCG, paired comparison of chunking and embedding changes, claim-level faithfulness, citation correctness, refusal when evidence is missing, LLM judge calibration against human labels, per-query cost and latency, CI regression thresholds, and test-set leakage control.\n---\n\nFollow RAG Retrieval and Grounding Evaluation Guardrails: score retrieval and generation separately. Keep a versioned golden set with document-level relevance labels, tags, a no-answer slice, and a frozen held-out slice. Report recall@k, MRR, and nDCG@k for the retriever alone at the k the prompt uses. Compare chunking, embedding, and re-ranker changes on the same corpus snapshot with paired per-case results. Check faithfulness claim by claim against retrieved context only, verify every citation points to a chunk in the prompt that supports the sentence, and measure correct- and false-refusal rates. Calibrate any LLM judge against human labels before it blocks. Track latency, tokens, and cost per query. Gate regressions in CI against a committed baseline, and never put golden questions into prompts, few-shot examples, or training data.\n```\n" + }, + { + "id": "llm-cost-latency-budget-guardrails", + "title": "LLM Token Budget, Cost & Latency Guardrails", + "whenToUse": "Adding or reviewing LLM calls, a model gateway, or a router and needing to know and bound what each call costs and how long it takes — token usage read from provider responses (including streaming final usage, cached and reasoning tokens), a versioned price table, per-call attribution to route, caller, and resolved model, max output, input, context, and agent-step caps, per-caller and global budgets with reserve-then-reconcile, retries and fallbacks counted against budgets, staged latency histograms with low-cardinality labels, a private usage store for reports, and cost judged next to quality.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "llm", + "cost", + "latency", + "budgets", + "observability", + "gateway" + ], + "formatNotes": "All three. Copilot: path-scoped `.github/instructions/llm-cost.instructions.md` with applyTo for `**/llm/**`, `**/gateway/**`, `**/*Client*.java`, `**/*Provider*.java`, `**/*Budget*.java`, `**/pricing/**`, and `**/application*.yml`. Optional `.github/skills/llm-cost-latency-budget-guardrails/SKILL.md`. Claude Code: `.claude/skills/llm-cost-latency-budget-guardrails/SKILL.md` plus a CLAUDE.md fact that every model call sets max output tokens, records provider usage with the resolved model and caller, prices it from the versioned table, and counts retries against budgets. Codex: an \"LLM cost and latency\" section in root `AGENTS.md` or a nested one in the gateway module. Pair with spring-boot-http-client-guardrails for timeouts and connection pools, spring-boot-micrometer-prometheus-guardrails for metric naming and cardinality, llm-eval-ci-regression-gate and rag-retrieval-eval-guardrails for the quality side of a model change. This skill owns usage metering, pricing, caps, budgets, retry accounting, and LLM latency stages.", + "body": "# Skill: LLM Token Budget, Cost, and Latency Guardrails\n\n## Goal\nEvery LLM call your service makes is metered, attributed, and bounded. You can say what a request, a feature, a tenant, and a day cost, where the time went (queueing, time to first token, generation, retries), and a runaway prompt, retry loop, or agent cannot burn through a month of budget overnight. Numbers come from the provider's usage fields and a versioned price table, not from guesses, and they live next to quality metrics so a cheaper model is never judged on cost alone.\n\n## When to apply\n- Adding the first LLM call to a service, or putting several features behind a shared gateway or router (Azure OpenAI, OpenAI-compatible providers, OpenRouter, Together, Modal, Anthropic, local models)\n- A bill or a rate-limit error arrived that nobody can explain by feature or caller\n- Adding streaming, tool calling, agents, retries, fallbacks, or long-context RAG, all of which multiply tokens quietly\n- Choosing between models or providers, or turning on prompt caching or batch processing\n- Setting per-tenant, per-user, or per-feature quotas, or a latency SLO for a chat or completion endpoint\n\n## Baseline rules\n1. **Record usage from the response, not from your own tokenizer.** Read the provider's usage block on every call: input and output tokens, plus cached-input and reasoning tokens when the provider reports them separately. For streaming, request the final usage chunk (for OpenAI-compatible APIs, `stream_options: {\"include_usage\": true}`) and handle streams that end without one. Estimate with a local tokenizer only for pre-flight budget checks, and label estimates as such.\n2. **Price with a versioned table.** Keep a price table in config or a data file keyed by provider, model, and token type (input, cached input, output, reasoning if billed separately), with an effective date and currency. Compute cost at write time and store the table version with the record. Never hard-code prices in business logic, and never mix list prices with your negotiated rates without saying which one a report uses.\n3. **Attribute every call.** Each usage record carries a request ID, trace ID, feature or route, caller identity (service, tenant, or user, as a stable ID, not an email), requested model alias, resolved provider and model version, attempt number, and outcome (ok, error class, refused, cancelled, budget-blocked). Without the resolved model, a fallback silently changes your cost and quality.\n4. **Bound each request before it is sent.** Set an explicit max output tokens on every call. Cap input size (characters or estimated tokens) per route, cap retrieved context and tool results before they are added to the prompt, and cap tool-call iterations and total tokens per agent run. Reject or trim with a clear error instead of letting the provider fail at the context limit.\n5. **Enforce budgets in layers.** Per request (max tokens), per run (agent step and token cap), per caller per window (tokens or cost per minute and per day), and a global daily spend ceiling with an alert well before it. Reserve an estimate before the call and reconcile with actual usage after it, so concurrent requests can't all pass a nearly-empty budget. Return a distinct, documented status (for example HTTP 429 with a reason and `Retry-After`) when a budget blocks.\n6. **Retries and fallbacks cost money too.** Retry only retryable errors (timeouts, 429, 5xx) with capped exponential backoff and jitter, honour `Retry-After`, and set a total attempt and total time limit per request. Count every attempt's tokens against the caller's budget. Don't retry client errors such as 400 or 401 on the next provider unless you have decided that on purpose and documented it. Cancel the upstream call when the client disconnects.\n7. **Measure latency in stages.** Record total duration, time to first token for streams, output tokens per second, gateway overhead (auth, redaction, routing), and retry or backoff wait, as histograms with p50, p95, and p99. Tag by route, provider, model, and outcome. A p95 that hides a 30 s retry chain is not a latency number.\n8. **Keep metric labels low-cardinality.** Model, provider, route, outcome, and token type are labels. Tenant can be a label only if the tenant count is small; user IDs, request IDs, and prompt text are never labels. Put per-request detail in the usage store or traces, not in metric dimensions. If you follow the OpenTelemetry GenAI semantic conventions (for example `gen_ai.client.token.usage` and `gen_ai.client.operation.duration`), pin the convention version, because it is still evolving.\n9. **Store usage durably and privately.** Write usage records to your own store (a table or an append-only log) that survives metric retention, so monthly reports and chargeback can be rebuilt. Store token counts, cost, timings, and IDs; do not store prompts or completions there by default, and redact them anywhere they are logged.\n10. **Report cost next to quality.** Model and prompt changes are judged on a fixed eval set with quality, cost per request, and p95 latency side by side. Prompt caching, smaller models, and context trimming count as wins only if the quality metrics hold.\n11. **Test the guards.** Unit-test cost calculation against the price table, budget reservation and reconciliation under concurrency, streaming with and without a final usage chunk, retry accounting, and the budget-blocked response. Add a CI check that fails when a route has no max output tokens set.\n\n## Workflow\n1. **Inventory.** List every place the code calls a model: client, route or feature, model alias, max tokens, streaming, tools, retries, fallbacks, and who the caller is.\n2. **Meter.** Add a single usage hook (client interceptor, gateway filter, or callback) that reads provider usage, resolves the model, computes cost from the price table, and writes a usage record plus metrics, per rules 1–3, 7, and 8.\n3. **Bound.** Add max output tokens, input and context caps, and agent step caps per route (rule 4). Make the defaults conservative and override per route in config.\n4. **Budget.** Add caller and global budgets with reserve-then-reconcile and a clear blocked response (rule 5). Start in report-only mode for a few days, compare against real traffic, then enforce.\n5. **Fix retries.** Bring retry and fallback policy in line with rule 6 and count every attempt.\n6. **Report.** Build the cost report from the usage store: per day, feature, caller, and model, with p95 latency and error rate next to it, and an alert on spend ceiling and on sudden token-per-request jumps.\n7. **Verify.** Run the tests in rule 11 and a short load test with a deliberately tiny budget to see the block, the alert, and the report all line up.\n\n## Tool-specific notes\n- **GitHub Copilot:** path-scoped `.github/instructions/llm-cost.instructions.md` with `applyTo` for `**/llm/**`, `**/gateway/**`, `**/*Client*.java`, `**/*Provider*.java`, `**/*Budget*.java`, `**/pricing/**`, and `**/application*.yml`. Put rules 1, 4, 5, and 8 at the top. Optional `.github/skills/llm-cost-latency-budget-guardrails/SKILL.md`. Ask agent mode for the inventory table before any change.\n- **Claude Code:** `.claude/skills/llm-cost-latency-budget-guardrails/SKILL.md` with the workflow above, plus a short `CLAUDE.md` fact: \"Every model call sets max output tokens, records provider usage with the resolved model and caller, prices it from the versioned price table, and counts retries against the caller's budget; prompts are never stored in the usage store.\"\n- **Codex:** an \"LLM cost and latency\" section in root `AGENTS.md`, or a nested `AGENTS.md` in the gateway or LLM module, with the commands that run the pricing and budget tests and the rule that price table changes go in their own commit with an effective date.\n\n## Anti-patterns\n- Counting tokens with a local tokenizer and calling it the bill\n- Streaming responses with no usage captured, so the busiest route reports zero cost\n- Prices hard-coded in a service class, or updated in place with no effective date\n- Usage records that show the alias `gpt-default` but not the model a fallback actually used\n- No max output tokens, so one prompt can generate until the context limit\n- An agent loop with no step or token cap\n- A daily budget checked after the call, so a burst of concurrent requests sails past it\n- Retrying a 400 on three providers and paying three times for the same bad request\n- User IDs or prompt text as metric labels\n- Prompts and completions written to the usage table \"for debugging\"\n- Switching to a cheaper model because the cost chart dropped, with no eval run\n\n## Done when\n- [ ] Every model call is in the inventory with route, caller, model alias, max tokens, streaming, tools, and retry policy\n- [ ] Usage records carry provider usage (including cached and reasoning tokens where reported), resolved model, caller, attempt, outcome, cost, and price table version\n- [ ] Streaming calls capture final usage, and missing usage is counted and alerted\n- [ ] Max output tokens, input and context caps, and agent step caps are set per route\n- [ ] Caller and global budgets use reserve-then-reconcile and return a documented blocked response\n- [ ] Retries and fallbacks are limited, honour `Retry-After`, and count against budgets\n- [ ] Latency is recorded in stages with p50, p95, and p99, using low-cardinality labels\n- [ ] A cost report per day, feature, caller, and model exists, with spend and token-spike alerts\n- [ ] Model and prompt changes are compared on quality, cost, and latency together\n- [ ] Pricing, budget, streaming, and retry-accounting tests pass in CI\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: llm-cost-latency-budget-guardrails\ndescription: Use when adding or reviewing LLM calls, a model gateway, or a router — record provider token usage (including streaming, cached, and reasoning tokens), price it from a versioned price table, attribute every call to route, caller, and resolved model, cap output, input, context, and agent steps, enforce per-caller and global budgets with reserve-then-reconcile, count retries and fallbacks against budgets, measure latency in stages with low-cardinality labels, keep a private usage store, and judge model changes on cost and quality together.\n---\n\nFollow LLM Token Budget, Cost, and Latency Guardrails: read token usage from every provider response, including the final usage chunk of a stream. Price calls from a versioned price table and store the table version with each record. Attribute each call to its route, caller, requested alias, resolved model, attempt, and outcome. Set max output tokens on every call and cap input, retrieved context, tool results, and agent steps. Enforce per-caller and global budgets by reserving before the call and reconciling after it, with a documented blocked response. Retry only retryable errors with capped backoff and count every attempt. Record total duration, time to first token, and retry wait as histograms with low-cardinality labels. Keep usage records without prompt text, and compare model or prompt changes on quality, cost, and latency together.\n```\n" } ] }