Add LLM token budget, cost and latency guardrails

promptbook/prompts.json

@@ -759,6 +759,26 @@
       ],
       "formatNotes": "All three. Point the tool at the ingest and chunking code, the embedding and vector-store or search client, the retriever and re-ranker, the prompt template and the class that builds the chat request with citations, the eval folder and golden set, and the CI workflow. Copilot: rag-eval path instructions plus those files in agent mode (or save it as `.github/prompts/evaluate-rag-retrieval-quality.prompt.md`). Claude Code: the rag-retrieval-eval-guardrails skill plus the files. Codex: AGENTS.md RAG evals section, then the diff.",
       "body": "Task: Evaluate the retrieval and grounding quality of the RAG feature I specify. Build or audit a golden question set with labelled evidence, score the retriever on its own, score answers for faithfulness, citation correctness, and refusal when evidence is missing, calibrate any LLM judge against human labels, and finish with a minimal diff that runs these evals in CI with regression thresholds.\n\nContext placeholders (fill from the repo or ask only if missing):\n- RAG module root: {{APP_PATH}}\n- Stack: {{STACK}}  (Spring AI | LangChain4j | LlamaIndex | LangChain | custom)\n- Retriever: {{RETRIEVER}}  (vector store and distance metric | keyword search | hybrid; plus re-ranker or none)\n- Chunking and embedding today: {{CHUNKING}}  (splitter, chunk size, overlap; embedding model and dimensions)\n- Top-k used in the prompt: {{TOP_K}}\n- Corpus snapshot: {{CORPUS}}  (where it lives and how it is versioned)\n- Golden set: {{GOLDEN}}  (e.g. evals/rag/golden.jsonl, or \"none yet\")\n- Change under test: {{CHANGE}}  (e.g. chunk 512→1024, new embedding model, add re-ranker, new prompt, new model, or \"baseline only\")\n- Judge model: {{JUDGE}}  (pinned version, or none)\n- Eval runner and CI: {{EVAL_RUNNER}}  (pytest | JUnit | Ragas or similar library | custom harness; GitHub Actions | Jenkins)\n- Budgets: {{BUDGETS}}  (e.g. p95 latency ≤ 2.5 s, cost per query ≤ $0.004)\n\nRequirements:\n\n1. Inventory\n   - Document the pipeline in {{APP_PATH}}: loaders, splitter and chunk settings, embedding model, index and filters, hybrid weights, {{TOP_K}}, re-ranker, prompt template and version, model, and where citations and refusals are produced\n   - List existing eval code, data, and metrics, and note anything unpinned (model aliases, corpus not versioned, judge unpinned)\n   - Flag any golden or fixture data that contains real customer data or PHI\n\n2. Golden set\n   - Each case: stable `id`, `question`, `relevant_ids` as stable document IDs plus span or required facts (graded 0–3 if possible), short reference answer or required facts, and tags (lookup, multi-hop, comparison, numeric, recent-change, no-answer, ambiguous, past-incident)\n   - At least 10% `no-answer` cases whose answer is not in {{CORPUS}}\n   - A frozen held-out slice that is not used for tuning\n   - Map document-level labels onto current chunks at eval time so a chunking change doesn't break labels\n   - If {{GOLDEN}} is missing, propose 30 starter cases from the corpus and mark them for human review; drop synthetic questions that copy chunk text nearly verbatim\n   - Don't rewrite or delete existing cases; flag questionable ones in the report\n\n3. Retrieval metrics (retriever only, no LLM)\n   - recall@k, MRR, and nDCG@k at {{TOP_K}} (and at 2×{{TOP_K}} to show headroom), context precision at k, and retrieved-context tokens, overall and per tag\n\n4. Generation metrics (end to end)\n   - Claim-level faithfulness against retrieved context only: supported-claim share and the count of answers with any unsupported claim\n   - Citation validity (every cited ID was in the prompt for that request, checked in code), citation precision, and citation recall\n   - Correct-refusal rate on no-answer cases and on cases with forced irrelevant context; false-refusal rate on answerable cases\n   - Answer correctness against the reference or required facts\n   - Per-query retrieval, re-rank, and generation latency (p50/p95), prompt, completion, and context tokens, and cost\n\n5. Judge calibration (only if a judge is used)\n   - Hand-label 100–200 answers with a written binary rubric per claim, split dev and held-out\n   - Report judge precision, recall, and Cohen's kappa on held-out labels; pin judge model, prompt, temperature, and parser\n   - Keep judge metrics warn-only until agreement is acceptable; the model under test never judges itself\n\n6. Compare {{CHANGE}}\n   - Same golden set, same corpus snapshot, same {{TOP_K}} unless top-k is the change; full re-embed for embedding changes\n   - Paired per-case comparison against the current configuration, per tag, with token, latency, and cost deltas\n   - Stage-tag every failure: not retrieved, ranked below k, in context but ignored, unsupported claim, wrong citation, should have refused\n\n7. CI gate\n   - Commit the accepted run as a baseline (separate commit, with the pins it came from)\n   - Per-metric floor and tolerance: recall@k, MRR, faithfulness, citation validity, correct-refusal rate; budgets from {{BUDGETS}} for p95 latency and cost per query\n   - Deterministic retrieval and citation-validity metrics block now; calibrated judge metrics block once step 5 passes\n   - Path-filter the job on retrieval, ingest, prompt, model config, and eval code; write a results file with pins and a short Markdown summary; exit non-zero on regression\n\n8. Leakage checks\n   - Confirm no golden question or reference answer appears in prompts, few-shot examples, fine-tuning data, or re-ranker or embedder training data; add a test that fails if one does\n\nConstraints:\n- Minimal diff. Don't change retrieval or prompt behaviour unless {{CHANGE}} asks for it; report findings instead.\n- Don't edit the golden set, labels, or baseline in the same commit as a retrieval, prompt, or model change.\n- Don't invent scores, labels, or corpus contents. Use placeholders and state assumptions.\n- Don't put real customer data, PHI, or secrets in fixtures, logs, or reports.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave index migrations and model swaps as follow-up notes unless they are {{CHANGE}}.\n\nOutput format:\n1. Pipeline summary with pins and unpinned items\n2. Golden set report: case counts per tag, no-answer share, held-out size, flagged cases\n3. Metrics table: metric | baseline | candidate | delta | per-tag notes (retrieval, generation, refusal, cost, latency)\n4. Judge calibration table (if used): metric | precision | recall | kappa | blocking yes/no\n5. Stage-tagged failure table: case id | stage | evidence | suggested fix\n6. Patch (eval harness, metrics, gate config, CI wiring, leakage test) plus follow-ups\n\nStart from {{APP_PATH}}. Score the retriever on its own before judging any answer.\n"
+    },
+    {
+      "id": "add-llm-token-budgets-and-cost-tracking",
+      "title": "Add LLM Token Budgets & Cost Tracking",
+      "whenToUse": "Meter, price, and bound the LLM calls in a service or gateway: inventory every model call, read token usage from provider responses (including streams), price it from a versioned table, attribute it to route, caller, and resolved model, add max-token, context, and agent-step caps, per-caller and global budgets with reserve-then-reconcile and a report-only mode, retry accounting, staged latency histograms, a cost report, and tests.",
+      "tools": [
+        "copilot",
+        "claude-code",
+        "codex"
+      ],
+      "tags": [
+        "llm",
+        "cost",
+        "latency",
+        "budgets",
+        "observability",
+        "gateway"
+      ],
+      "formatNotes": "All three. Point the tool at the model client or gateway code, the retry and fallback config, the application config with model aliases, the auth or caller-identity code, and the metrics setup. Copilot: llm-cost path instructions plus those files in agent mode (or save it as `.github/prompts/add-llm-token-budgets-and-cost-tracking.prompt.md`). Claude Code: the llm-cost-latency-budget-guardrails skill plus the files. Codex: AGENTS.md LLM cost and latency section, then the diff.",
+      "body": "Task: Add token budgets, cost tracking, and latency measurement to the LLM calls in the service I specify. Inventory every model call, meter usage from provider responses, price it from a versioned table, cap each request, add caller and global budgets, fix retry accounting, and finish with a minimal diff, tests, and a cost report.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Service or module root: {{APP_PATH}}\n- Stack: {{STACK}}  (Spring Boot + Spring AI | LangChain4j | plain HTTP client | Python SDK | gateway)\n- Providers and models: {{PROVIDERS}}  (e.g. Azure OpenAI deployments, OpenAI-compatible endpoints, OpenRouter, Anthropic)\n- Callers to attribute: {{CALLERS}}  (service, tenant, user, or feature; and where the identity comes from)\n- Price source: {{PRICES}}  (list prices with date, or negotiated rates; currency)\n- Budgets: {{BUDGETS}}  (e.g. 200k tokens per caller per day, $50 global per day, max 8 agent steps)\n- Latency SLO: {{SLO}}  (e.g. p95 time to first token ≤ 1.5 s, p95 total ≤ 8 s)\n- Usage store and metrics: {{STORE}}  (database table, append-only log, Micrometer/OpenTelemetry metrics backend)\n- Mode: {{MODE}}  (report-only | enforce)\n\nRequirements:\n\n1. Inventory\n   - Table of every model call in {{APP_PATH}}: file and method, route or feature, caller, model alias, max output tokens (or \"not set\"), streaming, tools or agent loop, retry and fallback policy\n   - Flag calls with no max output tokens, streams that never read usage, agent loops with no step cap, and retries of non-retryable errors\n\n2. Metering\n   - One usage hook (client interceptor, gateway filter, or callback), not per-call code\n   - Read provider usage: input, output, and, where reported, cached-input and reasoning tokens; for OpenAI-compatible streams request the final usage chunk and count streams that end without it\n   - Record: request ID, trace ID, route, caller ID (stable ID, never an email), requested alias, resolved provider and model version, attempt number, outcome, token counts, timings, cost, price table version\n   - Never store prompts or completions in the usage store; redact them in logs\n\n3. Pricing\n   - Versioned price table from {{PRICES}} keyed by provider, model, and token type, with effective date and currency\n   - Cost computed at write time; unit tests for every token type and for a model missing from the table (record it, flag it, don't price it as zero)\n\n4. Request caps\n   - Explicit max output tokens per route, input and retrieved-context caps, tool-result size cap, agent step and total-token cap per run\n   - A clear error or trim policy when a cap is hit; a CI or unit check that fails when a route has no max output tokens\n\n5. Budgets\n   - Per-caller per-window and global daily budgets from {{BUDGETS}}\n   - Reserve an estimate before the call, reconcile with actual usage after it, release on failure; safe under concurrency\n   - Blocked response: documented status (e.g. 429 with reason and Retry-After) and a metric; in report-only {{MODE}}, log and count would-be blocks without rejecting\n   - Alert at a threshold below the global ceiling and on sudden jumps in tokens per request\n\n6. Retries and fallbacks\n   - Retry only timeouts, 429, and 5xx, with capped exponential backoff and jitter, honouring Retry-After, with a total attempt and time limit\n   - Every attempt's tokens count against the caller's budget; record which provider and model each attempt used\n   - Cancel the upstream call when the client disconnects\n\n7. Latency\n   - Histograms for total duration, time to first token, output tokens per second, gateway overhead, and retry wait, tagged by route, provider, model, and outcome\n   - Labels stay low-cardinality: no user IDs, request IDs, or prompt text; if using OpenTelemetry GenAI conventions, pin the version\n   - Report against {{SLO}}\n\n8. Report and tests\n   - A query or endpoint that reports cost, tokens, p95 latency, and error rate per day, route, caller, and model from {{STORE}}\n   - Tests: pricing, reserve-and-reconcile under concurrency, streaming with and without final usage, retry accounting, blocked response, missing-price handling\n\nConstraints:\n- Minimal diff. Don't change prompts, models, or routing unless a finding requires it; report findings instead.\n- Don't invent prices, token counts, or traffic numbers. Use placeholders and state assumptions.\n- Don't put prompts, completions, secrets, or personal data in the usage store, metrics, or test fixtures.\n- Put price table changes in their own commit with an effective date.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave dashboard building and provider migrations as follow-up notes.\n\nOutput format:\n1. Inventory table with flagged calls\n2. Usage record schema and where it is written\n3. Caps and budgets table: route or caller | limit | window | current behaviour | new behaviour\n4. Retry and fallback policy before and after\n5. Patch (usage hook, price table, caps, budgets, retry changes, metrics, report query, tests)\n6. Sample report output with placeholder numbers clearly marked, plus follow-ups\n\nStart from {{APP_PATH}}. Meter before you limit: get honest usage numbers first, then add caps and budgets.\n"
     }
   ]
 }