Skip to content

Add RAG retrieval and grounding evaluation skill and prompt - #75

Merged
rmkr-dev merged 2 commits into
mainfrom
skillbook/rag-retrieval-eval-guardrails
Oct 24, 2026
Merged

rmkr-dev merged 2 commits into
mainfrom
skillbook/rag-retrieval-eval-guardrails

Conversation

@rmkr-dev

Copy link
Copy Markdown
Owner

Adds a Skillbook entry and a matching Promptbook prompt for measuring whether a RAG feature retrieves the right evidence and answers only from it.

Skill: rag-retrieval-eval-guardrails

  • Golden question set with document-level relevance labels (survives re-chunking), tags, a no-answer slice, and a frozen held-out slice
  • Retrieval scored on its own: recall@k, MRR, nDCG@k, and context precision at the k the prompt uses
  • Chunking, embedding, index, and re-ranker changes compared on the same corpus snapshot with paired per-case results
  • Claim-level faithfulness against retrieved context only, plus citation validity, precision, and recall
  • Correct- and false-refusal rates when evidence is missing
  • LLM judge calibrated against human labels (held-out agreement) before it can block
  • Per-query latency, tokens, and cost next to quality; CI thresholds against a committed baseline
  • Test-set leakage rules (no golden data in prompts, few-shot examples, or training data)
  • Copilot / Claude Code / Codex placement notes and a SKILL.md stub

Prompt: evaluate-rag-retrieval-quality

  • Inventory, golden set audit, retrieval and generation metrics, judge calibration, paired comparison for a change, stage-tagged failure table, CI gate, and a leakage test

Scope is split from llm-eval-ci-regression-gate (gate mechanics) and llm-app-prompt-injection-guardrails (security); this pair owns retrieval and grounding quality.

Catalogs go to 39 skills / 39 prompts. node scripts/site-check.mjs passes locally.

@rmkr-dev
rmkr-dev merged commit e3baa9b into main Oct 24, 2026
1 check passed
@rmkr-dev
rmkr-dev deleted the skillbook/rag-retrieval-eval-guardrails branch October 24, 2026 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant