Prajna1999 · Oct 27, 2026 · Oct 27, 2026 · Oct 27, 2026
diff --git a/backend/Benchmarking datasets/Content Safety/toxicity_test_combined.csv b/backend/Benchmarking datasets/Content Safety/toxicity_test_combined.csv
diff --git a/backend/Benchmarking datasets/FINDINGS.md b/backend/Benchmarking datasets/FINDINGS.md
@@ -0,0 +1,160 @@
+# Guardrails validator evaluation set: notes and findings
+
+Date: 2026-10-27. Workbook: `guardrails_validator_eval_set.xlsx`. Rebuild it with `build_eval_workbook.py`.
+
+## What's here
+
+```
+Benchmarking datasets/
+├── guardrails_validator_eval_set.xlsx   # combined eval set (generated, don't hand-edit data sheets)
+├── build_eval_workbook.py               # builds the workbook from the CSVs below
+├── FINDINGS.md                          # this file
+├── Content Safety/ Gender neutrality/ Privacy protection/ Scope Control/   # benchmark CSVs + prior run outputs
+└── Synthetic/                           # 8 synthetic CSVs + topic config + llm_critic rubric
+```
+
+Rebuild:
+
+```bash
+cd "backend/Benchmarking datasets"
+uv run --with openpyxl --with pandas python build_eval_workbook.py
+```
+
+## Coverage (3,766 cases, 12 validators)
+
+| Validator | Origin | Rows | FAIL | PASS | Notes |
+|---|---|---:|---:|---:|---|
+| `content_safety` | Benchmark (hasoc, sharechat, lexical) | 1,801 | 1,139 | 662 | `source_dataset` column kept |
+| `uli_slur_match` | Synthetic | 79 | 41 | 38 | `slur_severity` column |
+| `profanity_free` | Synthetic | 79 | 43 | 36 | |
+| `nsfw_text` | Synthetic | 74 | 38 | 36 | sentence-level traps |
+| `llamaguard_7b` | Synthetic | 66 | 37 | 29 | `policy` column, all 6 policies |
+| `pii_remover` | Benchmark | 419 | 256 | 163 | `pii_entities`, `entity_count` from target placeholders |
+| `gender_assumption_bias` | Benchmark | 600 | 300 | 300 | PASS rows = neutral rewrites, same as the runner |
+| `topic_relevance_llm` | Benchmark (education + healthcare) | 355 | 115 | 240 | `TR-EDU` / `TR-HEALTH` configs |
+| `topic_relevance` | Synthetic (agriculture) | 70 | 38 | 32 | new `TR-AGRI` allowed-topics config |
+| `ban_list` | Synthetic | 79 | 47 | 32 | `BAN-MATERNAL`, `BAN-EDU`; `banned_term_hit` column |
+| `answer_relevance_custom_llm` | Synthetic | 72 | 42 | 30 | `query` / `answer` columns + JSON `input_text` |
+| `llm_critic` | Synthetic | 72 | 43 | 29 | `failing_metrics` column; `CRIT-NGO` rubric |
+
+Each synthetic set mixes English, Hindi in Devanagari, and Hinglish. Each includes hard negatives: benign text that looks risky.
+
+## Workbook structure
+
+- **README**: column dictionary, how to run the set periodically, and data caveats.
+- **Index**: one row per validator with origin, positive-class meaning, recommended stage, runner script, coverage counts by label and language, and live TP/FP/FN/TN, accuracy, precision, recall and F1 formulas. The metrics fill in once `actual_result` is entered.
+- **Configs**: exact parameters behind every `config_ref`, including topic configs, ban lists and the critic rubric.
+- **12 validator sheets**: they share a schema: `case_id, validator, input_text, expected_result, label, expected_output, category, difficulty, language, domain, config_ref, source, notes`, then validator-specific columns, then the yellow run columns `actual_result`, `actual_output` and `match`.
+- **Run_Log**: one row per validator per run (date, git SHA, model, metrics, latency).
+
+Convention: `expected_result = FAIL` means the validator **should trigger** (ground truth). Labels follow intent, not what the validator is expected to output, so known misses stay in the set on purpose.
+
+## Latest run
+
+[2026-10-27](runs/2026-10-27/RUN_REPORT.md): 11 of 12 validators ran; LlamaGuard is blocked because the hub was shut down. The run found bugs in `gender_assumption_bias` (fixed below; F1 went from 0.81 to 0.93 and fix exact match from 0.31 to 0.62) and in `pii_remover` (ordinary words masked as `<IN_PAN>`, not fixed), and showed that `nsfw_text` blocks sexual-health questions. Re-run with `run_eval_workbook.py`.
+
+## Bugs found and fixed in this PR
+
+### 1. BanList fix left the first letter of every banned word
+
+The hub `guardrails_ai.ban_list.BanList` maps spaceless indices to 1-based positions, so every span starts one character late:
+
+- `"Can I get a sonography to know"` became `"Can I get a s to know"`.
+- `"this contains badword"` became `"this contains b"`.
+
+Fix: added `app/core/validators/ban_list.py`, a subclass with 0-based spans, and pointed `BanListSafetyValidatorConfig` and `app/evaluation/ban_list/run.py` at it.
+
+- Exact match on the synthetic ban-list FAIL rows (after collapsing whitespace) went from **0/37 to 34/37**.
+- Four integration tests in `test_guardrails_api_integration.py` had the buggy output in their expected values. They now expect the banned word removed completely.
+- `error_spans` now point at the banned word itself.
+
+### 2. Slur list entries with nukta letters never matched
+
+`LexicalSlur.normalize_for_matching` NFKC-normalizes the input, which decomposes precomposed nukta letters (ज़ → ज + ़). The list labels were only lowercased, so these 12 entries could never match:
+
+नाजायज़, लौड़ा, लौड़े, बहन की लोड़ी, भाड़े का टट्टू, लोड़ू, तवाइफ़, दो कौड़ी की औरत, भड़वा, बत्तमीज़, पिछवाड़ा, बाज़ारू औरत
+
+Fix: `load_slur_list` now NFKC-normalizes the labels. A regression test was added. `content_safety` reuses `LexicalSlur`, so it is fixed too.
+
+### 3. gender_assumption_bias rewrites were often wrong
+
+The eval run found that only 31% of rewrites matched the expected neutral sentence. All four causes are fixed in `gender_assumption_bias.py`:
+
+- **Identity rows flagged neutral text.** `woh → woh` and `attendant → attendant` changed nothing but still reported bias. These caused all 80 false positives. Rows that map a word to itself are now skipped.
+- **`her` was always `them`.** "checked her patient" became "checked them patient". `her` now becomes `their` when a noun follows and `them` otherwise.
+- **Devanagari substring matches.** `\b` treats vowel signs as non-word characters, so `छात्र` matched inside `छात्रा` (giving `विद्यार्थीा`), and `गया।` never matched. Word boundaries now include Devanagari vowel signs, virama and nukta.
+- **Lowercased output.** Replacements now keep the matched word's casing ("He" → "They").
+
+The word list also had `hers → their`; it's now `hers → theirs`.
+
+## Other validator findings (not fixed)
+
+**`uli_slur_match`**
+- The list contains many everyday words that cause false positives: mad, crazy, stupid, contempt, illiterate, homo, poop, kala, halwa, chakki (severity H), mc, dyan, hilana, chamcha, bai, pataka, kutta, साली, टट्टी.
+- Devanagari matches inside longer words, because vowel signs and halant aren't `\w` for the boundary check. For example, मूत्र (urine) is flagged as मूत, and सूअरों is flagged as सूअर.
+- Duplicate list entries (cunt, randi, saali, bhenchod) appear twice in the error message.
+- Victims quoting abuse they received are flagged. These are labelled PASS in the set.
+
+**`profanity_free` (alt-profanity-check)**
+- It misses all Devanagari profanity (prob ≈ 0.04) and most romanized-Hindi profanity.
+- It misses obfuscations such as `sh1t`, `f u c k`, `b!tch` and `a$$hole`.
+- False positives:
+  - "The cock crows" (0.99)
+  - "Our bitch is pregnant" (dog breeding, 0.98)
+  - "stupid weather" (0.99)
+  - clinical words: "penis swollen" (0.66), "vagina discharge" (0.52), "testicles" (0.53)
+
+**`ban_list`**
+- Fuzzy matching (Levenshtein 1, spaces ignored) causes false positives across word boundaries:
+  - Sonographer
+  - "gender checklist"
+  - monography
+  - "roof leaked; paper"
+  - "Approx. attendance"
+  - "cake certificate"
+- Hindi transliterations (सोनोग्राफी, "ling jaanch") and 2-edit typos (sonograhpy, sonografy) are missed.
+- Inflections leave suffixes behind (`Are sonographies safe?` → `Are es safe?`).
+
+**`nsfw_text`**
+- The default model is English-only, so about 10 Hindi and Hinglish FAIL rows are expected misses.
+- The PASS side deliberately includes clinical sexual-health wording, which is the main false-positive risk for maternal-health bots.
+
+**`answer_relevance_custom_llm`**
+- The strict default prompt is expected to reject short but correct answers (`relevant_concise`).
+- It is also expected to reject correct safe redirects (`safe_redirect`), such as a self-harm ask answered with the Tele-MANAS helpline.
+- These are labelled PASS. Use a custom prompt if redirects should pass.
+
+**`topic_relevance`**
+- LLMCritic scores 1–3 with threshold 2, so "partially related" (2) passes; only "clearly outside scope" (1) fails. Tangential asks labelled FAIL in the set are therefore hard cases.
+- Ambiguous agriculture-adjacent asks (gold loan, MGNREGA, a farmer's health insurance) are labelled per the `TR-AGRI` config and noted as hard cases.
+
+## Baselines from the existing runs (`*/output/`)
+
+| Validator | Dataset | Precision | Recall | F1 | Latency (mean) |
+|---|---|---:|---:|---:|---:|
+| `uli_slur_match` | content safety, all | 0.90 | 0.63 | 0.74 | 1.0 ms |
+| ↳ | hasoc / sharechat / lexical (recall) | | 0.19 / 0.33 / 1.00 | | |
+| `profanity_free` | content safety, all | 0.78 | 0.23 | 0.36 | 3.6 ms |
+| slur + profanity combined | content safety, all | 0.85 | 0.72 | 0.78 | |
+| `pii_remover` | PII set | 0.92 | 0.80 | 0.86 | 10.6 ms |
+| `gender_assumption_bias` | 300 biased + 300 neutral | 0.76 | 0.87 | 0.81 | 0.3 ms |
+| `topic_relevance_llm` | education + healthcare | 0.90 | 0.76 | 0.82 | |
+
+These ran before the slur-list fix. Re-run `content_safety` and `uli_slur_match` to measure the change.
+
+## Data caveats
+
+- **content_safety:** about 99% of the 601 `lexical` rows contain a term from the slur list, which is why slur recall is 1.00 there. Report metrics per `source_dataset`.
+- **content_safety:** language comes from the source datasets. Hindi rows without Devanagari script are tagged `hi-Latn`. Some rows the source labels "English" are actually Hinglish.
+- **gender_assumption_bias:** only 15 unique biased sentences, each repeated 20 times. Duplicates are flagged in `notes`. The set needs more variety, especially already-neutral sentences that contain no gendered rewrite.
+- **pii_remover:** per-entity precision is low for some types: `IN_PAN` 0.10 and `PHONE_NUMBER` 0.22 in the existing run. `JOBAREA` placeholders in `expected_output` have no matching recognizer (0/12 recall), so those rows can never fully match.
+- **topic_relevance_llm:** `language` is auto-detected with a keyword heuristic.
+- **Synthetic sets:** written by an LLM and spot-reviewed. Hard-case labels are a starting point; refine them with partner NGO data. Uncertain rows explain their label in `notes`.
+- **Formulas:** the workbook's formulas calculate when the file is opened in Excel or Google Sheets. Tools that read cached values (pandas, `openpyxl data_only`) see empty metric cells until then.
+
+## Suggested next steps
+
+1. Add runners for `answer_relevance_custom_llm` and `llm_critic`, which have none yet. Point the existing runners at the workbook sheets, or export each sheet to CSV.
+2. Re-run `uli_slur_match` and `content_safety` after the nukta fix, and log the results in Run_Log.
+3. Prune the generic words from the slur list, or move them to a context-dependent severity.
+4. Expand the gender set beyond 15 templates.