intake: re-rank reconcile so its suspects are worth reading - #215
Merged
Conversation
Leg 2 of PyAutoMind draft/feature/pyautomind/draft_staleness_detection_signals.md. Measured before touching anything, against a labelled set: PyAutoMind f25e154e, 148 prompts, five findings independently confirmed against upstream source. Reconcile flagged 96 of 148 (65%), 52 of them "high", and MISSED the largest true positive entirely. Not a missing signal -- every signal counted the same, so a completion record merely NAMING a prompt made it high, which described most of the backlog. BEFORE 96/148 (65%) 52 high biggest find not flagged AFTER 31/148 (21%) 9 high biggest find at RANK 2 What changed, each grounded in a specific confirmed finding rather than taste: Bare references no longer score. They are kept as evidence, because they are worth reading once a prompt is flagged, but on their own they were the single biggest noise source. Status: alone no longer flags. It is hand-set across the backlog and contributed a whole confidence band that said nothing. Rare tokens replace raw Jaccard, IDF-weighted with a fan-out bonus. The biggest find scores 0.25 Jaccard against its own completion record -- unreachable at any threshold -- because record stems are short and share nothing else. The real signal is that ONE rare token (kxs, in 7 of 947 records) appears in SIX record stems: a series that shipped in phases. Note the obvious first try, requiring two shared tokens, scores that case exactly 0. Shared rare identifiers, weighted by how many. Two is a coincidence; six is the record describing this prompt's deliverable. This is what a human grader reads. A record asserting the work SHIPPED scores on its own. One sentence -- "the 4 jax_substructure/ prompts shipped to main" -- retired four prompts in the sweep, and nothing else had flagged them. The instructive failure is recorded in the code. Matching a bare <work-type>/<target>/ series prefix pulled in one more true finding, and also falsely flagged test_mode_bypass_ordered_assertion_ties off references to four UNRELATED sibling prompts in the same folder -- a prompt confirmed NOT shipped, and exactly the mis-grade this tool must never make. The series match now requires the line to discuss the folder's prompts as a group, which keeps the jax_substructure win without the false positive. Two of the five findings remain out of reach and that is correct, not a gap: one had no completion record at all (its evidence sat inside a sibling PROMPT), the other left no Mind trace whatsoever. Chasing them from Mind-local evidence costs precision without gaining truth; they need the upstream leg. 8 new tests, each driving input that must trip or must not trip the leg, including the noise cases that motivated the change and the read-only contract. One test carries a note that the rare-token signal is inherently scale-relative -- an earlier version of it failed because a 5-record fixture cannot express "rare", not because the ranker was wrong. Full suite 275 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E3MuurHXi3xo9TLRpMLJA6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leg 2 of
PyAutoMind draft/feature/pyautomind/draft_staleness_detection_signals.md. Pairs with PyAutoMind#168 (leg 1, theCloses-when:/Blocked-by:gate keys) — independent, mergeable in either order.Measured first, against a labelled set
PyAutoMind
f25e154e— 148 prompts, five findings independently confirmed against upstream source during the draft/ sweep.highNot a missing signal — every signal counted the same, so a completion record merely naming a prompt made it
high, which described most of the backlog.What changed, each grounded in a confirmed finding
Bare references no longer score. Kept as evidence (worth reading once a prompt is flagged), but on their own they were the single biggest noise source.
Status:alone no longer flags. Hand-set across the backlog; contributed a whole confidence band that said nothing.Rare tokens replace raw Jaccard, IDF-weighted with a fan-out bonus. The biggest find scores 0.25 Jaccard against its own completion record — unreachable at any threshold — because record stems are short and share nothing else. The real signal is that one rare token (
kxs, in 7 of 947 records) appears in six record stems: a series that shipped in phases. Worth noting the obvious first try — requiring two shared tokens — scores that case exactly 0.Shared rare identifiers, weighted by count. Two is a coincidence; six is the record describing this prompt's deliverable. This is what a human grader actually reads.
A record asserting the work shipped scores on its own. One sentence — "the 4
jax_substructure/prompts shipped tomain" — retired four prompts in the sweep, and nothing else had flagged them.The instructive failure, recorded in the code
Matching a bare
<work-type>/<target>/series prefix pulled in one more true finding — and falsely flaggedtest_mode_bypass_ordered_assertion_tiesoff references to four unrelated sibling prompts in the same folder. That prompt is one the sweep confirmed is not shipped, and it's precisely the mis-grade this tool must never make. The series match now requires the line to discuss the folder's prompts as a group, keeping thejax_substructurewin without the false positive.Two findings stay out of reach, correctly
One had no completion record at all (its evidence sat inside a sibling prompt); the other left no Mind trace whatsoever — the fix shipped upstream without a record. Chasing them from Mind-local evidence costs precision without gaining truth. They need the upstream
--repoleg, which is not built here and is now better motivated.This also means the source prompt's "all five in the top band" acceptance criterion was unachievable and contradicted its own § Hard limit — corrected in PyAutoMind#168.
Contract unchanged
Read-only, always. Retiring a prompt stays a human act, asserted by a test.
Tests
8 new, each driving input that must trip or must not trip the leg — including the noise cases that motivated the change. One carries a note that the rare-token signal is inherently scale-relative: an earlier version failed because a 5-record fixture cannot express "rare", not because the ranker was wrong. Full suite 275 passed.
Generated by Claude Code