Repository navigation
perf(search): meaning hits for explicit syntax and inside a filter, messages - #278
Merged
Merged
Conversation
…essages Totals for cli-core #2254 on the owner's store, read-only: 0 of 220 hits break the query; explicit-syntax queries go from 0 to 10-17 meaning hits in 20; a narrow chat filter from 9 hits to 20. Cost of ranking a filter's messages: ~2 us a message listed, ~22 us a vector scored. Per-query rows stay in cli-private. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leemour
added a commit
to WireCatLabs/cli-core
that referenced
this pull request
Oct 31, 2026
…lter in --discover (#2254) ## Task Owner decision NEED-173 = B, plus RISK-61 and FIND-112 (wirecat-desktop journal). ## Why - `search messages --discover` with `-`, `NOT`, `OR`, groups, phrases or `exact:` ran strict search alone: no meaning hits. Notes have given such queries meaning hits that obey the whole query since 0.496.0 (#2223). - Meaning took the 100 nearest messages overall, then dropped those outside the filter, so a narrow filter lost hits. #2236 fixed the same for notes. - The follow-up step created a temporary `fts5vocab` table, which a read-only connection refuses. ## What - **Explicit syntax:** the words keep the whole query (`discovery.method: "lexical-strict"`). Meaning embeds only the wanted words. Its hits, and the replies and follow-ups that discovery adds, must match every field and every excluded word. This uses the same `fieldsOfQuery` / `meaningWords` as notes, moved to `src/search/lucene/meaning-parts.ts`. Only the "match everything" stand-in differs: `in:notes` for notes, `date:[* TO *]` for messages. A query with no word to embed (`date:today`, `Helix*`) stays strict. - **Inside a filter:** when the query's fields hold at most `RANK_WITHIN_MESSAGES` = 100,000 messages, every one of them that has a vector is scored (`QueriesStore.matchingPks`, `AccountVectors.scoreMessages`, `MeaningLane`'s new `within` argument). A larger filter, or one with row-by-row leaves, keeps the old way. - **Read-only:** rare stems are now counted with a bounded `MATCH … LIMIT rareAt + 1` query, not a temp vocab table. - The tests that pinned "explicit syntax means strict search" are replaced by tests showing that meaning hits obey negation, `NOT`, `OR` and groups. New tests: ranking inside a narrow filter (it fails with the threshold off), and follow-ups on a read-only connection (it fails on main with "attempt to write a readonly database"). ## Testing - Full Vitest suite: 8,492 passed. `pnpm lint`, `tsc` and `pnpm docs:check` are clean. - Owner's real store, read-only, through cli-core's services, without the reranker. Before = main plus the read-only fix only. - 6 explicit-syntax queries (`-date`, `NOT`, `OR`, a group of dates, `-chat`, `-(chat OR chat)`): 0 meaning hits before, 10–17 of 20 after. **0 hits break the query.** - `word chat:<1k-message chat>`: 9 hits / 3 meaning candidates before, 20 / 100 after. Another chat: 0 meaning hits before, 20 hits found by both words and meaning after. `word date>=30d` (91k messages, 957 with vectors): 10 candidates before, 100 after. `оплата date>=14d`: 5 meaning hits before, 10 after. - Cost: listing a filter takes ~2 µs a message (161 ms for 91k), and scoring ~22 µs a vector (174 ms for 7.7k). - Totals and method: [cli-testing #278](WireCatLabs/cli-testing#278) (`performance/search/messages-meaning-filters`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Totals for cli-core #2254:
search messages --discoverwith-,NOT,ORand groups, and ranking inside a narrow filter (RISK-61), on the owner's store, read-only. Per-query rows are in cli-private. Linked fromperformance/search/README.mdandPROCESS.md.🤖 Generated with Claude Code