feat(storage): restore HNSW graph from embeddings_hnsw across restarts - #54
Merged
Conversation
- Persist a versioned neighbors payload (version, m, efConstruction, levels) so Restore can validate stored graphs and reject foreign or legacy formats - HNSWIndex.Restore loads vectors + link lists from the graph table, requiring exact membership parity with the embeddings table - SearchHNSW now restores the persisted graph on first use (no recomputation after a restart) and falls back to a full Build when the stored graph is absent, stale, or built with different parameters - BuildHNSWIndex remains the force-rebuild path - Remove now-dead HNSWNode type (superseded by the payload) - Tests: restart restore, stale-graph fallback + re-persist, and parameter-mismatch rejection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 follow-up to #50/#53.
Problem: a restart lost the in-memory HNSW graph, so the first
SearchHNSWrecomputed the entire index from scratch.Solution:
{version, m, efConstruction, levels}so stored graphs can be validated (legacy rows from pre-restore builds safely fall back to a rebuild)HNSWIndex.Restoreloads vectors + link lists fromembeddings_hnsw, only accepting the graph when its node set exactly matches the embeddings table and the construction parameters matchSearchHNSWrestores on first use; falls back toBuildwhen the graph is absent, stale, or foreign — the fallback re-persists so subsequent restarts restore againBuildHNSWIndexkeeps its force-rebuild semanticsHNSWNodetypeTests: identical results across a simulated restart (new
Storeon the same DB), stale-graph rejection + rebuild fallback + re-persist, and construction-parameter mismatch rejection.go test ./...green.