Skip to content

fix(store): deterministic LFU cache eviction (tie-break by key) - #6707

Open
SnowingFox wants to merge 1 commit into
graphprotocol:masterfrom
SnowingFox:fix/lfu-cache-deterministic-eviction
Open

fix(store): deterministic LFU cache eviction (tie-break by key)#6707
SnowingFox wants to merge 1 commit into
graphprotocol:masterfrom
SnowingFox:fix/lfu-cache-deterministic-eviction

Conversation

@SnowingFox

Copy link
Copy Markdown

Root cause (issue #6706)

EntityCache's LFU cache (LfuCache in graph/src/util/lfu_cache.rs) evicts a different set of entries on every process for the same deployment, the same blocks and the same binary.

The chain, as analyzed in the issue:

  1. EntityCache.updates is a std::collections::HashMap (RandomState, seeded once per process), so as_modifications iterates it in a different order on every run.
  2. That iteration order becomes the LFU insertion order: entities are (re-)inserted into self.current as the loop runs (graph/src/components/store/entity_cache.rs).
  3. LfuCache uses a priority_queue::PriorityQueue whose eviction priority is (stale, Reverse<frequency>). Ties on that priority are the normal case, not the exception — within one block the overwhelming majority of entries sit at exactly (false, Reverse(1)).
  4. PriorityQueue::pop() breaks priority ties by heap position, which is determined by insertion order. So which tied entry is evicted depends on the random HashMap iteration order → different eviction sets per process.

The indexing result is unaffected (entity data, modification counts, PoI digests are identical across runs); what moves is provenance — which reads are served from the cache vs. from the store. That makes cache hit rate and store read counts non-reproducible between otherwise identical runs (see the issue's measurements).

The fix

Break eviction-priority ties deterministically on the cache key. The priority type becomes (stale, Reverse<frequency>, Reverse<K>). Keys are unique in the cache and K: Ord, so the priority is now a strict total order and PriorityQueue::pop() no longer falls back to the insertion-order-dependent heap layout for ties.

This fixes all callers of LfuCache at once (today that is EntityCache via the EntityLfuCache alias), regardless of the order in which they insert entries.

Test

Adds graph/tests/lfu_cache_determinism.rseviction_is_independent_of_insertion_order: insert the same six entries (equal weight, equal frequency → every candidate tied), in three different orders, evict down to a fixed max weight, and assert the same entries survive each time. On the old code the eviction set changes with insertion order (the test fails); on the new code it is deterministic.

Scope note

  • Indexing output (modifications, PoI digests) is unchanged — this only makes read-path counters (cache hit rate, store read volume) reproducible.
  • The tie-break policy is "evict the lexicographically smallest key among equal (stale, frequency) entries". The specific order is arbitrary; the important property is that it is deterministic.
  • Memory: the priority now holds one extra copy of the key per entry. EntityKey is interned/cheap to clone (an InputSchema handle + interned atoms + CausalityRegion(i32) + an interned Id), so the per-entry overhead is on the order of a small struct and is not counted toward max_weight. If that is a concern for very large caches, a caller-side fix (deterministically ordered iteration in as_modifications) could be layered on, but the cache-level fix is sufficient and covers every caller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant