Repository navigation
Add the loading cache core declares and the module that supplies it - #1365
Open
jwrosewell wants to merge 63 commits into
Open
jwrosewell wants to merge 63 commits into
jwrosewell wants to merge 63 commits into
Conversation
`PlatformLoadingCache` in `platform/loading_cache.rs` takes a key and a loader and returns the value, running the loader at most once per key at a time, so however many requests ask for a missing value at the same moment the source is asked once and every one of them continues with its value or its error. A failed load is not stored, so the next caller loads again. `NoLoadingCache` holds nothing and runs every load, and is what `RuntimeServices` carries when no module supplies a cache. Every key is a `StoreKey`, being the publisher's `ServerKeyPrefix`, the cache's name and the caller's key. The prefix is made in one place, `platform/server_key.rs`, from the publisher's domain in the loaded settings with a single leading `www.` folded and `ts.` in front, so two publishers sharing a store never share an entry. `PlatformValueCache` is the host's shared cache of small values with lifetimes, which a host supplies through `RuntimeServices` when it has one, with `lookup_or_reserve` handing the caller a `ValueReservation` where the host can make other callers for a key wait, and `remove`. A loading cache keeps its shared copies there where the host has one and in the key-value store otherwise. An `IntegrationBuilder` offers a loading cache with `with_loading_cache`, and `loading_cache_from` gives an adapter the one its linked builders offer, refusing two, so no adapter names the crate that supplies it. Core's tests get `MemoryLoadingCache`, which keeps what it loads for the lifetime the load gave and records every call.
A contributor's answer is loaded through `RuntimeServices::load_cached`, so the contributor is asked once however many requests want its rules at the same moment, and the copy in the host's shared store answers every instance until the contributor's refresh is due. The answer's lifetime in the cache is the contributor's refresh, and `[cache.robots-txt]` sets how long the copies last with `memory_seconds`, `store_seconds` and `store_idle_seconds`, defaulting to a minute, a week and 30 days. A memory lifetime may be 0, the other two must be at least 1, and none may be more than a year, which `LoadingCacheConfig::check` refuses as the settings load, naming the setting. The last answer a contributor gave is also kept apart from the cache for 30 days, in the host's value cache where it has one and in the key-value store otherwise, and read only when the contributor fails, because yesterday's rules are a better answer than none. Every key starts with the publisher's `ServerKeyPrefix`, so two publishers sharing a store never share an answer. The handler hands the settings through in place of the publisher's domain and the time, and a selection carries the name the contributor was declared under. The tests use `MemoryLoadingCache` and cover the one load for a held answer, the key naming the publisher, changed settings asking again, the kept answer served when the contributor fails, in a second instance and from a value cache, and a failure with nothing kept being an error that is not held.
Each adapter takes the cache from the builders it is given with `loading_cache_from` when it builds its state, and hands it to every request's services, so every request in a process shares the cache's memory copy and waits for one load. Axum, Cloudflare and Spin take it from the stock list followed by the builders a deployment passes, as they take every vendor module, and hold nothing without one. No adapter names the crate that supplies it. On Fastly the finishing steps' services carry the same cache as the request, so a lookup there shares the request's memory copy, and the entry point builds those services once from the state it has, or with the unavailable store and a cache that holds nothing when no state was built. Fastly supplies `FastlyValueCache` over Fastly's cache as the value cache, which needs nothing created or linked on the service. A transaction lookup reserves a key nothing holds for this instance and makes every other instance at the location wait for it to be filled, a stale entry is reserved for a fresh load rather than served, and a reservation dropped unfilled is released so a waiting instance takes it over. Each entry is written with a surrogate key, a hash of its key, which `remove` purges. Its tests run under Viceroy and cover the reservation, its release, a stale entry and the purge.
`crates/loading-cache/51degrees` implements `PlatformLoadingCache` with the published `fiftyone-caching` package, version 4.5.8-49, with its default features off so it builds for every host. Each cache is a least recently used copy in process memory with a short lifetime over the host's shared store, being its value cache where it has one and its key-value store otherwise, written in the package's entry format so a store that deletes late or keeps no lifetimes never returns an entry past its drop time. The memory layer is shared by every request in a process and loads a missing key once, covering the store read and the source together. The store layer is made for each call from that request's stores. A load a cache allows to run apart from its caller runs on a pool of threads on a native host and with the request's `wait_until` on Cloudflare, and every load runs in its caller on Fastly and Spin. The crate is the module `loading-cache.51degrees`, named by its folder, and `builder()` offers the cache whatever the settings say, because a loading cache is how a deployment caches rather than a module a document selects. The stock list carries the builder, so every adapter takes the cache from it with `loading_cache_from` and names no crate, and a test holds the list to offering one cache that keeps what it loads. The crate carries the maintainers table, the workspace lists it, and the Axum aliases lint and test it, where its own tests run natively. The crate's tests ask for one key from 16 callers at once on 16 threads with the pool, on 16 threads with loads in the caller, and on one thread with each load a task of its own, and the source runs once each time. Other tests cover a failure reaching every waiter and not being kept, the store answering a second process, the brief memory copy, idle expiry, a value's own lifetime, a reserving value cache being filled and released, two publishers kept apart, removal, the module's name and the builder's offer.
Spin's key-value store refuses a write with a lifetime, so a robots.txt answer kept for a later failure, and a loading cache entry in the module's key-value layer, are written without one there, as the held answer was before. The entry format carries each entry's own drop time, so a copy in a store that keeps no lifetimes still ends when it should, and a kept answer stays until it is replaced, which is what the configuration guide says of Spin. The robots.txt load carries the name the contributor was declared under as the static text the module context takes, and the handler hands it over from the selection. Tests cover the kept answer's lifetime where the store takes one, the answer kept where it refuses one, and a store that refuses lifetimes still answering a second process through the module.
The configuration guide gains "Loading caches" under the cache section, saying what is held, where the two copies live, how the keys start, which crate a stock build caches with and the `[cache.robots-txt]` lifetimes, and its robots.txt section says each answer is held through it. The integration guide gains "Supplying a loading cache", being how a crate offers one with `with_loading_cache` and how code loads through one with `load_cached`, a `CacheSpec`, a `StoreKey` and a `Loader`, and the builder table lists the offer. The example configuration shows `[cache.robots-txt]`, and the changelog records the cache, the lifetimes, the key prefix and the value cache under Unreleased.
The crate keeps a load running with the request's `wait_until` only where the Worker's handling of the request is wrapped in `scope_request`, and nothing on an adapter wraps it, so the crate's documentation and the changelog say so rather than that the load keeps running after the response on Cloudflare.
The item exists only on the Cloudflare target, so a link to it from the crate's documentation is reported broken on every other target's documentation build. The name is written as plain code instead.
The first build adds the published fiftyone-caching crate, hashlink and tokio-util to the lock, which a build with --locked refuses until they are committed.
A feature that holds answers in a loading cache names, in its own `[cache.<name>]` table, the module whose cache holds them, as the module's short name or its name in full. `loading_caches_from` resolves every feature's selection against the caches the linked builders offer, makes each module's cache once however many features select it, and refuses a name no builder offers a cache from, naming the feature and the modules that do. The services carry the selections as `LoadingCaches`, and a feature that selected none loads through `NoLoadingCache`. Before this the stock build's one offered cache held every feature's copies, and a deployment linking two caches was refused. Now each feature decides, so the asset cache and any later feature can choose a cache of its own. `[cache.robots-txt] module = "51degrees"` selects the stock cache for the robots.txt modules' answers. The guides, the example configuration and the changelog say so, and the adapters pass the selections where they passed the one cache.
# Conflicts: # CHANGELOG.md # crates/trusted-server-core/src/robots_txt/holding.rs # crates/trusted-server-core/src/robots_txt/mod.rs
A held copy that no longer reads as a contribution is removed from the loading cache and the module is asked again, where it used to answer 503 until the copy ended. The 51Degrees crate drops its unused thread pool, local spawner and Cloudflare request scope, so every load runs in its caller. The guides and changelog say holding is opt-in, the four adapters describe the loading cache field in one sentence, and the lock no longer moves prost onto itertools 0.13.
This was referenced Oct 15, 2026
RuntimeServices::load_cached handed the loading cache the request's own services, and the robots.txt holding handed each contributor the same, so the client address, connection details and host signals of whichever reader asked first reached a load whose answer every reader is served. RuntimeServices::without_reader clears those and keeps the stores, and both paths now take it.
…cluded On wasm32-unknown-unknown the factory kept one cache for the whole Worker isolate, so a request could wait on a load another request was running. The published caching crate says such a request fails unless every request runs in its request scope with its wait_until spawner, and nothing here does. The factory now makes a cache for each application state on every host. Cloudflare builds its state for each request, so its memory copy lasts one request and Workers KV holds the copy between requests.
The next branch removes the robots.txt route and the route tests that held a module's answer, so a store test inside that module would not outlive this branch. the_declared_store_is_the_key_value_store_a_request_is_given now lives in each adapter's tests/key_value_store.rs with its own store kept in memory, and builds the registry from the declaration the way EdgeZero does. The Spin test that the declared label is one spin.toml grants moves there too. The robots.txt route tests are left as they were.
The sentence said a deployment whose key-value store cannot be opened keeps no last answer, which stays true now that Cloudflare and Spin fail every request without their store, and the Cloudflare and Spin guides say that. The point goes back to its wording so the guide merges cleanly into the next branch, which adds a point after it.
# Conflicts: # CHANGELOG.md # docs/guide/configuration.md
The stock list in crates/trusted-server-modules no longer offers trusted_server_loading_cache_51degrees::builder(), and the crate no longer depends on the module crate, so no adapter and not the ts tool compiles the module or fiftyone-caching. The module crate stays in the tree for a deployment that links it through run_with on Fastly or routes_with_registrations on Axum, Cloudflare and Spin. the_stock_build_links_no_loading_cache fails if a builder in the stock list offers a loading cache. It replaces the three tests that pinned the stock cache, and the order test stops naming loading-cache.51degrees. The module crate's builder documentation no longer says the stock list carries it.
The configuration guide, the integration guide, the Cloudflare and Spin guides, the example settings and the changelog said that a stock build links loading-cache.51degrees, holds answers without a line in the document, or keeps copies in its key-value store. They now say that a stock build links no loading cache module, so no feature keeps a copy and every module a [cache.<name>] table names is refused, and that a deployment that wants copies kept links a loading cache module itself. The changelog adds an entry saying that a stock build refuses module = "51degrees", so a deployment whose document names the module links it first.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
crates/trusted-server-core/srccrates/loading-cache/51degreescrates/trusted-server-adapter-fastly/srccrates/trusted-server-integration-tests/testscrates/trusted-server-adapter-cloudflare/testsdocs/guidecrates/trusted-server-adapter-spin/testsCargo.lockcrates/trusted-server-adapter-cloudflare/srccrates/trusted-server-adapter-spin/srccrates/trusted-server-modules/srctrusted-server.example.tomlcrates/trusted-server-adapter-axum/srcCHANGELOG.mdCargo.tomlcrates/trusted-server-adapter-cloudflare/README.mdcrates/trusted-server-adapter-spin/README.md.cargo/config.tomlcrates/trusted-server-adapter-cloudflare/wrangler.tomlCloses
Closes #1282
Spec
Spec: loading-cache
Test plan
CI on this pull request at 23951e9 passed all 20 checks.
cargo test-fastly && cargo test-axumcargo clippy-fastly && cargo clippy-axumcargo fmt --all -- --checkcd crates/trusted-server-js/lib && npx vitest runcd crates/trusted-server-js/lib && npm run formatcd docs && npm run formatcargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1fastly compute serveChecklist
unwrap()in production code (clippy deniesunwrap_usedin CI)logmacros as AGENTS.md asks (notprintln!)Stacks on a middleware that prepares (#1364).
What changed after the 51Degrees PR review
Agreed at the review on 23 October, point 2. The 51Degrees loading cache is pluggable through modules, and the publisher asset feature becomes a module.
the_stock_build_links_no_loading_cachefails if a builder on it offers a loading cache.Agreed at the review on 23 October, point 1. No configuration or permission policy is compiled into the WASM, and a WASM without a policy does not start. Since #1345 a loaded domain carries its permission policy and every services graph needs one, so the tests below give theirs the sample policy.
Kept current
mainat 7aa5c99 and the changes made lower in the stack.What this pull request does
Core declares a loading cache,
PlatformLoadingCacheintrusted_server_core::platform, which loads a missing value once per key however many requests ask for it at the same moment, gives every request waiting on that load the value or the error, and holds nothing that failed. Each cache is named by aCacheSpecwith the lifetimes of its copies, a brief one in process memory and one in the host's shared store, and keys under aStoreKeythat starts with the publisher's prefix, so publishers sharing a store never share a copy.Each cache is selected by naming, in its own
[cache.<name>]table, a module that offers a loading cache. A table that names none takes the one loading cache module the build links. With none linked, or several and none named, the table getsNoLoadingCache, which holds nothing and runs every load. No stock build links a loading cache module, so in a stock build every table getsNoLoadingCache, and a table that names a module is refused. A name no linked crate offers a cache from is refused when the service starts, with the names that could be written. On Cloudflare, which builds the state for every request, that refusal is per request. Each robots.txt module's answer is the first thing held this way, so where a deployment links a loading cache module, a robots.txt module is asked again only when no copy young enough is held.A deployment that links the 51Degrees module selects it like this.
The implementation is a module crate,
crates/loading-cache/51degrees, maintained by 51Degrees, written with the publishedfiftyone-cachingcrate over the shared store each adapter gives a request. On Fastly that is Fastly's cache. On Cloudflare and Spin it is the key-value store each adapter declares, Workers KV and Spin'sdefaultstore, and because both adapters build their application state for every request, a memory copy there lasts one request and the store holds the copy between requests. Axum has no shared store, so its memory copy, which lasts for the process, is the only one. Every load runs in its caller. The module offers the cache from its builder, no stock list carries it, and an adapter takes the selections from the settings and the builders it is given withloading_caches_from, so no adapter names the crate. The testthe_stock_build_links_no_loading_cachefails if a builder in the stock list offers a loading cache.A load that several requests share is handed nothing of the reader whose request started it, because
RuntimeServices::load_cachedhands the cache its services throughRuntimeServices::without_connection. Each application state has a loading cache of its own, and a[cache.<name>]table whosemoduleno linked builder offers is refused at startup and byts config validate.Why
The same answer is asked for by every instance and every request, and a cache that lets every miss load at once asks a module's service as many times as there are requests. One load per key at a time, with the result shared by every waiter, is the pattern the other 51Degrees language packages use, and the crate that implements it is published for every 51Degrees service to use. Core keeps only the interface, so core names no vendor and depends on no package, and with the module out of the stock list no stock build depends on one either. At the 51Degrees PR review on 23 October, 51Degrees said it would check that the loading cache could be a module that a deployment links. As decided by 51Degrees after the review, the module stays in the repository and no stock build links it.
What breaks
[cache.robots-txt] modulenames. The robots.txt change (Add robots.txt rules from modules #1355) held each answer in the key-value store on Cloudflare and Spin.module = "51degrees"in a[cache.<name>]table, or any othermodule, is refused byts config validateand when the service starts. A deployment whose document names the module links it first, by handingtrusted_server_loading_cache_51degrees::builder()to its adapter throughrun_withon Fastly orroutes_with_registrationson Axum, Cloudflare and Spin.TRUSTED_SERVER_KVand Spin'sdefaultstore, so EdgeZero opens it for every request. A declared store is required, so a Worker without the binding, or a Spin component that cannot opendefault, fails every request before any route runs.How it was verified
No load has run against a real host store. Fastly's cache is checked on Viceroy. The page template does not load through the cache. Locally on Windows the fixes of 16 October (a1931fd to 0d49c46) passed their tests, clippy and
cargo fmt --all -- --check, and at 3564019 all 275 tests of the Fastly adapter passed under Viceroy.The test
the_stock_build_links_no_loading_cachefailed while the stock list still offered the 51Degrees module, with "a stock build should link no loading cache, and links ["loading-cache.51degrees"]", and passes without it. Locally on Windows, on this pull request's tree, these passed:cargo fmt --all -- --check, clippy-fastly, clippy-axum, clippy-cli, clippy-cloudflare, clippy-spin-native, clippy-cloudflare-wasm and clippy-spin-wasm, the core library's tests (2,864),cargo test-axum(1,032, 1 ignored),cargo test-cloudflare(59, 1 ignored),cargo test-spin(97, 1 ignored) and the Fastly adapter's library tests under Viceroy (281).References #1282.