Repository navigation
Conversation
`generated_blob_verifies_and_applies_origin_override` builds the envelope and then loads settings from it the way an adapter does at startup. It called the loader that knows core's own modules alone, so it refused the fixture's Prebid Server source once that implementation left core. It passes the stock list, as every adapter does. The integration workflow runs this binary's tests before the container tests, so the failure also hid whether those run.
The plain OpenRTB demand implementation is `auction-protocol.openrtb`, and it now lives in `crates/auction-protocol/openrtb`. The stock list offers it, so a `[demand.<name>]` table with `implementation = "auction-protocol.openrtb"` selects it as before and its `request_ext` and `imp_ext` settings do not change. It is the project's own, and its manifest says so. It was the last module in core's `integrations` folder, which now holds the registry, the builder a module registers with and the JavaScript asset proxy. Core's built-in list offers no auction implementation. What reads an ordinary OpenRTB response is core's and not this implementation's, so `parse_openrtb_response` moves beside the response extraction it wraps in `auction::openrtb`, and stays public for an implementation in a crate of its own. Core's tests of the plan compiler, the routing, the request driver and the page path ran on the plain implementation. They run on a stand-in compiled for core's tests only, `auction.plain-fixture`, which takes every eligible slot, inherits the auction's timeout, leaves its endpoint as written and writes the `request_ext` and `imp_ext` its table sets, so both golden requests stay in core unchanged. `demand_named` takes the implementation its sources run. Two of core's tests were the implementation's own and went with it, under names that say what they cover: - `openrtb_extensions_are_bounded_and_cannot_claim_reserved_fields` is `static_extensions_are_bounded_and_cannot_claim_reserved_fields`, - `standard_static_extensions_have_no_invented_bidder_param_location` is `static_extensions_have_no_invented_bidder_param_location`. The crate also holds both golden requests, signed and unsigned, byte for byte as core holds them, a test that its endpoint is left as written and the test that its name is its folder. The Prebid crate's two tests of a bidder that runs on both sides name the plain implementation and hand its builder to the validation they call.
…tegration-seam-impl
Core builds one `ModuleContext` where it calls modules, by borrowing what the request already holds, and every module called there borrows the same context. A module is handed a `ModuleCall`, which is the context together with the permissions that module declares, and its process function names what it needs as parameters. `ModuleCall::inject` passes each one in from the context. A parameter is the request (method, path, query, host and scheme), the reader's evidence, the connection, the response, the resolved permissions, the decoded consent, the settings, the services, the location, the device signals, the Edge Cookie identifier, the document state or the request's typed extensions. It may be an `Option` of one, which is `None` when the value cannot be passed, or a tuple of them. A function takes the module first and up to eight parameters, `inject_with` passes one argument of the caller's own after the module, and the function may be async. The location, the device signals and the Edge Cookie identifier each carry the permissions the module that produced them declares. Such a value is passed only to a module that declares every one of those permissions and is granted them on this request. A withheld value is `None` for an `Option` parameter, and a call that names the value itself is skipped, with the reason in the debug log. Core holds modules as trait objects, which cannot have a generic method, so each kind of module's trait method will take a `ModuleCall` and the module's implementation hands its own function to `inject`. That call is made where the module's type is known, so nothing is boxed, allocated or copied. Nothing calls the context yet. The commits that follow hand it to each kind of module. `PermissionSet` gains `difference`, and a `Display` that lists its Data Use identifiers, which the reason for a skipped call uses.
`DeviceModule::detect` takes a `ModuleCall`, and a device module names what it reads through `inject`. Classification now sees the request it classifies, being its method, address, host and scheme, and the settings, as well as the evidence and the services it had before. The Fastly entry point is the one place a device module is asked. It builds the context from the Fastly request before that request is converted, reading the host from the `Host` header and the scheme from the TLS mark the entry point has already set from Fastly's own metadata. The module it asks is borrowed, whether a registration supplied it or the `[device]` selector built it, so the request allocates nothing more than before. Classification runs before the permissions are resolved, so no value whose use needs a permission is passed to it. Signals stay infallible. A module whose function is not passed what it names answers `DeviceSignals::unknown`, an unknown device that does not look like a browser, and a test asks the built-in module with no evidence to show it. The built-in module converts in core, and the Fastly module and the seam probe's in their own crates. `module_context::test_support::request` gives a module's own tests a request, behind the `test-utils` feature.
`EdgeCookieModule::generate` takes a `ModuleCall`, and `resolve_from_client` takes a `ModuleCall` and the posted payload. A module names what it needs through `inject`, or `inject_with` for the payload. `IdentityInput` and `ClientResolveInput` are gone, because the permissions and consent they carried are in the context, along with the request the identifier is created for, its host and scheme, the settings, the location and the device signals. `EcContext` snapshots what the request resolved to beside the headers it already kept, under the same condition, so generation later in the request and orphan recovery hand the module the request they are for. The snapshot holds the host and the scheme where it held the path and the query alone. `candidate_id` takes the settings and builds the context from the snapshot and its own state. The resolve endpoint builds the context from the resolve request and the state read for it, and lends the module the normalized client IP that state holds, the same form the module sees when it generates. The built-in HMAC and host signal modules, the `client_fixed` module and the shared wrapper convert in core, and the seam probe's module in its own crate. A module whose function is not passed what it names produces no identifier. The HMAC module still refuses a request whose evidence carries no client IP. The test of the evidence a module reads at generate time also checks the host and address reach it, and the resolve round trip checks the module is handed the resolve request.
Permission assembly builds one module context per request, carrying what the request resolved to, the evidence, consent, settings, services and location, and every signal module is asked in it. `assemble_permissions` takes that context in place of the consent and evidence, and reads both from it. `SignalInput` keeps every field and gains the context, and `call()` gives a module its call into it, so `signal` can name anything else the request offers. `valid_signal` and `tdls` take a `ModuleCall`. A module declares `required_permissions`, none by default, though nothing gated is passed here because the permissions are what the run produces. `SignalInput::new` still gives an input for a module asked on its own, in `ModuleContext::empty`, and `with_context` puts one in a request's context. The request view on this path borrows from the request, so assembly allocates nothing more per request. `http_util` gives the host and scheme borrowed, and `RequestInfo::from_request` owns copies of them as before. The GPC, GPP, Model Terms, TCF and US Privacy modules read their signals and terms in their own functions, handed to the call, each naming only what it reads, being the consent record for the four IAB schemes and the evidence for the Model Terms. `ModuleContext` is `Copy`, being borrows alone, so a caller with no request builds on the empty context. The permissions inspector does, and the combined signal tests assemble in a context. A new core test shows a module answering for the host of the request it is handed.
`IntegrationProxy::handle` takes the route's `ModuleCall` and the request, in place of the settings and the services, and a route declares the permissions it needs with `required_permissions`, none by default. The registry builds the context for the request it dispatches from the request's resolved state, so a route can name the permissions, consent, location, device signals and Edge Cookie identifier resolved for the request, as well as the settings and the services. The identifier only reaches a route that declares the permission its use needs, as for every other kind of module. Each route hands its own function to `inject_with`, with the request as that function's own argument, and the function names only what it reads. Seven routes name the settings and the services, four name the services alone, and the APS renderer, which answers the same page to everyone, takes nothing from its call. The tests of a route call its function directly. The route takes the request, so the context holds a copy of what the request resolved to, being its method, address, host and scheme. It holds no copy of the headers, because the route has the request itself, so a route is not passed the evidence. The Axum, Cloudflare and Spin adapters dispatched a module route with an empty Edge Cookie context, which carried no permissions. They now resolve it for the request, as the publisher path on those adapters does and as Fastly does for every request. On a navigation the registry generates an identifier for the route's context when the permissions allow, as it does on Fastly. These three adapters do not run the Edge Cookie finalization that writes the cookie on a response, so on them that identifier lasts for the one request. A new test shows a route handed its request's permissions and identifier, and one that declares nothing seeing the permissions and not the identifier. `ModuleCall::empty` gives a call outside any request, for a test of a route that takes nothing.
The response, the document state and the request's typed extension map leave the context, with `ModuleResponse`, `ModuleExtensions` and the three builder methods that set them. Nothing here sets them. They belong to the middleware that follows and arrive with it, so every value the context offers has a caller that hands it. The table of what is absent where now says what this tree does. The evidence is absent in a route, which holds the request itself. The connection, the permissions and the consent are absent where the device is classified, and the permissions where the signal modules run. The settings and the services are absent outside any request, which is the empty context a module asked on its own is called in.
A new section sets out the module context for a module author: the function a module hands to its call, the values that function may name, what happens when one cannot be passed, the permissions a gated value needs, and what the context carries for each kind of module. The example is a route that names the settings and, when it may be passed, the location.
A deployment answers `GET /_ts/permissions` with a page, and `/_ts/permissions.json` with the same information as data, with no credential. It shows the permissions resolved for the request that asks, the signals that produced them and the terms the data is held under, so a publisher can point a reader, a regulator or an auditor at it and nobody has to take anybody's word for what was decided. The answer is the evaluation a page receives as `window.tsjs.permissions`. `PermissionState::page_value` is now the one place that shape is spelled, and `page_json` writes it. The endpoint adds what a person asking why a permission is not set needs and a page does not: whether storage was explicitly withdrawn, which signal modules the publisher configured and which of them contributed a signal, and the version that answered. The handler is core's, `inspect::permissions::handle_permissions`, and all four adapters route to it, so the answer is the same on each, which a parity test checks. It resolves the request's state as a page's is resolved and writes nothing for the reader. On Fastly it answers before the Edge Cookie lifecycle, so no cookie is set and no identity row is written. The answer is never stored, and is readable from any origin so another site can render it. A request from another origin carries no cookies, so that site reads the resolution of a request with no stored signal.
`inspect::config::build_view` gives the settings a deployment is running, after defaults are applied, with values masked as `XXXX` and every masked path listed. Anyone can read the scripts a page runs, and this is what lets anyone read the configuration that runs at the edge. Three kinds of value are masked. 1. Every secret, whatever the publisher's document says. The settings loader now records each leaf it fills from the secret store, with the value it wrote, and the view masks those paths, every leaf declared a secret, and then any string that still contains one of the values, wherever it sits, a key included. 2. Every value sensitive by default, unless the publisher shows it. `Redacted` values are, and so are the settings now written with `redacted::sensitive`, being the origin URLs, the origin host override, the Basic-auth path, and the store names and ids. 3. Everything the publisher hides. The publisher decides with a new `[inspect]` section, `show`, `hide` and `config = false`, each entry a path pattern such as `proxy.asset_routes[].origin_url`. A pattern that would not be honored as written refuses the configuration, both when a deployment is validated and when the settings load. That is a pattern that matches nothing, a `show` naming a secret, a pattern another pattern already covers, a `show` a `hide` covers, a pattern that changes nothing, and any pattern beside `config = false`. Marking is a mode of serialization. `redacted::to_marked_value` wraps each sensitive value under a marker key while it runs, and every other serialization, the configuration push included, is unchanged, which a test checks field by field. The section is left out of a stored document unless the publisher wrote one, which a test checks too, so an existing document gains nothing. The note at the top of `redacted.rs` said the build script compiles the file. It does not, on this branch or on `main`, so the note goes. `handle_config` answers `/_ts/config` and `/_ts/config.json`. No adapter routes to it yet.
`GET /_ts/config` answers the masked view of the running settings as a page, and `/_ts/config.json` as data, with no credential, on all four adapters. Each routes to core's `handle_config`, so the view is the same on each, which a parity test checks. It says nothing about the request, so on Fastly it is answered ahead of everything a request passes through, and the Edge Cookie lifecycle does not run. `[inspect] config = false` answers `404`. The API reference describes the endpoint, and the configuration guide describes `[inspect]`, what is always masked and what is masked by default.
`inspect::data::handle_data` answers `/_ts/data` with the identity graph record stored against the identifier in the request's own `ts-ec` cookie, being when the record was made, the consent recorded, the location and device class kept, and which partners hold an identifier of their own for the same browser. A deployment that writes an identifier to a browser can show the person using that browser what it keeps against it. The Edge Cookie is `HttpOnly`, so a script on the page cannot read the identifier, and the page does not undo that. 1. Only a browser opening the address as a page is answered, which the browser marks with `Sec-Fetch-Mode: navigate` and `Sec-Fetch-Dest: document`. Anything else is answered `403`. 2. The page is sent with `Content-Security-Policy: sandbox`, so the document has an origin of its own and a page that opened it cannot read it. 3. No identifier is shown. The Edge Cookie's value and each partner's identifier show as `XXXX`. 4. It is never stored, and no other origin is told it may read it. Checked in Chrome against a server answering by the same rule with the same headers. A page's `fetch` was refused, with and without the two headers set by the script, and so were a frame and an embedded object. A page that opened the address could not read it, and could once the sandbox policy was left out. A service worker registered for the whole site did read it, as it reads every navigation on the site, and so can whoever holds an identifier and sends it as the cookie. Neither finds an identifier in the page, which is why none is shown. The record is shown as it is stored. A test sets every field the record has and lists them, so a field added to the record fails until it is decided whether the field is an identifier. The page renderer takes a lede of its own, because this address has no `.json` form, and the mask moves up to `inspect` now that two pages use it. `ec::request_cookie_ec` hands out the identifier a request's cookie carries. No adapter routes to the handler yet.
`GET /_ts/data` routes to core's `handle_data` on all four adapters. Fastly keeps the identity graph, so it hands the handler the graph and the Edge Cookie module the deployment selects. It answers ahead of the Edge Cookie lifecycle, so nothing is written for the reader. Axum, Cloudflare and Spin keep no identity graph, so each answers that nothing is held, and a parity test checks that the three refuse and answer alike. The API reference describes the endpoint, who can read the page and why no identifier is in it.
…form A module changes a page through four hook traits today. The integration seam design replaces them with one middleware contract (section 3.7), and asks that each converted implementation reproduces its recorded output. This is the recording, made by the hooks as they are. `html_processor::test_support::assert_page_is_recorded`, behind `test-utils`, runs a page through the document pipeline of a deployment that writes a module's settings and links its builder, and compares what a reader receives with a file, byte for byte. With `TS_RECORD_PAGES` set it writes the file instead. The script bundle's `?v=` hash is replaced by a fixed word, so a recording changes when a module's page change does and not when a browser module is rebuilt. Eleven module crates and core's own script proxy each gain one test and three files under `src/fixtures`, being the settings a deployment writes, the page the origin sends and the page a reader receives. A `.gitattributes` line keeps the two pages' line endings as committed, as the seam probe does for its script. Between them the recordings pass through all twenty hook implementations in this tree, being nine attribute rewriters, three script rewriters, one stream processor and seven head injectors. Two of the head injectors act on a request their module marked, the Google Publisher Tag diagnostics and DataDome's tag, and the recordings are of a request with no mark, so the first writes nothing and the second writes its tag. Two things the recordings show as they are. The Google Tag Manager page holds `'&l='+l` where the origin's snippet holds `'&l='+l`, because a rewritten script is written back escaped. And Prebid's script removal takes out a link whose address matches a script pattern, as it takes out a script.
When a script rewriter replaces a script's text, the HTML processor wrote the replacement with `ContentType::Text`, which escapes `&`, `<` and `>`. A browser decodes no entity inside a `<script>`, so the page ran the escaped text as it stood. `main` does the same. It reached every inline script the Google Tag Manager module rewrites. The standard snippet's `'&l='+l` came out as `'&l='+l`, so a page that names its own data layer asked for a parameter called `amp;l` and its container read the default data layer. A loader of the page's own holding `&&`, `<` or `>` became a syntax error. It reached a `script#__NEXT_DATA__` the Next.js module rewrites as well, wherever the payload holds one of the three characters unescaped. The replacement is written with `ContentType::Html`, which is inserted as it is. A rewriter is handed the text as the page wrote it, so that is the right content type in any element. The recording of the Google Tag Manager page changes by the one line that showed the fault, from `'&l='+l` to `'&l='+l`, and no other recording changes. Four tests, each failing before the change. One in the HTML processor swaps a host inside a script holding `&&`, `<`, `>` and a query string. Two in the Google Tag Manager module run the standard snippet with a data layer name of its own, and a loader holding operators, through the page pipeline. One in the Next.js module runs a payload holding a query string and a value with all three characters. This comes before the page hooks convert, because the middleware contract hands each handler a script's text as the one before it left it and writes it back as it is, and converting a hook must leave its recorded page as it was.
The Google Tag Manager module holds a script's text back while a marker
might still arrive in a later part, and writes it with the script's last
part. A part that arrived after the held text had stopped looking like a
marker was kept as it came, so it went out at once, ahead of the text
held before it, and the script reached the page out of order. `main`
does the same.
The parser hands a script's text over in parts split where a `<` might
close the script, so this needs no help from the network. Before this
change,
footer.innerHTML = '<b>Search with google</b> <i>or</i> <u>another</u> engine';
came out as
footer.innerHTML = '<> <i>or</i> <u>another</u> engine';b>Search with google</b
because the part ending on `google` is held and the parts after it are
not. It also happened when the origin's bytes broke after `google`, or
after a longer start of either marker such as `googletag`, in any inline
script that carried on past the next break.
When nothing held can become a marker any more, what was held is now
written ahead of the part that showed it, and the rest of the script
passes through as before. No recording changes.
Three tests, each failing before the change. One page arrives whole and
writes markup, one arrives in parts with no character that needs
escaping, and one is a script that is held, never rewritten, and holds
`&&` and `<`, which needed the change before this one as well.
A module changes a page through four hook traits, and every hook of a
selected module runs on every page. This adds the other way to make a
page change, which the modules convert to in the commits that follow. A
middleware is a page change a module supplies and the operator's
settings place. It runs only on the pages a `[[fetch]]` entry names it
for, in the order that entry gives.
[[fetch]]
media_type = "text/html"
path = "/news/"
middleware = ["tag.example.cleanup", "tag.example"]
[[fetch]]
media_type = "text/html"
middleware = ["tag.example"]
A page takes the first entry that covers it. An entry places and orders,
and switches nothing on. The section that selects a module still decides
whether it runs, and a middleware reads its settings from its module's
own table.
`trusted_server_core::middleware` holds the contract. A middleware is
asked once for each document, and answers with any of head markup,
markup for straight after the script bundle, element handlers, text
handlers and a stream processor.
What core does with it.
1. The registry takes `.with_middleware(...)` from a registration. At
startup it refuses two registrations that supply one name, a name no
entry could write, and an entry naming a middleware that no running
module supplies, listing the names that could be written. A
middleware no entry names is logged as a warning, because a module
may be selected for what else it does.
2. The settings read `[[fetch]]` and refuse an entry that could not do
what it says, when they load and at deploy validation. That is a
media type other than `text/html`, a path that is not a request path
prefix, an entry that names nothing or names one middleware twice, a
key an entry does not read, and an entry an earlier one already
covers.
3. The publisher path carries the request path to where a document's
processor is made, and the chain for that path is planned there.
4. The HTML processor carries the plan out after the hooks of the same
kind. Every text handler, a hook's or a middleware's, is now handed a
script as the one before it left it. Before, a second rewriter that
matched the same script was handed the origin's text, and its
replacement overwrote the first's.
A fetch middleware is told nothing about the reader, because what it
leaves is what a shared template stores. The entries are part of the
settings a template's fingerprint covers, so changing them selects a
template of its own.
A middleware chooses its selectors for each document, so one that does
not parse cannot be refused at startup. The response fails with a 500
that names the middleware, through both finalizers, where the parser
would otherwise panic.
The seam probe supplies a middleware from outside core, with one of each
handler, and the Axum and Fastly adapters' probe tests cover an entry at
startup.
Tests cover the entries, the registry's checks, what the document
pipeline does with each part of a plan, a whole request through both
finalizers at a covered path and at another, and a stored shared
template. Eleven of them were watched failing with the code broken in
six places, being the chain dropped from the publisher path, the chain
planned for the root path whatever was asked for, text handlers handed
the origin's text, the registry's check removed, and each of the two
settings checks removed.
The guides describe `[[fetch]]` and the contract, and the documentation
test compiles the guide's middleware fixture as it compiles the existing
one.
No stock module supplies a middleware yet, so no page changes.
The commit before this one places a middleware on the page as the origin
sent it, where what it writes is stored with a shared template. Some
page changes belong to one reader. This adds the second phase. A
`[[serve]]` entry names the middleware that run on each reader's copy,
whether the page came from the store or from the origin, and nothing
they write is stored.
[[serve]]
media_type = "text/html"
middleware = ["tag.example.reader"]
A middleware says which phase it runs in. Startup refuses an entry that
names one in the other phase, and the message says which phase it runs
in.
A serve middleware is handed what its module's request hooks left on the
reader's request. A request that carries any was already kept off the
shared template path, so a page from the store has none to hand over.
A page leaves from four places, and the serve entry runs at each.
1. A page that is not stored runs it on the same pass as the fetch
entry, after it, so a streamed page is still streamed.
2. A page transformed as a shared template runs it once the template is
stored and this reader's copy is assembled.
3. A page from the store runs it after assembly in the buffered
finalizer.
4. A page from the store runs it piece by piece in the streaming
finalizer, so the head still goes out before the auction is awaited.
A serve middleware works on a page core has already rewritten, and
nothing of core's own is written twice. Its head markup goes straight
before the script bundle's tag, after the fetch entry's, and its markup
for after the bundle goes straight after the tag.
A page from the store keeps no record of its origin's host. A serve
middleware is told the host from the settings, so it is told the same
on a hit as on a miss.
A serve entry that cannot run fails the response with a 500 at each of
the four places, before a byte of the page is sent.
The seam probe supplies a serve middleware from outside core, and the
Axum adapter's probe tests cover a `[[serve]]` entry at startup, one
that names a serve middleware and one that names a fetch middleware.
Tests drive a serve middleware through a whole request at each of the
four places and through both finalizers, with an auction and without,
and read the stored template to see it holds what the fetch entry wrote
and nothing the serve entry wrote. Each of the four places was then
broken in turn, and tests failed for each. Tests also failed with the
host read from the page's own record, and with the serve entry run
inside the transform of a page that is stored.
The guides describe both phases.
The Axum adapter's probe tests already build an adapter with a `[[serve]]` entry that names the probe's serve middleware, and are refused one that names its fetch middleware. The Fastly adapter is the primary deployment target, so its probe tests now make the same two checks where that adapter builds its state.
A page takes one entry, so a deployment that runs several modules names all their middleware in the one entry that covers those pages. An operator who instead writes an entry for each module was refused with advice to put the entry with the longer path first, which does not help when both entries cover the same pages. That case now says that the second entry covers the same pages as the first, and that the middleware for those pages belong in one entry's list.
The modules convert their page hooks to middleware in the commits that follow, and each module's tests then need the same four things. `AttributeRewrite::each` gives one handler for each of several attributes, all asking one decision, which is the shape nine of the attribute hooks have, a script's address judged in `src` and in `href`. `middleware::test_support::context` is what a test hands a middleware it calls directly. `html_processor::test_support::place_on_every_page` writes the entry that covers every page, and `create_page_processor` builds the processor of a test page with the middleware both phases' entries name. The recording helper builds its processor through it, so a test and a recording run a page the same way.
The GPT module embeds `gpt_bootstrap.js` as it is on disk and writes it into the head of every page. A checkout that rewrites line endings, as a Windows checkout does under `core.autocrlf`, changed the bytes of the script that is served, so a deployment built there served a different page from one built anywhere else. The recorded page showed it. The Axum tests failed on the Windows runner in `the_page_a_reader_receives_is_the_recorded_one`, and giving the file CRLF line endings on a machine where the test passes makes the same test fail. The file is pinned to LF in the crate's `.gitattributes`, as the seam probe pins the script it embeds.
Testlight pointed a `src` or `href` that names `testlight.js` at its shim through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `testing.testlight`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_scripts` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware. This is the first module to convert, so the changelog gains the entry for the whole change, which is breaking for a deployment that selects a converted module and writes no entry. The configuration guide gains the table of the middleware the stock modules supply, with this one row.
Permutive pointed a `src` or `href` that is its SDK's address at the first-party endpoint through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `audience.permutive`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_sdk` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware.
lockr pointed a `src` or `href` that is its SDK's address at the first-party endpoint through an attribute hook, on every page of a deployment that selected it. It now supplies that change as the middleware `identity.lockr`, which runs on the pages a `[[fetch]]` entry names it for. The decision is the one the hook made, and it still does nothing unless `rewrite_sdk` is set. The recorded page is unchanged. Its settings gain the entry that places the middleware.
Didomi wrote `window.__tsjs_didomi` into the head through a head injector, on every page of a deployment that selected it. The script hands the browser module the path Didomi is served under. It is now the middleware `cmp.didomi`, which runs on the pages a `[[fetch]]` entry names it for, and writes the same script. The recorded page is unchanged. Its settings gain the entry that places the middleware.
Sourcepoint changed a page through two hooks, on every page of a deployment that selected it. A head injector wrote the browser module's settings and, with `rewrite_sdk` set, the trap on `window._sp_`. An attribute hook pointed a `src` or `href` on Sourcepoint's CDN at the first-party path, also only with `rewrite_sdk` set. Both are now the one middleware `cmp.sourcepoint`, which runs on the pages a `[[fetch]]` entry names it for and makes the same two changes under the same setting. The recorded page is unchanged. Its settings gain the entry that places the middleware.
Core's version_header module is removed, with its call from the finalize step of the Axum, Cloudflare and Spin adapters and from the Fastly adapter's health probe and its settings fallback, so no response carries x-ts-version unless a [[serve]] entry names the service_version middleware. That middleware now writes the git version the build compiled in from TRUSTED_SERVER__GIT_VERSION or local git as X-TS-Version, when the build knows one, and the version the adapter reports for the service as X-TS-Service-Version, when it reports one. HEADER_X_TS_FASTLY_VERSION, which nothing wrote, goes with the module. The health tests of the three adapters prove the probe carries no version, the middleware's own tests prove each header is written when its version is known and nothing when neither is, and the publisher tests prove both headers reach a buffered page and neither reaches a page no entry names the middleware for. The configuration guide, the getting started guide, the EdgeZero guide and the changelog say the same.
# Conflicts: # docs/guide/configuration.md
# Conflicts: # docs/guide/configuration.md
jwrosewell
marked this pull request as ready for review
October 14, 2026 12:35
This was referenced Oct 14, 2026
Collaborator
|
Moved into this repository as #1366, one branch of the stack, so the review continues there. This pull request is closed in its favor and its comments stay here for the record. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on the loading cache (#1312). The additions that follow the seven pull requests open here are listed in #777.
Compare
split/24-loading-cachewithsplit/25-robots-txt-middlewareto see only this pull request's change.
What this pull request does
/robots.txtwas answered by a handler of core's own, registered as a route on each of the four adapters with a cache policy of its own, while every other change to what a deployment serves is a middleware named by an entry. It is nowrobots_txt, a middleware of core's own, which runs ontext/plainand prepares.Its preparation does what the route did. A refusal of every crawler needs nothing and gives the refusal document, and otherwise each selected module is asked through its holding and the file is assembled. The action replaces the origin's
text/plainfile with it. Where the origin answers404, or200with a page, as a single-page application's catch-all does, the creating rule (#1309) makes the file. Any other answer passes as the origin gave it, so a staging site behind a login that answers401sends the crawler that401and not the refusal, though every response still carriesX-Robots-Tag. With no answer held for a module that fails, the preparation refuses with503andRetry-After, and nothing is stored.The registry makes the middleware itself, once every registration is in, because the modules it asks are known only then and a middleware cannot reach the registry.
[robots-txt]is refused when the settings load, and so byts config validate, unless a[[fetch]]entry fortext/plainwithpath = "/robots.txt"namesrobots_txt, because only a[[fetch]]entry makes the file where the origin has none. An entry naming it on another path or media type is refused too, because it would replace that file with robots.txt.The name is
robots_txtwith an underscore, unlikerobots-txtfor the section and the module folder andads-txt.sellersfor the stock file middleware, because it is core's own middleware and not a module under a type folder, and core's own names are written with underscores, asjs_asset_proxy,refuse_allandallow_allare.Why
#1301 added the route. This undoes its commit 3701fbc in full, which is about 600 of the 692 lines removed here: the route, the handler and eleven adapter tests. The two are separate pull requests because the middleware needs the preparation step (#1311) and the creating rule (#1309), which came after #1301, and #1301 is kept as the change that made robots.txt modules, whatever serves the file.
What breaks
handle_robots_txt, the response builder and the route's ownCache-Control: public, max-age=3600go. The file takes the deployment's cache policy for the path, as every file does, so where the response is stored a module is asked once per stored copy's life, not once per refresh.The route goes from the four adapters, with its path constant in each named-route table and the served-check that existed for it alone. Each adapter's tests now prove the path is no route of this server and that a refusal still tags every response.
A
[robots-txt]section needs the[[fetch]]entry above.How it was verified
CI passed on
895e66c30on the fork. Not exercised: no file has been served through an adapter against a real origin. The file is proven through the publisher path with a stub origin, and each adapter's tests prove the path is no route of this server.References #1283.