Skip to content

Make a file the origin does not have from the middleware named for its path - #1309

Closed
Jamesr51d wants to merge 568 commits into
IABTechLab:mainfrom
jwrosewell:split/22-creating
Closed

Jamesr51d wants to merge 568 commits into
IABTechLab:mainfrom
jwrosewell:split/22-creating

Conversation

@Jamesr51d

@Jamesr51d Jamesr51d commented Oct 12, 2026 •

Copy link
Copy Markdown

Stacks on the consent prompt phases (#1308). The additions that follow the seven pull requests open here are listed in #777.
Compare split/21-cmp-phases with split/22-creating
to see only this pull request's change.

What this pull request does

When the origin answers 404 Not Found for a path under the path of a [[fetch]] entry for a media type other than HTML, core discards the origin's body and headers and runs that entry's middleware on an empty document of the entry's media type, and its output is the file. The prefix is compared as an entry's path always is, so path = "/ads.txt" also makes /ads.txt.bak. The made file carries only Content-Type. It is not stored as a template, which is HTML alone, and no cache policy applies unless an asset policy names the path, so it is made again on every request. So a publisher whose origin has no ads.txt gets one made of the sellers lines alone.

[ads-txt]
modules = ["sellers"]

[ads-txt.sellers]
lines = ["ssp.example, 12345, DIRECT, f08c47fec0942fa0"]

[[fetch]]
media_type = "text/plain"
path = "/ads.txt"
middleware = ["ads-txt.sellers"]

The rule

A well known file is made where middleware is configured for its path, and is the origin's answer where none is.

  • An entry for HTML, or without a path, makes nothing, because a page the origin does not have is the origin's answer.
  • The entry taken is the one with the longest path covering the request, of those for a media type other than HTML, so neither an HTML entry nor one for a shorter path shadows it.
  • Any answer from the origin other than 404 stands, an error among them.
  • A GET and a HEAD are answered alike, the HEAD with no body.

The middleware is an ordinary one, given an empty document. A middleware that adds to a file writes the same on an empty one, with nothing before its first line, which the sellers module already did.

How it was verified

CI passed on 82c1a196e on the fork. Not exercised: no file has been made through an adapter against a real origin. The rule is checked through the publisher path with a stub origin answering 404, and the made file is streamed through the entry's middleware.

References #1280.

The Fastly entry point built each request's device signals with
`build_device_module`, which knows only `builtin` and `fastly`. With
`[device] module` naming a module a registration supplies, the registry
resolved that module and the adapter put it in the request's services, and
nothing ever asked it to classify a request, so the built-in User-Agent
module answered in its place.

The entry point now hands `derive_device_signals` the module the registry
resolved. A registered module is shown the User-Agent and the cookies, which
carry what a page gathers in the browser. With no registered module the
built-in or the Fastly module answers as before, shown the User-Agent alone.

Two tests in the adapter show a registered module classifying the request
and seeing its cookies, and the built-in module answering when nothing is
registered. `build_device_module`'s doc comment said a name it does not know
is refused by the settings, which is the registry's job, and says so.

Fastly is the one adapter that classifies a request. The Axum, Cloudflare
and Spin adapters resolve the selected device module and set no device
signals, on `main` as here.
…ition branch

The merged-in entry point hands the device module the registry resolved to
`derive_device_signals`. That function lives in `lib.rs` on this branch,
so the change is made there and `main.rs` stays the thin binary. A ninth
round trip test selects the probe's device module and shows it classifying a
request through the function the entry point calls, with the built-in module
answering when the selector is unset.
The probe supplied an ad server, so one half of the auction seam was reached
from a crate core does not know and the other was held by tests inside core
alone. Its builder now registers a demand source as well, one that sends the
standard request and reads every answer as no bid, which is enough to show
that everything a demand source needs is reachable from outside core.

An Axum test starts the adapter and runs deploy validation with the probe's
demand source selected, and both refuse the same settings without the
probe's builder. The probe's own tests pin both implementation names to its
module name and show the demand source refusing a setting it does not read.
…ition branch

The merged-in probe supplies a demand source, so a tenth round trip test
builds this adapter's state with it selected and shows the same settings
refused without the probe's builder, as the ad server's test does.
The Fastly adapter becomes a library with a thin binary, so a deployment
that ships an integration crate writes its own binary and passes that
crate's builder to `run_with`. The seam's round trip then runs on Fastly
under Viceroy from a crate core does not know, as it already does on Axum.

The branch was built and checked apart from this one so the seam could be
read without it. It is merged here because the seam's design asks for a
crate outside core to register on every adapter, and Fastly was the one
adapter where that could not be shown.
…ords

`KvNetwork` said a low cluster count "indicates an individual or
household". A count of connections is a fact about a network, and reading
a low one as an individual describes a device record as a record about a
person. It now says a small network, such as a home connection.

`KvEntry::ids` now says what it holds, which is each partner's identifier
for the same browser on the same device, never an identifier for a person.

Comments only, with no change in behavior.
The crate's description and the workspace manifest said only the Axum
adapter's tests depend on the probe. The Fastly adapter's tests drive the
same round trip, so both say so, and the description points at both test
files.
…e crate

The Osano consent mirror is the module `cmp.osano`, and it now lives in the
crate that name describes, `crates/cmp/osano`. Core no longer declares or
lists it. The settings that select and configure it do not change.

`crates/trusted-server-modules` lists the modules a stock build ships, in
the order their hooks run. Each adapter builds its state with that list
followed by the builders a deployment added, and loads its settings against
the same list. The `ts` tool registers the list with
`register_deploy_integrations`, because `TrustedServerAppConfig` validates
through a trait that takes no arguments, so `ts config validate` and
`ts config push` run a stock module's own rules.

Core's tests of how a selected module's table is read used the
configuration types of Osano and Testlight as examples, and now use two
example modules of their own. The two tests that needed Osano's builder
inside core run in Osano's crate with that builder. Core's shared test
settings are offered under the `test-utils` feature, so a module crate's
tests start from the same settings core's do.
An integration that leaves core still bounds a request body and an upstream
response, registers a backend, strips the headers an upstream must not set
on a first-party response, matches a host against the allowed domains, marks
a synthesized response private, reads a list setting and rewrites a bare
host. Those were private to the crate because every caller lived in it.
They are public, each with the reason on it.

The platform test doubles are offered under the `test-utils` feature, where
the adapters already take core's other fixtures, so a moved integration's
tests keep the stub HTTP client and stores they were written against.
The feature turns on the platform library's feature of the same name,
because the doubles are built on that library's own.
The Lockr identity module is `identity.lockr`, and it now lives in
`crates/identity/lockr`. The stock list offers it in the place its hooks
ran before, and the settings that select and configure it do not change.

Three of core's tests used Lockr as their example of a built-in module.
The duplicate id test and the standalone script test use core's own
modules, the JavaScript asset proxy and the creative browser module. The
test that the documented settings template's module tables validate moves
to the stock list crate, where each module's settings type is in reach,
and the two template helpers it shares with core's own tests are offered
from core's test support.
The Permutive audience module is `audience.permutive`, and it now lives in
`crates/audience/permutive`. The stock list offers it in the place its
hooks ran before, and the settings that select and configure it do not
change.
The Sourcepoint consent management module is `cmp.sourcepoint`, and it now
lives in `crates/cmp/sourcepoint`. The stock list offers it in the place
its hooks ran before, and the settings that select and configure it do not
change.

Its tests build a registry with the crate's own builder, through
`IntegrationRegistry::with_registrations`, which is offered under the
`test-utils` feature for that. The test that a selected consent mirror's
browser script loads with the page moves here from core, as Osano's half
of it did.
The Didomi consent management module is `cmp.didomi`, and it now lives in
`crates/cmp/didomi`. The stock list offers it in the place its hooks ran
before, and the settings that select and configure it do not change.

Core's test that a module's table survives the configuration blob round
trip used Didomi's settings as its example. It uses an example module of
its own, with a flag set away from its default.
The Google Tag Manager module is `tag.google-tag-manager`, and it now
lives in `crates/tag/google-tag-manager`. The stock list offers it in the
place its hooks ran before, and the settings that select and configure it
do not change.

One of its tests runs the HTML pipeline over the publisher page fixture
core's own HTML tests use. Core's test support offers that page, so the
half megabyte file stays in one place.
The Testlight example integration is `testing.testlight`, and it now lives
in `crates/testing/testlight`. The stock list offers it in the place its
hooks ran before, and the settings that select and configure it do not
change.

It reads the Edge Cookie a request carries through
`edge_cookie::recognized_ec_id`, so that module is public. The test that
serves a page through the HTML pipeline and checks Testlight's script is
replaced moves with it. The registry test that lists the modules a
deployment does supply names a probe builder of its own.
The Next.js module is `framework.nextjs`, and it now lives in
`crates/framework/nextjs`, with its six source files and two fixtures. The
stock list offers it in the place its hooks ran before, and the settings
that select and configure it do not change.

Five of core's page pipeline tests used Next.js as their example of an
integration that rewrites script payloads in two passes, a script rewriter
that leaves placeholders and a stream processor that fills them. They test
core's part of that (the document state, the hold on an unresolved group,
the auction seam, the template cache), so they stay in core and select a
stand-in that does the same two passes with no vendor's format. The
stand-in is compiled for core's tests only. The test of Next.js's own
overflow handling moves to its crate, and the registry test that used
Next.js as an integration with no browser script uses a probe.

Two of Google Tag Manager's tests check that it leaves another module's
script rewrite alone, with Next.js as that module, so its crate takes the
Next.js crate as a test dependency. The stock list crate gains the test
that every stock module refuses a setting it does not know, which core's
own list can no longer show for a module that has left it.
The Google Publisher Tags module is `ad-tag.google`, and it now lives in
`crates/ad-tag/google` with the bootstrap script it injects. The stock list
offers it, and the settings that select and configure it do not change. Its
diagnostics module, `ad-tag.google.diagnostics`, belongs to the same crate
and stays in core until the page path can carry a module's own decision for
a request.

Five of core's tests used this module as their example of an integration
that tags a page: the HTML processor's test of the attribute an injector
asks for on the publisher bundle tag, the streaming test of the head
arriving before the origin finishes, and the JavaScript asset proxy's three
tests of its place ahead of an integration's own rewriter. They select a
second stand-in compiled for core's tests only, which inserts a head
script, asks for a bundle attribute and rewrites the address of its own
script. The core test that a selected module with invalid settings fails
startup uses an example module of its own.

The browser tests and the initial render harness read the bootstrap script
from its new place.
Two modules decide something about one request and act on it later, and the
page path named both. It called the Google Publisher Tags diagnostics module
to prepare the request and to finish the response, carried that module's
decision to the HTML processor, which wrote its two scripts, and carried a
flag for DataDome's tag beside it. Neither module can leave core while core
does that for it.

A request preparer or a request filter now leaves a value in the request's
`IntegrationRequestState` under its integration id. The page path copies
those values into the document's state before parsing starts, so the
module's own head injector reads them, and hands them to the response
finalizer a builder declares with `with_response_finalizer`. A head injector
gets a second place to write, straight after the script bundle, which is
where the diagnostics module loads its standalone script. The diagnostics
module and DataDome both use these and core calls neither.

A request that carries any value keeps to the origin path, so its document
is never read from a shared template and never stored as one, and its HTML
response is sent `private, no-store`. That was true of both modules before,
each by its own route. It is now one rule that holds for any module, checked
where the request is routed and again where the document is built.

The registry runs the request preparers once for a request, so the page
path asks it to prepare a request that reached it unprepared and names no
module to do it.

The token an auction publishes with its winning bids was made when the
diagnostics module ran. A builder now declares with `with_auction_token`
that its browser script reads the token, the auction plan records whether a
selected module does, and the page path asks the orchestrator.

Core's tests of all this use a third stand-in compiled for its tests only.
It marks a document navigation that carries its query, writes a head script
and its standalone module's tag for a marked request, and sets a cookie on
the response. The template cache tests that set a diagnostics decision by
hand, the template rendering tests and the auction token tests use it. The
tests that drive either module itself still do, and hand the page path the
request state where they handed it a decision or a flag.
The diagnostics module is `ad-tag.google.diagnostics`, the second module of
the crate at `crates/ad-tag/google`, and it now lives there beside the
Google Publisher Tags module. The stock list offers it last, where its hooks
ran before, and the settings that select it do not change. It prepares the
request, writes its two scripts and finishes the response through the hooks
any module has, so core calls nothing of its own.

Its tests went with it, and four were added for what it now does through
those hooks: that only a decision which acts is left for the page path,
that the finalizer acts only on such a decision, and that a document gets
both scripts in order when diagnostics is active and neither when it is
not. The test that the built-in list attached its request preparer is now a
test that the stock list does.

Core's tests that used the module as their example use core's stand-in: the
HTML processor's test of what is written around the bundle, the registry's
two tests of a standalone browser module, and the page path's tests of a
query a module acts on, a warm template, an inactive ad stack and a
standalone module's own path. The Fastly adapter's test that a late filter
cannot make such a response public marks the response the way a module's
finalizer does and names no module.

One of core's template cache tests depended on this module without saying
so. The module's preparer writes the Cookie header of every request out
again as it strips its own cookie, dropping a field it cannot read, and the
test expected a request with such a field to be stored. The stand-in now
reserves a cookie and does the same, so the test still shows that the cache
classifies a request as a preparer left it. A stock build behaves as
before, because the stock list still runs the module's preparer for every
request.

The page path's two request classifiers the module uses, for a prefetch and
for a crawler, are public. The changelog's list of moved modules had been
garbled by the full stop in Next.js and reads correctly again.
A setting that holds a secret holds the name of its key in a pushed
configuration and the secret once the settings load. Core named DataDome's
two such settings in three places: the list of secret paths, the step that
clears a reference nothing uses before the lookup, and the deploy check that
a reference in use names a key. A module with a secret could not leave core
while core listed its settings by name.

A builder now declares them with `with_secret_settings`, each as a path
inside the module's own table and a rule for when the table as written puts
it to use. Core does the three things from the builders it is given, for
any module. DataDome declares its server-side key as in use when protection
is on, and its test bypass credential when the bypass is on as well, which
are the rules core applied for it.

The settings load takes the declarations from the builders it already
validates against, so a module a deployment added has its secrets looked up
as a stock one does. A leaf listed twice is looked up once, because the
fixed list and the load can both name a module core itself offers, and a
second lookup would read the secret as the name of a key.

A module's table sits under the name its section selects it by, which may
be the short name or the full one. The paths are listed and cleared at both,
where the code this replaces looked only at `bot-protection.datadome`.
DataDome is the module `bot-protection.datadome`, and it now lives in
`crates/bot-protection/datadome` with its Protection API filter and its
protection scope. The stock list offers it in the place its hooks ran, and
the settings that select and configure it do not change. It leaves its tag
suppression for one request, and declares its two secret settings, through
the hooks any module has.

Its 59 tests went with it and eight were added from what core tested for it:
that a suppressed document keeps and rewrites the publisher's own tag, that
an ordinary one gets the client configuration, that the settings load looks
up both secrets when both are in use and clears each one that is not, the
deploy check of an invalid test bypass, and the refusal of a removed
setting. The stock list crate checks that a registered list puts DataDome's
two settings among the leaves a push treats as naming a key.

Core's tests that used DataDome use core's stand-in, or the neutral marker
they were about. The stand-in gains a request filter that leaves its mark
for a header, so core still tests state a filter leaves reaching the
document, which is the route DataDome takes. The scheduling test's origin
document gains a head. It asserted that DataDome's tag was absent from a
document that had no head for any tag to be written into, so it could not
fail, and it now asserts that what a module left on the request is written.
The tests of a personalized response say personalized where they said
suppressed, and no longer set DataDome's marker beside the neutral one.
Core's configuration tests use a module that exists only in the test.

Core's settings no longer remove DataDome's two deprecated secret store
selectors from its table as they are read. DataDome accepts and discards
both, as it did, so a configuration that carries one still loads, and a
pushed configuration now keeps them.
`ts audit` turns the integrations it finds on a page into the section that
selects each one's module and the name written there. It took each module's
name from `trusted_server_core::integrations`, so it stopped compiling when
the first module left core, and no local check showed that because the `ts`
tool cannot be built on the machine this branch was made on.

The stock list crate now answers the question by integration id from the
builders it already lists, with a test of every id the generator knows, and
the generator asks it. The generator names no module of a vendor's. Prebid
is the one exception, because it is registered through the auction plan and
no builder carries its name, and it stays in core for now.
Prebid and APS each register a page integration that depends on the
compiled auction plan, so the registry called both by name before it ran
any builder, recorded Prebid's id by hand, and listed Prebid's module name
among the ones a section may select. Deploy validation called Prebid's two
checks by name for the same reason. Neither module can leave core while
core does that.

A builder now registers from the plan with `with_plan_registration` and
checks its settings against the plan with `with_plan_validator`. A
registration from the plan is made whether or not a section selects the
module, because the function decides from the plan and the settings whether
the module runs, and its hooks go ahead of every module a section selects,
which is where Prebid's and APS's ran. Prebid is a builder like any other,
selected in `[auction]`, and APS's implementation builder carries its
registration. The registry and the deploy check name neither.

The page path read the identifier a renderer picks its bid by with APS's
type tag and APS's payload key. The code that builds a renderer descriptor
now states which payload key holds that identifier, with `picking_bid_by`,
and the page path asks the descriptor. A descriptor built without the
statement has no such identifier whatever its payload holds, so a key of the
same name under another renderer's tag is still not read, which a test
pins. The statement is no part of what is sent to a page.
The demonstration ad server is the implementation `ad-server.mock`, and it
now lives in `crates/ad-server/mock`. The stock list offers it, so
`[ad-server] module = "mock"` selects it as before and its settings do not
change. Its 22 tests went with it.

Core's tests of the ad server seam selected the mock because it was the one
ad server there was. They select a stand-in compiled for core's tests only,
`ad-server.fixture`, which reads an endpoint and a timeout, refuses any
other setting and answers every request with no bid. That covers the plan
compiler's tests of how a name resolves within `[ad-server]` and how a
table's settings are checked, the registry's refusal of an auction
implementation named in a section, and three tests that built a mock
provider only because they needed a provider.

One orchestrator test ran APS through the mock ad server and checked that
the winner kept its seat, its bidder name and its renderer. It is a test of
the two together, so it is in the stock list crate, where both are on
offer, and it drives the orchestrator through its public interface.

The bid dimension helpers the mock ad server admits a bid with are public,
so an auction implementation in a crate of its own applies the rules core's
response reader applies.
The Amazon Publisher Services demand implementation is `auction.aps`, and
it now lives in `crates/auction/aps`. The stock list offers it, so a
`[demand.<name>]` table with `implementation = "auction.aps"` selects it as
before and its settings do not change. Its builder registers the page
support the APS renderer needs when the auction plan selects an APS source.

Its 38 tests went with it, and 24 more came from core, where APS had been
the example the engine was tested with:

- three tests of the renderer descriptor APS writes,
- eight that run an APS source through the orchestrator and the response
  reader,
- four of the plan compiler and the page registration,
- six of the request APS builds, with both golden requests carried byte for
  byte,
- three of the deploy check of APS's own settings.

Those tests run against core's engine from APS's crate, so core offers what
they need under its `test-utils` feature. That is the helpers that build an
auction context, a demand table and a plan, two drivers that build a
request for a plan's first source and read a response as that source, and
the backend double in the platform test doubles.

Core's own tests of routing, the plan compiler, the request builder, the
response format and the page's bid map use core's plain OpenRTB source, or
a renderer descriptor of the test's own tagged `example`, in place of APS.

`ERROR_TYPE_HTTP_STATUS` and `ignored_bidder_params_count` are public, so
an implementation in a crate of its own reports a failed call and counts
ignored bidder parameters the way core does.
Jamesr51d and others added 23 commits October 13, 2026 16:30
The wasm clock lint reaches core's build script, which runs on the host.
Core's version_header module is removed, with its call from the finalize
step of the Axum, Cloudflare and Spin adapters and from the Fastly adapter's
health probe and its settings fallback, so no response carries x-ts-version
unless a [[serve]] entry names the service_version middleware. That
middleware now writes the git version the build compiled in from
TRUSTED_SERVER__GIT_VERSION or local git as X-TS-Version, when the build
knows one, and the version the adapter reports for the service as
X-TS-Service-Version, when it reports one. HEADER_X_TS_FASTLY_VERSION, which
nothing wrote, goes with the module.

The health tests of the three adapters prove the probe carries no version,
the middleware's own tests prove each header is written when its version is
known and nothing when neither is, and the publisher tests prove both
headers reach a buffered page and neither reaches a page no entry names the
middleware for. The configuration guide, the getting started guide, the
EdgeZero guide and the changelog say the same.
# Conflicts:
#	docs/guide/configuration.md
@jwrosewell

Copy link
Copy Markdown
Collaborator

Moved into this repository as #1363, one branch of the stack, so the review continues there. This pull request is closed in its favor and its comments stay here for the record.

@jwrosewell jwrosewell closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants