diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md new file mode 100644 index 0000000..44e813f --- /dev/null +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -0,0 +1,1223 @@ +(adr-0001-typed-vanilla-doctest-core)= + +# ADR 0001: A typed, vanilla-compatible doctest core + +Status: Proposed +Date: 2026-08-02 + +## Context + +### What ships today + +`doctest_docutils` re-implements the *finding* half of {func}`doctest.testfile` +over a docutils or MyST doctree and keeps CPython's *running* half. +`pytest_doctest_docutils` wraps that in a pytest plugin. + +**Released gp-libs already has per-block identity, and no sharing unit at all.** +`DocutilsDocTestFinder._find` walks the doctree and appends one +{class}`doctest.DocTest` per matched node, named `page.md[k]` where `k` is the +document-order index. The collector yields one {class}`pytest.DoctestItem` per +test. Each test is built by handing `globs` to +{meth}`doctest.DocTestParser.get_doctest`, and `DocTest.__init__` **copies** the +mapping — so every block runs against its own isolated namespace. + +That is the real starting point, and it frames the problem precisely: the +granularity this design wants to *preserve* is already shipped. What is missing +is any unit coarser than a block — no groups, no phases, no way for a narrative +page to build state across the prose that explains it. + +**The plugin blocks the plugin whose internals it imports.** +`pytest_configure` calls `config.pluginmanager.set_blocked("doctest")`, and the +same module then imports that plugin's private helpers. This survives only +because `_pytest/fixtures.py` has no `pytest_plugin_unregistered` handler, so the +already-parsed `doctest_namespace` fixture outlives unregistration. + +### What a shared namespace costs + +[PR #87](https://github.com/git-pull/gp-libs/pull/87) is an open, unmerged +attempt at the first problem: it adds Sphinx-style groups, a merge step, phase +ordering, skip lifting, an exec-mode runner and an xdist scheduler. **None of it +has shipped in any release, and none of it is on trunk.** It is described here as +a design under review, not as the status quo, because what it had to build to +work is the evidence this record turns on. + +Three costs are worth naming, because a clean-room design must either pay them +again or explain why it does not. + +**Merging blocks fights line-number fidelity.** A merged group is one `DocTest` +with one `docstring` and one `lineno`, and both doctest's `%03d` gutter and +pytest's `repr_failure` reconstruct locations by slicing that single string. So +the blocks must be laid out on a synthetic page with blank-line padding and a +clamp, and a wholly-skipped block must be lifted back out to report at all. + +**Prompt-free `{testcode}` needs a CPython private.** The per-example loop +[`DocTestRunner.__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) +hard-codes `"single"` +([`Lib/doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400)), +and `sphinx.ext.doctest` gets around that by rebinding `doctest.compile` +process-wide and never restoring it +([`sphinx/ext/doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310)) — +unavailable to a library that loads into every pytest session that installed it. +PR #87's answer clones the mangled method's code object into a fresh +{class}`types.FunctionType` whose globals map `compile` to a local helper. That +carries a latent defect: those globals are a snapshot of `vars(doctest)` taken at +import, so a later rebind of a module-level name in `doctest` is invisible to the +clone while remaining visible to the stock runner, and two runners in one process +disagree. + +**A live shared mapping forces an xdist fork.** Only execnet-serializable +builtins cross a worker boundary, so a shared namespace must either be merged +into one `DocTest` or kept on one worker. Keeping it there means a scheduler, and +the only affinity primitive in all of xdist is +[`LoadScopeScheduling._split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284), +a pure function on node-id strings. The controller never collects; it learns the +suite only as node ids arriving from workers. So PR #87 re-derives "these ids +share state" from strings, and re-implements +[`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26) +including its quirks. + +### The conflation underneath + +All three costs follow from one thing. **The granularity of test identity and the +granularity of shared state are different axes, and every surveyed design couples +them.** PR #87's two settings are the clearest illustration: `merged` gives one +`DocTest` and one node id per group; `per-block` gives N `DocTest`s, N node ids +and one live mapping — a node id that raises `NameError` when selected alone. +Sybil ships the second shape without acknowledging it (see [](#prior-art)). + +## Decision + +**One pytest item owns one shared-state group; inside it, each source block +remains a real, independent {class}`doctest.DocTest`.** + +The decoupling is not "group versus block". It is **scheduling identity versus +diagnostic identity**: pytest schedules the group, while each `DocTest` keeps its +own source location, examples and failure gutter. + +One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block +`DocTest`s run in phase order against one live `globs` mapping that never leaves +the item. + +The execution shape is partly precedented. `sphinx.ext.doctest` runs several +`DocTest`s against one shared group namespace — but only for the *test* phase. +All of a group's `testsetup` blocks are combined into a **single** simulated +`DocTest` named `f"{group.name} (setup code)"`, and likewise cleanup; only +`group.tests` is one `DocTest` per block +([`sphinx/ext/doctest.py:525-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L525-L556)). + +So this design is per-block in all three phases where Sphinx is per-block in one, +and — more importantly — Sphinx produces no selectable, reportable unit for any +of them: every ordinary test block in a group shares one `DocTest.name`, which is +why `SphinxDocTestRunner` overrides a private stdlib method to swallow the +resulting `IndexError`. Mapping the group onto one {class}`pytest.Item` while +each block keeps its own identity is the contribution. + +With the default checker that buys, for free and with no override of +`repr_failure` or `reportinfo`: per-block failure locations, per-block gutters, +and per-block "location unknown". +Meanwhile `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode are structurally +incapable of splitting the shared state, because there is only one item to +schedule. + +**A per-block `SKIPPED` outcome is not among them.** +{class}`pytest.TestReport`'s `outcome` is one scalar per item, so a group holding +one all-`SKIP` block and one passing block reports `PASSED` with the skip erased. +Signalling the skip instead flips the *whole* group to `SKIPPED`. Surfacing it +per block requires pytest's builtin-but-experimental `subtests` plugin, appears +in the terminal gutter and `-rs` only at `verbosity_subtests >= 1`, and never +becomes a separate ``, `--lf` entry or rerunnable unit. See +[](#the-outcome-contract). + +### The three facts this rests on + +Each was verified by executing it, not by reading it. + +**1. pytest reads failure locations per failure, not per item.** +[`DoctestItem.repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317) +iterates the failure list and reads `failure.test.filename`, `failure.test.lineno` +and `example.lineno` inside the loop +([`_pytest/doctest.py:337-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L337-L344)): + +```python +for failure in failures: + example = failure.example + test = failure.test + filename = test.filename + if test.lineno is None: + lineno = None + else: + lineno = test.lineno + example.lineno + 1 +``` + +With one `DocTest` per block, every failure therefore carries its own `filename` +and `lineno` for free. A block reached through `.. include::` reports the +*included* file. A block docutils could not locate carries `lineno=None` and +takes pytest's honest `EXAMPLE LOCATION UNKNOWN` branch **without poisoning its +siblings**. The default-checker path requires no override of `repr_failure` or +`reportinfo`. + +This is what makes merging unnecessary: the synthetic page, its blank-line +padding and its clamp exist only to reconstruct locations from a single spliced +docstring, and there is no spliced docstring here. + +**2. `_DocTestRunner__run` is an ordinary attribute override.** Name mangling +rewrites the *call site* at compile time, so the `self.__run(...)` lookup inside +`run()` +([`Lib/doctest.py:1571`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1571)) +resolves through the instance's MRO like any other attribute. A subclass that +defines `_DocTestRunner__run` takes over the loop, with `run()` untouched. + +This page is itself a doctest, so the claim is checked on every run: + +```{doctest} +>>> import doctest +>>> import io +>>> fired = [] +>>> class Runner(doctest.DocTestRunner): +... def _DocTestRunner__run(self, test, compileflags, out): +... fired.append(test.name) +... return super()._DocTestRunner__run(test, compileflags, out) +>>> example = doctest.Example("1 + 1\n", "2\n") +>>> test = doctest.DocTest([example], {}, "demo", "demo.py", 0, None) +>>> Runner().run(test, out=io.StringIO().write) +TestResults(failed=0, attempted=1) + +The subclass method ran, and `run()` was never overridden: + +>>> fired +['demo'] +``` + +Keeping `run()` as stdlib's matters: it owns the save-and-restore of +`sys.stdout`, `pdb.set_trace`, `linecache.getlines`, `sys.displayhook`, +`_colorize.can_colorize` and the `PYTHON_COLORS`/`FORCE_COLOR` environment +variables, all in its own `finally` +([`Lib/doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +That contract is not reproduced; it is inherited. + +**3. Making the item the sharing unit dissolves the *affinity* problem.** A live +mapping never crosses a process boundary, so there is nothing for xdist to split, +under any `--dist` mode. No affinity primitive, no scheduler substitution, no +`parse_tx_spec_config` fork, no node-id string sniffing. + +The identical-collection requirement is untouched by this and still binds at any +granularity. It is approached separately through deterministic projection over +the complete source closure, normalized settings and a frozen registry — which +is why `:skipif:` is carried through collection unevaluated. + +(the-outcome-contract)= + +### The outcome contract + +One item means one item outcome. That is a real cost of this design and it is +stated here rather than discovered later. + +| Signal | Granularity | Notes | +|---|---|---| +| Failure location, `want`/`got`, gutter | **per block** | default `repr_failure` iterates failures and reads each one's own `DocTest`; custom checkers use the same locations in the narrow rendering branch | +| `EXAMPLE LOCATION UNKNOWN` | **per block** | a block with `lineno=None` does not affect its siblings | +| `passed` / `failed` / `skipped` | **per item** | `TestReport.outcome` is one scalar | +| JUnit `` | **per item** | node reporters are keyed by node id | +| `--lf`, `-k`, `--deselect`, rerun unit | **per item** | | + +The skip case is the sharp edge, and it cuts both ways. If the item swallows a +block's skip, a group with one all-`SKIP` block and one passing block reports +`PASSED` and the skip leaves no record — no count, no `-rs` line, no JUnit +``. If the item raises instead, the whole group reports `SKIPPED` even +though a sibling passed. pytest's own doctest plugin takes the second horn only +when *every* example is skipped, via `_check_all_skipped`. + +This design takes the same position over the test phase: **skip the item when +every `Phase.TEST` block is skipped; otherwise report partial skips as typed block +detail, not as a pytest outcome.** Setup and cleanup are infrastructure and do +not contribute a passed or skipped test. No extra reports are synthesized. + +"Typed block detail" needs a channel, or implementers will re-invent skip lifting +or write to stderr. The channel is a `GroupResult` — one `BlockResult` per block, +each carrying phase, outcome, gate reason and location — attached to the item and +rendered in two places: the failure longrepr when the group fails, and a +terminal-summary line at `-rs`. It is **not** visible at default verbosity, and +it never becomes a JUnit `` entry. + +That is a real product loss relative to lifting a gated block into its own item, +and it is accepted deliberately: **a gated block inside a mixed group gives up its +selectable skip row.** The information survives; the addressable unit does not. + +`subtests` — a builtin pytest plugin since 9.0, exporting `pytest.Subtests` and +`pytest.SubtestReport` — can emit per-block outcomes, and is the only sanctioned +mechanism that can. It is not adopted here: pytest documents it as experimental, +its output is invisible at default verbosity, and it produces no separate JUnit +entry, so it would buy terminal detail at the cost of depending on an unstable +surface. Revisit if it stabilizes. + +### Layers + +Dependencies flow from the hosts toward small foundations. No foundational +layer imports a host, and configuration never owns discovered capabilities. + +```text +contracts settings model + \ | / + +--------- registry --------+ + | + markup + | + project + | + runner + | + direct / pytest / Sphinx hosts +``` + +| Layer | Owns | Must not know | +|---|---|---| +| `contracts` | Public protocols and immutable contribution records: `DocumentParser`, `ExecutionProfile`, `ExecutionRuntime`, `CheckerFactory`, `Contributor` and `Registrar` | Sphinx, pytest, xdist and host lifecycle objects. Only stdlib and public parser types cross this boundary | +| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings`. `None` sentinels at the resolve boundary make a future default change announceable | registries, pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | +| `model` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan`, `RunContext` and the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only | +| `registry` | A private mutable builder and the public immutable `RegistrySnapshot` consumed by every later layer | host lifecycle objects after the snapshot is frozen | +| `markup/` | Text → `(blocks, diagnostics)`. **The whole docutils vocabulary**: which node classes each kind may arrive as, the `BlockAttributes` stamp, line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration, per-kind `option_spec` | Groups as a runtime concept, `DocTest`, pytest, pairing | +| `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | +| `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing, run-time `:skipif:`, the profile's context manager, and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | +| `pytest_doctest_docutils` | Options, `Document(pytest.Module)`, `DocutilsItem`, group `globs` lifetime, the outcome contract, built-in-plugin composition, surfacing diagnostics | docutils node classes, MyST configuration, grouping rules | + +`pytest_doctest_docutils._compat` is the only module that imports +`_pytest.doctest`, behind a pinned support matrix. See +{doc}`0006-pytest-private-api-compatibility`. + +### Settings and the frozen registry + +Settings have **lifetimes**, not just precedence. "Resolve exactly once at session +start" cannot be true of document front matter, which does not exist until that +document is parsed. Three scopes: + +| Scope | Owns | Resolved | +|---|---|---| +| `SessionSettings` | defaults plus normalized host configuration | once, before collection or direct execution | +| `DocumentSettings` | the front-matter overlay a page is permitted to set | per document, after parsing | +| block / example policy | directive options, then inline `# doctest:` flags | per block, at projection and run | + +Not every field shares one ladder, so the precedence is stated per axis. For +option flags it follows Sphinx: **runner defaults → directive or output +`:options:` → inline flags.** + +Two fields move out of the core entirely. **Encoding** belongs to the source +loader, because `DocumentParser` already receives `str`. **Report style** belongs +to the host adapter. And wildcard resolution and name minting are *invariants*, +not user-configurable knobs — exposing them would let a project produce node ids +no other project can read. + +`ProjectionSettings.ungrouped` defaults to `"default"`. An unlabelled runnable +block therefore joins the page's `default` group unless the user explicitly asks +for block isolation. This clean-slate default follows Sphinx's author vocabulary +and makes state-sharing opt-out rather than a project-specific surprise. + +The **registry** is a separate input resolved after `SessionSettings`. "Frozen" +means the public `RegistrySnapshot` contains immutable mappings and records; the +mutable builder is private and discarded. Registering after the host freezes its +snapshot is an error. Keeping these values separate matters under xdist: settings +are normalized user input, while the snapshot is the capability set discovered in +that process. + +**Contribution and the snapshot are public; mutation is not.** The stated goal is +an extendable, pluggable core, so a small host-neutral contributor protocol ships +in v1. Front ends, block kinds, execution profiles and checkers all feed one +builder and every consumer receives the same `RegistrySnapshot`. Host-specific +registration timing is a separate decision; the core contract does not import a +pytest hook or a Sphinx application. + +The direct, pytest and Sphinx lifecycles, including the xdist consistency check, +are specified in {doc}`0007-host-plugin-registration-lifecycle` so this record +does not mistake a host bootstrap policy for a core dependency. + +### Item lifecycle + +The custom item is load-bearing, and half-reusing {class}`pytest.DoctestItem` +reintroduces the exact bug this design exists to avoid. The contract, stated so +an implementer cannot get it wrong by omission: + +0. **The carrier.** {class}`pytest.DoctestItem` reads `self.dtest` in + `setup()`, `reportinfo()` and `_check_all_skipped()`, so the subclass must + define it even though a group holds many tests. `self.dtest` is a synthetic + **zero-example** `DocTest` for the group, and its `globs` **is** the canonical + live mapping — the same object every `RunnableBlock`'s test is given. That + makes the inherited `setup()` inject fixtures into exactly the right place + with no override of the injection itself. + +1. **Collection** builds one `GroupPlan` per (document, group) and one item per + plan. An empty plan yields no item. +2. **`setup()`** starts an attempt. In order: clear the live mapping **in place**; + restore the plan's `seed`, `extraglobs` and `__name__`; then call + `super().setup()` so fixtures inject into that same object. Clearing in place + rather than rebinding is what keeps `item.globs is run.globs` true for every + block, and what stops attempt two of a `--reruns` run from reading attempt + one's mutations. Each `RunnableBlock` is materialized against this object — + its `DocTest.globs` assigned *after* construction, because `DocTest.__init__` + copies. +3. **`runtest()`** is overridden. It must not delegate to + `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` + defaulting to `True` — that would empty the shared mapping after the first + block. It calls `run_group()`, which materializes and runs each + `RunnableBlock` in phase order with `clear_globs=False`, evaluates `:skipif:` + and `:pyversion:` against the live mapping and interpreter, + finalizes each paired `want` from its gated `ExpectedOutput`, and wraps the + body in a `try`/`finally` so cleanup runs whether or not the body raised. When + cleanup *also* fails, the body's failure is the one raised; cleanup's is + recorded in the `GroupResult`. + + The `OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` + conversion and `continue_on_failure` are reimplemented here, because + `PytestDoctestRunner` is nested inside a factory and cannot be imported. +4. **Outcome** follows [](#the-outcome-contract): only `Phase.TEST` blocks + determine pass versus skip. A plan with no test block yields no item. Setup and + cleanup are infrastructure: an error there may fail or abort the item, but a + successful setup is not a passed test and a cleanup skip cannot erase the test + result. Skip the item only when every test block is skipped. + `runtest()` must also keep `_disable_output_capturing_for_darwin()`, which + the inherited implementation calls before running and which has nothing to do + with grouping. + +5. **Failure projection** flattens every `Failed.failures` tuple in block order + and raises pytest's `MultipleDoctestFailures`. With the default pytest checker, + `repr_failure` is inherited unchanged and therefore preserves pytest's exact + output. A contributed checker must supply both `check_output` and + `output_difference`; the item takes one small custom-rendering branch so the + checker that decided the failure also explains it. `repr_failure` does not + render `GroupResult`. Partial-skip detail goes to a report section and the + terminal summary; see [](#the-outcome-contract). + +6. **Reporting across processes.** The controller never sees the item — it + receives serialized `TestReport` dictionaries. So `pytest_runtest_makereport` + copies a **versioned, JSON-safe** block summary onto the report. The summary + contains only the schema version, group name, block name, phase, outcome, + counts and structured skip reason — never exceptions or live objects. pytest + serializes arbitrary report attributes and xdist reconstructs them + controller-side. The rich `GroupResult`, and every exception in it, stays + worker-local. + +### Vocabulary + +Goal (e) — speaking doctest's, pytest's *and* Sphinx's idioms — is mostly a +naming problem, because the three overload the same nouns with different +referents. Each term below is decided once and used only that way. + +| Term | doctest | pytest | Sphinx | Decision | +|---|---|---|---|---| +| `globs` | the dict examples exec in; [`DocTest.__init__` stores a **copy**](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | — | assigned to `test.globs` after construction, run with `clear_globs=False` | Keep `globs` for the mapping | +| namespace | — | [`doctest_namespace`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) means *injected names* | — | **Not** used for the sharing unit; pytest owns the word | +| group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | +| scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The real question is what an *unlabelled* block joins, so the setting is `ungrouped = "default" | "block"`, not a `share` axis | +| test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | +| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | +| directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | +| optionflags | an int bitmask; `register_optionflag` | `doctest_optionflags` ini | `:options:` plus `doctest_default_flags` | Keep verbatim. `register_optionflag` is the one genuinely cross-library extension point | +| setup / cleanup | `setUp`/`tearDown` on the suite builders | fixtures | `testsetup`/`testcleanup` directives | Author-facing names stay Sphinx's; `phase` is the internal ordering axis; a fixture is never "setup" | +| name | a dotted path; `__lt__` compares it as **text** | node id is `parent.nodeid + "::" + name` | the *group* name, shared by every block in it | `DocTest.name` is a unique machine-independent id. The absolute path lives only in `filename` | + +### Data model + +```python +class Phase(enum.IntEnum): + SETUP = 0 + TEST = 1 + CLEANUP = 2 + + +# --- contracts: public, host-neutral extension seams ---------------------- + + +Failure: t.TypeAlias = doctest.DocTestFailure | doctest.UnexpectedException + + +class RuntimeOutcome(t.NamedTuple): + results: doctest.TestResults + failures: tuple[Failure, ...] + + +class RuntimeSettings(t.NamedTuple): + optionflags: int + continue_on_failure: bool + checker: doctest.OutputChecker + + +class CheckerFactory(t.Protocol): + def __call__(self) -> doctest.OutputChecker: ... + + +class ExecutionRuntime(t.Protocol): + def run(self, test: doctest.DocTest) -> RuntimeOutcome: ... + + +class ExecutionProfile(t.Protocol): + def open( + self, settings: RuntimeSettings + ) -> contextlib.AbstractContextManager[ExecutionRuntime]: ... + + +# --- parsed: inert, produced by extraction, owns no semantics ------------- + + +class ParsedBlock(t.NamedTuple): + kind: str # registered BlockKind name + source: str # dedented body, verbatim author text + path: pathlib.Path # the file the text lives in, not the collected document + line: int | None # None when docutils could not recover one + document_order: int # position among blocks AND outputs; the pairing key + block_ordinal: int # position among runnable blocks; the identity key + groups: tuple[str, ...] # declared verbatim; () and ("*",) unresolved here + options: t.Mapping[int, bool] # plain int keys, exactly as doctest produces + skipif: str | None # UNEVALUATED + pyversion: str | None # UNEVALUATED PEP 440 specifier + hidden: bool + + +class ParsedOutput(t.NamedTuple): + """A `testoutput` body. Not a block: it never runs.""" + + text: str + path: pathlib.Path + line: int | None + document_order: int # shares one sequence with ParsedBlock for pairing + groups: tuple[str, ...] + options: t.Mapping[int, bool] + skipif: str | None # a gated output means its testcode expects nothing + pyversion: str | None + + +class BlockKind(t.NamedTuple): + phase: Phase + profile_name: str # resolved against the frozen registry, not held here + pairs_with: str | None + grouped: bool + + +# --- projected: one per block, with everything the runner needs ------------ + + +class ExpectedOutput(t.NamedTuple): + text: str + options: t.Mapping[int, bool] + skipif: str | None # when truthy at run time, `want` becomes "" + pyversion: str | None # when disallowed at run time, `want` becomes "" + + +class ExampleRecipe(t.NamedTuple): + """Everything needed to rebuild one stock `doctest.Example`.""" + + source: str + want: str + exc_msg: str | None + lineno: int # 0-based, relative to the block's docstring + indent: int + options: t.Mapping[int, bool] + + +class ProjectedBlock(t.NamedTuple): + """A RECIPE. Holds no `DocTest`, because a `DocTest` is mutable.""" + + phase: Phase + name: str # the minted test name + block_ordinal: int # stable among runnable blocks before filtering + examples: tuple[ExampleRecipe, ...] # a prompt block yields SEVERAL + docstring: str # what pytest's failure renderer slices + filename: str + lineno: int | None # the block's own line; examples are relative to it + options: t.Mapping[int, bool] # block-level directive :options: + profile_name: str # resolved against the frozen registry per attempt + skipif: str | None # UNEVALUATED; gated in run_group() + pyversion: str | None # UNEVALUATED; gated in run_group() + expected: ExpectedOutput | None # paired testoutput, itself gateable + + +class GroupPlan(t.NamedTuple): + group: str + blocks: tuple[ProjectedBlock, ...] # in phase order; STRUCTURALLY immutable + seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt + + +# --- run: one attempt materializes fresh stdlib objects -------------------- + + +class RunnableBlock(t.NamedTuple): + """Materialized per attempt. Its `DocTest` is never retained by a plan.""" + + recipe: ProjectedBlock + test: doctest.DocTest # fresh stock objects, built this attempt + runtime: ExecutionRuntime # from the profile factory, this attempt + + +class RunContext: + """One execution attempt. Owns the live mapping; a plan never does.""" + + plan: GroupPlan + globs: dict[str, t.Any] # cleared in place and reseeded per attempt + runtimes: dict[str, ExecutionRuntime] # one per profile used by the group + + +# --- results: a discriminated union, so invalid states cannot be built ----- + + +class Counts(t.NamedTuple): + attempted: int + skipped: int # a PASSING block can still carry skipped examples + + +class SkipReason(t.NamedTuple): + kind: t.Literal["skipif", "inline-flag", "pyversion", "profile"] + detail: str # the gate expression, the flag, the specifier + + +class Passed(t.NamedTuple): + block: ProjectedBlock + counts: Counts + + +class Failed(t.NamedTuple): + block: ProjectedBlock + counts: Counts + # PLURAL: continue_on_failure yields several from one block + failures: tuple[Failure, ...] + + +class Skipped(t.NamedTuple): + block: ProjectedBlock + counts: Counts + reason: SkipReason + + +class Errored(t.NamedTuple): + block: ProjectedBlock + error: BaseException # a gate that raised, or a runtime that would not start + + +BlockResult: t.TypeAlias = Passed | Failed | Skipped | Errored + + +class GroupResult(t.NamedTuple): + group: str + blocks: tuple[BlockResult, ...] + primary: BaseException | None # what runtest() re-raises + secondary: tuple[BaseException, ...] # e.g. cleanup failing after the body +``` + +`ParsedBlock` carries no `want`, because neither owner of a `want` is the parsed +block: for a prompt-form block it is *inside* `source` and +{class}`doctest.DocTestParser` extracts it at projection, and for a +`testcode`/`testoutput` pair it is a separate `ParsedOutput`. Conflating the two +was what made "projection owns pairing" untrue. + +`document_order` is one monotonic sequence shared by runnable blocks and +`ParsedOutput` records. Pairing therefore follows the source stream even when an +output sits between two runnable candidates. `block_ordinal` counts runnable +blocks only and survives gating, filtering and wildcard expansion, so adding or +removing expected output cannot rename every later test. Both `:skipif:` and +`:pyversion:` remain data until the run boundary; collection never evaluates +either gate. + +`BlockKind` names a profile rather than holding one, so a public type never +contains a private implementation. The profile name and the block kind's own +registration name resolve against the frozen registry. + +**A plan holds no `DocTest`.** {class}`doctest.DocTest` is mutable — the design +assigns `globs` to it after construction, and a run mutates that mapping — so a +plan retaining one would not be a recipe, it would be last attempt's state. Under +`--reruns` that is the false-green this design exists to prevent. `ProjectedBlock` +therefore carries the *ingredients*, and `RunContext` materializes fresh stock +`Example` and `DocTest` objects for every attempt. + +**The ingredients are per example, not per block.** One prompt block routinely +yields several {class}`doctest.Example` objects, each with its own `source`, +`want`, `exc_msg`, `lineno`, `indent` and `options` — three, for a block whose +last statement raises: + +```{doctest} +>>> import doctest +>>> src = ">>> x = 1\n>>> x + 1\n2\n>>> int('z')\nTraceback (most recent call last):\nValueError: bad\n" +>>> test = doctest.DocTestParser().get_doctest(src, {}, "blk", "p.md", 0) +>>> len(test.examples) +3 +>>> [(e.lineno, e.want.strip()) for e in test.examples] +[(0, ''), (1, '2'), (3, 'Traceback (most recent call last):...')] +``` + +A single `source` and one `lineno` cannot represent that, and `docstring` is +separately required by pytest's known-location failure renderer. Hence +`ExampleRecipe` and `ProjectedBlock.docstring`: the recipe reproduces exactly what +{meth}`doctest.DocTestParser.get_doctest` produced, rather than approximating it. + +The same applies to a gated `testoutput`: when its gate is truthy the output is +**absent**, not empty. Its text *and* its output-specific options both disappear, +which is what Sphinx does, and which a pre-built `want=""` with retained options +would get wrong. + +`GroupPlan` is **structurally** immutable, not deeply so. Its tuples cannot be +rebound, but `seed` is a `Mapping` whose *values* are arbitrary user objects. Each +attempt shallow-copies it into the live mapping, which is exactly what +`DocTest.__init__` does with `globs` — matching doctest's own namespace semantics +rather than inventing a deeper guarantee the ecosystem does not provide. + +**Results are a discriminated union**, not one record with nullable fields, so +"passed with an exception attached" is unrepresentable rather than merely +unlikely. `Errored` exists because a gate that raises, or a runtime that will not +start, is none of pass, fail or skip. + +Four result details are load-bearing: + +- **`Failed.failures` is plural.** Under `continue_on_failure` one block reports + several failures; a singular field silently keeps the first. +- **`Passed` carries counts.** A block can pass *and* have skipped examples — + `failed=0 attempted=2 skipped=1` — and a result type without counts loses the + skip entirely, which is the same information ADR 0001's outcome contract + promises to surface. +- **`Skipped` carries counts too.** A whole-block gate attempts zero examples, + while an all-`SKIP` doctest has parsed examples and reports them skipped. The + reason alone cannot distinguish those cases. +- **`SkipReason` is typed.** A skip originates from `:skipif:`, an inline + `# doctest: +SKIP`, `:pyversion:`, or a profile declining to run — and "the gate + expression" describes only the first. + +**Exception precedence is phase-aware, not a single ladder.** Grouping +{exc}`KeyboardInterrupt`, a debugger quit, {exc}`pytest.skip`, `xfail` and +`pytest.exit` into one "control-flow" tier is unsafe. A cleanup skip must not erase +a real test failure, while a session exit must never be converted into block data. + +| Class | Examples | Rule | +|---|---|---| +| process, debugger or session abort | {exc}`KeyboardInterrupt`, `SystemExit`, `bdb.BdbQuit`, `pytest.exit` | always propagates from every phase; cleanup still runs, but cannot replace it | +| host outcome from setup or test | `pytest.skip`, `pytest.xfail` | propagates as the host outcome after cleanup | +| doctest mismatch from test | `DocTestFailure`, `UnexpectedException` | retained in source order and projected to the host after cleanup | +| runtime error from setup or test | a gate that raises, or a profile that cannot start | recorded as `Errored`; becomes the primary failure when no abort or host outcome exists | +| cleanup exception or cleanup host outcome | any exception, including `pytest.skip` or `pytest.xfail` | recorded as `secondary` when a primary exists; otherwise becomes the item failure, never a skip or xfail | + +Profile runtimes are entered through {class}`contextlib.ExitStack`, so a partial +startup unwinds deterministically in reverse. Classifying which exceptions are +pytest outcomes is the pytest adapter's job; the core knows only the phase. + +`ParsedBlock.line` being nullable is load-bearing, not defensive. A bare `>>>` block +nested in a `.. note::`, a list item or a block quote reports `line=None, +source=None` from docutils, and an `.. include::`-ed block numbers against the +*included* file. Both propagate to `DocTest.lineno=None` and pytest's honest +"location unknown", rather than to a fabricated number. + +`ProjectedBlock` carries phase and gate because `run_group()` owns phase +sequencing, run-time `:skipif:` evaluation and a cleanup `finally` — and cannot do +any of the three from a bare tuple of `DocTest`s. A `DocTest` carries no phase and +no gate, so the recipe has to. Tagging each entry also makes the ordering +self-describing rather than a convention a comment asserts. + +**A paired `want` is not known until run time.** Sphinx accepts +`:skipif:` on a `testoutput`, and when that output is gated away its `testcode` +still runs — expecting *empty* output. So the `want` of a paired block is a +function of a gate evaluated at run time, and the plan must carry the paired +output as data (`ExpectedOutput`, itself gated) with the `DocTest` finalized in +`run_group()`. Freezing `want` at projection time silently runs the wrong +assertion. + +**A wildcard block is projected separately per group it joins.** Projection +clones the recipe and mints a group-qualified name for each destination. Each +`RunContext` then builds its own `DocTest` from its own recipe. Reusing one +`ProjectedBlock` would make its name ambiguous; sharing one materialized +`DocTest` would be worse, because `DocTest.globs` is mutable and the second +group's assignment would win. + +**The gate's evaluation namespace is a deliberate divergence.** Sphinx evaluates +each `:skipif:` in a fresh context seeded with `doctest_global_setup`; this design +evaluates it against the live group mapping, after fixture injection and after +earlier blocks have run. That is more useful — a gate can consult a fixture — and +it is not what `sphinx-build` does. Recorded rather than hidden. + +**`ExecutionProfile` is an immutable factory; `ExecutionRuntime` is per attempt.** +A group can mix prompt, `exec` and async blocks, so there is no single per-group +profile. The profile is chosen per block and names a factory; `RunContext` creates +one runtime *per distinct profile* the group uses, and blocks sharing a profile +share its runtime. An async runtime therefore owns one event loop for the whole +group, which is what lets awaited state cross block boundaries. + +It is not a `Literal["single", "exec"]`, because a second execution policy already +exists in this repository: [PR #59](https://github.com/git-pull/gp-libs/pull/59) +adds top-level `await`, which needs `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` and an +event-loop lifetime a mode string cannot express. The runtime's context manager is +what `run_group()` enters, so that lifetime is served without overriding `run()` +and stdlib's save-and-restore `finally` stays inherited. + +**The ordinary lane does not use the owned loop at all.** For prompt-form blocks — +the overwhelming majority — the runner is a plain reporter subclass over CPython's +*untouched* per-example loop. `_DocTestRunner__run` is invoked only for extended +profiles: `exec` bodies, top-level await, and whatever comes next. Ordinary +doctests are then compatible **by construction** rather than by differential +testing, and {doc}`0002-runner-conformance-across-cpython`'s harness shrinks to +guarding the extended lane. + +**A checker owns both comparison and explanation.** The default pytest +registration constructs pytest's checker, preserving `ALLOW_UNICODE`, +`ALLOW_BYTES`, `NUMBER` and its inherited failure representation exactly. A +contributed `CheckerFactory` constructs a fresh checker for each runtime. The +same instance performs `check_output()` and `output_difference()`; using pytest's +private `_get_checker()` only at rendering time would let one checker reject the +example and another explain why. + +**Which docutils node classes a kind may arrive as is a front-end concern, not a +`BlockKind` field.** `testsetup`, `testcleanup` and any `:hide:` block are +emitted as {class}`docutils.nodes.comment`, not `literal_block` +([`sphinx/ext/doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93)), +and a walker restricted to `literal_block` silently loses every one of them while +the page still renders. That requirement is real, but it belongs to `markup/` +alongside the rest of the node vocabulary — putting it on `BlockKind` would drag +docutils into a layer declared stdlib-only. + +### Typing + +The runtime objects are stdlib's, unconditionally. Precision lives in a parallel +layer that never changes what is constructed. + +- **Parsing and extraction are two seams, not one.** A single + `DocumentParser.parse(text, path)` cannot serve Sphinx, because a Sphinx extension + already *has* a doctree and a raw re-parse is not the same tree. So: + + ```python + class DocumentParser(t.Protocol): + """Text -> doctree. Plural implementations: _rst, _myst, third-party.""" + + suffixes: t.ClassVar[frozenset[str]] + + def parse( + self, text: str, path: pathlib.Path, *, settings: ParseSettings + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: ... + + + def extract_blocks( + doctree: nodes.document, *, settings: ParseSettings + ) -> ParseResult: ... + ``` + + `extract_blocks` is deliberately a plain function, not a `Protocol`: exactly + one extractor is the *point* of the split, and a second implementation would + reintroduce the standalone-versus-Sphinx divergence it exists to prevent. + Standalone reST and MyST use both halves; a Sphinx extension calls only the + extractor, on the doctree it already resolved. + + **`DocumentParser` is not a {class}`doctest.DocTestParser`.** The two + signatures are incompatible — stdlib's is `parse(self, string, name='')` + returning alternating `str` and `Example`, and `get_doctest` depends on exactly + that + ([`Lib/doctest.py:657`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L657), + [`:696`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L696)). + So there are three lanes, not one: + + | Lane | Contract | + |---|---| + | plain text and strings | the exact `DocTestParser` contract, unmodified | + | reST / MyST | `DocumentParser` → doctree → `extract_blocks` | + | Python objects | a `DocTestFinder`-shaped adapter | + + Anything promising `DocFileSuite(parser=...)` compatibility is a separate + stdlib-shaped façade over the first lane, not the markup lane wearing a + stdlib name. + + This is not a Sphinx builder. A builder owns discovery, an `env`, an `outdir` + and a reporting format; an extractor is a pure function from a doctree to + blocks, and {doc}`0001-typed-vanilla-doctest-core`'s rejection of a builder + stands. + +- **Python object discovery is not a front end.** Finding doctests in a module's + docstrings takes an *object*, not `(text, path)`, and stdlib already has the + right shape for it. It is a {class}`doctest.DocTestFinder`-shaped adapter, and + putting a `_python` module in `markup/` was a category error. + +- **`Protocol` for markup seams; nominal classes only for stdlib façades.** + `DocumentParser` is a `Protocol` so a third party can supply one structurally. + It must not subclass {class}`doctest.DocTestParser`: their `parse()` signatures + and return types are incompatible, so the apparent typeshed accommodation is + itself an invalid override. The optional `DocFileSuite` façade instead owns a + separate nominal `DocTestParser` adapter with the exact stdlib signature. + Runtime passability still comes from matching the called method: a finder whose + `find()` takes a string first cannot be handed to `DocTestSuite`, which passes a + module, and subclassing does not fix that. + + One genuine nominal edge does exist: `DocTestSuite` sorts its results, and + `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom + finder must return real `DocTest` objects. +- **`TypedDict` at the docutils boundary.** `BlockAttributes` types what a + directive stamps on a node, with one narrowing accessor that validates once. + This is where `Any` currently enters: `str(node.get("test") or ...)` and + `dict(node.get("options") or {})` are runtime coercions paid for a static hole, + and typeshed's docutils stub makes it worse — its `get(key, failobj: _T) -> _T` + claims `_T` even when the key is present holding something else. +- **`t.Literal` for closed vocabularies**, derived from one source of truth so a + public signature and a config field cannot diverge. +- **Plain `int` keys for optionflags.** {class}`enum.IntFlag` was considered and + rejected; see [](#alternatives-rejected). +- **`py.typed` ships.** The project already runs mypy strict over `src` and + `tests`, and the marker file does not exist, so every consumer sees the package + as untyped. This is a packaging defect independent of the rest of this ADR. + +### What "vanilla-compatible" promises + +The phrase is worth decomposing, because it covers several different promises of +several different strengths. + +| Surface | Promise | +|---|---| +| `Example` / `DocTest` runtime types | **Exact.** Stock instances, never subclassed for metadata | +| plain-text parsing | **Exact.** The stdlib lane uses `DocTestParser` unmodified | +| option flags and checkers | **Exact.** `register_optionflag` and the stdlib checker contract, with pytest's `ALLOW_UNICODE`/`ALLOW_BYTES`/`NUMBER` reachable | +| prompt-block execution | **Exact.** CPython's own per-example loop, unmodified | +| `DocTestFinder`-shaped Python-object discovery | **Shaped**, as a separate adapter | +| `DocFileSuite` / `DocTestSuite` | **Façade only**, over the stdlib lane. A group plan cannot be expressed through an API that returns one `DocTest` per parser call | +| `{testcode}`, async, groups, phases, Sphinx gates | **Deliberate extension.** No stdlib equivalent to be compatible with | +| pytest collection, fixtures, reporting | pytest's own contracts, composed with rather than replaced | +| Sphinx **node** vocabulary | **Exact.** The `BlockAttributes` stamp is byte-compatible | +| Sphinx **execution** | **Not promised.** See below | + +**The Sphinx promise is narrow, and this record narrows it deliberately.** What is +offered is an *extractor over a Sphinx-resolved doctree* — a pure function from a +doctree to blocks, callable from an extension. +{doc}`0007-host-plugin-registration-lifecycle` defines how Sphinx extensions +contribute capabilities, but not a Sphinx execution lifecycle or result channel. +Inventing those would be the builder that [](#alternatives-rejected) turns down. +The record promises doctree consumption and nothing more. + +## Constraints + +The design is pinned by facts about three upstreams. Each was verified at the tag +cited. The full derivation is in `notes/analyses/`. + +### CPython `doctest` (v3.14.2) + +| Constraint | Anchor | +|---|---| +| A failure's file line is `test.lineno + example.lineno + 1`; `Example.lineno` is 0-based within the containing string | [`doctest.py:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | +| `DocTest.__init__` **copies** the globs mapping, so a shared mapping must be assigned after construction and run with `clear_globs=False` | [`doctest.py:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | +| Never sort collected tests. `__lt__` compares `(name, filename, lineno, id(self))`; `name` leads, so a name carrying its position as text sorts `page.md[10]` before `page.md[1]` however correct `lineno` is. `filename` and `lineno` only break ties among equal names | [`doctest.py:596-603`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596-L603) | +| The per-example loop is name-mangled; the supported in-loop seams are the four `report_*` methods and the injected checker | [`doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | +| `run()` mutates global interpreter state for its duration and restores in `finally`; it is neither reentrant nor thread-safe | [`doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) | +| Each example compiles under `""` in `"single"` mode with `dont_inherit=True`; `"exec"` suppresses expression echo, emptying every `want` | [`doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | +| `TestResults` is a 2-field namedtuple carrying `skipped` as an extra instance attribute; a third tuple field breaks every `failures, tries = runner.run(...)` unpack | [`doctest.py:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) | +| Custom flag names must be registered at import; ints are `1 << len(OPTIONFLAGS_BY_NAME)` and an unregistered name makes a page fail to **parse** | [`doctest.py:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) | + +`report_skip` does not exist at v3.14.2 — the runner has only `report_start`, +`report_success`, `report_failure` and `report_unexpected_exception`. A runner +that owns the loop must probe for it rather than assume it. + +### pytest (9.1.1) + +| Constraint | Anchor | +|---|---| +| `repr_failure` reads each failure's own `test` — the fact this design is built on | [`doctest.py:317-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) | +| `DoctestItem.setup()` does `self.dtest.globs.update(globs)`, so the mapping must be mutable and survive collection → setup → run | [`doctest.py:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | +| `runtest()` calls `run(self.dtest, out=failures)` with `clear_globs` defaulting to `True` — which would empty a shared mapping after the first block | [`doctest.py:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | +| `PytestDoctestRunner` is defined *inside* `_init_runner_class()` and is not importable, so the `OutcomeException` re-raise, `BdbQuit` → `outcomes.exit` and `continue_on_failure` handling do **not** come for free | [`doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181) | +| A page must be a `pytest.Module` with `obj = None` as a **class** attribute, or the `Module` machinery tries to import the `.rst`/`.md` file | [`doctest.py:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | +| Conftest autouse fixtures reach page items through `FixtureManager.pytest_plugin_registered`, **not** through a collector calling `parsefactories` — that call is `DoctestModule`-only, for fixtures defined in the collected `.py` itself | [`fixtures.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/fixtures.py), [`doctest.py:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | +| `_is_doctest` claims any `.txt`/`.rst` **initial path before consulting `--doctest-glob`**, so `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob | [`doctest.py:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | +| An empty `DocTest` must not be yielded as an item | [`doctest.py:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | + +### pytest-xdist (v3.8.0) + +| Constraint | Anchor | +|---|---| +| Every worker must collect identical node ids in identical order. Violation is not an exception — the scheduler logs `**Different tests collected, aborting run**` and the session executes zero tests | [`load.py:259`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259), [`loadscope.py:359`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) | +| The only affinity primitive is `_split_scope(nodeid) -> str`; `loadfile` and `loadgroup` are two-line overrides of it, and `load`/`worksteal` have no scope concept at any layer | [`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284), [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) | +| `xdist_group` is honoured only when the *worker's own* `--dist` is `loadgroup`, and works by appending `@name` to `item._nodeid` | [`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) | +| `parse_tx_spec_config` builds a list, so a negative multiplier contributes zero specs — `xspeclist.extend([spec] * num)`, not a sum | [`workermanage.py:26-37`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) | + +Making the item the sharing unit removes the need to satisfy the second and third +at all. **The first still binds at any granularity** — identical collection is +required whether a page yields one item or fifty — and is approached instead +through determinism over source closure, normalized settings and a frozen +registry, not through a purity claim collection cannot make. The fourth is why a +worker-count fork is not worth carrying: `pytest_xdist_setupnodes(config, specs)` +hands over the already-expanded spec list and never raises. + +There is a fifth hazard the single-item shape also removes, worth naming because +it has no guard otherwise: a worker crash re-runs only the *uncompleted* items of +a work unit on a fresh process, so blocks 3..N of a shared group would run +against an empty mapping. Worker restarts are on by default. + +### Sphinx (v8.2.3, the version this project resolves) + +| Constraint | Anchor | +|---|---| +| `testsetup`, `testcleanup` and `:hide:` blocks are emitted as `nodes.comment` | [`doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | +| A `:skipif:`-gated node is dropped during collection, with no outcome, id or count | [`doctest.py:449-450`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | +| `:options:` is **not in `TestcodeDirective.option_spec`**, so writing it on a `testcode` is an unknown-option error that drops the block — a loud rejection, not a silent discard | [`doctest.py:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180), [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111) | +| `:pyversion:` **is** in `TestcodeDirective.option_spec` and is silently ignored there — the real silent loss on a testcode | [`doctest.py:177`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L177) | +| Cleanup does **not** run when setup fails: the group returns early | [`doctest.py:554-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L554-L556) | +| `is_allowed_version(spec, version)` takes the specifier **first** | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | +| `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310), [`:549`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L549) | + +Sphinx 9.0 changed the fallback only for a bare doctest node with no `groups` +attribute: it now uses `doctest_test_doctest_blocks` +([`v9.0.0:463`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L463)). +An unargumented directive still stamps `groups=["default"]` +([`v9.0.0:94-98`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L94-L98)). +Compatibility therefore distinguishes directive-produced nodes from bare +`doctest_block` nodes instead of claiming the author-facing default changed. + +## Tensions + +Each is a genuine conflict where satisfying one goal costs another. "Both" is not +an answer; the position taken and its price are recorded. + +**A vanilla `DocTest` cannot carry the front-end's metadata.** It has exactly +`(examples, globs, name, filename, lineno, docstring)`. *Position:* all extension +metadata stays on `ProjectedBlock`, `RunContext` and the result records. Stock +`DocTest` and `Example` objects remain exact compatibility objects, not metadata +carriers. *Price:* a consumer holding only the stdlib object sees only stdlib +semantics; it must retain the core recipe to inspect groups, profiles or gates. + +Putting metadata on an `Example` subclass is rejected because +{meth}`doctest.Example.__eq__` gates on exact type identity +([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)), +so a bare subclass is unequal to a stock `Example` with identical fields in both +directions while hashing the same. Restoring equality with an {func}`isinstance` +override then over-corrects: two *unrelated* subclasses compare equal to each +other, and to any third party's bare subclass. + +```{doctest} +>>> import doctest +>>> def tagged(name): +... ns = {"__eq__": lambda s, o: isinstance(o, doctest.Example) +... and s.source == o.source, "__hash__": doctest.Example.__hash__} +... return type(name, (doctest.Example,), ns) +>>> Exec, Await = tagged("Exec"), tagged("Await") +>>> Exec("1\n", "1\n") == Await("1\n", "1\n") +True +``` + +A block's execution policy is **uniform across its examples**, so it belongs on +`ProjectedBlock`, not on the examples. The runner establishes an *active +execution request* immediately before delegating to stdlib `run()` and clears it +in a `finally`; `_DocTestRunner__run` reads it. Stock `Example` objects stay +stock, and nothing in the compatibility kernel is subclassed for metadata at all. + +**Node-id granularity versus shared state.** *Position:* decouple them — N +`DocTest`s under one node id. *Price:* selecting a group runs all its blocks; +there is no id that names block three alone. That is honest: no surveyed +implementation makes a node id a promise of independent runnability, and the +proposed shared per-block shape would expose ids that raise `NameError` when +selected without their predecessors. + +**Sphinx's skip versus pytest's skip.** *Position:* pytest's meaning, doctest's +mechanism — set `SKIP`, never drop the node. *Price:* a page carrying a gated +block gains an item relative to `sphinx-build -b doctest`, and `--collect-only` +must not evaluate the gate, which is why `:skipif:` is carried through collection +unevaluated and run in `runtest()`. + +**Collection determinism versus `--collect-only` fidelity.** *Position:* +collection evaluates no *author-supplied* Python — `:skipif:` is carried through +unevaluated and run in `runtest()`. *Price:* `--collect-only` no longer shows +which blocks will skip. + +(the-collection-contract)= + +Collection is **not** a pure function of (bytes, argv, ini), and claiming so +would be wrong on five counts: `.. include::` reads transitive files, docutils +directive implementations execute during parsing, the directive registry is +process-global, MyST plugins change the tree, and the frozen registry is itself +an input — assembled from installed plugins and conftests, neither of which is +argv or ini. The defensible contract is +determinism over **complete source closure + normalized settings + frozen +registry** — all three defined above. Deferring the author's gate removes the +largest divergence risk; it does not make xdist divergence structurally +impossible. + +**Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards +an orphan `testoutput`, silently discards a `testoutput` following a `doctest` +block, silently overwrites a duplicate `testoutput`, and silently ignores +`:pyversion:` on a `testcode`. *Position:* keep the behaviour, add a diagnostic +with a stable code for each of the four. *Price:* a page warns under pytest and +is silent under `sphinx-build`; the results still match. + +**Guaranteed cleanup versus Sphinx's setup-failure short-circuit.** When setup +fails, Sphinx returns before the cleanup phase, skipping page `testcleanup` +blocks and `doctest_global_cleanup` alike. *Position:* run cleanup +unconditionally in a `try`/`finally`, because a page that spawns a server in +setup and fails mid-way should not leak it. *Price:* a page whose setup fails +leaves different residue under pytest than under `sphinx-build`, and that is a +deliberate divergence rather than an oversight. + +**Owning the loop versus tracking CPython.** *Position:* own it, because +constraint 4 makes it the only way to control compile mode without a +process-global patch or a code-object clone. *Price:* a version shim and a +conformance harness. See {doc}`0002-runner-conformance-across-cpython`. + +## What this avoids + +Only one item here exists on trunk today: `set_blocked("doctest")` and its +unblock path, which {doc}`0006-pytest-private-api-compatibility` replaces. The +rest are machinery [PR #87](https://github.com/git-pull/gp-libs/pull/87) has to +build to make a shared namespace work, and which this shape never needs: + +- the merge step, its blank-line padding and its `max()` clamp +- skip lifting, which pulls a wholly-skipped block back out of a running group +- the code-object clone with its stale `vars(doctest)` snapshot +- the worker-count fork of `parse_tx_spec_config` +- the node-id string sniffing that infers "these ids share state" +- the scheduler substitution and both xdist hooks + +**The result is not smaller.** It lands roughly flat against a finder that does +the same job. +Under this project's rule that every function carries a NumPy docstring with a +working doctest, splitting one long method into five functions costs prose it did +not previously pay. The value is fewer hazards, not fewer lines: a code-object +clone with a stale globals snapshot, a fork of an upstream parser, node-id string +sniffing, and a method that branches on string literals to decide what a block is +all disappear. Gate on shape instead — a function-length ceiling, a module-length +ceiling, and the `import-linter` contract on the leaf. + +(prior-art)= + +## Prior art + +| Project | Bet | Outcome | +|---|---|---| +| [Sybil 10.0.1](https://github.com/simplistix/sybil/tree/10.0.1) | A document is a flat sequence of non-overlapping character spans; every format is a regex lexer; zero runtime dependencies | Format independence at no dependency cost, and a non-overlap invariant that raises on double collection. But one mutable namespace per document with [one independently selectable item per span](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py), so `-k` on a later example raises `NameError`. [Node ids are positional](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (`line:4,column:1`), so adding a paragraph renames every downstream test. No group support at all — a regex cannot see directive options | +| [xdoctest v1.3.2](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | Abandon stdlib compatibility; own the parser via `ast`/`tokenize`; make directives structured objects | `ast` parsing and structured directives are real advances — [`REQUIRES`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) carries *why* a block skipped, which a bool cannot. But it is now building compatibility back, and its permissive got/want defaults silently change tests users wrote for stdlib. It unregisters pytest's doctest plugin outright | +| [pytest-examples v0.0.18](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | Emit the canonical form rather than parse it; rewrite expected output in place | Check-mode and update-mode collapse into one path. Its absolute Python string indices enable source rewriting, although one indent scalar does not invert dedent in general. It composes with pytest by contributing no collector at all — the cheapest correct integration in the survey | +| [typeshed](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | Annotate the 2001 API faithfully | Hands back `Any` at exactly the three extensible points — `globs`, `**options`, `optionflags: int`. Declares `DocTestRunner.test: DocTest` unconditionally although runtime assigns it only inside `run()`, so the stub type-checks a crash | + +The collective lesson: **namespace scope is not test identity, markup parsing is +not Python parsing, and runtime compatibility is not static precision.** Every +project conflated at least two, and the first conflation is the one that produces +silent wrong answers rather than inconvenience. + +(alternatives-rejected)= + +## Alternatives rejected + +**Prefix replay** — re-executing a group's predecessors so any block can be +selected standalone. It rebinds `getfixture` to the *replaying* item's request, +so a replayed predecessor resolves different fixture instances: a results +difference, not a performance one. It re-executes gated and deselected blocks +with no node id, in exactly the environment the gate says they must not run in. +It is superlinear precisely where shared groups exist. And it assumes +idempotence, while downstream setup blocks spawn servers and create +repositories. + +**One `DocTest` per group whose `docstring` is the whole file.** Attractive — +`lineno=0` and true file lines with no padding — but broken at the two shapes +docutils cannot locate. A nested block reports `line=None`, and an included file +numbers against itself, so one group-wide docstring maps both to confidently +wrong lines, and pytest's per-`DocTest` "location unknown" signal becomes +structurally inexpressible. Per-block `DocTest`s deliver the same benefits with +none of this. + +**{class}`enum.IntFlag` as the optionflag surface.** The runtime hash equality is +real and irrelevant to the claims made from it. With a third-party flag +registered — and pytest lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES` and +`NUMBER` in [`_get_flag_lookup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385) — +iteration and `repr` silently under-report members outside the enum. And +`Mapping` is invariant in its key type, so `Mapping[Flag, bool]` fails mypy +strict against `dict[int, bool]` in both directions, requiring casts at exactly +the boundaries the change was supposed to clean. + +**A registered `EXEC` optionflag to carry compile mode.** It would put compile +mode in the same user-writable namespace as `ELLIPSIS`. A `doctest_optionflags = +EXEC` in any ini, or a stray `# doctest: +EXEC`, compiles in exec mode, which +suppresses expression echo so every `want` compares against empty output and the +suite passes vacuously. Execution policy is selected by +`ProjectedBlock.profile_name`, outside the user-writable optionflag namespace. + +**Sybil's non-overlap-raises invariant.** Two `.. include::` directives naming +the same file produce blocks over identical source spans. That is a legitimate +page, and an invariant that raises would reject it. Double collection becomes a +diagnostic carrying both provenances. + +**Reimplementing `_pytest.doctest`'s helpers instead of importing them.** +Mis-costed by roughly threefold: `_get_checker` alone returns a checker +implementing `ALLOW_UNICODE`, `ALLOW_BYTES` and `NUMBER` with float-precision +handling. Since the plugin still reads `doctest_optionflags` from the built-in +plugin, a project setting `NUMBER` would either raise or — worse — have the bit +accepted and silently do nothing. + +**A Sphinx builder.** New scope in a rewrite that must not grow, and both drafts +that proposed one had it deliberately diverging from `sphinx-build -b doctest` on +`:skipif:` and the silent-loss cases. Shipping a builder that disagrees with the +tool it replaces is worse than shipping none. + +**An import-time guard that raises.** A `pytest11` plugin raising at import +aborts the whole session, taking down suites whose majority of tests never touch +a doctest. It also checks the wrong property: `hasattr(DocTestRunner, +"_DocTestRunner__run")` is true in exactly the scenario it claims to prevent, +while the thing that actually changed within the supported range — +`__record_outcome`'s arity — is invisible to it. This becomes a differential +conformance test in CI. + +## Consequences + +### Positive + +- Failure locations are correct by construction, including through `.. include::`; + the default-checker path needs no `repr_failure` or `reportinfo` override. +- A block docutils cannot locate degrades to an honest disclaimer instead of a + fabricated line, and does not affect its siblings. +- Every `--dist` mode works, because there is no shared state to split. +- No CPython code-object clone, and no process-global rebinding of anything. +- Collection runs no author-supplied Python, so `--collect-only` has no side + effects and the largest source of worker divergence is removed. +- Grouping is one pure function with no docutils, pytest or filesystem dependency, + and is testable without any of them. +- A new block kind is a registration, not an edit to a method branching on string + literals. +- Parse diagnostics become values with stable codes rather than stderr writes and + mid-parse aborts. + +### Tradeoffs + +- The per-example loop is this project's to maintain across supported + interpreters, including two version-shaped divergences. +- A page containing Sphinx `{testcode}` blocks produces `DocTest`s a stock runner + cannot run, because `compile("a = 1\nb = 2\n", "", "single")` raises. + Prompt-form blocks — the overwhelming majority — run perfectly on an unmodified + runner, and a test asserts it. +- The plugin now *requires* the built-in doctest plugin rather than blocking it, + so `-p no:doctest` is an error rather than a degraded mode. +- The line count does not fall. + +### Risks + +**Runner drift.** A CPython refactor that inlines the loop into `run()` would +silently route execution back to stdlib — invisible for prompt-form blocks, +immediately broken for `{testcode}`. Mitigated by the conformance harness in +{doc}`0002-runner-conformance-across-cpython`, gating on capability probes rather +than a `sys.version_info` ladder. + +**pytest private API.** Four private helpers and a subclassed item. Mitigated by +quarantining them in one module behind a pinned matrix; see +{doc}`0006-pytest-private-api-compatibility`. + +**Foreign directive registration.** `Sphinx.add_directive` overrides existing +registrations unconditionally, so `sphinx.ext.doctest` loaded in the same +interpreter can replace these directive classes. Mitigated by reading +`BlockAttributes` off the node — byte-compatible with what Sphinx stamps — rather +than depending on this project's own classes having run. + +**Over-suppressed diagnostics.** Suppressing one code too many turns a broken page +into a silent zero-test page, which is worse than a mid-parse abort. Mitigated by +the narrow default set in {doc}`0004-diagnostics-as-data`. + +## Relationship to other ADRs + +This ADR fixes the architecture. Six decisions it defers get their own records: +{doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven +equivalent), {doc}`0003-rejecting-per-block-items` (why shared per-block items +are rejected), {doc}`0004-diagnostics-as-data` (what is reported and what is +suppressed), {doc}`0005-line-recovery-for-nested-blocks` (the optional last +step), {doc}`0006-pytest-private-api-compatibility` (the quarantine and its +matrix), and +{doc}`0007-host-plugin-registration-lifecycle` (host registration and freeze +points). + +## Final position + +The core produces real {class}`doctest.DocTest` objects holding real +{class}`doctest.Example` objects. `Example.source` is the stdlib-normalized +executable body — prompts and indentation stripped, trailing newline added, the +stripped column recorded in `Example.indent` — not a synthesized wrapper and not +the author's verbatim text, which is what `ParsedBlock.source` holds. Everything +else — groups, phases, pairing, diagnostics, distribution — is a layer above that +fact, and no layer reaches around another. + +The unit that shares a `globs` mapping is the unit pytest schedules. That is the +one invariant every other property in this document follows from, and it is not +negotiable for a convenience elsewhere. diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md new file mode 100644 index 0000000..afd1889 --- /dev/null +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -0,0 +1,112 @@ +(adr-0002-runner-conformance-across-cpython)= + +# ADR 0002: Runner conformance across CPython versions + +Status: Draft +Date: 2026-08-02 + +## Context + +{doc}`0001-typed-vanilla-doctest-core` decides that the runner owns the +per-example loop by defining `_DocTestRunner__run` in a subclass, rather than +cloning CPython's code object or rebinding `doctest.compile` process-wide. + +Owning the loop means owning the private state it writes into, and that state has +changed shape inside this project's supported interpreter range. Three +divergences are known: + +**The outcome accumulator changed name and arity.** On 3.10 through 3.12 it is +`__record_outcome(self, test, f, t)` writing into `self._name2ft`; on 3.13 and +later it is `__record_outcome(self, test, failures, tries, skips)` writing into +`self._stats` +([`Lib/doctest.py:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485)). +A loop that calls the wrong one leaves `summarize()` reporting zeros for a +passing file — a silent, total failure of the reporting path. + +**`TestResults` gained a third value that is not a tuple field.** It carries +`skipped` as an extra instance attribute +([`Lib/doctest.py:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), +so `TestResults(f, a, skipped=s)` works on 3.13+ and raises on earlier versions. + +**`report_skip` does not exist at v3.14.2.** The runner has only `report_start`, +`report_success`, `report_failure` and `report_unexpected_exception` +([`Lib/doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)). +It appears in later prereleases, so a loop must probe rather than assume in +either direction. + +A fourth risk has no current instance but would be silent: a CPython refactor +that inlines the loop into `run()` would route execution back to stdlib. That is +invisible for prompt-form blocks and immediately broken for `{testcode}`. + +## Question + +How is an owned per-example loop proven equivalent to the interpreter's own, +continuously, without a `sys.version_info` ladder? + +## Direction + +A differential conformance harness, run in CI on every supported interpreter, +gating the build step that lands the runner. + +**Scoped to the extended lane.** {doc}`0001-typed-vanilla-doctest-core` runs +ordinary prompt blocks on CPython's untouched per-example loop, so those need no +differential proof — they *are* the reference. The owned `__run` is invoked only +for `exec` bodies, top-level await and future profiles, and that is what this +harness guards. It is a smaller obligation than an unconditionally owned loop, +and it is the reason owning the loop is affordable at all. + +A fixed case matrix — pass, fail, unexpected exception, `SyntaxError`, all +examples skipped, partially skipped, `FAIL_FAST`, `REPORT_ONLY_FIRST_FAILURE`, +`IGNORE_EXCEPTION_DETAIL`, and an exec-mode body — is run through both this +runner and a stock {class}`doctest.DocTestRunner`, asserting the captured +`report_*` text, `summarize()` output, the accumulator contents, and the result +as `(failed, attempted, skipped)`. + +**Assert the triple, not `TestResults` equality.** `TestResults` is a two-field +namedtuple carrying `skipped` off-tuple, so `==` compares only two of the three +values and a skip-count regression passes silently. `attempted` is also +incremented *before* the `SKIP` check, so a skip that wrongly executes moves +neither counter — it is invisible to both the tuple and to `summarize()` at zero +failures, and only the `report_*` text distinguishes it. + +The exec-mode case is the one the two runners are *meant* to disagree on, and it +still compares against stock. `compile()` raises on a multi-statement body, but +that call sits inside the loop's own `try` +([`Lib/doctest.py:1398-1408`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1398-L1408)), +so a stock {class}`doctest.DocTestRunner` catches the `SyntaxError` and records +it as an unexpected exception rather than propagating it: one +`report_unexpected_exception` call, `TestResults(failed=1, attempted=1)`, and +`_stats` at `(1, 1, 0)`. Only {class}`doctest.DebugRunner` — and pytest's runner +beneath it — converts that into a raise, as +{exc}`doctest.UnexpectedException`. So the case is asserted as a pair: stock +records the failure, this runner records a pass. A regression that silently +reverts to `"single"` mode shows up as the two converging. + +**What else belongs in the matrix, and what does not.** Add `report_*` hook +events — the only channel that distinguishes a skip which wrongly *executed*, +since `attempted` increments before the `SKIP` check and neither counter moves — +and repeated runs of one test, which exercise accumulator arithmetic across +calls. + +Cross-block `FAIL_FAST` and cleanup aggregation stay out. Both are properties of +`run_group()` rather than of the per-example loop, so a stock runner offers +nothing to compare them against; they belong to +{doc}`0001-typed-vanilla-doctest-core`'s item-lifecycle tests. A +{exc}`pytest.skip` raised inside an example and a debugger exit are likewise +pytest-layer concerns, testable only through a pytest session. + +Version handling is by capability probe, never by version comparison, so a +backport, a vendored interpreter or a fork behaves correctly rather than by +coincidence. {doc}`0001-typed-vanilla-doctest-core` rejects an import-time guard +that raises: a `pytest11` plugin that aborts at import takes down suites whose +majority of tests never touch a doctest. + +## Open + +- Whether the harness asserts on `report_*` text verbatim, or on a normalized + form — verbatim is stricter and will churn when CPython adjusts wording. +- Whether a probe failure degrades to stdlib's loop with a diagnostic, or fails + the affected items loudly. Degrading is silent for prompt-form blocks, which is + the argument against it. +- The floor: whether supporting 3.10's `_name2ft` shape is worth its shim once + that version reaches end of life. diff --git a/docs/adrs/0003-rejecting-per-block-items.md b/docs/adrs/0003-rejecting-per-block-items.md new file mode 100644 index 0000000..2a273dd --- /dev/null +++ b/docs/adrs/0003-rejecting-per-block-items.md @@ -0,0 +1,79 @@ +(adr-0003-rejecting-per-block-items)= + +# ADR 0003: Rejecting per-block items over a shared mapping + +Status: Draft +Date: 2026-08-02 + +## Context + +[PR #87](https://github.com/git-pull/gp-libs/pull/87) proposes two settings that +together choose how a page's blocks are collected. One of them, +`doctest_docutils_namespace_items = per-block`, keeps a node id for every block +of a shared page and hands those blocks one live `globs` mapping rather than +merging them into a single test. + +**Neither setting has shipped.** Both live on an open branch, in no release and +on no tag. There is nothing to deprecate, and this record does not propose a +deprecation — it records why the shape should not ship. + +## The shape, and why it is attractive + +`per-block` answers a real complaint about merging. Merging a group into one +`DocTest` collapses N node ids into one, merges fixture lifetime across the whole +group, and makes the failure gutter span the page. Keeping one id per block fixes +all three, and on a large documentation tree the difference is the bulk of the +suite's visible granularity. + +## Why it should not ship + +**A node id that cannot be selected is not a node id.** Selecting block three of +a stateful page raises `NameError`, because the blocks that bound the names it +reads did not run. The id promises an addressable unit and does not deliver one. + +**A live mapping cannot cross a process.** Only execnet-serializable builtins +reach an xdist worker, so the shape needs a scheduler that keeps a page whole — +and the only affinity primitive in xdist is +[`_split_scope(nodeid) -> str`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284). +Under a user-typed `--dist load` there is no scheduler to influence, so the +options collapse to refusing the run. + +**A live mapping cannot survive an item running twice.** A retry re-runs a block +against globals it already mutated, so an expectation true only on the second +attempt reports as a pass. Guarding that means refusing reruns. + +**A worker crash re-runs only the uncompleted tail** of a work unit, on a fresh +process — so blocks 3..N of a shared group run against an empty mapping, and +worker restarts are on by default. This one has no guard at all. + +Those four are why the branch also carries a worker-count fork, a page-inference +heuristic over node-id strings, a scheduler substitution, a scheduler refusal and +a rerun refusal. The guards are the cost of the shape, not incidental. + +## Decision + +Do not ship per-block items over a shared mapping, under this or any spelling. + +{doc}`0001-typed-vanilla-doctest-core` reaches the same granularity goal from the +other side: one item per group, holding one `DocTest` per block. That gives +per-block failure locations, gutters and "location unknown" without a shared +mapping ever becoming schedulable, so none of the four guards is needed. + +What it does not give is a per-block *outcome* or a per-block *node id*. That +limit is honest and is recorded in {doc}`0001-typed-vanilla-doctest-core`'s +outcome contract, rather than papered over with an id that raises when used. + +## Consequences + +Because nothing shipped, there is no migration path to write, no deprecation +warning to add and no downstream grep to run. + +## Open + +- Whether a human-facing block *label* — in failure text and the report header, + never as a node id — is worth adding later, so a reader can find the failing + block without the design promising `-k` isolation. Not in a first version. +- Whether `--doctest-docutils-namespace-scope` should be renamed to + `--doctest-docutils-share` before or after this architecture lands. + {doc}`0001-typed-vanilla-doctest-core` settles the vocabulary; the rename is + independently schedulable and, since neither spelling has shipped, cheap. diff --git a/docs/adrs/0004-diagnostics-as-data.md b/docs/adrs/0004-diagnostics-as-data.md new file mode 100644 index 0000000..45ed820 --- /dev/null +++ b/docs/adrs/0004-diagnostics-as-data.md @@ -0,0 +1,105 @@ +(adr-0004-diagnostics-as-data)= + +# ADR 0004: Diagnostics as data + +Status: Draft +Date: 2026-08-02 + +## Context + +Parsing a page currently writes docutils reporter output straight to stderr, +interleaved with pytest's own output and attributable to nothing. Two failure +modes follow from the default settings. + +A level-4 message raises `SystemMessage` mid-parse and aborts collection of the +file, so one malformed construct takes down a page whose other blocks are fine. + +More quietly, a `.. doctest::` block carrying an unknown option collects **zero** +tests and the session exits green. A page that checks nothing reports the same +way as a page that passes. + +{doc}`0001-typed-vanilla-doctest-core` gives the front-end layer a second return +value for this: `Diagnostic(level, code, message, path, line)`. + +**Two mechanism assumptions in the first draft were wrong, and the fix is not +cosmetic.** + +*There is no stable code to key on.* A docutils `system_message` carries a level +and text, and nothing semantically stable. So codes exist only for diagnostics +**this project emits**; docutils-originated messages arrive code-less and have to +be *classified* before they can be suppressed or promoted. The classifier is the +open question below, and it cannot be "key on the code", because for these +messages there is none. + +*An observer does not silence the stream.* Attaching one is additive: the message +still reaches the warning stream. Turning reporter output into values needs three +settings together — `halt_level` above 4 (both to avoid the mid-parse abort and +because a halting message bypasses observer notification entirely), +`report_level` at 5 or `warning_stream` disabled to stop the write, and then the +observer. + +## Question + +Which diagnostics are shown by default? + +The naive answer — show everything — was measured against this project's own +`docs/` and produces well over a hundred messages per run, almost all of them +`Unknown interpreted text role` and `Unknown directive type` for roles and +directives that Sphinx supplies and a bare-docutils parse structurally cannot +resolve. Those are false positives. Emitting them is noise-as-policy, and users +would learn to ignore the channel that also carries real errors. + +The opposite error is worse: suppressing one code too many turns a broken page +into a silent zero-test page, which is the exact condition this ADR exists to +surface. + +## Direction + +**Treat unknown roles and unknown directives differently.** They are not the same +risk, and the first draft's symmetric treatment was the mistake. + +An **unknown role** is inline markup. It cannot swallow a code block, so a +bare-docutils parse seeing `:mod:` in a Sphinx project is noise and is suppressed +by default. + +An **unknown body-owning directive** is a collection error. It swallows its body +unparsed, so a page whose doctests live inside one collects zero tests and exits +green — which is the failure diagnostics-as-data exists to prevent. Suppressing it +by default trades a loud, correct error for a silent wrong answer. A project with +legitimate foreign containers registers them as known vocabulary; that is an +explicit act, not a default. + +For a Sphinx project the question does not arise: the extractor consumes an +already-resolved doctree, in which every registered directive has run. + +"By code" remains the intent for everything else; the classifier that assigns a +code to a docutils message is unsettled, which is why this record stays `Draft`. + +Every diagnostic raised by this project's own layers defaults to visible, and +`level="error"` from those layers fails collection with the file and line named. +A page whose only block fails to parse, and a page with a malformed `:options:` +value, must both produce a collection error rather than collecting nothing and +passing. + +Expose promotion and suppression by code so a project can tune the set without a +global on/off switch. + +## Open + +- Whether diagnostics surface as {class}`pytest.PytestWarning` subclasses, giving + `-W error::` control for free, or as a dedicated report section. +- **What classifies a code-less docutils message.** The options are an owned, + version-pinned message-text table with a test that fails on upstream rewording + (and which must handle two dialects — reST's `Unknown directive type "x".` at + ERROR/3 versus MyST's `Unknown directive type: 'x'` at WARNING/2), or + pre-empting at the source by overriding the directive-dispatch path so the + unknown case never becomes a reporter message at all. This is the decision + ADR 0004 cannot ship without. +- **How a project registers a legitimate foreign container**, so that an unknown + body-owning directive it genuinely does not care about stops erroring. This is + the escape hatch the default requires, and it needs a spelling. +- Whether a near-miss to a registered name (`.. doctset::` for `.. doctest::`) + earns a distinct, more helpful message than the generic unknown-directive + error. Cheap, and the typo is the common case. +- Whether the CLI (`python -m doctest_docutils`) and the pytest plugin share one + formatter or two. diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md new file mode 100644 index 0000000..784f7c7 --- /dev/null +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -0,0 +1,119 @@ +(adr-0005-line-recovery-for-nested-blocks)= + +# ADR 0005: Line recovery for nested blocks + +Status: Draft +Date: 2026-08-02 + +## Context + +docutils does not report a usable line for every node, and what it reports +differs by front-end and by version. + +At **docutils 0.21.2** — which this project does not pin but does resolve, via +its Sphinx and myst-parser constraints — a bare `>>>` block +nested in a `.. note::`, a list item, a block quote or a `{tab}` directive +reports `line=None, source=None`. A top-level reStructuredText `doctest_block` +reports its **last** line. A MyST fence reports its **first** line. An +`.. include::`-ed block numbers against the *included* file. + +{doc}`0001-typed-vanilla-doctest-core` handles all of this honestly rather than +approximately: `ParsedBlock.line` is nullable, the per-front-end meaning is +normalized inside the front-end that knows it, `ParsedBlock.path` carries the +file the text actually lives in, and a block with no recoverable line propagates +`DocTest.lineno=None` into pytest's `EXAMPLE LOCATION UNKNOWN` branch. + +That is correct but not maximal. A nested block's failure says the location is +unknown when the parser knew it and threw it away. + +## The mechanism this record originally proposed does not work + +The idea was to substitute `docutils.parsers.rst.Parser.state_classes` per +parser instance, on the reasoning that `state_classes` is an instance attribute +and therefore scoped to one parse. + +It fails on two counts, both checked: + +**It does not reach nested blocks.** A nested parse builds its state machine from +`nested_sm_kwargs`, so a substitution applied only to the top-level +`state_classes` never reaches the constructs that need it — which are exactly the +constructs with the missing lines. + +**It is not scoped.** `RSTState.nested_sm_cache` is a shared *class* attribute, +so substituted classes leak into subsequent parses that did not ask for them. +The claim that substitution is "fully scoped to one parse with no process-global +mutation" is wrong. + +## Direction + +**Raise the docutils floor instead** — but that is **support policy, not core +architecture**. Nothing in {doc}`0001-typed-vanilla-doctest-core` depends on the +answer: a nullable line is the honest representation either way, and the floor +only decides how often it is `None`. This record can stay open indefinitely +without blocking the design. + +docutils 0.22 fixed the underlying defect upstream: a nested block reports a real +line, and top-level and nested blocks agree on reporting the **first** line +rather than the last. Every case this record was invented to work around is +resolved by the floor, with no probe, no substitution and no fallback path. + +Getting there is three moves, and the second is upstream of this repository: + +1. **Raise `requires-python` to `>= 3.11`.** Sphinx 9.0 is the first release that + permits docutils 0.22, and it declares `requires-python >= 3.11`. Dropping 3.10 + also touches the classifiers, the mypy and ruff target versions, and the CI + matrix. +2. **Ship a `gp-sphinx` release that widens its `sphinx < 9` cap.** This is the + binding constraint today, and it is not in this repository. With the cap in + place, a resolver asked for `docutils >= 0.22` reports the requirements + unsatisfiable. +3. **Then** declare `docutils >= 0.22, < 0.23`. An open-ended floor breaks the + moment a resolver reaches 0.23. + +Resolved versions per interpreter, with `docutils >= 0.22` requested: + +| Python | docutils | myst-parser | Sphinx | +|---|---|---|---| +| 3.10 | 0.23 | 0.13.6 | 3.5.3 — a degenerate backtrack, not viable | +| 3.11 | 0.22.4 | 5.1.0 | 9.0.4 | +| 3.12–3.14 | 0.22.4 | 5.1.0 | 9.1.0 | + +Today's lock resolves Sphinx 8.1.3 on Python 3.10 and 8.2.3 elsewhere, and +neither permits docutils 0.22. + +## Consequences + +The nullable `ParsedBlock.line` stays. It is not a workaround for this defect; +it is the honest representation of a front-end that may legitimately not know, +and `.. include::` attribution still needs `ParsedBlock.path` regardless of +version. + +The line-convention normalization in `markup/` gets *simpler* at the new floor — +both reStructuredText and MyST report the first line — but the normalization +layer stays, because the conventions still differ below the floor and a front-end +is the right place to know which it is dealing with. + +Every line-convention claim elsewhere in these records is version-qualified. +A statement about "docutils" that does not name a version is a bug in the +statement. + +## Open + +- **The blocking question: does this project drop Python 3.10?** Everything else + here is downstream of that, and it is a support-matrix decision that outlives + this record. Until it is settled, "raise the floor" is a direction, not a + decision — which is why this record's status stays `Draft`. +- Whether to raise the floor at all or support both, since docutils 0.21.2 is + what resolves today. Supporting both means keeping the normalization branch and + documenting two behaviours for the same page. +- Sequencing with the `gp-sphinx` cap. That release has to land first, and this + repository does not control it. +- Whether the Sphinx move belongs in this record or its own. Sphinx **9.0** + changed the fallback group for a bare, unstamped `doctest_block` from + `['default']` to `[doctest_test_doctest_blocks]`; directives always stamp + `groups`, so unargumented *directives* are unaffected. 9.x also differs in + fail-fast and result propagation. All of that is semantics beyond line numbers. +- Whether tests should pin exact `(path, line)` for a bare block nested in a + `.. note::`, a list item, a block quote and a `{tab}` directive. They should — + they are the regression net for the floor, and this repository already has + `{tab}` coverage from the GH-48 regression. diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md new file mode 100644 index 0000000..175bba4 --- /dev/null +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -0,0 +1,118 @@ +(adr-0006-pytest-private-api-compatibility)= + +# ADR 0006: pytest private API compatibility + +Status: Draft +Date: 2026-08-02 + +## Context + +The plugin currently calls `config.pluginmanager.set_blocked("doctest")` and then +imports that same blocked plugin's private helpers, while continuing to read four +ini and CLI options the blocked plugin declared. It works only because +`_pytest/fixtures.py` has no `pytest_plugin_unregistered` handler, so the +already-parsed `doctest_namespace` fixture outlives unregistration. That is an +undocumented behaviour bet on for every collected page. + +{doc}`0001-typed-vanilla-doctest-core` inverts the relationship: the built-in +doctest plugin is not blocked, it is *required*. Its checker, its failure repr, +its `--doctest-report` formatting and `doctest_namespace` are all worth keeping, +and reimplementing them was measured as costing roughly three times what it was +budgeted at — `_get_checker` alone returns a checker implementing +`ALLOW_UNICODE`, `ALLOW_BYTES` and `NUMBER` with float-precision handling +([`_pytest/doctest.py:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662)). + +Not blocking it exposes a live defect the block was masking. +[`_is_doctest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) +claims any `.txt` or `.rst` **initial path before consulting `--doctest-glob`**: + +```python +def _is_doctest(config: Config, path: Path, parent: Collector) -> bool: + if path.suffix in (".txt", ".rst") and parent.session.isinitpath(path): + return True + globs = config.getoption("doctestglob") or ["test*.txt"] + return any(fnmatch_ex(glob, path) for glob in globs) +``` + +So `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob +configuration, and `pytest_collect_file` is not `firstresult` — the directory +collector yields the results of *every* implementation for a path. Declining the +path is therefore not sufficient; the duplicate has to be removed. + +**Narrowing `--doctest-glob` cannot help**, because `_is_doctest` returns `True` +for an `.rst` initial path *before* it consults the glob at all. + +**And removing the duplicate late is too late.** `DoctestTextfile.collect()` +reads and parses the page inside `collect()`, so by the time +`pytest_collection_modifyitems` runs, the built-in has already produced an item — +or already reported a collection error, which deselection cannot retract. + +## Question + +What private surface is depended on, and how does a pytest release that changes +it fail? + +The current surface is `_get_checker`, `get_optionflags`, +`_get_continue_on_failure`, `_get_report_choice` and `MultipleDoctestFailures`. +Not everything in the quarantine is equally risky: {class}`pytest.DoctestItem` is +**public** — exported from `pytest` — so subclassing it is ordinary API use. The +collector class filtered out of the multicall result is private, and that filter +is the part that needs a version matrix. `_init_runner_class` is explicitly *not* usable: +`PytestDoctestRunner` is defined inside it +([`_pytest/doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181)) +and is unreachable by name, which is why the `OutcomeException` re-raise, +`BdbQuit` → `outcomes.exit` and `continue_on_failure` handling must be +reimplemented rather than inherited. + +## Direction + +Quarantine every private import in one module, `pytest_doctest_docutils._compat`, +with a pinned support matrix. + +**Filter the built-in's collector out of the `pytest_collect_file` result, in a +hook wrapper.** The directory collector consumes the multicall result directly, +and returning a modified result from a wrapper is documented and supported. +Filtering there removes the duplicate *before* the built-in collector parses +anything, so neither the duplicate item nor its collection error is ever produced +— which late deselection cannot achieve. + +**Use the old-style `hookwrapper=True` with `outcome.force_result()`.** New-style +`wrapper=True` is gated on **pluggy ≥ 1.2**, not on pytest 8 — it works fine +under pytest 7 with a new enough pluggy. But pytest 7 declares only +`pluggy>=0.12,<2.0`, so a resolver may legally install pluggy 1.0 or 1.1, where +`wrapper=True` raises `TypeError` *while importing the plugin* — a session-wide +abort, which this record and {doc}`0001-typed-vanilla-doctest-core` both forbid. +Old style needs no floor at all and was verified working on both pytest 7 and 9. + +Whichever spelling is used, **name the minimum supported pytest**. The CI matrix +floor is 7 and the package declares no pytest dependency, so today the support +statement exists only in the workflow file. + +**Fail on an unsupported pytest only when an affected document is collected**, +not at plugin registration. A `pytest11` plugin that raises at import takes down +sessions whose majority of tests never touch a doctest, and +{doc}`0001-typed-vanilla-doctest-core` and +{doc}`0002-runner-conformance-across-cpython` both reject session-wide startup +failures for the same reason. The error names the pytest version and the missing +symbol, and it names the document that triggered it. + +CI carries a job pinned to the minimum supported pytest and one tracking its +prerelease. + +Registry construction is not a pytest-private-API concern. The host-neutral +contract, pytest hookspec, Sphinx adapter and xdist manifest are specified in +{doc}`0007-host-plugin-registration-lifecycle`. + +## Open + +- Whether requiring the built-in plugin should be stated as a hard dependency. + `-p no:doctest` already fails today with a raw `ValueError` about an unknown + option, so this is not a regression — but the message should become actionable. +- Whether the probe should accept a *newer* pytest it has not been tested against, + or refuse it. Refusing is safer and more annoying; for a private-API quarantine + with a small matrix, safer probably wins. +- What the filtering wrapper should do when the built-in's collector is the *only* + one for a path — that is the ordinary `--doctest-glob` case this plugin has no + business touching, so the filter must be scoped to paths it actually claims. +- Whether any of these helpers can be promoted upstream, which would delete the + quarantine entirely. diff --git a/docs/adrs/0007-host-plugin-registration-lifecycle.md b/docs/adrs/0007-host-plugin-registration-lifecycle.md new file mode 100644 index 0000000..be3b515 --- /dev/null +++ b/docs/adrs/0007-host-plugin-registration-lifecycle.md @@ -0,0 +1,194 @@ +(adr-0007-host-plugin-registration-lifecycle)= + +# ADR 0007: Host plugin registration lifecycle + +Status: Draft +Date: 2026-08-02 + +## Context + +{doc}`0001-typed-vanilla-doctest-core` makes block kinds, document parsers, +execution profiles and output checkers extensible. The core needs one typed +contribution contract, but its hosts discover contributors at different times: +direct callers already have an explicit iterable, pytest loads installed plugins +and conftests in stages, and Sphinx loads extensions before it reads doctrees. + +Treating settings and registrations as one object obscures that difference. +Settings are normalized user input. Registrations are discovered capabilities, +and xdist may discover them in more than one process. The mutable construction +mechanism must not leak into parsing or execution. + +## Decision + +The public boundary consists of `Contributor`, `Registrar`, immutable registration +records and `RegistrySnapshot`. A private builder is the only mutable object. It +accepts contributions, validates them and produces a snapshot; every parser, +projector and runner receives that snapshot explicitly. + +```python +T = t.TypeVar("T") + + +class Provider(t.NamedTuple): + name: str + version: str | None + + +class Registration(t.NamedTuple, t.Generic[T]): + name: str + value: T + provider: Provider + + +class Contributor(t.Protocol): + provider: Provider + + def contribute(self, registrar: Registrar) -> None: ... + + +class Registrar(t.Protocol): + def add_block_kind( + self, name: str, kind: BlockKind, *, replace: bool = False + ) -> None: ... + + def add_document_parser( + self, name: str, parser: DocumentParser, *, replace: bool = False + ) -> None: ... + + def add_execution_profile( + self, name: str, profile: ExecutionProfile, *, replace: bool = False + ) -> None: ... + + def add_output_checker( + self, name: str, factory: CheckerFactory, *, replace: bool = False + ) -> None: ... + + +class RegistrySnapshot(t.NamedTuple): + block_kinds: t.Mapping[str, Registration[BlockKind]] + document_parsers: t.Mapping[str, Registration[DocumentParser]] + execution_profiles: t.Mapping[str, Registration[ExecutionProfile]] + output_checkers: t.Mapping[str, Registration[CheckerFactory]] + + +def build_registry( + contributors: t.Iterable[Contributor] = (), +) -> RegistrySnapshot: ... +``` + +The snapshot fields are read-only `MappingProxyType` views over private copies, +not mutable dictionaries typed as `Mapping`. The builder is discarded after +`freeze()`. While applying each `Contributor`, the builder gives it a registrar +bound to that contributor's `Provider`; registrations cannot claim a different +origin. A contributor retaining that registrar cannot retain a mutation path: +every method raises `RegistryClosedError` after the snapshot is made. + +### Names, collisions and order + +Registration names are case-sensitive ASCII identifiers matching +`[a-z][a-z0-9_.-]*`. The same name may exist in different categories. Within one +category a duplicate is an error naming the category, incumbent provider and +challenger provider unless the challenger passes `replace=True`. Replacement +retains the incumbent's insertion position, so an explicit override cannot +silently reorder parser or profile selection. + +Built-ins register first. Contributor order is then the order supplied by the +host, and calls within a contributor retain program order. The snapshot preserves +that order. Any selection rule that needs precedence uses this declared sequence; +it never sorts by an implementation object's representation or module path. + +For document parsers, overlapping suffix claims are also collisions. They are +accepted only when the challenger uses the incumbent parser's name and passes +`replace=True`; two differently named parsers cannot both win `.md` by incidental +plugin load order. + +## Host adapters + +### Direct API + +Direct callers pass contributors to `build_registry()`. The function registers +built-ins, applies the iterable once, freezes and returns `RegistrySnapshot`. +There is no entry-point scan or process-global default in the core API. + +### pytest + +The pytest adapter publishes its hookspec in `pytest_addhooks`: + +```python +class DoctestCoreHooks: + @pytest.hookspec + def pytest_doctest_core_contributors( + self, + ) -> Contributor | t.Iterable[Contributor] | None: + """Return doctest-core contributors before collection.""" +``` + +Its `pytest_configure(trylast=True)` implementation invokes the hook, flattens +its non-`None` results in pluggy's hook-call order, builds the registry and stores +the snapshot on pytest's stash. Installed plugins and initial conftests are +already registered at that point. The snapshot is therefore ready before +`pytest_sessionstart`, when xdist starts controller nodes, and before collection. + +Nested conftests load during collection and are outside this lifecycle. A +`pytest_plugin_registered` guard detects a late plugin implementing the hookspec +and raises `pytest.UsageError` naming that plugin and the closed registration +phase. Fixtures and unrelated hooks in nested conftests remain valid. + +### Sphinx + +The Sphinx adapter exposes +`add_doctest_core_contributor(app, contributor)`. Extensions call it from their +`setup(app)` function. At `config-inited`, after extension setup and before any +document is read, the adapter emits a `doctest-core-contributors` event, appends +the `Contributor` objects returned by its listeners to the queued contributors, +builds the registry and freezes the snapshot on the application. + +The adapter function is the order-independent path. An extension that connects +directly to the custom event must list the doctest-core extension before itself, +because Sphinx cannot connect a listener to an event that has not been declared. +Calling the adapter after `config-inited` raises `RegistryClosedError` with the +extension name. + +This lifecycle makes the extractor usable on Sphinx-resolved doctrees. It does +not add a builder or claim parity with `sphinx-build -b doctest` execution. + +## xdist consistency + +The controller sends a JSON-safe manifest through `workerinput` from +`pytest_configure_node`. Each worker builds its own snapshot during +`pytest_configure` and compares before collection. The manifest has a schema +version and contains: + +- a JSON-safe projection of normalized `SessionSettings` +- every registry category, name, provider and provider version in declared order +- `doctest.OPTIONFLAGS_BY_NAME`, sorted by flag name + +A mismatch aborts the session with the controller and worker manifests. This is +an extension-set consistency check, not proof that two workers are semantically +identical. Equal provider names and versions do not prove equal source code, and +the manifest does not hash included documents, directive implementations or MyST +plugins. + +Version 1 therefore supports homogeneous worker environments. Equal source +closure and equal installed provider code are preconditions, while xdist's own +identical-collection check remains authoritative for node ids. Stronger support +for deliberately heterogeneous SSH or socket workers would require content or +environment attestation and is deferred. + +## Consequences + +- Core extension authors implement one `Contributor` regardless of host. +- Settings remain serializable inputs; discovered objects remain in the registry. +- Parse and execution code cannot mutate capabilities after collection starts. +- pytest and Sphinx own timing and diagnostics in their native idioms without + leaking their lifecycle types into the core. +- Replacement is possible but visible, attributed and deterministic. +- Supporting heterogeneous xdist workers is explicitly outside the first + contract rather than implied by a weak manifest. + +## Open + +- Whether a later release should add opt-in entry-point discovery to the direct + adapter. The core function remains explicit either way. +- Whether provider code hashes are useful enough to justify the packaging and + editable-install edge cases they introduce. diff --git a/docs/adrs/index.md b/docs/adrs/index.md new file mode 100644 index 0000000..be6a504 --- /dev/null +++ b/docs/adrs/index.md @@ -0,0 +1,39 @@ +(adrs)= + +# Architecture Decision Records + +Significant design decisions for `doctest_docutils` and +`pytest_doctest_docutils`, and their rationale. + +These records govern the shape of the doctest engine: what it produces, what it +may reach into, and what vocabulary it speaks. A record states the context that +forced a decision, the decision itself, what it costs, and what it rules out — +so a later reader can tell a deliberate constraint from an accident. + +Supporting structural research lives in `notes/analyses/`. It decides nothing and +is cited by these records as evidence. + +## Conventions + +**Numbering** is sequential and permanent. A record is never renumbered, and a +superseded one is marked rather than deleted. + +**Status** is one of `Draft`, `Proposed`, `Accepted`, `Superseded by NNNN`. + +**Source links are pinned.** Every citation of an external project names a git +tag, or a commit reachable from that project's trunk where it publishes no tags. +Line anchors are only used on a pinned ref, because they are meaningless without +one, and a `blob/master` link rots silently — the file moves, lines shift, and +the anchor lands on unrelated code while still resolving. + +```{toctree} +:maxdepth: 1 + +0001-typed-vanilla-doctest-core +0002-runner-conformance-across-cpython +0003-rejecting-per-block-items +0004-diagnostics-as-data +0005-line-recovery-for-nested-blocks +0006-pytest-private-api-compatibility +0007-host-plugin-registration-lifecycle +``` diff --git a/docs/index.md b/docs/index.md index aba9e4c..006ef63 100644 --- a/docs/index.md +++ b/docs/index.md @@ -74,6 +74,7 @@ modules/doctest_docutils/index modules/pytest_doctest_docutils/index modules/linkify_issues/index project/index +adrs/index history GitHub ``` diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md new file mode 100644 index 0000000..233184a --- /dev/null +++ b/notes/analyses/00-taxonomy.md @@ -0,0 +1,89 @@ +# Taxonomy: the axes a doctest engine is classified on + +Nine axes. Every system in these notes takes a position on each, and most of the +disagreements between them reduce to a different position on one axis rather than +a different philosophy. + +## The axes + +| # | Axis | Positions | +|---|---|---| +| 1 | **Sharing unit vs. selection unit** | same object · different objects, acknowledged · different objects, unacknowledged | +| 2 | **Test identity** | author-declared name · symbol-derived · ordinal among extracted blocks · source-coordinate-derived (line/column or byte range) | +| 3 | **Runtime object model** | stdlib `DocTest`/`Example` · own model with a bridge · own model, no bridge | +| 4 | **Document model** | real parse tree · flat character spans · regex over text · none | +| 5 | **Option representation** | `int` bitmask · structured state · enum | +| 6 | **Extension mechanism** | callable aliases · nominal subclassing · named registry · `Protocol` · none | +| 7 | **Relationship to pytest's doctest plugin** | compose · block/unregister · replace by instruction · no collector | +| 8 | **Got/want strictness** | stdlib defaults · permissive defaults · no want at all | +| 9 | **Direction of data** | read-only · read plus write-back | + +## Where each system sits + +| System | 1 sharing/selection | 2 identity | 3 object model | 4 document model | +|---|---|---|---|---| +| CPython `doctest` | same (one `DocTest` per docstring) | dotted symbol path | *is* the model | none — line regex over a string | +| `_pytest.doctest` | same (one item per `DocTest`) | `path::module.qualname` | stdlib, unchanged | none — delegates | +| `sphinx.ext.doctest` | same (one group, no selectable unit) | group name, shared by every test block | stdlib; per test block, combined for setup and cleanup | real doctree | +| Sybil | **different, unacknowledged** | positional `line:N,column:N` | stdlib `Example`, one-line `DocTest` fork | flat character spans | +| xdoctest | same (one `DocTest` per docstring) | `Callname:N` | own, with a late bridge back | none for `.rst`/`.txt` | +| pytest-examples | none — no implicit sharing | positional `path:start-end` | none | regex over fences | +| `doctest_docutils` released | same object (one block, one item, isolated copied globals) | `page.md[k]` ordinal | stdlib | real doctree | +| PR #87 (proposed) | configurable; `per-block` is different-and-guarded | group name, or `page.md[k]` | stdlib | real doctree | +| ADR 0001 | **different, decoupled by construction** | group name, or `page.md[k]` | stdlib | real doctree | + +| System | 5 options | 6 extension | 7 vs. pytest doctest | 8 strictness | 9 direction | +|---|---|---|---|---|---| +| CPython `doctest` | `int` bitmask + registry | nominal subclassing | n/a | strict | read-only | +| `_pytest.doctest` | `int` + name lookup | subclass its classes | *is* it | strict | read-only | +| `sphinx.ext.doctest` | `int` via `:options:` | directive subclassing | unaware | strict | read-only | +| Sybil | `int` | callable aliases | replace by instruction (`-p no:doctest`) | strict | read-only | +| xdoctest | structured `TypedDict` + bridge | two registries, else fork | unregisters it | **permissive** | read-only | +| pytest-examples | n/a | none | no collector — composes trivially | no want at all | **write-back** | +| `doctest_docutils` released | `int` | directive subclassing | blocks it, imports its privates | strict | read-only | +| ADR 0001 | `int` + registry | `Protocol` + nominal + `BlockKind` registry | **compose; require it** | strict | read-only | + +## What the matrix shows + +**Axis 1 is the only one where a wrong answer is silent.** Every other axis +produces inconvenience — a renamed test, a conversion layer, an extra knob. Axis 1 +produces a `NameError` in a test the user believed they could select, or a false +green under `--reruns`. Sybil sits in the unacknowledged column and its +documentation never mentions it. + +**Axes 1 and 2 are independent, and everyone treated them as one.** The full +product space is four cells: + +| | one node id | N node ids | +|---|---|---| +| **one `DocTest`** | PR #87's `merged` | — (incoherent) | +| **N `DocTest`s** | `sphinx.ext.doctest` for its *test* phase only, and with *no* ids; ADR 0001 adds the pytest identity | Sybil, PR #87's `per-block`, released `doctest_docutils` (no sharing) | + +The bottom-right cell is where the silent failure lives — but only when the +blocks share state. Released `doctest_docutils` sits there safely because its +blocks share nothing: each gets its own copied `globs`. + +The bottom-left cell is where the design goes. Sphinx already *executes* that +shape — but only for ordinary test blocks: all of a group's setup blocks are +combined into one simulated `DocTest`, and likewise cleanup. It also produces no +addressable unit for any of them, since every test block shares one +`DocTest.name`. So the design is per-block in three phases where Sphinx is +per-block in one, and the pytest identity is new either way. + +**Axis 3 has an empirical answer.** xdoctest is the controlled experiment for +abandoning the stdlib object model, and it is now building the bridge back. The +cost of divergence is paid years later, in knobs that exist only to restore the +default that was abandoned. + +**Axis 4 is decided by host fidelity, not by lexing power.** A regex *can* parse +directive arguments and options — Sybil's directive lexers do it. What a regex +cannot give you is the same tree Sphinx renders from, and that is what makes a +page behave identically under `sphinx-build` and under pytest. Sybil having no +group concept, and telling users to clear the namespace instead, is a design +choice rather than a limit of its lexer. + +**Axis 7 correlates with hostility.** Two of the surveyed projects disable +pytest's doctest plugin — one in `pytest_configure`, one by telling users to pass +`-p no:doctest`. A `pytest11` plugin loads into sessions belonging to people who +never asked for it, and the plugin it disables is the one whose checker, failure +repr and `doctest_namespace` fixture it wants to keep. diff --git a/notes/analyses/10-cpython-doctest.md b/notes/analyses/10-cpython-doctest.md new file mode 100644 index 0000000..aa75509 --- /dev/null +++ b/notes/analyses/10-cpython-doctest.md @@ -0,0 +1,154 @@ +# CPython `doctest` + +Pinned at [`v3.14.2`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py). + +## Classification + +A four-stage pipeline of unannotated classes, whose documented extension surface +is six classes plus three injection slots and four reporting hooks. The per-example +loop — the thing every extender eventually wants — is not among them. + +## Core data structures + +```text +Example source, want, exc_msg, lineno, indent, options + | lineno is 0-based, relative to the start of the containing string + v +DocTest examples, globs, name, filename, lineno, docstring + | globs is COPIED by __init__; __lt__ compares name as TEXT + v +TestResults namedtuple(failed, attempted), with `skipped` as an EXTRA attribute +``` + +Three properties of these are load-bearing for anything built on top: + +**`DocTest.__init__` copies the globs mapping** +([`:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565)). +Passing a shared dict through the constructor has no effect whatsoever. A shared +mapping must be assigned to `test.globs` after construction, and the runner must +be given `clear_globs=False` or it empties the mapping in its `finally`. + +**`__lt__` compares `(name, filename, lineno, id(self))`** +([`:596-603`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596-L603)). +`name` leads, so any name carrying a position as text sorts `[10]` before `[1]` +however correct `lineno` is; the later terms only break ties among equal names. +This fails silently: every test passes, in the wrong order. Released gp-libs hits +it — `find()` calls `tests.sort()` over blocks named `page.md[k]`, so an +eleven-block page runs its eleventh block second. + +**`TestResults` carries `skipped` outside the tuple** +([`:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), +with a `repr` that falls back to the plain namedtuple form when it is zero. +Promoting it to a third field would break every `failures, tries = runner.run(...)` +unpack in the ecosystem, including `doctest._test()` itself. + +## Data flow + +```text +source string + | DocTestParser.parse -> list[str | Example], alternating, + | covering the NORMALIZED input + | DocTestParser.get_doctest -> DocTest + v +DocTestFinder.find(obj) -> list[DocTest] + | recurses into __test__, tracks a seen-map by id() + v +DocTestRunner.run(test, compileflags, out, clear_globs) + | saves sys.stdout, pdb.set_trace, linecache.getlines, + | sys.displayhook, _colorize.can_colorize; pops PYTHON_COLORS + | and FORCE_COLOR from os.environ; restores all in finally + | + +--> __run(test, compileflags, out) <- name-mangled + for each example: + merge test-level and example-level optionflags + SKIP? -> continue BEFORE report_start; still counts as attempted + compile(source, "" % (test.name, n), + "single", flags, dont_inherit=True) + exec in test.globs + OutputChecker.check_output(want, got, flags) + report_success / report_failure / report_unexpected_exception + __record_outcome(...) +``` + +The `parse()` contract is not incidental. It must return alternating `str` and +`Example` reconstructing the input, because `script_from_examples()` walks the +`str` pieces to build the prose comments in a debugging script. `Example.indent` +is computed against the original string — after `expandtabs`, before dedent — so a +post-dedent indent shifts every reported column. + +## Extension seams + +| Seam | Kind | Documented | +|---|---|---| +| `parser=` object with the `DocTestParser` methods | structural at runtime, nominal in typeshed | yes | +| `test_finder=` object with the `DocTestFinder` methods | structural at runtime, nominal in typeshed | yes | +| `checker=` object with `check_output` / `output_difference` | structural at runtime, nominal in typeshed | yes | +| `report_start`, `report_success`, `report_failure`, `report_unexpected_exception` ([`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)) | subclass hook | yes | +| `register_optionflag` / `OPTIONFLAGS_BY_NAME` ([`:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153)) | process-global registry | yes | +| `setUp` / `tearDown` on `DocTestSuite` / `DocFileSuite` | callable param | yes | +| `__run` ([`:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344)) | **name-mangled** | no | +| `__record_outcome` ([`:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485)) | **name-mangled** | no | +| `__patched_linecache_getlines` ([`:1501`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501)) | **name-mangled** | no | +| `_load_testfile` ([`:245`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245)), `_EXAMPLE_RE` ([`:618`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618)) | private | no | + +The name-mangled three are the interesting entry. Mangling rewrites the *call +site* at compile time, so `self._DocTestRunner__run(...)` inside `run()` is an +ordinary attribute lookup that resolves through the subclass's MRO. Defining +`_DocTestRunner__run` in a subclass therefore takes over the loop — verified on +3.14.2 with `run()` untouched. It is not an override point by *design*, but it is +one by *mechanism*, and that distinction is what lets a downstream own the loop +without cloning a code object or patching a module global. + +`register_optionflag` is the only genuinely cross-library extension point in the +module. Its ints are `1 << len(OPTIONFLAGS_BY_NAME)`, so they are +registration-order dependent, typeshed hard-codes the builtin values, and an +unregistered flag name makes a page fail to **parse** rather than to run. + +## Configuration + +There is none, in the modern sense. Behaviour is set by optionflags, which arrive +from three places with a fixed precedence: the runner's constructor, the +`DocTest`'s per-example `options` dict, and the inline `# doctest: +FLAG` comment +parsed out of the example source. `set_unittest_reportflags` mutates a module +global. `doctest.master` accumulates results across invocations — the +documentation calls it advanced tomfoolery. + +## What it cannot do + +- **Run a multi-statement body.** `"single"` mode rejects it, and `"exec"` mode + suppresses expression echo, which empties every `want`. This one fact is the + origin of every downstream monkeypatch of `doctest.compile`. +- **Report a per-example result as a value.** Outcomes exist only as counters and + as text pushed through `out`. pytest works around this by repurposing `out` from + a write-callable into a *list*; Sphinx works around it by not producing machine- + readable results at all. +- **Share a globs mapping across `DocTest`s** without the caller assigning + `test.globs` post-construction and passing `clear_globs=False`. +- **Be reentrant or thread-safe.** `run()` mutates interpreter globals for its + duration ([`:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +- **Report a skip as an outcome.** `SKIP` short-circuits before `report_start`. + There is no `report_skip` at v3.14.2; it appears in later prereleases, so a + downstream loop must probe rather than assume. + +## Anchors + +- [`TestResults`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) · + [`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) · + [`_load_testfile`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245) +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) · + [`DocTest.__lt__`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) +- [`DocTestParser`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L609) · + [`_EXAMPLE_RE`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618) · + [`DocTestFinder`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) +- [`report_*` hooks](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) · + [`__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) · + [`compile(..., "single", ...)`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) +- [`__record_outcome`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485) · + [`__patched_linecache_getlines`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501) · + [`run()` global-state save/restore](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) +- [`OutputChecker`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) · + [`DebugRunner`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1874) · + [`testfile`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2091) · + [`DocTestSuite`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) · + [`DocFileSuite`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) +- Typed contract: [`typeshed stdlib/doctest.pyi`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) diff --git a/notes/analyses/11-pytest-doctest.md b/notes/analyses/11-pytest-doctest.md new file mode 100644 index 0000000..b365dac --- /dev/null +++ b/notes/analyses/11-pytest-doctest.md @@ -0,0 +1,134 @@ +# `_pytest.doctest` + +Pinned at [`9.1.1`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py). + +## Classification + +A deliberately thin adapter. It owns two collectors and one item, and delegates +every piece of domain knowledge to stdlib `DocTest`/`Example`/`DocTestFailure`. It +overrides exactly the seams it needs and invents no parallel abstraction. It is +the reference implementation of how to integrate with `doctest` rather than +replace it, and the model this project's pytest layer should resemble. + +## Core data structures + +```text +DoctestItem(Item) dtest: DocTest, runner: DocTestRunner, fixture_request + | obj = None (class attribute) +DoctestTextfile(Module) obj = None; one DocTest for the whole file +DoctestModule(Module) one DocTest per docstring; parsefactories for + fixtures defined in the collected .py itself +MultipleDoctestFailures carries a list; the workaround for stdlib having no + per-example result value +ReprFailDoctest (ReprFileLocation, lines) pairs +``` + +`obj = None` as a *class* attribute +([`:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421)) +is what keeps `Module` from trying to import a `.txt`/`.rst` file. Subclassing +`Module` rather than `File` is what makes `scope="module"` fixtures resolve +against the page — a page collector *is* the module scope. + +`parsefactories` +([`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556)) +is `DoctestModule`-only, and it collects fixtures defined *in the `.py` being +collected*. It is **not** what makes conftest autouse fixtures apply — those are +registered through `FixtureManager.pytest_plugin_registered` when the conftest is +loaded, independently of any collector. A page collector needs no `parsefactories` +call to see them. + +## Data flow + +```text +pytest_collect_file(file_path, parent) [:126] + | .py -> DoctestModule (when --doctest-modules) + | else -> DoctestTextfile (when _is_doctest) + v +Collector.collect() -> DoctestItem.from_parent(...) per non-empty DocTest + | an EMPTY DocTest is never yielded [:451] + v +DoctestItem.setup() [:288] + | fixture request is filled, then: self.dtest.globs.update(globs) + | -> the mapping must be MUTABLE and must survive collection + v +DoctestItem.runtest() [:295] + | _check_all_skipped(self.dtest) -> outcomes.skip if every example is SKIP + | self.runner.run(self.dtest, out=failures) + | ^^^ `out` is a LIST, not a write-callable. + | clear_globs defaults to True. + | raise MultipleDoctestFailures(failures) + v +DoctestItem.repr_failure(excinfo) [:317] + for failure in failures: [:337] + lineno = test.lineno + example.lineno + 1 [:344] +``` + +Two details in that flow decide a great deal for anything built on it. + +**`out` is repurposed as a list.** The most important consumer of stdlib's runner +deliberately violates typeshed's `_Out = Callable[[str], object]`, marked with a +`# type: ignore[arg-type]`, so that `report_failure` can append rather than write. +Any claim that a "typed vanilla core" can narrow `out` honestly has to reckon with +the fact that the ecosystem's largest caller does not. + +**`repr_failure` reads each failure's own `test`.** Locations are computed +per failure inside the loop, not once per item. That is what makes N `DocTest`s +under one item report N correct locations with no override — the fact ADR 0001 is +built on. + +## Extension seams + +| Seam | Kind | +|---|---| +| `--doctest-modules`, `--doctest-glob`, `--doctest-continue-on-failure`, `--doctest-report`, `doctest_optionflags`, `doctest_encoding` | ini/CLI, declared by the always-loaded plugin | +| `doctest_namespace` session fixture ([`:721`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721)) | fixture | +| `_get_flag_lookup` ([`:385`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385)) | private; lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES`, `NUMBER` | +| `_get_checker` ([`:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662)) | private; returns the checker implementing those flags | +| `_get_continue_on_failure` ([`:410`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L410)), `_get_report_choice` ([`:703`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L703)) | private | +| `PytestDoctestRunner` ([`:181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181)) | **unreachable** — defined inside `_init_runner_class()` ([`:178`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178)) | + +That last row matters more than it looks. `PytestDoctestRunner` is where the +`OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` conversion, and +the `continue_on_failure` buffering live. Because it is defined inside a function +and never bound at module scope, a downstream cannot import or subclass it. Any +design that assumes those behaviours "come for free" by subclassing `DoctestItem` +is wrong: the item supplies the *plumbing*, but the runner supplies the +*behaviour*, and only the plumbing is reachable. + +## Configuration + +`pytest_addoption` in this module declares six settings that the plugin reads back +through helpers. A third-party plugin may **read** them but must not re-declare +them — re-adding an existing option raises at option-parsing time. Conversely, +suppressing the built-in plugin before `pytest_configure` (`-p no:doctest`) +removes the options entirely, and any downstream read of them then fails. + +## What it cannot do + +- **Collect a page with directives.** It has no document model at all; a `.rst` + file is one string handed to `DocTestParser`. +- **Share state across items.** `runtest()` runs with `clear_globs=True`. +- **Decline a path it has claimed.** `_is_doctest` + ([`:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152)) + returns `True` for any `.txt`/`.rst` **initial path before consulting + `--doctest-glob`**, and `pytest_collect_file` is not `firstresult`, so a + third-party collector claiming the same path gets its items collected + *alongside* — not instead of — the built-in's. + +## Anchors + +- [`pytest_collect_file`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L126) · + [`_is_doctest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) · + [`_is_setup_py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L141) · + [`_is_main_py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L155) +- [`_init_runner_class`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178) · + [`PytestDoctestRunner`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181) · + [`MultipleDoctestFailures`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) +- [`DoctestItem`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L251) · + [`setup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) · + [`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) · + [`repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) +- [`get_optionflags`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L401) · + [`_check_all_skipped`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) · + [`DoctestTextfile`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420) · + [`DoctestModule`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md new file mode 100644 index 0000000..d2f6e7b --- /dev/null +++ b/notes/analyses/12-pytest-xdist.md @@ -0,0 +1,157 @@ +# pytest-xdist + +Pinned at [`v3.8.0`](https://github.com/pytest-dev/pytest-xdist/tree/v3.8.0). + +## Classification + +A controller/worker distribution layer over execnet. All scheduling is integer +indices into a per-worker collection list, and the controller **never collects**. +That single asymmetry is the source of every constraint xdist imposes on a plugin +that wants to keep related tests together. + +## Core data structures + +```text +controller worker (one per process) + NodeManager -> specs: list[str] session collects normally + Scheduling implementation reports node ids back as STRINGS + node2collection: dict[node, list[str]] + node2pending: dict[node, list[int]] <- integer indices, not ids + collection: list[str] <- the agreed id list +``` + +The controller's entire model of the suite **during scheduling** is a list of +node-id strings that arrived from a worker. It has no items, no marks, no fixtures +and no knowledge of what any test does. A plugin that needs "these tests share +state" therefore cannot tell the controller so directly — it can only encode the +fact *into the node id* or infer it from string shape. + +**Reporting is a separate channel with a different shape.** After execution the +controller receives serialized `TestReport` dictionaries, and pytest serializes +arbitrary extra attributes on a report, which xdist reconstructs controller-side. +So a worker *can* ship structured per-block detail to the controller — as +JSON-safe data on the report, never as an object hanging off the item. Confusing +the two channels is what makes a controller-side summary look impossible when it +is not. + +## Data flow + +```text +pytest_cmdline_main xdist promotes -n N into --dist load (tryfirst) + | +pytest_sessionstart -> NodeManager.setup_nodes + | pytest_xdist_setupnodes(config, specs) <- specs already expanded + v +each worker collects independently + | + +-> pytest_xdist_node_collection_finished(node, ids) + | + v +Scheduling.add_node_collection(node, ids) + | every worker's list must be IDENTICAL, in the same ORDER + | mismatch -> log "**Different tests collected, aborting run**" + | and assign nothing. Zero tests execute. + v +Scheduling.schedule() -> send integer index batches to workers + | + v (on worker crash) + only the UNCOMPLETED items of the crashed work unit are re-sent + to a FRESH worker with FRESH process state +``` + +The abort path is the constraint that matters most. It is not an exception and it +is not loud in the usual sense: the scheduler logs a line, assigns nothing, and +the session ends having run nothing. Collection is not a pure function of files, +argv and ini: included files, directive implementations, MyST plugins and the +discovered registry are inputs too. A timestamp, PID, hostname, unstable iteration +order or evaluated `:skipif:` can make workers diverge when any of those values +affects identity or order. + +A registry manifest can expose differing extension sets before collection. It +cannot prove equal source closure or equal provider code, so it supplements rather +than replaces xdist's identical-node-id check. + +The crash path is the second. Because only uncompleted items of a work unit are +retried, a group whose blocks 1-2 ran before the crash has blocks 3..N re-run +against an empty process, producing a `NameError` cascade attributed to the wrong +cause. Worker restarts are on by default. + +## Extension seams + +| Seam | Kind | +|---|---| +| `pytest_xdist_make_scheduler(config, log)` | hook — substitute a `Scheduling` implementation | +| `pytest_xdist_node_collection_finished(node, ids)` | hook — observe the agreed id list | +| `pytest_xdist_setupnodes(config, specs)` | hook — receives the already-expanded spec list; never raises | +| `pytest_xdist_auto_num_workers(config)` | hook | +| `LoadScopeScheduling._split_scope(nodeid) -> str` ([`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284)) | subclass hook — the only affinity primitive *inside the shipped schedulers* | +| `@pytest.mark.xdist_group(name)` | marker, honoured only under `--dist loadgroup` | + +`pytest_xdist_make_scheduler` is the broader seam: a plugin may substitute an +entire `Scheduling` implementation, which is strictly more control than +`_split_scope` alone. That does not rescue shared state, because the substitution +happens controller-side and the controller only ever sees node-id strings — but +"the only affinity seam in the codebase" overstates it, and the honest claim is +narrower: *within the shipped schedulers*, `_split_scope` is the only affinity +primitive. + +`_split_scope` is worth stating plainly: it is a pure function from a node-id +string to a scope string, and both shipped grouping modes are two-line overrides +of it — [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) +returns the file part, [`loadgroup.py:24`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) +returns the `@`-suffix. `load` and `worksteal` have **no scope concept at any +layer**: `load` slices `pending[:num]` and `worksteal` steals a raw suffix. + +So under a user-typed `--dist load`, a plugin has exactly three options: refuse +the run, substitute the scheduler, or make the group not need protecting. There +is no "declare affinity and let the chosen scheduler honour it" API. + +The `xdist_group` marker is narrower than it appears. It is applied +[worker-side](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254), +and only when the worker's own literal `--dist` string is `loadgroup`; it works by +appending `@` to `item._nodeid`. A controller-side scheduler substitution +never reaches a worker, so no `@` suffix is written and every item becomes its own +scope — strictly worse than plain `load`. Node ids copied from a `loadgroup` run +also do not select under `-n0`. + +## Configuration + +`-n`, `--dist`, `--tx`, `--maxprocesses`, `--max-worker-restart`. Worker counts +come from +[`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37), +which builds a *list*: + +```python +xspeclist.extend([xspec[i + 1 :]] * num) +``` + +List multiplication by a negative number yields an empty list, so a negative +multiplier contributes **zero** specs. A re-implementation that sums the integer +instead contributes a negative number, and `--tx -1*popen --tx 2*popen` then +counts 1 where xdist counts 2 — a divergence whose failure direction is +permissive. + +`parse_tx_spec_config` raises `pytest.UsageError` when a run names no environment, +so it cannot be called defensively. `pytest_xdist_setupnodes(config, specs)` is +the safe source of the same information: it receives the already-expanded list, +fires during `pytest_sessionstart` — strictly before `pytest_xdist_make_scheduler` +— and never raises. + +## What it cannot do + +- **Ship a Python object between processes.** Only execnet-serializable builtins + cross. A live `globs` mapping cannot be shared, which is the whole reason a + shared doctest namespace is a distribution problem. +- **Tell the controller what an item is.** The controller sees strings. +- **Preserve process state across a worker restart** for the uncompleted tail of a + work unit. + +## Anchors + +- [`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`loadfile`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) · + [`loadgroup`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) +- [`load.schedule` abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) · + [`loadscope.schedule` abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) +- [`xdist_group` node-id append](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) diff --git a/notes/analyses/13-pytest-asyncio.md b/notes/analyses/13-pytest-asyncio.md new file mode 100644 index 0000000..fdb76ea --- /dev/null +++ b/notes/analyses/13-pytest-asyncio.md @@ -0,0 +1,133 @@ +# pytest-asyncio + +Pinned at [`v1.4.0`](https://github.com/pytest-dev/pytest-asyncio/tree/v1.4.0). + +Read here as an **idiom exemplar**, not for async semantics. It is a mature, +widely-installed `pytest11` plugin that solves the same shape of problem this +project has: a per-item resource with a configurable lifetime, an opt-in mode, and +a default it needed to change without breaking anyone. + +## Classification + +A hook-driven behaviour plugin that **does** own its item class. It defines +`PytestAsyncioFunction(Function)` +([`:506`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L506)) +with four concrete subclasses — `Coroutine`, `AsyncGenerator`, +`AsyncStaticMethod`, `AsyncHypothesisTest` — and a `pytest_pycollect_makeitem` +hookwrapper +([`:689-723`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L689-L723)) +that substitutes them for every collected async `Function`. + +That substitution-by-hookwrapper is itself the pattern worth noting: it swaps the +item class without owning collection, so pytest still decides *what* is a test +and the plugin only decides *how* it runs. + +## Core data structures + +```text +Mode(str, enum.Enum) AUTO | STRICT [:82] +PytestAsyncioSpecs its own hookspec namespace [:90] + pytest_asyncio_loop_factories(config, item) -> Mapping | None firstresult +_ScopeName reuses pytest's scope vocabulary verbatim +``` + +`Mode` inherits `str`, but a conversion layer still exists and is exactly where +drift would occur: `_get_asyncio_mode` +([`:222-232`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222-L232)) +reads the CLI value, falls back to the ini value, and calls `Mode(val)` inside a +`try`, translating a `ValueError` into a {exc}`pytest.UsageError` that lists the +valid modes. It is *called* from several sites, so the value is not literally +resolved once per session — but the conversion and its error message live in one +named function, and that is the transferable part. + +Declaring its own `HookspecMarker("pytest")` namespace is the interesting one. A +third party extends pytest-asyncio by implementing a hook, not by subclassing +anything and not by mutating a registry — which sidesteps both the nominal-typing +trap and the process-global-registry trap. + +## Data flow + +```text +pytest_addoption --asyncio-mode + asyncio_mode ini [:108] + | asyncio_default_fixture_loop_scope [:137] + | asyncio_default_test_loop_scope [:143] + | every one declared with default=None + v +pytest_configure validate scopes; addinivalue_line for the marker [:295] + | an unset default is DETECTED, not silently assumed + v +_get_asyncio_mode(config) -> Mode, resolved once [:222] + | + v +in AUTO mode: item.add_marker("asyncio") + | => marker presence becomes the single question downstream asks + v +fixture/loop resolution by scope, then pyfunc call wrapping +``` + +## Extension seams + +| Seam | Kind | +|---|---| +| `asyncio_mode` ini + `--asyncio-mode` CLI | configuration | +| `@pytest.mark.asyncio` | marker, registered via `addinivalue_line` | +| `asyncio_default_fixture_loop_scope`, `asyncio_default_test_loop_scope` | configuration, reusing pytest's scope names | +| `pytest_asyncio_loop_factories` | its own `firstresult` hookspec | +| `@pytest_asyncio.fixture(loop_scope=...)` | decorator, stamping `_loop_scope` on the function | + +## What is worth stealing + +**The `default=None` sentinel, used selectively.** Of the six options +`pytest_addoption` declares +([`:108-147`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108-L147)), +three carry `None` and three carry their effective default — so this is a +technique applied where it earns its keep, not a blanket rule. + +It earns its keep on the options whose default the project intends to move. A +`None` lets the plugin distinguish "the user chose the current default" from "the +user has not chosen", which is what makes a future change *announceable* — only +the second group is warned. `pytest_configure` does exactly that for an unset +`asyncio_default_fixture_loop_scope` +([`:296-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L296-L301)). + +This project has the same problem coming: ADR 0001 settles the vocabulary as +`ungrouped = "default" | "block"`, and any future move of that default needs the +same mechanism. + +**Normalize, then query once.** In `AUTO` mode the plugin literally adds the +marker it would otherwise have to special-case, so downstream code has one +question with one answer shape. The alternative — branching on mode at every read +site — is what produces the "two components disagree about the current setting" +class of bug. + +**Reuse the host's vocabulary rather than inventing a parallel one.** Loop scope +uses pytest's own `function`/`class`/`module`/`package`/`session` ladder and its +scope names verbatim. It does not invent a third word for lifetime. Compare +PR #87's `namespace_scope`/`namespace_items`, which collides with two pytest +concepts at once. + +**Session-wide errors raise {exc}`pytest.UsageError`.** An invalid `asyncio_mode` +and an invalid loop-scope ini value both raise it, and both are reached from +session-level config in `pytest_configure`. A misspelled session setting stops the +session, which is the right blast radius for a value that would otherwise +mis-apply to every item. + +The mirror rule — per-item errors raising something narrower — is *not* something +this plugin demonstrates cleanly, so do not cite it as precedent. Marker parsing +is one function with one blast radius. The session half is the transferable part. + +## What it cannot tell us + +Its resource — an event loop — is cheap to create, has no cross-process identity +problem, and never needs to be scheduled onto a particular worker. It therefore +has nothing to say about the distribution question that dominates this project's +design, and its scope model should not be copied on that axis. + +## Anchors + +- [`Mode`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L82) · + [`PytestAsyncioSpecs`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) +- [`pytest_addoption`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108) · + [`_get_asyncio_mode`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222) · + [`pytest_configure`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L295) +- [`_make_asyncio_fixture_function`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L210) diff --git a/notes/analyses/14-asyncio.md b/notes/analyses/14-asyncio.md new file mode 100644 index 0000000..685a449 --- /dev/null +++ b/notes/analyses/14-asyncio.md @@ -0,0 +1,111 @@ +# `asyncio` — the standard library's own pluggable architecture + +Pinned at [`v3.14.2`](https://github.com/python/cpython/tree/v3.14.2/Lib/asyncio). + +`asyncio` has nothing to do with doctests. It is here because it is the standard +library's worked example of a *deliberately* pluggable subsystem, written by +roughly the same community and shipped in the same tree as `doctest`. Setting the +two side by side answers a question the other notes cannot: when CPython wants an +extension point, what does it build — and why does `doctest` have almost none? + +## Classification + +A layered subsystem with four distinct seam kinds, none of which `doctest` uses: +an abstract base class defining the contract, duck-typed callback interfaces, a +policy indirection for selecting an implementation, and a context-manager runner +that owns lifecycle. + +## Core data structures + +```text +Handle / TimerHandle a scheduled callback [events.py:34, :141] +AbstractEventLoop the CONTRACT, ~90 methods [events.py:254] +BaseEventLoop(AbstractEventLoop) the shared implementation [base_events.py:417] +Future a result slot with callbacks [futures.py:31] +Task(Future) a coroutine driven by a loop [tasks.py:56] +Runner context manager owning a loop [runners.py:21] +BaseProtocol / Protocol / BufferedProtocol / DatagramProtocol / SubprocessProtocol + what YOU implement [protocols.py:9, :66, :109, :162, :177] +BaseTransport / ReadTransport / WriteTransport / Transport / ... + what the LOOP implements [transports.py:9, :46, :72, :148] +``` + +## The four seam kinds + +**1. An abstract base as a published contract.** `AbstractEventLoop` +([`events.py:254`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254)) +names every method an event loop must provide, separately from +`BaseEventLoop`, which implements most of them. A third party writing uvloop +implements the *contract*, not a subclass of the shipped implementation. Compare +`doctest`, where the contract and the implementation are the same class, so +`DocTestFinder(parser=...)` demands the class rather than the shape. + +**2. Paired duck-typed roles.** `Protocol` is what the user writes; `Transport` is +what the loop provides. Neither is registered anywhere, neither is checked with +`isinstance`, and the split is by *direction of the call*: the transport is called +by you, the protocol is called by the loop. This is the cheapest possible +extension mechanism — two documented method vocabularies — and it has carried +third-party HTTP, TLS and subprocess stacks for a decade. + +**3. A policy indirection, and its retirement.** `get_event_loop_policy` and +`set_event_loop_policy` sit at +[`events.py:804`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) +and [`:817`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817), +now delegating to private `_get_event_loop_policy` / `_set_event_loop_policy` +([`:798`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L798), +[`:808`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L808)) — +the public spellings are on the way out. That is the most instructive thing in +this file. A process-global, mutable indirection for "which implementation should +this program use" was shipped, was widely misused, and is being replaced by +passing the choice explicitly: `asyncio.run(main, loop_factory=...)` +([`runners.py:169`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169)). + +The lesson transfers directly. A process-global mutable registry is the seam you +regret. docutils' directive table is the same shape and has the same problems — +see [`16-docutils-myst.md`](16-docutils-myst.md). + +**4. A runner that owns lifecycle.** `Runner` +([`runners.py:21`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21)) +is a context manager that creates the loop, runs the work, cancels stragglers +([`:207`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207)) +and shuts down cleanly. Global state that must be restored lives in one `finally` +owned by one object. + +`doctest.DocTestRunner.run` does exactly this for `sys.stdout`, `pdb.set_trace`, +`linecache.getlines`, `sys.displayhook` and `PYTHON_COLORS` +([`doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +It is the one place `doctest` and `asyncio` agree on architecture, and it is +precisely why ADR 0001 takes over `__run` but leaves `run()` alone: the lifecycle +owner should keep owning the lifecycle. + +## Cross-cutting: what `doctest` would look like with `asyncio`'s seams + +| `asyncio` | `doctest` equivalent | Present? | +|---|---|---| +| `AbstractEventLoop` as a separate contract | an ABC or `Protocol` for finder/parser/checker | no — the class *is* the contract | +| `Protocol`/`Transport` duck-typed roles | `OutputChecker` is close: a documented method vocabulary, injected | partly | +| policy indirection | none | no | +| `Runner` owning lifecycle | `DocTestRunner.run`'s save/restore | yes | +| explicit `loop_factory=` replacing global policy | `parser=`, `checker=`, `test_finder=` injection | yes, and it is the healthy part | + +The gap is the first row, and it is the concrete reason +`DocutilsDocTestFinder` cannot be handed to `DocTestSuite(test_finder=...)` today +despite exposing a differently shaped `find()`. ADR 0001's answer is to keep the +contracts separate: structural `DocumentParser` implementations for markup, an +exact `DocTestParser` lane for strings, and a `DocTestFinder`-shaped adapter for +Python objects. Nominal subclassing is used only when the replacement preserves +the nominal method signature. + +## Anchors + +- [`AbstractEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) · + [`BaseEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/base_events.py#L417) +- [`get_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) · + [`set_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) +- [`BaseProtocol`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L9) · + [`Transport`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L148) +- [`Runner`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21) · + [`run(main, *, loop_factory=None)`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169) · + [`_cancel_all_tasks`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207) +- [`Future`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/futures.py#L31) · + [`Task`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/tasks.py#L56) diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md new file mode 100644 index 0000000..e73f415 --- /dev/null +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -0,0 +1,157 @@ +# `sphinx.ext.doctest` + +Pinned at [`v8.2.3`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py), +the version this project resolves. Sphinx 9.0 changed only the fallback for a +bare doctest node with no `groups` attribute: it now uses +`doctest_test_doctest_blocks` +([`v9.0.0:463`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L463)). +An unargumented Sphinx directive still stamps `groups=["default"]` +([`v9.0.0:94-98`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L94-L98)), +so its group did not change. + +## Classification + +A semantic fork. It invents its own document-level model — groups, phases, five +directives — and then converts all of it back into stdlib `doctest.DocTest` +objects for execution. It is the source of every author-facing spelling this +project supports, and the reason those spellings must be honoured exactly. + +It is also builder-coupled: nothing in the pipeline is callable without a built +Sphinx app, and it produces no machine-readable results at all. + +## Core data structures + +```text +TestCode code, type, filename, lineno, options [:235] + `type` in {testsetup, testcleanup, doctest, testcode, testoutput} +TestGroup name, setup: list, tests: list, cleanup: list [:200] + add_code(code, prepend=False) [:207] +DocTestBuilder(Builder) [:292] + self.type: "single" | "exec" <- mutable, read by a + process-global compile patch +SphinxDocTestRunner(doctest.DocTestRunner) [:257] +``` + +`TestGroup.tests` holds heterogeneous entries — `[code]` for a bare block, +`[code, output]` for a paired testcode/testoutput. `add_code` is where the silent +losses live: an orphan `testoutput` is discarded; a `testoutput` following a +`doctest` block is discarded, because a doctest entry has length 1 and fails the +`len(latest_test) == 2` guard; and a second `testoutput` *replaces* the first. + +A fourth silent loss is in the directives rather than `add_code`: `:pyversion:` is +declared on **both** `testcode` +([`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180)) +and `testoutput` +([`:184-190`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L184-L190)), +and honoured on neither — the version gate runs only for `doctest`. An author who +writes it on either gets no error and no gate. + +`:options:` on a `testcode` is **not** a silent loss. It is absent from +`TestcodeDirective.option_spec` entirely, so writing it is an unknown-option +error that drops the block — loud, and visible in the build output. + +## Data flow + +```text +directive run() [:66] + | parse group names from the optional argument + | trim `# doctest:` flags out of the RENDERED code, keep the original + | in node["test"] + | nodetype = nodes.comment when name in {testsetup, testcleanup} + | or "hide" in options [:92-93] + | stamp node["testnodetype"], ["groups"], ["options"], ["skipif"] + v +doctree + | +DocTestBuilder.test_doc(docname, doctree) [:428] + | for node in doctree.findall(condition): + | if self.skipped(node): continue <- GATED BLOCK IS DROPPED [:449-450] + | code = TestCode(...) + | "*" in groups -> add to every group + | else groups[name].add_code(code) + v +per group: ns = {} + | setup codes -> ONE simulated DocTest containing N Examples + | each ordinary or paired test -> ONE DocTest + | cleanup codes -> ONE simulated DocTest containing N Examples + | all have test.globs = ns after construction, since __init__ copies + | runner.run(test, out=..., clear_globs=False) + | self.type flipped to "exec" for setup, cleanup, testcode [:549, :608] + | to "single" for ordinary doctests [:580] + | + | if setup fails -> RETURN. Cleanup does not run. [:554-556] + v +six builder counters + text streamed to outdir/output.txt +``` + +## Extension seams + +| Seam | Kind | +|---|---| +| `TestDirective` subclassing, with `option_spec` ([`:66`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66)) | subclass | +| The node attribute stamp — `testnodetype`, `groups`, `options`, `skipif`, `test` | implicit protocol | +| `doctest_global_setup`, `doctest_global_cleanup`, `doctest_test_doctest_blocks`, `doctest_default_flags` | Sphinx confvals | +| `is_allowed_version(spec, version)` ([`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45)) | function | + +The node attribute stamp is the important one, and it is undocumented. It is the +only decoupled interface in the module: any directive that emits a +`literal_block` or `comment` carrying `testnodetype` participates. It is also +what makes a *third-party* collector — this project's — able to read a page +Sphinx's own directives produced, and vice versa. Reading attributes off the node +rather than trusting one's own directive classes is the only defence against +`Sphinx.add_directive` overriding a registration unconditionally. + +## Semantics this project must match exactly + +| Rule | Anchor | +|---|---| +| `testsetup`, `testcleanup` and `:hide:` render as `nodes.comment` | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | +| `:options:` is accepted only on `doctest` and `testoutput`; on a `testcode` it is an unknown-option error, not a discard | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111), [`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180) | +| Cleanup does **not** run when setup fails — the group returns early | [`:554-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L554-L556) | +| A gated block is dropped during collection — no outcome, id or count | [`:449-450`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | +| `*` means every group the document declares; an unargumented directive stamps `default` | [`:94-98`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L94-L98), [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) onward | +| Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200-L226) | +| `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | +| `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) | + +The last two are where this project deliberately diverges. The argument order was +a real defect in the local helper. The `compile` rebinding is unavailable to a +library that loads into every pytest session, which is the entire origin of +ADR 0001's decision to own the per-example loop instead. + +Two more are rejected on purpose. Sphinx's gated-block drop destroys the node id, +the count and the `-rs` line, where pytest users reasonably expect a `SKIPPED` +outcome with a reason. And the setup-failure short-circuit leaves a page's +`testcleanup` unrun, which for a page that spawns a server in setup means a leak. + +**One thing Sphinx already does that is worth stating plainly:** it runs one +`DocTest` *per block* against one shared group namespace. That execution shape is +not novel to ADR 0001. What Sphinx lacks is any selectable, reportable identity +for those blocks — they all share one `DocTest.name`, which is the defect below. + +## What it cannot do + +- **Run outside a Sphinx build.** `DocTestBuilder` binds an `env`, a `config`, an + `outdir`, a `sys.path` mutation and an open file handle. +- **Produce results a caller can inspect.** Six ints and text to a file. No failure + can be mapped back to its node without re-parsing prose. +- **Distinguish two blocks in one group.** Every block in a group shares + `DocTest.name`, which is why `SphinxDocTestRunner` overrides a private stdlib + method to swallow the resulting `IndexError` + ([`:257`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257)). + +## Anchors + +- [`is_allowed_version`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) · + [`TestDirective`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66) · + [`comment nodetype rule`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) +- [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) · + [`add_code`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L207) · + [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) +- [`SphinxDocTestRunner`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257) · + [`DocTestBuilder`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L292) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) · + [`skipped-node drop`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) · + [`type = "exec"` for testcode](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L548) +- [User-facing contract](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/doc/usage/extensions/doctest.rst) diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md new file mode 100644 index 0000000..0ea4cea --- /dev/null +++ b/notes/analyses/16-docutils-myst.md @@ -0,0 +1,148 @@ +# docutils and MyST-Parser + +docutils pinned at `docutils-0.21.2` (canonical repository is on SourceForge; the +GitHub copies are third-party mirrors, so anchors here name file and symbol rather +than a permalink). MyST-Parser pinned at +[`v5.1.0`](https://github.com/executablebooks/MyST-Parser/tree/v5.1.0). + +## Classification + +The parsing floor. Two front-ends producing one node model, with two different +line-number conventions and one shared, process-global, unscoped extension +registry. + +## Core data structures + +```text +docutils.nodes.Element attributes: dict[str, Any] + literal_block a rendered code block + comment what testsetup/testcleanup/:hide: become + doctest_block a bare >>> block in reStructuredText + .line int | None + .source the file the text lives in (None when unset) + .rawsource the pre-render source, when the node kept it + +docutils.parsers.rst.Parser + .state_classes an INSTANCE attribute, therefore substitutable + per parse — but NOT scoped in practice: see below + +myst_parser.parsers.docutils_.Parser(RstParser) [v5.1.0:235] + .settings_spec = (..., create_myst_settings_spec(), *RstParser.settings_spec) + [v5.1.0:241-245] +``` + +## The two line conventions + +**These are docutils 0.21.2 behaviours.** docutils 0.22 fixed the nested case +upstream and made top-level and nested blocks agree on the **first** line, so any +claim here that does not name a version is a bug in the claim. ADR 0005 +(`docs/adrs/0005-line-recovery-for-nested-blocks.md`) covers the floor question. + +| Front-end | Construct | `.line` reports (0.21.2) | +|---|---|---| +| reStructuredText | top-level `doctest_block` | its **last** line | +| reStructuredText | any block nested in a directive, list item or block quote | `None`, with `.source` also `None` | +| MyST | fenced block | its **first** line | +| either | a block reached through `.. include::` | numbered against the **included** file | + +All four are real and all four have to be normalized by the front-end that knows +which it is. A collector that assumes one convention mis-anchors the other's +blocks; a collector that reads `.line` as a number crashes on the nested case. +This is why `ParsedBlock.line` in ADR 0001 is nullable and `ParsedBlock.path` is +separate from the collected document. + +## The directive registry + +`docutils.parsers.rst.directives` keeps one module-level dict consulted by both +the reStructuredText parser and MyST's `run_directive`. It has no per-document +scoping, no versioning, and no ownership. + +Three failure modes follow, and all three have been observed: + +1. **It can be rebound, not merely mutated.** Sphinx's `docutils_namespace()` + restores a snapshot by rebinding the module attribute, so any registration made + inside that context is discarded *and the dict's object identity changes*. A + registration guard that caches a boolean is wrong; membership must be + re-checked against the live dict. +2. **Registrations are overwritten silently.** `Sphinx.add_directive` overrides an + existing name unconditionally, with only a warning. `sphinx.ext.doctest` loaded + in the same interpreter therefore replaces a forked directive class with one + that has different option handling — including the reversed + `is_allowed_version` argument order. +3. **A missing registration is silent.** An unregistered directive parses to a + docutils error node, the page still renders, and the collector finds zero + tests. This is the GH-48 failure shape, and it is the reason ADR 0004 treats + diagnostics as data. + +The defence is not to win the registry. It is to read the *node attributes* — +`testnodetype`, `groups`, `options`, `skipif`, `test`, `hide` — which are +byte-compatible with what `sphinx.ext.doctest` stamps, so a page survives either +class having produced it. + +The registry is a smaller instance of the pattern `asyncio` is currently retiring; +see [`14-asyncio.md`](14-asyncio.md). + +## MyST configuration + +`myst_parser.parsers.docutils_.Parser` subclasses `RstParser` and composes its own +settings spec from `create_myst_settings_spec()` +([`v5.1.0:208`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208), +[`:241-245`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245)). +Driving MyST through that `Parser` is what makes the `myst_*` docutils settings — +including `myst_enable_extensions` and `myst_fence_as_directive` — reachable, and +what makes front-matter configuration merge. + +Constructing an `MdParserConfig` by hand and calling `md_parser.render()` against +a bare `make_document()` skips all of it. Colon-fence directives do not exist, a +plain ```` ```python ```` fence is only picked up by a prompt sniff, and the +line-length guard and MyST transforms never run. Those omissions become a decision +rather than an accident once the front-end owns its own configuration. + +`myst_fence_as_directive` is narrower than it sounds. It runs the fence through +the directive of the **same name**, so a ```` ```python ```` fence looks for a +directive called `python`. It does not rename `python` to `testcode`; a project +wanting that must register a `python` directive or alias itself. + +## Reporter behaviour + +Default settings send reporter output to stderr and raise `SystemMessage` at +`halt_level`, aborting mid-parse. + +Turning messages into values takes **three** settings, not one. `attach_observer` +is *additive*: the observer receives the message and the warning stream still gets +written. So all of `halt_level` above 4 (both to avoid the abort and because a +halting message bypasses observer notification entirely), `report_level` at 5 or +`warning_stream` disabled to stop the write, and the observer itself. + +A `system_message` carries a level and text and **nothing semantically stable** — +no code. So a downstream that wants to suppress or promote by category has to +*classify* the message, and cannot key on an attribute docutils does not provide. +That is the open problem in ADR 0004. + +## What it cannot do + +- **Scope a directive registration** to one parse, one document or one thread. +- **Scope a `state_classes` substitution either.** `state_classes` is an instance + attribute, which makes substitution *look* parse-local — but a nested parse + builds its machine from `nested_sm_kwargs`, so a top-level substitution never + reaches a nested block, and `RSTState.nested_sm_cache` is a shared **class** + attribute that leaks substituted classes into later parses. This is why ADR 0005 + abandoned the mechanism. +- **Report a line for every node.** See the table above. +- **Type its own attribute channel.** `Element.attributes` is `dict[str, Any]`, and + typeshed's stub for `get(key, failobj: _T) -> _T` is actively wrong — it claims + `_T` even when the key is present holding something else. One narrowing accessor + at the parse boundary is cheaper and safer than a coercion at every read site. + +## Anchors + +- MyST: [`docutils_.Parser`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L235) · + [`create_myst_settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208) · + [`settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) · + [`MdParserConfig`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) +- Sphinx's registry snapshot/rebind: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/util/docutils.py) · + unconditional override: [`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/application.py) +- docutils: `docutils/parsers/rst/directives/__init__.py` (`_directives`), + `docutils/parsers/rst/states.py` (`state_classes`, `doctest_block` line + assignment), `docutils/utils/__init__.py` (`Reporter.attach_observer`, + `system_message`), at `docutils-0.21.2`. diff --git a/notes/analyses/17-prior-art.md b/notes/analyses/17-prior-art.md new file mode 100644 index 0000000..ef55ecf --- /dev/null +++ b/notes/analyses/17-prior-art.md @@ -0,0 +1,174 @@ +# Prior art: Sybil, xdoctest, pytest-examples, typeshed + +Four projects that solved some part of this problem differently. Each is read for +one specific question, and each answers it — two of them by counterexample. + +--- + +## Sybil + +Pinned at [`10.0.1`](https://github.com/simplistix/sybil/tree/10.0.1). + +**The bet:** a document is a flat sequence of non-overlapping character spans, not +a parse tree, and everything else — markup format, language, assertion semantics, +test runner — is a plugin over that primitive. It parses nothing itself: no +docutils, no myst-parser, no CommonMark implementation, and zero runtime +dependencies. Every format is a regex `Lexer` producing `Region(start, end, +lexemes)` spans over raw text. + +**Data model.** `Region` is a half-open span with three payload slots — `lexemes`, +`parsed`, `evaluator` — and lives in two undocumented-by-type states: lexed +(lexemes only) and parsed (evaluator attached). `Document` is text plus path plus +regions plus **one `namespace: dict`**. `Example` joins document and region at run +time and holds a *reference* to that namespace. `Sybil` itself is pure +configuration. + +**What is genuinely good.** `Document.add` bisect-inserts and **raises +`ValueError` on any overlap**. That single invariant gives a total order for free +and converts the silent class of parser bug — a block dropped or collected twice — +into a loud error at collection time. It is the best structural decision in the +codebase. + +Its public testing helpers (`check_lexer`, `check_parser`, `check_sybil`) are +documented *and* used by the project's own runnable documentation, so the +extension guide is under test. That is rare and worth copying. + +**The fatal flaw.** One mutable namespace per document, and +[one independently selectable pytest item per region](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py). +`pytest -k` on an example whose predecessor bound a name raises `NameError`. The +documentation never mentions this; there is no discussion of `xdist`, parallelism +or deselection anywhere in it. This is axis 1 of +[`00-taxonomy.md`](00-taxonomy.md), in the unacknowledged column. + +**The second flaw.** [Node ids are positional](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) — +`line:{line},column:{column}`. Adding a paragraph above an example renames every +downstream test, breaking `--lf`, `--nf`, deselect files, xfail lists and CI flake +history. For a *documentation* test runner, prose above examples is the thing that +changes most often. + +**What it does not do.** Groups. Sybil has no group concept and directs users to +clear the namespace instead — but that is a *design choice*, not a lexing limit: +its directive lexers do parse directive arguments and options. + +So the argument for paying the docutils dependency is not "a regex cannot see +`:skipif:`". It is **host fidelity**: collecting from the same doctree Sphinx +renders means a page behaves the same under `sphinx-build` and under pytest, and +this project already depends on docutils regardless. Nesting and source +attribution are not the argument either — Sybil locates a nested block exactly, +where docutils 0.21.2 reports `None`. + +**Verdict on the invariant.** The non-overlap check does not survive contact with +docutils: two `.. include::` directives naming the same file legitimately produce +blocks over identical source spans. Adopt the *idea* — detect double collection — +as a diagnostic carrying both provenances, not as an exception. + +--- + +## xdoctest + +Pinned at [`v1.3.2`](https://github.com/Erotemic/xdoctest/tree/v1.3.2). + +**The bet:** a doctest is Python source that happens to live in a docstring, so +parse it with `ast`/`tokenize` rather than a line regex, and abandon stdlib +compatibility to fix the design. + +**What is genuinely good.** + +- `ast`-based parsing really is better than `_EXAMPLE_RE` for Python. +- Directives are **structured objects** rather than an int bitmask, and + [`REQUIRES`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) + carries a *set of unmet requirements* — so a skip knows it skipped because + `module:torch` was absent. A bool discards exactly the information the reader + wants, and this is the single most transferable idea in the survey. +- Per-part synthetic filenames plus a filename-to-block map, so an exception raised + inside a function that an *earlier* block defined is attributed to the defining + block. Directly applicable to grouped namespaces. + +**The counterexample it provides.** It is now building stdlib compatibility back, +in modules that did not exist at the tagged release. Getting stdlib semantics out +of its own intake seam requires setting `REQUIRE_WANT`, +`deferred_output_matching=False` **and** `compile_mode='single'` — three knobs to +undo one abandoned default, paid years later. This is the empirical answer to +"should the core be vanilla?" + +**Two things not to copy.** Its got/want defaults are permissive — +`ELLIPSIS`, `NORMALIZE_WHITESPACE` and `NORMALIZE_REPR` all default true — which +silently changes the meaning of tests users wrote for stdlib. And unknown +directives and parse errors `warnings.warn` and vanish, so a typo weakens a test +instead of failing it. A test that reports green while checking nothing is worse +than no test. + +It also unregisters pytest's doctest plugin outright, which is hostile in a +`pytest11` package. + +--- + +## pytest-examples + +Pinned at [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18). + +**The bet:** invert the contract — the author does not write expected output, the +runner writes it. A block is a module exec'd once with `print` captured, and the +output is rendered back into the source file. + +**What is genuinely good.** + +- **Emitting the canonical form instead of parsing it collapses check-mode and + update-mode into one code path.** That is a real structural insight. +- **Absolute source offsets plus a recorded indent** are most of what a data model + needs to rewrite source. They are Python *string* indices, not byte offsets, and + a single indent scalar does not invert a dedent in general — a block whose lines + carry differing leading whitespace, or a tab, does not round-trip. Keeping the + offsets is still the difference between "we could add `--update-examples` later" + and "we would have to redesign the data model first". +- **It composes with pytest by contributing no collector at all.** Examples are + `parametrize` params, so marks, `-k`, fixtures and xfail all work unmodified. + This is the cheapest correct integration in the entire survey. + +**What not to copy.** It hard-codes a stack depth as a magic integer — the depth of +another library's internal call stack — and forges frames through +`ctypes.pythonapi.PyFrame_New`. Its write-back splices at collection-time offsets +with no staleness check, so editing a file while it runs corrupts the file. And +its update mode is a session-scoped in-process two-phase commit, so `-x` or a crash +writes nothing and says nothing — the worst outcome for a tool whose selling point +is rewriting your files. + +**Rule extracted.** Any write-back must content-hash the region at collection and +refuse on mismatch. An unconditional splice at stale offsets is data loss, not a +race. + +--- + +## typeshed + +Pinned at [`8c7256c`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) +(no tags on this repository; commit reachable from trunk). + +**Read for:** what a typed `doctest` actually costs, and where the type system +gives up. + +The stub annotates every public name and then hands back `Any` at exactly the +three extensible points: `globs: dict[str, Any]` (correct for values, since they +are user objects), `**options: Any` on the three suite builders (incorrect — the +accepted keys are exactly known), and `optionflags: int` everywhere (so +`optionflags=4096` type-checks). + +Two facts matter for anything claiming to be a "typed vanilla core": + +**The de-facto contract is the stub, not the source.** Downstream projects run +mypy against it. Narrowing `parse()` to `list[Example]`, dropping the `bool` +overload on `find`, or typing `out` as `Callable[[str], None]` stops type-checking +for existing typed callers whose code runs fine. + +**Its largest consumer violates it deliberately.** `_pytest.doctest` repurposes +`out` from a write-callable into a *list*, with a `# type: ignore[arg-type]`. Any +honest typing of that parameter has to accommodate the ecosystem's actual usage. + +The stub also has a false negative worth knowing: `DocTestRunner.test: DocTest` is +declared unconditionally, while runtime assigns it only inside `run()`. A stub that +type-checks a crash is worse than no stub for that attribute. + +**Rule extracted.** Keep the runtime objects structurally identical to stdlib's and +put precision in a parallel layer — `TypedDict` over the node-attribute channel, +`Protocol`s for the seams, `Literal` for closed vocabularies. Do not narrow a +signature typeshed publishes wider. diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md new file mode 100644 index 0000000..eeb6a6f --- /dev/null +++ b/notes/analyses/20-data-structures.md @@ -0,0 +1,96 @@ +# Cross-cutting: the data structures, lined up + +Every system in these notes ends up representing the same four things: a *unit of +source*, a *unit of execution*, a *unit of shared state*, and a *unit of result*. +The disagreements are entirely about which of those four are the same object. + +## The four units + +| System | source unit | execution unit | shared-state unit | result unit | +|---|---|---|---|---| +| CPython `doctest` | `Example` | `DocTest` | `DocTest.globs` | `TestResults` (2 ints + an attribute) | +| `_pytest.doctest` | `Example` | `DocTest` = one `Item` | `DocTest.globs`, wiped per item | `MultipleDoctestFailures` → `ReprFailDoctest` | +| `sphinx.ext.doctest` | `TestCode` | one `DocTest` per ordinary/paired test; setup and cleanup are each combined into one simulated `DocTest` | `ns`, assigned post-construction to every `DocTest` | six builder counters + text to a file | +| Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | +| xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | +| pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | +| ADR 0001 | `ParsedBlock` | `DocTest` per block | one `globs` per **group**, on the `Item` | `BlockResult`/`GroupResult`, projected to stdlib's | + +Reading across the "shared-state unit" column against the "execution unit" column +is the whole design problem. Sphinx and Sybil both put the shared state at a +coarser granularity than the execution unit. Sphinx gets away with it by having +no selectable unit at all — it is a builder, not a test runner, so there is +nothing for a `-k` to split. Sybil does not: it hands out one pytest item per +span over one shared mapping, which is the failure. + +Three units are easy to conflate for Sphinx specifically, so keep them apart: the +**runner call** is per ordinary test but combined per setup or cleanup phase, the +**shared state** is per group, and the **result** is six counters on the builder. + +## Field-by-field: what a source unit carries + +| Field | `Example` | `TestCode` | `Region` | `CodeExample` | `ParsedBlock` (ADR 0001) | +|---|---|---|---|---|---| +| source text | `source` | `code` | via `lexemes` | `source` | `source` | +| expected output | `want` | paired separately | — | written, not read | a separate `ParsedOutput` | +| line | `lineno` (0-based, string-relative) | `lineno` | computed from span | `start_line` | `line` (nullable) | +| string offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | +| dedent scalar | `indent` | — | `Lexeme.offset` | `indent` | — (deferred) | +| document order | list position | list position | span order | list position | `document_order`, shared with outputs | +| identity order | example index | — | positional node id | parametrization id | `block_ordinal`, runnable blocks only | +| kind | — | `type` | inferred from evaluator | `prefix_tags()` | `kind` | +| group | — | via `TestGroup` | — | — | `groups` | +| options | `options` | `options` | — | — | `options` | +| gate | — | `skipif` on the node | — | — | `skipif` and `pyversion` (unevaluated) | +| file | on the `DocTest` | `filename` | on the `Document` | `path` | `path` | +| compile mode | — | on the *builder*, mutable | — | always exec | on `ProjectedBlock` | + +Three observations. + +**Only pytest-examples carries source offsets and a recorded dedent.** They are +Python string indices rather than byte offsets, and one indent scalar does not +generally invert a dedent — but carrying them at all is the difference between a +read-only tool and one that can rewrite expected output later. ADR 0001 defers +them, which is a decision to be revisited rather than a decision made. + +**Compile mode is on the wrong object everywhere except ADR 0001.** Sphinx keeps +it as mutable builder state read through a process-global patch; stdlib hard-codes +it in the loop. ADR 0001 puts it on the projected *block*, which is where it +actually belongs — a block's execution policy is uniform across its examples, so +putting it on an `Example` subclass would both over-specify and drag the +compatibility kernel into carrying metadata. + +**A nullable line is not unique to ADR 0001.** Sphinx's own `get_line_number` +returns `None` — its docstring says "get the real line number or admit we don't +know" — for a block whose source is a stripped docstring. What ADR 0001 adds is +not the nullability but the *propagation*: `lineno=None` reaches pytest's +`EXAMPLE LOCATION UNKNOWN` branch per block, without a sibling's known line +masking it. + +## What a result unit carries + +None of these systems produce a per-example result *value*: + +- stdlib returns `TestResults(failed, attempted)` with `skipped` bolted on as an + instance attribute, and pushes everything else through `out` as text. +- pytest works around that by repurposing `out` into a list so `report_*` can + append, then rebuilds a location by slicing `test.docstring`. +- Sphinx works around it by not producing machine-readable results at all. + +This absence is why a merged `DocTest` needs a synthetic page with blank-line +padding: the only channel for a location is `(test.lineno, test.docstring, +example.lineno)`, so a group holding many blocks must fabricate a docstring in +which those arithmetic relations still hold. + +Giving each block its own `DocTest` removes the need for the fabrication rather +than improving it — the three fields are then already true. + +## Anchors + +- [`Example` / `DocTest` / `TestResults`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) +- [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) · + [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) +- [`MultipleDoctestFailures`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) · + [`repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) +- [Sybil `Region`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · + [Sybil `Document`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/document.py) diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md new file mode 100644 index 0000000..e2ff665 --- /dev/null +++ b/notes/analyses/21-data-flows.md @@ -0,0 +1,114 @@ +# Cross-cutting: the data flows, lined up + +Four pipelines that all end in the same `exec()`, drawn to the same scale so the +divergence points are visible. + +## The pipelines + +```text +CPython doctest + string ─► DocTestParser.parse ─► DocTestFinder.find ─► DocTestRunner.run + └─► __run ─► exec + +pytest --doctest-glob + path ─► pytest_collect_file ─► DoctestTextfile.collect ─► DoctestItem + (one DocTest for the whole file) ├─ setup(): globs.update(fixtures) + ├─ runtest(): runner.run(clear_globs=True) + └─ repr_failure(): per-failure location + +sphinx.ext.doctest + doctree ─► test_doc ─► condition filter ─► skipped? DROP + │ + ├─► TestCode ─► TestGroup{setup, tests, cleanup} + │ + └─► per group: ns={}; test.globs=ns (post-construction) + 3 runners (setup/test/cleanup), clear_globs=False + self.type flipped single/exec around each run + └─► doctest.compile (PROCESS-GLOBAL PATCH) ─► exec + +ADR 0001 + path ─► markup.parse_file ─► (Blocks, Diagnostics) + │ + ├─► project() ─► GroupPlan{group, blocks[], seed} + │ pure: no docutils, no pytest, no filesystem, + │ no user code — :skipif: passes through unevaluated + │ + └─► Document(pytest.Module) ─► DocutilsItem (one per GROUP) + ├─ setup(): globs cleared, then fixtures injected + ├─ runtest(): run_group() runs each block's DocTest + │ in phase order, clear_globs=False + │ :skipif: evaluated HERE + │ DocutilsRunner.__run ─► exec (mode from data) + └─ repr_failure(): inherited; reads each block's own DocTest +``` + +## Where they diverge + +**At the gate.** Sphinx evaluates `:skipif:` during collection and *drops* the +node. pytest cannot express that — an item must exist to have an outcome — so +`doctest_docutils` marks the block `SKIP` instead. ADR 0001 moves the evaluation +to `runtest()`, which additionally buys collection purity: with no user code +running at collection, worker collections cannot diverge and `--collect-only` +cannot have side effects. + +**At the globs assignment.** Every system that shares state has to assign +`test.globs` *after* `DocTest.__init__`, because the constructor +[copies](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565). +Sphinx does this explicitly. pytest does not share at all, and its +[`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) +runs with `clear_globs` defaulting to `True` — so any design that inherits +`runtest()` unchanged and expects sharing gets its mapping emptied after the first +block. That is a trap with no diagnostic; the symptom is a `NameError` in block +two. + +**At the compile call.** stdlib hard-codes `"single"`. Sphinx flips a mutable +builder attribute and reads it through a process-global rebinding of +`doctest.compile` that is never restored. PR #87 proposes cloning the +mangled loop's code object to get a private version of that rebinding. ADR 0001 puts the policy on the projected *block*, materializes a runtime per +profile, and reads it in a loop it owns — and only for extended profiles, since +ordinary prompt blocks run on CPython's untouched loop — the only one +of the four that neither mutates process state nor copies a code object. + +**At the location.** stdlib computes `test.lineno + example.lineno + 1`. Sphinx +gives every block in a group the same `DocTest.name` and pays for it by overriding +a private method to swallow an `IndexError`. PR #87 fabricates a +synthetic page so the arithmetic stays true across merged blocks. ADR 0001 gives +each block its own `DocTest`, so the arithmetic is true without fabrication. + +## The order-of-operations facts + +Three orderings are load-bearing and each has a failure mode with no error +message. + +**Fixtures are injected in `setup()`, into the mapping, in place.** +`DoctestItem.setup()` does `self.dtest.globs.update(globs)` +([`:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293)). +A design that clears and rebinds the mapping around that call either discards the +injected names — so `getfixture` raises `NameError` — or reuses the previous +attempt's mutations. Under `--reruns`, the second is a false green: an expectation +true only on attempt two reports as a pass. + +**Phase order and page order are different orders.** A group hands its blocks over +as setup, tests, cleanup; a reader meets them in whatever order the page writes +them. Any layout that anchors reported lines on the *run* order reports examples +against whichever block came first in that sequence — and can point past the end of +the file. Sphinx sidesteps this by not reporting useful lines at all. + +**Collection order must be numeric, never lexicographic.** `DocTest.__lt__` +compares names as text +([`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596)), +and names carry positions as text, so any accidental `sorted()` runs `page.md[10]` +before `page.md[1]`. Every test still passes, in the wrong sequence — and for a +group sharing state, the wrong sequence is the bug. + +## Anchors + +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) · + [`__lt__`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) · + [`__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) · + [`compile(..., "single", ...)`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) +- [`DoctestItem.setup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) · + [`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) · + [gated-node drop](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md new file mode 100644 index 0000000..e32a615 --- /dev/null +++ b/notes/analyses/22-extension-seams.md @@ -0,0 +1,113 @@ +# Cross-cutting: extension seams + +Five mechanisms appear across these systems. They are not equally good, and the +ranking is not a matter of taste — each has an observed failure mode. + +## The five mechanisms + +| Mechanism | Where | Failure mode | +|---|---|---| +| **Nominal subclassing** — the *type checker* demands the class, the interpreter does not | stdlib `doctest` (via typeshed), `sphinx.ext.doctest` directives | Typed callers reject structurally valid objects, but subclassing is invalid when the replacement method has a different signature. stdlib performs no `isinstance` check on `parser` or `test_finder`; the pressure comes from `doctest.pyi`. One genuine runtime edge: `DocTestSuite` sorts, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom finder must return real `DocTest`s | +| **Callable aliases** — `Evaluator = Callable[[Example], str \| None]` | Sybil | Types nothing. You cannot express "this lexer emits `source` and `arguments`", so a mismatched pairing raises `KeyError` at run time. Sybil's own source comments say the payload "could likely be a `TypedDict`" | +| **Named registry** — a module dict plus a `register_*` function | `doctest.register_optionflag`, docutils directives, xdoctest's two facades | Process-global mutable state. Order-dependent, unscoped, and silently overwritable | +| **`Protocol`** — structural typing | xdoctest's `StdlibExampleLike` | None inherent; but a `Protocol` alone does not satisfy a nominal consumer | +| **Hookspec** — the host's own plugin protocol | pytest, pytest-asyncio's `PytestAsyncioSpecs` | Requires a host with a plugin system; not available to a library core | + +## Why `register_optionflag` works and the docutils registry does not + +Both are process-global mutable dicts. One is fine and one is a recurring bug +source, and the difference is instructive. + +[`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) +is **append-only and idempotent**: registering a name that exists returns the +existing bit. It cannot be overwritten, so two libraries registering `NUMBER` agree +rather than fight. Its only sharp edge is ordering — ints are +`1 << len(OPTIONFLAGS_BY_NAME)` — which matters because typeshed hard-codes the +builtin values, so a flag registered *before* the builtins would change every +stdlib constant. Registering at import of the core rather than from a plugin hook +is what keeps that ordering stable, and it is also required because an +unregistered name makes a page fail to **parse**. + +The docutils directive table is **overwrite-by-default and rebindable**. Sphinx's +`docutils_namespace()` restores a snapshot by rebinding the module attribute, so +the dict's *identity* changes and a cached boolean guard is wrong. +`Sphinx.add_directive` overwrites unconditionally with only a warning. The result +is that a directive class you registered may not be the one that ran. + +**The rule extracted:** a global registry is acceptable when registration is +append-only and idempotent, and a liability when it is last-writer-wins. Where the +registry is someone else's and last-writer-wins, do not depend on having won — +depend on the *data* both writers produce. That is why ADR 0001 reads +`BlockAttributes` off the node rather than trusting its own directive classes to +have run: the attribute set is byte-compatible with what `sphinx.ext.doctest` +stamps, so either winner is fine. + +`asyncio` reached the same conclusion about global indirection from the other +direction and is retiring `set_event_loop_policy` in favour of an explicit +`loop_factory=` argument — see [`14-asyncio.md`](14-asyncio.md). + +## The nominal/structural trap requires separate adapters + +Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: +DocTestParser = ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. The +interpreter does not perform an `isinstance` check, but it still calls the exact +stdlib methods. A markup parser whose `parse(text, path, *, settings)` returns a +doctree cannot override `DocTestParser.parse(string, name)` returning alternating +strings and examples. A mypy probe rejects that override, and subclassing does not +make the runtime calls compatible. + +The solution is two contracts, not one class wearing two names: + +```python +class DocumentParser(t.Protocol): + suffixes: t.ClassVar[frozenset[str]] + + def parse( + self, text: str, path: pathlib.Path, *, settings: ParseSettings + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: ... + + +class StdlibParserFacade(doctest.DocTestParser): + def parse( + self, string: str, name: str = "" + ) -> list[str | doctest.Example]: ... +``` + +The markup `DocumentParser` has reStructuredText and MyST implementations. Plain +strings use the exact stdlib parser lane, and Python objects use a separate +`DocTestFinder`-shaped adapter. A nominal façade is useful only where it preserves +the nominal API's signature. + +## What deserves a seam, and what does not + +The project rule is that a new public API waits until a caller outside the module +needs it. Applied to the candidates that came up: + +| Candidate | Verdict | +|---|---| +| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a record. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind` | +| Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()`. One factory supplies the checker used for both comparison and failure explanation | +| `DocumentParser` protocol | **Yes.** Two markup implementations ship initially: reStructuredText and MyST. Plain text and Python objects use the separate stdlib-shaped lanes | +| stdlib parser and finder façades | **Yes, but separate.** They preserve `DocTestParser` and `DocTestFinder` signatures instead of subclassing them with incompatible markup or object-discovery methods | +| `ExecutionProfile` | **Yes, and contributable.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. The type and its immutable registration are public; the mutable builder is not | +| A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | +| `RegistrySnapshot` and a builder | **Yes, with asymmetric visibility.** The snapshot and immutable registration records are public inputs; the mutable builder is private. Settings remain a separate value. Direct, pytest and Sphinx adapters own their freeze points in ADR 0007 | +| Entry-point plugin discovery | **Optional, behind the contributor protocol.** Discovery is not itself nondeterministic, but matching manifests only compare declared extension sets. Version 1 requires homogeneous provider code and source closure; heterogeneous-worker attestation is deferred | + +The distinction those two rows turn on is worth stating once: **discovery is not +the hazard; divergent inputs are.** A registry populated with matching names and +versions can still resolve to different code, and matching registries can still +parse different included files. ADR 0007's manifest catches declared extension +drift; [`12-pytest-xdist.md`](12-pytest-xdist.md)'s identical-collection check +remains authoritative for node ids. + +## Anchors + +- [`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) · + [`report_*` hooks](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) · + [`OutputChecker`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`pytest_xdist_setupnodes` consumers](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) +- [`PytestAsyncioSpecs`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) +- [`AbstractEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) · + [`set_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md new file mode 100644 index 0000000..5ce0aa1 --- /dev/null +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -0,0 +1,146 @@ +# Cross-cutting: namespace scope and test identity + +The one axis where a wrong answer is silent. Every other design choice in these +notes produces inconvenience — a renamed test, a conversion layer, an extra knob. +This one produces a `NameError` in a test the user believed they could select, or a +green run that should have been red. + +## The two questions, which are not the same question + +1. **Which blocks share a `globs` mapping?** (scope) +2. **Which blocks can be selected, reported and scheduled independently?** + (identity) + +Everything downstream — `-k`, `--lf`, `--deselect`, `-x`, `--reruns`, every +`--dist` mode, JUnit rows, flake history — depends on the second. Everything a +narrative page needs depends on the first. + +## The product space + +| | one node id | N node ids | +|---|---|---| +| **one `DocTest`** | PR #87's `merged` | incoherent | +| **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids at all; ADR 0001 gives the shape a pytest identity | Sybil; PR #87's `per-block` | + +Sphinx belongs in the bottom row, with a qualification: it builds one `DocTest` +per *ordinary test* block against one shared group namespace, while combining all +setup blocks into one simulated `DocTest` and all cleanup into another. Its "one +node id" is really *no* id — every test block in a group shares one +`DocTest.name`, which is why `SphinxDocTestRunner` overrides a private stdlib +method to swallow the resulting `IndexError`. So the execution shape is partly +well-trodden; the contribution is making it addressable. + +The bottom-right cell is where the silent failure lives. One surveyed project +ships it, and one open proposal implements it with guards. + +Sybil is there **unacknowledged**: one `Document.namespace` shared by reference, +one pytest item per region. `pytest -k` on an example whose predecessor bound a +name raises `NameError`, and nothing in its documentation says so. + +PR #87's proposed `per-block` mode would sit there **acknowledged and guarded** — +the guards being an xdist scheduler substitution, a scheduler refusal and a +run-twice refusal. It has not shipped; +{doc}`ADR 0003 <../../docs/adrs/0003-rejecting-per-block-items>` rejects the +shape. Those guards are the reason `_worker_count`, `_shared_page`, `_is_page` +and `_splitting_scheduler` exist at all. + +The bottom-left cell gives per-block reporting *and* an unsplittable sharing unit, +and it needs no guards, because there is nothing to split. + +## Why the guards are expensive + +The constraints come from [`12-pytest-xdist.md`](12-pytest-xdist.md) and are all +structural, not incidental: + +- A live mapping is a Python object; only execnet-serializable builtins cross a + worker boundary. +- The controller never collects, so it cannot ask "which items share state" — it + sees node-id strings and nothing else. Any protection must be *inferred from + string shape* or *encoded into the id*. +- The only affinity primitive *inside the shipped schedulers* is + [`_split_scope(nodeid) -> str`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284). + A plugin may substitute a whole `Scheduling` via `pytest_xdist_make_scheduler`, + but that still reasons only over node-id strings. `load` and `worksteal` have no + scope concept at any layer, so under a user-typed `--dist load` the options are + refuse, substitute, or do not need protecting. +- `xdist_group` is applied + [worker-side](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) + and only when the worker's own `--dist` is literally `loadgroup`. A + controller-side substitution never reaches a worker. +- A worker crash re-runs only the *uncompleted* items of a work unit, on a fresh + process. Blocks 3..N then run against an empty mapping. Restarts are on by + default. +- A retry (`--reruns`, `--count`) re-runs a block against globals it already + mutated, so an expectation true only on attempt two reports **PASS**. + +Every one of these evaporates when the item is the sharing unit. + +## Test identity: never source-coordinate-derived + +Two of the surveyed projects derive node ids from **source coordinates** — +Sybil's +[`line:{line},column:{column}`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) +and pytest-examples' `path:start-end`. + +For a *documentation* test runner that is indefensible, because prose above +examples is the thing that changes most often. Adding a sentence renames every +downstream test, which breaks `--lf`, `--nf`, checked-in deselect files, xfail +lists and CI flake history. pytest-examples compounds it: the same string is the +dedupe key for its write-back, so an identity collision becomes file corruption. + +The rule: **author-declared name first, stable ordinal as fallback, and the +fallback shape invariant across configuration.** The test to apply is concrete — +adding a sentence to a page must rename zero tests. + +An **ordinal among the extracted blocks** is not a source coordinate and is fine: +`page.md[3]` is unchanged by a paragraph inserted above it, which is exactly the +edit that renames a `line:N` id. Released `doctest_docutils` names its tests that +way already, and the rule preserves it. + +Two corollaries: + +- `DocTest.name` must be machine-independent. Embedding an absolute path makes a + checked-in `--deselect` resolve only on the machine that produced it, and puts a + home directory in JUnit XML. +- Column is not worth carrying. It adds churn and disambiguates nothing once names + exist. + +## What a node id does *not* promise + +Worth stating plainly, because it is the honest limit of the recommended design: +**a per-block node id over shared mutable state cannot truthfully promise +independent execution.** Selecting block two of a stateful page raises +`NameError` under Sybil, under PR #87's `per-block`, and under any scheme of that +shape. Ids over *isolated* state — released `doctest_docutils`, or a page whose +blocks declare no group — promise independence and keep the promise. + +The choice is therefore not between "selectable blocks" and "unselectable blocks". +It is between an id that *claims* to be selectable and is not, and an id whose +granularity honestly matches what can be run alone. `merged` and ADR 0001 both +choose the latter; they differ only in whether the reporting granularity has to +match the selection granularity, and ADR 0001's answer is that it does not. + +## Fixture lifetime falls out of this + +A page collected as a `pytest.Module` **is** the module scope, so +`@pytest.fixture(scope="module")` already has page lifetime — no shim required. +Sybil reaches the same outcome through a `getparent` override that returns the +file collector when pytest asks for `Module`. + +The corollary is a trap for any design that shares state across items without +sharing the item: fixtures do not follow. A block that stashes a fixture-derived +object under a name keeps answering after that fixture has been finalized, because +the name outlives the object's lifetime. Making the item the sharing unit aligns +the two — the mapping and the fixtures have the same lifetime because they have the +same owner. + +## Anchors + +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) +- [`runtest` with `clear_globs=True`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) · + [`setup` globs update](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`xdist_group` append](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) · + [collection-mismatch abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) +- [Sybil `identify`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) · + [Sybil pytest integration](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py) diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md new file mode 100644 index 0000000..e404049 --- /dev/null +++ b/notes/analyses/90-bibliography.md @@ -0,0 +1,163 @@ +# Bibliography + +Every external anchor cited by `docs/adrs/` and these notes, in one place. All +links are pinned to a tag, or — where a project publishes no tags — to a commit +reachable from trunk. Line anchors are only meaningful on a pinned ref and are not +used anywhere else. + +## CPython — `v3.14.2` + +[`python/cpython @ v3.14.2`](https://github.com/python/cpython/tree/v3.14.2) + +### `Lib/doctest.py` + +| Symbol | Anchor | Cited for | +|---|---|---| +| `TestResults` | [`:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) | 2-field namedtuple; `skipped` is an extra attribute | +| `register_optionflag` | [`:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) | the one append-only, idempotent cross-library registry | +| `_load_testfile` | [`:245`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245) | private, reached by `doctest_docutils` today | +| `DocTest.__init__` | [`:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | **copies** the globs mapping | +| `DocTest.__lt__` | [`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | compares names as text | +| `DocTestParser` | [`:609`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L609) | the injectable parser | +| `_EXAMPLE_RE` | [`:618`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618) | private, used for prompt sniffing | +| `DocTestFinder` | [`:844`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) | the type typeshed names; accepted structurally at runtime | +| `report_*` hooks | [`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | the four supported in-loop seams; no `report_skip` here | +| `__run` | [`:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | name-mangled loop; overridable by mechanism | +| `compile(..., "single", ...)` | [`:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | the hard-coded mode `{testcode}` cannot use | +| `__record_outcome` | [`:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485) | arity and accumulator differ across supported versions | +| `__patched_linecache_getlines` | [`:1501`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501) | parses the `` filename shape back | +| `run()` save/restore | [`:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) | global interpreter state; not reentrant | +| `summarize` | [`:1590`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1590) | reads the accumulator the owned loop must write | +| `OutputChecker` | [`:1690`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) | the documented checker seam | +| `DebugRunner` | [`:1874`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1874) | `report_*` overriding as the sanctioned loop control | +| `testfile` | [`:2091`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2091) | the API `testdocutils` mirrors | +| `DocTestSuite` | [`:2467`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) | no `isinstance` on `test_finder`; sorts, so results must be real `DocTest`s | +| `DocFileSuite` | [`:2570`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) | no `isinstance` on `parser` | + +Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blob/v3.14.2/Doc/library/doctest.rst). + +### `Lib/asyncio/` + +| Symbol | Anchor | +|---|---| +| `AbstractEventLoop` | [`events.py:254`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) | +| `get_event_loop_policy` / `set_event_loop_policy` | [`events.py:804`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) · [`:817`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) | +| `BaseEventLoop` | [`base_events.py:417`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/base_events.py#L417) | +| `BaseProtocol` / `Protocol` | [`protocols.py:9`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L9) · [`:66`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L66) | +| `BaseTransport` / `Transport` | [`transports.py:9`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L9) · [`:148`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L148) | +| `Runner` / `run` / `_cancel_all_tasks` | [`runners.py:21`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21) · [`:169`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169) · [`:207`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207) | +| `Future` / `Task` | [`futures.py:31`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/futures.py#L31) · [`tasks.py:56`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/tasks.py#L56) | + +## pytest — `9.1.1` + +[`pytest-dev/pytest @ 9.1.1`](https://github.com/pytest-dev/pytest/tree/9.1.1) · +[`src/_pytest/doctest.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `pytest_collect_file` | [`:126`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L126) | not `firstresult` | +| `_is_setup_py` / `_is_main_py` | [`:141`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L141) · [`:155`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L155) | privates imported today | +| `_is_doctest` | [`:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | claims initpaths **before** `--doctest-glob` | +| `MultipleDoctestFailures` | [`:172`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) | the missing per-example result value, worked around | +| `_init_runner_class` / `PytestDoctestRunner` | [`:178`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178) · [`:181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181) | **unreachable by name** | +| `DoctestItem` | [`:251`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L251) | the subclassed item | +| `setup` | [`:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | `globs.update(...)` in place | +| `runtest` | [`:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | `clear_globs` defaults to `True` | +| `repr_failure` | [`:317-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) | reads each failure's own `test` | +| `_get_flag_lookup` | [`:385`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385) | lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES`, `NUMBER` | +| `get_optionflags` | [`:401`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L401) | read, not re-declared | +| `_get_continue_on_failure` | [`:410`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L410) | private helper imported today | +| `DoctestTextfile` | [`:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | `obj = None` as a class attribute | +| `_check_all_skipped` | [`:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | fires only once the item is running | +| `DoctestModule` / `parsefactories` | [`:500`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) · [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | fixtures defined in the collected `.py`; **not** conftest autouse, which arrives via `FixtureManager.pytest_plugin_registered` | +| `subtests` | [`src/_pytest/subtests.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/subtests.py) | builtin since 9.0; the only sanctioned sub-item outcome mechanism, and experimental | +| `_get_checker` | [`:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662) | the checker that would have to be reimplemented | +| `_get_report_choice` | [`:703`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L703) | private helper | +| `doctest_namespace` | [`:721`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) | the fixture that survives plugin blocking today | + +## pytest-xdist — `v3.8.0` + +[`pytest-dev/pytest-xdist @ v3.8.0`](https://github.com/pytest-dev/pytest-xdist/tree/v3.8.0) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `parse_tx_spec_config` | [`workermanage.py:26-37`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) | list `extend`, so a negative multiplier contributes zero | +| `LoadScopeScheduling._split_scope` | [`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) | the only affinity primitive | +| `LoadFileScheduling._split_scope` | [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) | two-line override | +| `LoadGroupScheduling._split_scope` | [`loadgroup.py:24`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) | two-line override | +| collection-mismatch abort | [`load.py:259`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) · [`loadscope.py:359`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) | logs and runs zero tests | +| `xdist_group` node-id append | [`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) | worker-side, `loadgroup` only | + +## pytest-asyncio — `v1.4.0` + +[`pytest-dev/pytest-asyncio @ v1.4.0`](https://github.com/pytest-dev/pytest-asyncio/tree/v1.4.0) · +[`pytest_asyncio/plugin.py`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `Mode` | [`:82`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L82) | `str` enum so ini, CLI and internal value are one object | +| `PytestAsyncioSpecs` | [`:90`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) | its own hookspec namespace | +| `pytest_addoption` | [`:108`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108) | every option `default=None` | +| `_make_asyncio_fixture_function` | [`:210`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L210) | stamping scope on the function | +| `_get_asyncio_mode` | [`:222`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222) | resolve once, query once | +| `pytest_configure` | [`:295-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L295-L301) | detecting an unset default via the sentinel | + +## Sphinx — `v8.2.3` + +[`sphinx-doc/sphinx @ v8.2.3`](https://github.com/sphinx-doc/sphinx/tree/v8.2.3) · +[`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `is_allowed_version(spec, version)` | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | specifier first — the reverse of the local helper | +| `TestDirective` | [`:66`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66) | the directive base and its option handling | +| comment nodetype rule | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | `testsetup`/`testcleanup`/`:hide:` become `nodes.comment` | +| `:options:` gating | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111) | accepted only on `doctest` and `testoutput` | +| `TestGroup` / `add_code` | [`:200`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) · [`:207`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L207) | phase ordering; three silent-loss cases | +| `TestCode` | [`:235`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) | the parsed unit | +| `SphinxDocTestRunner` | [`:257`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257) | overrides a private method to swallow an `IndexError` | +| `DocTestBuilder` | [`:292`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L292) | builder coupling | +| `doctest.compile` rebinding | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) | process-global, never restored | +| `test_doc` | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) | group resolution and `*` | +| gated-node drop | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | no outcome, id or count | +| `type = "exec"` for testcode | [`:548`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L548) | the mode flip | + +Documentation: [`doc/usage/extensions/doctest.rst`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/doc/usage/extensions/doctest.rst). +Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/util/docutils.py), +[`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/application.py). + +## MyST-Parser — `v5.1.0` + +[`executablebooks/MyST-Parser @ v5.1.0`](https://github.com/executablebooks/MyST-Parser/tree/v5.1.0) + +| Symbol | Anchor | +|---|---| +| `create_myst_settings_spec` | [`parsers/docutils_.py:208`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208) | +| `Parser(RstParser)` | [`parsers/docutils_.py:235`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L235) | +| `settings_spec` | [`parsers/docutils_.py:241-245`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) | +| `MdParserConfig` (`myst_enable_extensions`, `myst_fence_as_directive`) | [`config/main.py`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) | + +## docutils — `docutils-0.21.2` (the version this project pins) + +The canonical repository is on +[SourceForge](https://sourceforge.net/p/docutils/code/); the GitHub copies are +third-party mirrors and are not linked here. Anchors name file and symbol at the +tagged release: + +| File | Symbol | Cited for | +|---|---|---| +| `docutils/parsers/rst/directives/__init__.py` | `_directives` | the process-global, rebindable registry | +| `docutils/parsers/rst/states.py` | `state_classes`, `doctest_block` line assignment | per-instance substitutability; last-line convention | +| `docutils/utils/__init__.py` | `Reporter.attach_observer`, `system_message` | observation separable from display | +| `docutils/nodes.py` | `Element.attributes`, `literal_block`, `comment`, `doctest_block` | the untyped attribute channel | + +Typed surface: [`typeshed stubs/docutils`](https://github.com/python/typeshed/tree/8c7256c/stubs/docutils). + +## Prior art + +| Project | Ref | Key anchors | +|---|---|---| +| Sybil | [`10.0.1`](https://github.com/simplistix/sybil/tree/10.0.1) | [`sybil.py:155-157`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (positional ids) · [`document.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/document.py) (one namespace, non-overlap invariant) · [`integration/pytest.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py) (one item per region) · [`region.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · [`testing.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/testing.py) (public extension-test helpers) | +| xdoctest | [`v1.3.2`](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | [`directive.py:58`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) (`REQUIRES` carries its reason) · [`plugin.py`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/plugin.py) (unregisters pytest's doctest plugin) | +| pytest-examples | [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | [`find_examples.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/find_examples.py) · [`run_code.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/run_code.py) · [`modify_files.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/modify_files.py) (Python string offsets, recorded indent, unguarded splice) | +| typeshed | [`8c7256c`](https://github.com/python/typeshed/tree/8c7256c) | [`stdlib/doctest.pyi`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | diff --git a/notes/analyses/README.md b/notes/analyses/README.md new file mode 100644 index 0000000..72b5187 --- /dev/null +++ b/notes/analyses/README.md @@ -0,0 +1,69 @@ +# Doctest ecosystem structural analyses + +Structural analysis of how the systems `doctest_docutils` sits between are built — +their core data structures, data flows, extension seams, and configuration models. +These are research notes. They inform the ADRs in `docs/adrs/` and decide nothing +themselves. + +The question they exist to answer: a doctest engine that must be vanilla-compatible +at the core, pluggable, usable as a pytest plugin, usable with docutils and +myst-parser, *and* speak all three communities' idioms is standing on three +upstreams with three separate extension models and three overlapping vocabularies. +What exactly does each of them require, and where do they contradict each other? + +## Method + +1. **Portable citations first.** Every external reference is a deep link to a + specific file **pinned at a git tag** — never `main`, `master`, `HEAD` or a bare + SHA. Line anchors are only used on a pinned ref, because they are meaningless + without one. These links are the reproducible source surface for the analysis. +2. **Source review second.** Confirm and deepen against checked-out source with + `rg`/`fd`. Local notes may inform drafting; tracked notes must be readable and + verifiable without a workstation path. +3. **Execute the load-bearing claims.** Anything an ADR rests on is run, not read. + Where a note says "verified", a snippet was executed and its output recorded. + +## Pinned versions + +| Project | Repo | Ref | +|---|---|---| +| CPython (`doctest`, `asyncio`) | `python/cpython` | `v3.14.2` | +| pytest | `pytest-dev/pytest` | `9.1.1` | +| pytest-xdist | `pytest-dev/pytest-xdist` | `v3.8.0` | +| pytest-asyncio | `pytest-dev/pytest-asyncio` | `v1.4.0` | +| Sphinx | `sphinx-doc/sphinx` | `v8.2.3` — what this project resolves. `v9.0.0` is cited only for the bare-node group fallback change | +| MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` on Python ≥ 3.11; `v4.0.1` below | +| Sybil | `simplistix/sybil` | `10.0.1` | +| xdoctest | `Erotemic/xdoctest` | `v1.3.2` | +| typeshed | `python/typeshed` | `8c7256c` (no tags; commit reachable from trunk) | +| docutils | SourceForge (the GitHub clones are third-party mirrors) | `docutils-0.21.2` | + +## Files + +- [`00-taxonomy.md`](00-taxonomy.md) — the design axes, as a classification matrix. +- Per-system structural docs: [`10-cpython-doctest.md`](10-cpython-doctest.md), + [`11-pytest-doctest.md`](11-pytest-doctest.md), + [`12-pytest-xdist.md`](12-pytest-xdist.md), + [`13-pytest-asyncio.md`](13-pytest-asyncio.md), + [`14-asyncio.md`](14-asyncio.md), + [`15-sphinx-ext-doctest.md`](15-sphinx-ext-doctest.md), + [`16-docutils-myst.md`](16-docutils-myst.md), + [`17-prior-art.md`](17-prior-art.md). +- Cross-cutting: [`20-data-structures.md`](20-data-structures.md), + [`21-data-flows.md`](21-data-flows.md), + [`22-extension-seams.md`](22-extension-seams.md), + [`23-namespace-scope-and-test-identity.md`](23-namespace-scope-and-test-identity.md). +- [`90-bibliography.md`](90-bibliography.md) — every pinned anchor cited by the + ADRs, in one place. + +Each per-system doc follows the same section order — classification · core data +structures · data flow · extension seams · configuration · what it cannot do · +anchors — so the systems are directly comparable, and the cross-cutting docs can +line them up column by column. + +`14-asyncio.md` is included even though `asyncio` has nothing to do with doctests. +It is the stdlib's own worked example of a pluggable architecture built out of +protocols, an abstract base, a policy indirection and a runner, by roughly the same +people and in roughly the same era as `doctest`'s extension model. Reading the two +side by side is the cheapest available answer to "what does the standard library +consider a good seam, and why does `doctest` have so few of them?"