From 107bd8c82772985254c1fb3eccef7e48f48db35e Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 10:10:00 -0500 Subject: [PATCH 01/24] docs(adrs): Add the architecture decision record scaffold why: Design decisions for the doctest engine have lived in commit bodies and pull request threads, where a later reader cannot tell a deliberate constraint from an accident. The project needs one place that records what was decided, what forced it, and what it rules out. what: - Add docs/adrs/index.md stating what a record is for - Fix numbering as sequential and permanent, and name the four statuses a record can carry - Require pinned source links, since a blob/master anchor rots silently onto unrelated code while still resolving - Wire the section into the docs toctree --- docs/adrs/index.md | 27 +++++++++++++++++++++++++++ docs/index.md | 1 + 2 files changed, 28 insertions(+) create mode 100644 docs/adrs/index.md diff --git a/docs/adrs/index.md b/docs/adrs/index.md new file mode 100644 index 0000000..273990e --- /dev/null +++ b/docs/adrs/index.md @@ -0,0 +1,27 @@ +(adrs)= + +# Architecture Decision Records + +Significant design decisions for `doctest_docutils` and +`pytest_doctest_docutils`, and their rationale. + +These records govern the shape of the doctest engine: what it produces, what it +may reach into, and what vocabulary it speaks. A record states the context that +forced a decision, the decision itself, what it costs, and what it rules out — +so a later reader can tell a deliberate constraint from an accident. + +Supporting structural research lives in `notes/analyses/`. It decides nothing and +is cited by these records as evidence. + +## Conventions + +**Numbering** is sequential and permanent. A record is never renumbered, and a +superseded one is marked rather than deleted. + +**Status** is one of `Draft`, `Proposed`, `Accepted`, `Superseded by NNNN`. + +**Source links are pinned.** Every citation of an external project names a git +tag, or a commit reachable from that project's trunk where it publishes no tags. +Line anchors are only used on a pinned ref, because they are meaningless without +one, and a `blob/master` link rots silently — the file moves, lines shift, and +the anchor lands on unrelated code while still resolving. diff --git a/docs/index.md b/docs/index.md index aba9e4c..006ef63 100644 --- a/docs/index.md +++ b/docs/index.md @@ -74,6 +74,7 @@ modules/doctest_docutils/index modules/pytest_doctest_docutils/index modules/linkify_issues/index project/index +adrs/index history GitHub ``` From 524f879e1874562059493fda96303da1bfcf583b Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 10:11:04 -0500 Subject: [PATCH 02/24] docs(adrs[0001]): Record the doctest core architecture why: The finder is one 320-line method spanning parsing, grouping and naming; the runner reaches CPython's name-mangled loop through a code-object clone; the plugin blocks pytest's doctest plugin and then imports its privates; and a fork of xdist guards state xdist cannot see. All four follow from one conflation: test identity and shared state are separate axes, coupled here into a single setting. what: - Decide the item is the group and the DocTest is the block, so N DocTests report N locations under one unsplittable node id - Record the three facts it rests on, each verified by running it: repr_failure reads locations per failure, _DocTestRunner__run is an ordinary attribute override, and an item-scoped mapping never crosses a process - Fix the vocabulary where doctest, pytest and Sphinx collide on globs, namespace, group, scope, skip and name - Tabulate the upstream constraints, each anchored at a pinned tag - Record what was rejected and why, including prefix replay, IntFlag option surfaces and a whole-file docstring - Stub the five deferred decisions as 0002 through 0006 --- docs/adrs/0001-typed-vanilla-doctest-core.md | 551 ++++++++++++++++++ .../0002-runner-conformance-across-cpython.md | 71 +++ ...0003-retiring-per-block-namespace-items.md | 54 ++ docs/adrs/0004-diagnostics-as-data.md | 63 ++ .../0005-line-recovery-for-nested-blocks.md | 59 ++ .../0006-pytest-private-api-compatibility.md | 81 +++ docs/adrs/index.md | 11 + 7 files changed, 890 insertions(+) create mode 100644 docs/adrs/0001-typed-vanilla-doctest-core.md create mode 100644 docs/adrs/0002-runner-conformance-across-cpython.md create mode 100644 docs/adrs/0003-retiring-per-block-namespace-items.md create mode 100644 docs/adrs/0004-diagnostics-as-data.md create mode 100644 docs/adrs/0005-line-recovery-for-nested-blocks.md create mode 100644 docs/adrs/0006-pytest-private-api-compatibility.md diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md new file mode 100644 index 0000000..51a4c9f --- /dev/null +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -0,0 +1,551 @@ +(adr-0001-typed-vanilla-doctest-core)= + +# ADR 0001: A typed, vanilla-compatible doctest core + +Status: Proposed +Date: 2026-08-02 + +## Context + +`doctest_docutils` re-implements the *finding* half of {func}`doctest.testfile` +over a docutils or MyST doctree, keeps CPython's *running* half, and adds a +grouping layer — namespace, phase, merge, lift — that exists in neither parent. +`pytest_doctest_docutils` wraps that in a pytest plugin. + +The two modules work. They are also the join point of three separate +vocabularies and three separate extension models, and the seams between them +were never named. Four structural costs follow. + +**The finder is one method.** `DocutilsDocTestFinder._find` performs format +dispatch, node filtering, group resolution, wildcard expansion, name-collision +detection, `:skipif:` evaluation, `testoutput` pairing, test construction, +merging, splitting and naming, in one body with two nested closures, with +configuration threaded through as positional parameters. There is no boundary +between *document → blocks*, *blocks → tests*, and *tests → items*, so a change +to any one of them is a change to all three. + +**It reaches through name-mangled CPython privates.** Sphinx's prompt-free +`{testcode}` form needs `compile()` in `"exec"` mode, but the per-example loop +[`DocTestRunner.__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) +hard-codes `"single"` +([`Lib/doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400)). +`sphinx.ext.doctest` solves this by rebinding `doctest.compile` process-wide and +never restoring it +([`sphinx/ext/doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310)) — +unavailable to a library that loads into every pytest session that installed it. +`_exec_mode_run` therefore clones the mangled method's code object into a fresh +{class}`types.FunctionType` whose globals map `compile` to a local helper, guarded +by probing `co_freevars` and `co_names` and degrading with a logged error. That +mechanism also carries a latent defect: its globals are a snapshot of +`vars(doctest)` taken at import, so a later rebind of a module-level name in +`doctest` is invisible to the clone while remaining visible to the stock runner, +and two runners in one process disagree. + +**It blocks the plugin whose internals it imports.** `pytest_configure` calls +`config.pluginmanager.set_blocked("doctest")`, and the same module then imports +that plugin's private helpers — `_is_setup_py`, `_is_main_py`, `_get_checker`, +`_get_continue_on_failure` — and reads four ini and CLI options the blocked +plugin declared. This survives only because `_pytest/fixtures.py` has no +`pytest_plugin_unregistered` handler, so the already-parsed `doctest_namespace` +fixture outlives unregistration. + +**It forks pytest-xdist to protect state xdist cannot see.** Because a live +`globs` mapping is a Python object and only execnet-serializable builtins cross a +worker boundary, a shared namespace must either be merged into one +{class}`doctest.DocTest` or kept on one worker. Keeping it there means a +scheduler, and the only affinity primitive in all of xdist is +[`LoadScopeScheduling._split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284), +a pure function on node-id strings. The controller never collects; it learns the +suite only as node ids arriving from workers. So `_shared_page` and `_is_page` +re-derive "these ids share state" from strings, and `_worker_count` re-implements +[`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26) +including its quirks. + +Underneath all four is one conflation. **The granularity of test identity and the +granularity of shared state are different axes, and the current design couples +them.** `namespace_items = merged` gives one `DocTest` and one node id per group; +`per-block` gives N `DocTest`s, N node ids, and one live mapping — a node id that +raises `NameError` when selected alone. Every surveyed project makes the same +mistake or a worse one (see [](#prior-art)). + +## Decision + +**The pytest item is the group. The {class}`doctest.DocTest` is the block.** + +One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block +`DocTest`s run in phase order against one live `globs` mapping that never leaves +the item. + +This occupies the cell no surveyed project occupies: N `DocTest`s, one node id. +Per-block failure locations, per-block gutters, per-block `SKIPPED`, per-block +"location unknown" — while `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode +are structurally incapable of splitting the shared state, because there is only +one item to schedule. + +### The three facts this rests on + +Each was verified by executing it, not by reading it. + +**1. pytest reads failure locations per failure, not per item.** +[`DoctestItem.repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317) +iterates the failure list and reads `failure.test.filename`, `failure.test.lineno` +and `example.lineno` inside the loop +([`_pytest/doctest.py:337-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L337-L344)): + +```python +for failure in failures: + example = failure.example + test = failure.test + filename = test.filename + if test.lineno is None: + lineno = None + else: + lineno = test.lineno + example.lineno + 1 +``` + +With one `DocTest` per block, every failure therefore carries its own `filename` +and `lineno` for free. A block reached through `.. include::` reports the +*included* file. A block docutils could not locate carries `lineno=None` and +takes pytest's honest `EXAMPLE LOCATION UNKNOWN` branch **without poisoning its +siblings**. No override of `repr_failure` or `reportinfo` is required. + +This is what retires `_merge_blocks`: the synthetic merged page, its blank-line +padding and its `max(..., len(lines))` clamp exist only to reconstruct locations +from a single spliced docstring. + +**2. `_DocTestRunner__run` is an ordinary attribute override.** Name mangling +rewrites the *call site* at compile time, so the `self.__run(...)` lookup inside +`run()` +([`Lib/doctest.py:1571`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1571)) +resolves through the instance's MRO like any other attribute. A subclass that +defines `_DocTestRunner__run` takes over the loop, with `run()` untouched. + +This page is itself a doctest, so the claim is checked on every run: + +```{doctest} +>>> import doctest +>>> import io +>>> fired = [] +>>> class Runner(doctest.DocTestRunner): +... def _DocTestRunner__run(self, test, compileflags, out): +... fired.append(test.name) +... return super()._DocTestRunner__run(test, compileflags, out) +>>> example = doctest.Example("1 + 1\n", "2\n") +>>> test = doctest.DocTest([example], {}, "demo", "demo.py", 0, None) +>>> Runner().run(test, out=io.StringIO().write) +TestResults(failed=0, attempted=1) + +The subclass method ran, and `run()` was never overridden: + +>>> fired +['demo'] +``` + +Keeping `run()` as stdlib's matters: it owns the save-and-restore of +`sys.stdout`, `pdb.set_trace`, `linecache.getlines`, `sys.displayhook`, +`_colorize.can_colorize` and the `PYTHON_COLORS`/`FORCE_COLOR` environment +variables, all in its own `finally` +([`Lib/doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +That contract is not reproduced; it is inherited. + +**3. Making the item the sharing unit dissolves distribution.** A live mapping +never crosses a process boundary, so there is nothing for xdist to split, under +any `--dist` mode. No affinity primitive, no scheduler substitution, no +`parse_tx_spec_config` fork, no node-id string sniffing. + +### Layers + +Dependencies flow one way, from the leaf toward the hosts. No layer may import a +layer above it. + +```text +blocks inert data: Block, BlockKind, Phase, Diagnostic, Example + | (stdlib imports only) +markup/ _rst, _myst, _text, _python -> (blocks, diagnostics) + | +project grouping, pairing, phase order, naming -> GroupTest + | +runner owns the per-example loop; phase sequencing + | +settings one frozen Settings, resolved once + | +pytest_doctest_docutils collection, items, globs lifetime, reporting +``` + +| Layer | Owns | Must not know | +|---|---|---| +| `blocks` | `Block`, `BlockKind` registry, `Phase`, `Diagnostic`, `BlockAttributes`, `Example(doctest.Example)` carrying compile mode | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | +| `markup/` | Text → `(blocks, diagnostics)`. Line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration | Groups as a runtime concept, `DocTest`, pytest, pairing | +| `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | +| `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | +| `settings` | One frozen `Settings` resolved once, with `None` sentinels at the resolve boundary so a future default change is announceable | pytest's `Config`, argparse, ini format, Sphinx's `app` | +| `pytest_doctest_docutils` | Options, `Document(pytest.Module)`, `DocutilsItem`, group `globs` lifetime, built-in-plugin dedup, surfacing diagnostics | docutils node classes, MyST configuration, grouping rules | + +`pytest_doctest_docutils._compat` is the only module that imports +`_pytest.doctest`, behind a pinned support matrix. See +{doc}`0006-pytest-private-api-compatibility`. + +### Vocabulary + +Goal (e) — speaking doctest's, pytest's *and* Sphinx's idioms — is mostly a +naming problem, because the three overload the same nouns with different +referents. Each term below is decided once and used only that way. + +| Term | doctest | pytest | Sphinx | Decision | +|---|---|---|---|---| +| `globs` | the dict examples exec in; [`DocTest.__init__` stores a **copy**](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | — | assigned to `test.globs` after construction, run with `clear_globs=False` | Keep `globs` for the mapping | +| namespace | — | [`doctest_namespace`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) means *injected names* | — | **Not** used for the sharing unit; pytest owns the word | +| group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | +| scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The block-vs-document axis becomes `share` | +| test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | +| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | pytest's meaning, doctest's mechanism. Sphinx's drop is deliberately rejected | +| directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | +| optionflags | an int bitmask; `register_optionflag` | `doctest_optionflags` ini | `:options:` plus `doctest_default_flags` | Keep verbatim. `register_optionflag` is the one genuinely cross-library extension point | +| setup / cleanup | `setUp`/`tearDown` on the suite builders | fixtures | `testsetup`/`testcleanup` directives | Author-facing names stay Sphinx's; `phase` is the internal ordering axis; a fixture is never "setup" | +| name | a dotted path; `__lt__` compares it as **text** | node id is `parent.nodeid + "::" + name` | the *group* name, shared by every block in it | `DocTest.name` is a unique machine-independent id. The absolute path lives only in `filename` | + +### Data model + +```python +class Phase(enum.IntEnum): + SETUP = 0 + TEST = 1 + CLEANUP = 2 + + +class Block(t.NamedTuple): + kind: str # registered BlockKind name + source: str # dedented body, verbatim + want: str | None # from a paired testoutput + path: pathlib.Path # the file the text lives in, not the collected document + line: int | None # None when docutils could not recover one + position: int # 0-based document pre-order index; the naming key + groups: tuple[str, ...] # declared verbatim; () and ("*",) unresolved here + options: t.Mapping[int, bool] # plain int keys, exactly as doctest produces + skipif: str | None # UNEVALUATED + hidden: bool + + +class BlockKind(t.NamedTuple): + name: str + phase: Phase + mode: t.Literal["single", "exec"] + pairs_with: str | None + grouped: bool + node_types: tuple[str, ...] # docutils tagnames; "comment" for testsetup/:hide: + option_spec: t.Mapping[str, t.Callable[[str], object]] + + +class GroupTest(t.NamedTuple): + group: str + tests: tuple[doctest.DocTest, ...] # one per block, in phase order + globs: dict[str, t.Any] # the live mapping, shared by every test above +``` + +`Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block +nested in a `.. note::`, a list item or a block quote reports `line=None, +source=None` from docutils, and an `.. include::`-ed block numbers against the +*included* file. Both propagate to `DocTest.lineno=None` and pytest's honest +"location unknown", rather than to a fabricated number. + +`BlockKind.node_types` is likewise load-bearing: `testsetup`, `testcleanup` and +any `:hide:` block are emitted as {class}`docutils.nodes.comment`, not +`literal_block` +([`sphinx/ext/doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93)). +A walker restricted to `literal_block` silently loses every one of them while the +page still renders. + +### Typing + +The runtime objects are stdlib's, unconditionally. Precision lives in a parallel +layer that never changes what is constructed. + +- **`Protocol` for the seams, nominal subclassing for the concrete classes.** + `Frontend` is a `Protocol` so a third party can supply one structurally; the + shipped parser also subclasses {class}`doctest.DocTestParser` so it stays + passable to `doctest.testfile(parser=...)` and + {func}`doctest.DocFileSuite`. Typeshed's signatures demand the class, not the + shape — `DocTestFinder.__init__(parser: DocTestParser = ...)` — so a compatible + `find()` alone is not enough, which is why `DocutilsDocTestFinder` cannot be + passed to `DocTestSuite(test_finder=...)` today. +- **`TypedDict` at the docutils boundary.** `BlockAttributes` types what a + directive stamps on a node, with one narrowing accessor that validates once. + This is where `Any` currently enters: `str(node.get("test") or ...)` and + `dict(node.get("options") or {})` are runtime coercions paid for a static hole, + and typeshed's docutils stub makes it worse — its `get(key, failobj: _T) -> _T` + claims `_T` even when the key is present holding something else. +- **`t.Literal` for closed vocabularies**, derived from one source of truth so a + public signature and a config field cannot diverge. +- **Plain `int` keys for optionflags.** {class}`enum.IntFlag` was considered and + rejected; see [](#alternatives-rejected). +- **`py.typed` ships.** The project already runs mypy strict over `src` and + `tests`, and the marker file does not exist, so every consumer sees the package + as untyped. This is a packaging defect independent of the rest of this ADR. + +## Constraints + +The design is pinned by facts about three upstreams. Each was verified at the tag +cited. The full derivation is in `notes/analyses/`. + +### CPython `doctest` (v3.14.2) + +| Constraint | Anchor | +|---|---| +| A failure's file line is `test.lineno + example.lineno + 1`; `Example.lineno` is 0-based within the containing string | [`doctest.py:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | +| `DocTest.__init__` **copies** the globs mapping, so a shared mapping must be assigned after construction and run with `clear_globs=False` | [`doctest.py:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | +| Never sort collected tests by name: `__lt__` compares names as text, and a name carries its position as text, so `page.md[10]` sorts before `page.md[1]` | [`doctest.py:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | +| The per-example loop is name-mangled; the supported in-loop seams are the four `report_*` methods and the injected checker | [`doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | +| `run()` mutates global interpreter state for its duration and restores in `finally`; it is neither reentrant nor thread-safe | [`doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) | +| Each example compiles under `""` in `"single"` mode with `dont_inherit=True`; `"exec"` suppresses expression echo, emptying every `want` | [`doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | +| `TestResults` is a 2-field namedtuple carrying `skipped` as an extra instance attribute; a third tuple field breaks every `failures, tries = runner.run(...)` unpack | [`doctest.py:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) | +| Custom flag names must be registered at import; ints are `1 << len(OPTIONFLAGS_BY_NAME)` and an unregistered name makes a page fail to **parse** | [`doctest.py:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) | + +`report_skip` does not exist at v3.14.2 — the runner has only `report_start`, +`report_success`, `report_failure` and `report_unexpected_exception`. A runner +that owns the loop must probe for it rather than assume it. + +### pytest (9.1.1) + +| Constraint | Anchor | +|---|---| +| `repr_failure` reads each failure's own `test` — the fact this design is built on | [`doctest.py:317-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) | +| `DoctestItem.setup()` does `self.dtest.globs.update(globs)`, so the mapping must be mutable and survive collection → setup → run | [`doctest.py:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | +| `runtest()` calls `run(self.dtest, out=failures)` with `clear_globs` defaulting to `True` — which would empty a shared mapping after the first block | [`doctest.py:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | +| `PytestDoctestRunner` is defined *inside* `_init_runner_class()` and is not importable, so the `OutcomeException` re-raise, `BdbQuit` → `outcomes.exit` and `continue_on_failure` handling do **not** come for free | [`doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181) | +| A page must be a `pytest.Module` with `obj = None`, and the collector must call `parsefactories` | [`doctest.py:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421), [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | +| `_is_doctest` claims any `.txt`/`.rst` **initial path before consulting `--doctest-glob`**, so `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob | [`doctest.py:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | +| An empty `DocTest` must not be yielded as an item | [`doctest.py:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | + +### pytest-xdist (v3.8.0) + +| Constraint | Anchor | +|---|---| +| Every worker must collect identical node ids in identical order. Violation is not an exception — the scheduler logs `**Different tests collected, aborting run**` and the session executes zero tests | [`load.py:259`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259), [`loadscope.py:359`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) | +| The only affinity primitive is `_split_scope(nodeid) -> str`; `loadfile` and `loadgroup` are two-line overrides of it, and `load`/`worksteal` have no scope concept at any layer | [`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284), [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) | +| `xdist_group` is honoured only when the *worker's own* `--dist` is `loadgroup`, and works by appending `@name` to `item._nodeid` | [`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) | +| `parse_tx_spec_config` builds a list, so a negative multiplier contributes zero specs — `xspeclist.extend([spec] * num)`, not a sum | [`workermanage.py:26-37`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) | + +Making the item the sharing unit removes the need to satisfy the first three at +all. The fourth is why `_worker_count` is deleted rather than fixed. + +### Sphinx (v9.1.0) + +| Constraint | Anchor | +|---|---| +| `testsetup`, `testcleanup` and `:hide:` blocks are emitted as `nodes.comment` | [`doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | +| A `:skipif:`-gated node is dropped during collection, with no outcome, id or count | [`doctest.py:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | +| `:options:` is accepted only on `doctest` and `testoutput`; a `testcode`'s own options are discarded in favour of its output block's | [`doctest.py:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111), [`:207`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) | +| `is_allowed_version(spec, version)` takes the specifier **first** — the reverse of this project's local helper | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | +| `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310), [`:548`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) | + +## Tensions + +Each is a genuine conflict where satisfying one goal costs another. "Both" is not +an answer; the position taken and its price are recorded. + +**A vanilla `DocTest` cannot carry the front-end's metadata.** It has exactly +`(examples, globs, name, filename, lineno, docstring)`. *Position:* metadata that +the runner needs rides on a nominal {class}`doctest.Example` subclass — +`doctest.Example` has no `__slots__`, so attributes survive +{func}`copy.copy`, {mod}`pickle`, and a third party's naive +`DocTest(examples, globs, name, filename, lineno, docstring)` rebuild, because +that rebuild reuses the same `Example` objects. Metadata the runner does *not* +need — groups, wildcards, pairing — never touches a stdlib object and dies in the +projection layer. *Price:* one subclass to explain, and a rule that nothing may +smuggle through `DocTest.name`. + +**Node-id granularity versus shared state.** *Position:* decouple them — N +`DocTest`s under one node id. *Price:* selecting a group runs all its blocks; +there is no id that names block three alone. That is honest: no surveyed +implementation makes a node id a promise of independent runnability, and the +current `per-block` mode ships ids that raise `NameError` when selected. + +**Sphinx's skip versus pytest's skip.** *Position:* pytest's meaning, doctest's +mechanism — set `SKIP`, never drop the node. *Price:* a page carrying a gated +block gains an item relative to `sphinx-build -b doctest`, and `--collect-only` +must not evaluate the gate, which is why `:skipif:` is carried through collection +unevaluated and run in `runtest()`. + +**Collection purity versus `--collect-only` fidelity.** *Position:* collection +evaluates no user Python, so it is a pure function of (bytes, argv, ini). *Price:* +`--collect-only` no longer shows which blocks will skip. + +**Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards +an orphan `testoutput`, silently overwrites a duplicate one, and silently drops a +`testcode`'s own `:options:`. *Position:* keep the behaviour, add a diagnostic +with a stable code. *Price:* a page warns under pytest and is silent under +`sphinx-build`; the results still match. + +**Owning the loop versus tracking CPython.** *Position:* own it, because +constraint 4 makes it the only way to control compile mode without a +process-global patch or a code-object clone. *Price:* a version shim and a +conformance harness. See {doc}`0002-runner-conformance-across-cpython`. + +## What this deletes + +`_find`'s monolith · `_merge_blocks` and its padding and clamp · +`_split_skipped_blocks` and `_lifted_name` · `_exec_mode_run`, `_compile_source` +and `_ExecSource` · `_worker_count` · `_shared_page` and `_is_page` · +`_splitting_scheduler` and both xdist hooks · the `_init_runner_class` fork · +`set_blocked("doctest")` and its unblock path. + +**The result is not smaller.** It lands roughly flat against the current source. +Under this project's rule that every function carries a NumPy docstring with a +working doctest, splitting one long method into five functions costs prose it did +not previously pay. The value is fewer hazards, not fewer lines: a code-object +clone with a stale globals snapshot, a fork of an upstream parser, node-id string +sniffing, and a method that branches on string literals to decide what a block is +all disappear. Gate on shape instead — a function-length ceiling, a module-length +ceiling, and the `import-linter` contract on the leaf. + +(prior-art)= + +## Prior art + +| Project | Bet | Outcome | +|---|---|---| +| [Sybil 10.0.1](https://github.com/simplistix/sybil/tree/10.0.1) | A document is a flat sequence of non-overlapping character spans; every format is a regex lexer; zero runtime dependencies | Format independence at no dependency cost, and a non-overlap invariant that raises on double collection. But one mutable namespace per document with [one independently selectable item per span](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py), so `-k` on a later example raises `NameError`. [Node ids are positional](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (`line:4,column:1`), so adding a paragraph renames every downstream test. No group support at all — a regex cannot see directive options | +| [xdoctest v1.3.2](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | Abandon stdlib compatibility; own the parser via `ast`/`tokenize`; make directives structured objects | `ast` parsing and structured directives are real advances — [`REQUIRES`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) carries *why* a block skipped, which a bool cannot. But it is now building compatibility back, and its permissive got/want defaults silently change tests users wrote for stdlib. It unregisters pytest's doctest plugin outright | +| [pytest-examples v0.0.18](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | Emit the canonical form rather than parse it; rewrite expected output in place | Check-mode and update-mode collapse into one path, and byte offsets plus an invertible dedent scalar are what a data model needs to rewrite source. Composes with pytest by contributing no collector at all — the cheapest correct integration in the survey | +| [typeshed](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | Annotate the 2001 API faithfully | Hands back `Any` at exactly the three extensible points — `globs`, `**options`, `optionflags: int`. Declares `DocTestRunner.test: DocTest` unconditionally although runtime assigns it only inside `run()`, so the stub type-checks a crash | + +The collective lesson: **namespace scope is not test identity, markup parsing is +not Python parsing, and runtime compatibility is not static precision.** Every +project conflated at least two, and the first conflation is the one that produces +silent wrong answers rather than inconvenience. + +(alternatives-rejected)= + +## Alternatives rejected + +**Prefix replay** — re-executing a group's predecessors so any block can be +selected standalone. It rebinds `getfixture` to the *replaying* item's request, +so a replayed predecessor resolves different fixture instances: a results +difference, not a performance one. It re-executes gated and deselected blocks +with no node id, in exactly the environment the gate says they must not run in. +It is superlinear precisely where shared groups exist. And it assumes +idempotence, while downstream setup blocks spawn servers and create +repositories. + +**One `DocTest` per group whose `docstring` is the whole file.** Attractive — +`lineno=0` and true file lines with no padding — but broken at the two shapes +docutils cannot locate. A nested block reports `line=None`, and an included file +numbers against itself, so one group-wide docstring maps both to confidently +wrong lines, and pytest's per-`DocTest` "location unknown" signal becomes +structurally inexpressible. Per-block `DocTest`s deliver the same benefits with +none of this. + +**{class}`enum.IntFlag` as the optionflag surface.** The runtime hash equality is +real and irrelevant to the claims made from it. With a third-party flag +registered — and pytest lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES` and +`NUMBER` in [`_get_flag_lookup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385) — +iteration and `repr` silently under-report members outside the enum. And +`Mapping` is invariant in its key type, so `Mapping[Flag, bool]` fails mypy +strict against `dict[int, bool]` in both directions, requiring casts at exactly +the boundaries the change was supposed to clean. + +**A registered `EXEC` optionflag to carry compile mode.** It would put compile +mode in the same user-writable namespace as `ELLIPSIS`. A `doctest_optionflags = +EXEC` in any ini, or a stray `# doctest: +EXEC`, compiles in exec mode, which +suppresses expression echo so every `want` compares against empty output and the +suite passes vacuously. Compile mode is an attribute on the `Example` subclass, +unreachable from user configuration. + +**Sybil's non-overlap-raises invariant.** Two `.. include::` directives naming +the same file produce blocks over identical source spans. That is a legitimate +page, and an invariant that raises would reject it. Double collection becomes a +diagnostic carrying both provenances. + +**Reimplementing `_pytest.doctest`'s helpers instead of importing them.** +Mis-costed by roughly threefold: `_get_checker` alone returns a checker +implementing `ALLOW_UNICODE`, `ALLOW_BYTES` and `NUMBER` with float-precision +handling. Since the plugin still reads `doctest_optionflags` from the built-in +plugin, a project setting `NUMBER` would either raise or — worse — have the bit +accepted and silently do nothing. + +**A Sphinx builder.** New scope in a rewrite that must not grow, and both drafts +that proposed one had it deliberately diverging from `sphinx-build -b doctest` on +`:skipif:` and the silent-loss cases. Shipping a builder that disagrees with the +tool it replaces is worse than shipping none. + +**An import-time guard that raises.** A `pytest11` plugin raising at import +aborts the whole session, taking down suites whose majority of tests never touch +a doctest. It also checks the wrong property: `hasattr(DocTestRunner, +"_DocTestRunner__run")` is true in exactly the scenario it claims to prevent, +while the thing that actually changed within the supported range — +`__record_outcome`'s arity — is invisible to it. This becomes a differential +conformance test in CI. + +## Consequences + +### Positive + +- Failure locations are correct by construction, including through `.. include::`, + with no `repr_failure` or `reportinfo` override. +- A block docutils cannot locate degrades to an honest disclaimer instead of a + fabricated line, and does not affect its siblings. +- Every `--dist` mode works, because there is no shared state to split. +- No CPython code-object clone, and no process-global rebinding of anything. +- Collection is a pure function of (bytes, argv, ini), so worker divergence is + structurally impossible and `--collect-only` runs no user code. +- Grouping is one pure function with no docutils, pytest or filesystem dependency, + and is testable without any of them. +- A new block kind is a registration, not an edit to a method branching on string + literals. +- Parse diagnostics become values with stable codes rather than stderr writes and + mid-parse aborts. + +### Tradeoffs + +- The per-example loop is this project's to maintain across supported + interpreters, including two version-shaped divergences. +- A page containing Sphinx `{testcode}` blocks produces `DocTest`s a stock runner + cannot run, because `compile("a = 1\nb = 2\n", "", "single")` raises. + Prompt-form blocks — the overwhelming majority — run perfectly on an unmodified + runner, and a test asserts it. +- The plugin now *requires* the built-in doctest plugin rather than blocking it, + so `-p no:doctest` is an error rather than a degraded mode. +- Retiring `namespace_items = per-block` is a real feature removal. +- The line count does not fall. + +### Risks + +**Runner drift.** A CPython refactor that inlines the loop into `run()` would +silently route execution back to stdlib — invisible for prompt-form blocks, +immediately broken for `{testcode}`. Mitigated by the conformance harness in +{doc}`0002-runner-conformance-across-cpython`, gating on capability probes rather +than a `sys.version_info` ladder. + +**pytest private API.** Four private helpers and a subclassed item. Mitigated by +quarantining them in one module behind a pinned matrix; see +{doc}`0006-pytest-private-api-compatibility`. + +**Foreign directive registration.** `Sphinx.add_directive` overrides existing +registrations unconditionally, so `sphinx.ext.doctest` loaded in the same +interpreter can replace these directive classes. Mitigated by reading +`BlockAttributes` off the node — byte-compatible with what Sphinx stamps — rather +than depending on this project's own classes having run. + +**Over-suppressed diagnostics.** Suppressing one code too many turns a broken page +into a silent zero-test page, which is worse than a mid-parse abort. Mitigated by +the narrow default set in {doc}`0004-diagnostics-as-data`. + +## Relationship to other ADRs + +This ADR fixes the architecture. Five decisions it defers get their own records: +{doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven +equivalent), {doc}`0003-retiring-per-block-namespace-items` (the deprecation +path), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed), +{doc}`0005-line-recovery-for-nested-blocks` (the optional last step), and +{doc}`0006-pytest-private-api-compatibility` (the quarantine and its matrix). + +## Final position + +The core produces real {class}`doctest.DocTest` objects holding real +{class}`doctest.Example` objects, and `Example.source` is the author's verbatim +text. Everything else — groups, phases, pairing, diagnostics, distribution — is a +layer above that fact, and no layer reaches around another. + +The unit that shares a `globs` mapping is the unit pytest schedules. That is the +one invariant every other property in this document follows from, and it is not +negotiable for a convenience elsewhere. diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md new file mode 100644 index 0000000..bd7882e --- /dev/null +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -0,0 +1,71 @@ +(adr-0002-runner-conformance-across-cpython)= + +# ADR 0002: Runner conformance across CPython versions + +Status: Draft +Date: 2026-08-02 + +## Context + +{doc}`0001-typed-vanilla-doctest-core` decides that the runner owns the +per-example loop by defining `_DocTestRunner__run` in a subclass, rather than +cloning CPython's code object or rebinding `doctest.compile` process-wide. + +Owning the loop means owning the private state it writes into, and that state has +changed shape inside this project's supported interpreter range. Three +divergences are known: + +**The outcome accumulator changed name and arity.** On 3.10 through 3.12 it is +`__record_outcome(self, test, f, t)` writing into `self._name2ft`; on 3.13 and +later it is `__record_outcome(self, test, failures, tries, skips)` writing into +`self._stats` +([`Lib/doctest.py:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485)). +A loop that calls the wrong one leaves `summarize()` reporting zeros for a +passing file — a silent, total failure of the reporting path. + +**`TestResults` gained a third value that is not a tuple field.** It carries +`skipped` as an extra instance attribute +([`Lib/doctest.py:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), +so `TestResults(f, a, skipped=s)` works on 3.13+ and raises on earlier versions. + +**`report_skip` does not exist at v3.14.2.** The runner has only `report_start`, +`report_success`, `report_failure` and `report_unexpected_exception` +([`Lib/doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)). +It appears in later prereleases, so a loop must probe rather than assume in +either direction. + +A fourth risk has no current instance but would be silent: a CPython refactor +that inlines the loop into `run()` would route execution back to stdlib. That is +invisible for prompt-form blocks and immediately broken for `{testcode}`. + +## Question + +How is an owned per-example loop proven equivalent to the interpreter's own, +continuously, without a `sys.version_info` ladder? + +## Direction + +A differential conformance harness, run in CI on every supported interpreter, +gating the build step that lands the runner. + +A fixed case matrix — pass, fail, unexpected exception, `SyntaxError`, all +examples skipped, partially skipped, `FAIL_FAST`, `REPORT_ONLY_FIRST_FAILURE`, +`IGNORE_EXCEPTION_DETAIL` — is run through both this runner and a stock +{class}`doctest.DocTestRunner`, asserting equality of `TestResults`, the captured +`report_*` text, `summarize()` output, and the accumulator contents. + +Version handling is by capability probe, never by version comparison, so a +backport, a vendored interpreter or a fork behaves correctly rather than by +coincidence. {doc}`0001-typed-vanilla-doctest-core` rejects an import-time guard +that raises: a `pytest11` plugin that aborts at import takes down suites whose +majority of tests never touch a doctest. + +## Open + +- Whether the harness asserts on `report_*` text verbatim, or on a normalized + form — verbatim is stricter and will churn when CPython adjusts wording. +- Whether a probe failure degrades to stdlib's loop with a diagnostic, or fails + the affected items loudly. Degrading is silent for prompt-form blocks, which is + the argument against it. +- The floor: whether supporting 3.10's `_name2ft` shape is worth its shim once + that version reaches end of life. diff --git a/docs/adrs/0003-retiring-per-block-namespace-items.md b/docs/adrs/0003-retiring-per-block-namespace-items.md new file mode 100644 index 0000000..e9c7787 --- /dev/null +++ b/docs/adrs/0003-retiring-per-block-namespace-items.md @@ -0,0 +1,54 @@ +(adr-0003-retiring-per-block-namespace-items)= + +# ADR 0003: Retiring `namespace_items = per-block` + +Status: Draft +Date: 2026-08-02 + +## Context + +`doctest_docutils_namespace_items` selects between `merged` (the default: a group +collects as one test) and `per-block` (each block keeps its own node id while the +blocks share one live `globs` mapping). + +{doc}`0001-typed-vanilla-doctest-core` makes the setting unnecessary by +decoupling the two axes it was invented to trade between: one pytest item per +group, with one {class}`doctest.DocTest` per block inside it. That gives +`per-block`'s per-block failure locations, gutters and `SKIPPED` reporting +without `per-block`'s shared live mapping. + +It also removes `per-block`'s cost. A live mapping is a Python object, so it +neither crosses a worker process nor survives an item running twice. Guarding it +is the sole reason `_worker_count`, `_shared_page`, `_is_page`, +`_splitting_scheduler` and both xdist hooks exist. + +The setting's remaining distinguishing feature is a node id per block — and those +ids raise `NameError` when selected alone, because selecting one block does not +run the blocks that bound the names it reads. A node id that cannot be selected +is not a node id. + +## Question + +`per-block` is a shipped CLI option with an ini twin and its own how-to sections. +How does it get retired without breaking a user who set it? + +## Direction + +Announce it as a breaking change with the migration path stated plainly, and keep +the option accepting `per-block` for one minor release, mapped to `merged` with a +{class}`pytest.PytestDeprecationWarning`. + +Before landing, confirm no downstream consumer sets it. +`--doctest-docutils-modules` and `--no-doctest-docutils-modules` keep their exact +current spelling and `dest`, because downstream projects carry them in `addopts`. + +## Open + +- Whether the deprecation shim warns once per session or once per page. +- Whether `--doctest-docutils-namespace-scope` is renamed to `--doctest-docutils-share` + in the same release or a later one. {doc}`0001-typed-vanilla-doctest-core` + decides the vocabulary — `scope` is reserved for pytest's fixture-lifetime + ladder — but the rename is independently schedulable. +- Whether the `pytest11` entry point rename (from `sphinx`, which is what + `-p no:sphinx` disables today) ships in the same release as this removal or its + own. diff --git a/docs/adrs/0004-diagnostics-as-data.md b/docs/adrs/0004-diagnostics-as-data.md new file mode 100644 index 0000000..0eb82a1 --- /dev/null +++ b/docs/adrs/0004-diagnostics-as-data.md @@ -0,0 +1,63 @@ +(adr-0004-diagnostics-as-data)= + +# ADR 0004: Diagnostics as data + +Status: Draft +Date: 2026-08-02 + +## Context + +Parsing a page currently writes docutils reporter output straight to stderr, +interleaved with pytest's own output and attributable to nothing. Two failure +modes follow from the default settings. + +A level-4 message raises `SystemMessage` mid-parse and aborts collection of the +file, so one malformed construct takes down a page whose other blocks are fine. + +More quietly, a `.. doctest::` block carrying an unknown option collects **zero** +tests and the session exits green. A page that checks nothing reports the same +way as a page that passes. + +{doc}`0001-typed-vanilla-doctest-core` gives the front-end layer a second return +value for this: `Diagnostic(level, code, message, path, line)`, produced by +setting `halt_level` past the abort threshold and attaching a reporter observer. +Suppression and promotion key on the stable `code`, never on message text. + +## Question + +Which diagnostics are shown by default? + +The naive answer — show everything — was measured against this project's own +`docs/` and produces well over a hundred messages per run, almost all of them +`Unknown interpreted text role` and `Unknown directive type` for roles and +directives that Sphinx supplies and a bare-docutils parse structurally cannot +resolve. Those are false positives. Emitting them is noise-as-policy, and users +would learn to ignore the channel that also carries real errors. + +The opposite error is worse: suppressing one code too many turns a broken page +into a silent zero-test page, which is the exact condition this ADR exists to +surface. + +## Direction + +Suppress by code, narrowly, and only for the two classes a bare-docutils parse +cannot judge: unknown roles and unknown directives. + +Every diagnostic raised by this project's own layers defaults to visible, and +`level="error"` from those layers fails collection with the file and line named. +A page whose only block fails to parse, and a page with a malformed `:options:` +value, must both produce a collection error rather than collecting nothing and +passing. + +Expose promotion and suppression by code so a project can tune the set without a +global on/off switch. + +## Open + +- Whether diagnostics surface as {class}`pytest.PytestWarning` subclasses, giving + `-W error::` control for free, or as a dedicated report section. +- Whether an unknown-directive diagnostic should be promoted when the directive + name is one this project registers — that case is not Sphinx supplying it, it + is a registration that did not happen, which is the GH-48 failure mode. +- Whether the CLI (`python -m doctest_docutils`) and the pytest plugin share one + formatter or two. diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md new file mode 100644 index 0000000..f24616b --- /dev/null +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -0,0 +1,59 @@ +(adr-0005-line-recovery-for-nested-blocks)= + +# ADR 0005: Exact line recovery for nested blocks + +Status: Draft +Date: 2026-08-02 + +## Context + +docutils does not report a usable line for every node, and what it does report +differs by front-end. + +A bare `>>>` block nested in a `.. note::`, a list item, a block quote or a +`{tab}` directive reports `line=None, source=None`. A top-level reStructuredText +`doctest_block` reports its **last** line; a MyST fence reports its **first**. An +`.. include::`-ed block numbers against the *included* file. + +{doc}`0001-typed-vanilla-doctest-core` handles all three honestly rather than +approximately. `Block.line` is nullable, the per-front-end meaning is normalized +inside the front-end that knows it, `Block.path` carries the file the text +actually lives in, and a block with no recoverable line propagates +`DocTest.lineno=None` into pytest's `EXAMPLE LOCATION UNKNOWN` branch. + +That is correct but not maximal. A nested block's failure says the location is +unknown when the information exists in the parser and was discarded. + +## Question + +Can the exact line be recovered for a nested block without a fragile dependency? + +The mechanism identified is per-instance substitution of +`docutils.parsers.rst.Parser.state_classes`. `state_classes` is an instance +attribute, so substitution is fully scoped to one parse with no process-global +mutation — unlike the directive registry, which has no such scoping. + +It also depends on undocumented docutils structure, and a refactor upstream would +break it as `None` spans rather than as an exception. + +## Direction + +Land it last, as a strictly optional improvement behind a feature probe. + +When the expected structure is absent, fall back to `node.line` normalization and +emit `line=None`. Degrading to an honest disclaimer is acceptable; degrading to a +fabricated number is not, and that is the whole reason this is separable from +{doc}`0001-typed-vanilla-doctest-core` rather than part of it. + +Pin with tests asserting exact `(path, line)` for a bare block nested in each of +the four constructs, plus a test that forces the probe to fail and asserts the +fallback yields `lineno=None`. + +## Open + +- Whether the same technique gives MyST nested blocks anything, or whether + markdown-it's token stream already carries enough. +- Whether `.. include::` line attribution needs the same treatment or is already + correct once `Block.path` is respected. +- Whether the probe result is reported anywhere, so a user can tell which mode + produced a given report. diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md new file mode 100644 index 0000000..943e9d9 --- /dev/null +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -0,0 +1,81 @@ +(adr-0006-pytest-private-api-compatibility)= + +# ADR 0006: pytest private API compatibility + +Status: Draft +Date: 2026-08-02 + +## Context + +The plugin currently calls `config.pluginmanager.set_blocked("doctest")` and then +imports that same blocked plugin's private helpers, while continuing to read four +ini and CLI options the blocked plugin declared. It works only because +`_pytest/fixtures.py` has no `pytest_plugin_unregistered` handler, so the +already-parsed `doctest_namespace` fixture outlives unregistration. That is an +undocumented behaviour bet on for every collected page. + +{doc}`0001-typed-vanilla-doctest-core` inverts the relationship: the built-in +doctest plugin is not blocked, it is *required*. Its checker, its failure repr, +its `--doctest-report` formatting and `doctest_namespace` are all worth keeping, +and reimplementing them was measured as costing roughly three times what it was +budgeted at — `_get_checker` alone returns a checker implementing +`ALLOW_UNICODE`, `ALLOW_BYTES` and `NUMBER` with float-precision handling +([`_pytest/doctest.py:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662)). + +Not blocking it exposes a live defect the block was masking. +[`_is_doctest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) +claims any `.txt` or `.rst` **initial path before consulting `--doctest-glob`**: + +```python +def _is_doctest(config: Config, path: Path, parent: Collector) -> bool: + if path.suffix in (".txt", ".rst") and parent.session.isinitpath(path): + return True + globs = config.getoption("doctestglob") or ["test*.txt"] + return any(fnmatch_ex(glob, path) for glob in globs) +``` + +So `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob +configuration, and `pytest_collect_file` is not `firstresult` — the directory +collector yields the results of *every* implementation for a path. Declining the +path is therefore not sufficient; the duplicate has to be removed. + +## Question + +What private surface is depended on, and how does a pytest release that changes +it fail? + +The current surface is `_get_checker`, `get_optionflags`, +`_get_continue_on_failure`, `_get_report_choice`, `MultipleDoctestFailures`, and +a `pytest.DoctestItem` subclass. `_init_runner_class` is explicitly *not* usable: +`PytestDoctestRunner` is defined inside it +([`_pytest/doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181)) +and is unreachable by name, which is why the `OutcomeException` re-raise, +`BdbQuit` → `outcomes.exit` and `continue_on_failure` handling must be +reimplemented rather than inherited. + +## Direction + +Quarantine every private import in one module, `pytest_doctest_docutils._compat`, +with a pinned support matrix and an import-time probe that raises a named error at +plugin registration — naming the pytest version and the missing symbol — rather +than failing somewhere in the middle of collection. + +Claim `.rst` and `.md` paths and deselect the built-in plugin's duplicate items in +a `pytest_collection_modifyitems` hookwrapper, firing `pytest_deselected` so the +reported counts stay consistent. + +CI carries a job pinned to the minimum supported pytest and one tracking its +prerelease. + +## Open + +- Whether requiring the built-in plugin should be stated as a hard dependency. + `-p no:doctest` already fails today with a raw `ValueError` about an unknown + option, so this is not a regression — but the message should become actionable. +- Whether deselecting is preferable to narrowing the built-in's globs. Deselecting + is explicit and reportable; narrowing is quieter but mutates another plugin's + configuration. +- Whether the probe should accept a *newer* pytest it has not been tested against, + or refuse it. Refusing is safer and more annoying. +- Whether any of these helpers can be promoted upstream, which would delete the + quarantine entirely. diff --git a/docs/adrs/index.md b/docs/adrs/index.md index 273990e..ed763f1 100644 --- a/docs/adrs/index.md +++ b/docs/adrs/index.md @@ -25,3 +25,14 @@ tag, or a commit reachable from that project's trunk where it publishes no tags. Line anchors are only used on a pinned ref, because they are meaningless without one, and a `blob/master` link rots silently — the file moves, lines shift, and the anchor lands on unrelated code while still resolving. + +```{toctree} +:maxdepth: 1 + +0001-typed-vanilla-doctest-core +0002-runner-conformance-across-cpython +0003-retiring-per-block-namespace-items +0004-diagnostics-as-data +0005-line-recovery-for-nested-blocks +0006-pytest-private-api-compatibility +``` From 0eac0a7c8bad6d6b5d85da0f2eebb568596480f8 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 10:11:17 -0500 Subject: [PATCH 03/24] notes(analyses): Add the structural research behind ADR 0001 why: ADR 0001 asserts constraints about six upstream projects. Those assertions need a reviewable derivation, or the next person to question one has to redo the reading. Keeping it beside the record rather than inside it also keeps the record about the decision. what: - Add a per-system structural doc for CPython doctest, _pytest.doctest, pytest-xdist, pytest-asyncio, sphinx.ext.doctest, and docutils with myst-parser, each in the same section order so they compare - Add asyncio as the stdlib's own worked example of a pluggable architecture, for contrast with doctest's four seams - Add prior art on Sybil, xdoctest, pytest-examples and typeshed - Add cross-cutting docs on data structures, data flows, extension seams, and the namespace-scope versus test-identity axis - Add a bibliography collecting every anchor in one place - Pin every citation to a tag, or to a trunk-reachable commit where a project publishes none --- notes/analyses/00-taxonomy.md | 79 +++++++++ notes/analyses/10-cpython-doctest.md | 151 ++++++++++++++++ notes/analyses/11-pytest-doctest.md | 127 +++++++++++++ notes/analyses/12-pytest-xdist.md | 136 ++++++++++++++ notes/analyses/13-pytest-asyncio.md | 112 ++++++++++++ notes/analyses/14-asyncio.md | 109 ++++++++++++ notes/analyses/15-sphinx-ext-doctest.md | 127 +++++++++++++ notes/analyses/16-docutils-myst.md | 130 ++++++++++++++ notes/analyses/17-prior-art.md | 167 ++++++++++++++++++ notes/analyses/20-data-structures.md | 86 +++++++++ notes/analyses/21-data-flows.md | 113 ++++++++++++ notes/analyses/22-extension-seams.md | 106 +++++++++++ .../23-namespace-scope-and-test-identity.md | 127 +++++++++++++ notes/analyses/90-bibliography.md | 162 +++++++++++++++++ notes/analyses/README.md | 69 ++++++++ 15 files changed, 1801 insertions(+) create mode 100644 notes/analyses/00-taxonomy.md create mode 100644 notes/analyses/10-cpython-doctest.md create mode 100644 notes/analyses/11-pytest-doctest.md create mode 100644 notes/analyses/12-pytest-xdist.md create mode 100644 notes/analyses/13-pytest-asyncio.md create mode 100644 notes/analyses/14-asyncio.md create mode 100644 notes/analyses/15-sphinx-ext-doctest.md create mode 100644 notes/analyses/16-docutils-myst.md create mode 100644 notes/analyses/17-prior-art.md create mode 100644 notes/analyses/20-data-structures.md create mode 100644 notes/analyses/21-data-flows.md create mode 100644 notes/analyses/22-extension-seams.md create mode 100644 notes/analyses/23-namespace-scope-and-test-identity.md create mode 100644 notes/analyses/90-bibliography.md create mode 100644 notes/analyses/README.md diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md new file mode 100644 index 0000000..0f6bf22 --- /dev/null +++ b/notes/analyses/00-taxonomy.md @@ -0,0 +1,79 @@ +# Taxonomy: the axes a doctest engine is classified on + +Nine axes. Every system in these notes takes a position on each, and most of the +disagreements between them reduce to a different position on one axis rather than +a different philosophy. + +## The axes + +| # | Axis | Positions | +|---|---|---| +| 1 | **Sharing unit vs. selection unit** | same object · different objects, acknowledged · different objects, unacknowledged | +| 2 | **Test identity** | author-declared name · symbol-derived · positional (line/column or byte range) | +| 3 | **Runtime object model** | stdlib `DocTest`/`Example` · own model with a bridge · own model, no bridge | +| 4 | **Document model** | real parse tree · flat character spans · regex over text · none | +| 5 | **Option representation** | `int` bitmask · structured state · enum | +| 6 | **Extension mechanism** | callable aliases · nominal subclassing · named registry · `Protocol` · none | +| 7 | **Relationship to pytest's doctest plugin** | compose · block/unregister · replace by instruction · no collector | +| 8 | **Got/want strictness** | stdlib defaults · permissive defaults · no want at all | +| 9 | **Direction of data** | read-only · read plus write-back | + +## Where each system sits + +| System | 1 sharing/selection | 2 identity | 3 object model | 4 document model | +|---|---|---|---|---| +| CPython `doctest` | same (one `DocTest` per docstring) | dotted symbol path | *is* the model | none — line regex over a string | +| `_pytest.doctest` | same (one item per `DocTest`) | `path::module.qualname` | stdlib, unchanged | none — delegates | +| `sphinx.ext.doctest` | same (one group, one runner pass) | group name, shared by every block | stdlib, constructed per group | real doctree | +| Sybil | **different, unacknowledged** | positional `line:N,column:N` | stdlib `Example`, one-line `DocTest` fork | flat character spans | +| xdoctest | same (one `DocTest` per docstring) | `Callname:N` | own, with a late bridge back | none for `.rst`/`.txt` | +| pytest-examples | none — no implicit sharing | positional `path:start-end` | none | regex over fences | +| `doctest_docutils` today | configurable; `per-block` is different-and-guarded | group name, or `page.md[k]` | stdlib | real doctree | +| ADR 0001 | **different, decoupled by construction** | group name, or `page.md[k]` | stdlib | real doctree | + +| System | 5 options | 6 extension | 7 vs. pytest doctest | 8 strictness | 9 direction | +|---|---|---|---|---|---| +| CPython `doctest` | `int` bitmask + registry | nominal subclassing | n/a | strict | read-only | +| `_pytest.doctest` | `int` + name lookup | subclass its classes | *is* it | strict | read-only | +| `sphinx.ext.doctest` | `int` via `:options:` | directive subclassing | unaware | strict | read-only | +| Sybil | `int` | callable aliases | replace by instruction (`-p no:doctest`) | strict | read-only | +| xdoctest | structured `TypedDict` + bridge | two registries, else fork | unregisters it | **permissive** | read-only | +| pytest-examples | n/a | none | no collector — composes trivially | no want at all | **write-back** | +| `doctest_docutils` today | `int` + a forked lookup | directive subclassing | blocks it, imports its privates | strict | read-only | +| ADR 0001 | `int` + registry | `Protocol` + nominal + `BlockKind` registry | **compose; require it** | strict | read-only | + +## What the matrix shows + +**Axis 1 is the only one where a wrong answer is silent.** Every other axis +produces inconvenience — a renamed test, a conversion layer, an extra knob. Axis 1 +produces a `NameError` in a test the user believed they could select, or a false +green under `--reruns`. Sybil sits in the unacknowledged column and its +documentation never mentions it. + +**Axes 1 and 2 are independent, and everyone treated them as one.** The full +product space is four cells: + +| | one node id | N node ids | +|---|---|---| +| **one `DocTest`** | Sphinx, `doctest_docutils` `merged` | — (incoherent) | +| **N `DocTest`s** | *unoccupied until ADR 0001* | Sybil, `doctest_docutils` `per-block` | + +The bottom-right cell is where the silent failure lives. The bottom-left cell — +per-block `DocTest`s under one item — gives per-block reporting *and* an +unsplittable sharing unit, and no surveyed project occupies it. + +**Axis 3 has an empirical answer.** xdoctest is the controlled experiment for +abandoning the stdlib object model, and it is now building the bridge back. The +cost of divergence is paid years later, in knobs that exist only to restore the +default that was abandoned. + +**Axis 4 is decided by axis 6.** A regex cannot see `:skipif:` or a group name, +which is why Sybil has no group concept at all and tells users to clear the +namespace instead. If directive options are part of the product, the document +model must be a parse tree. + +**Axis 7 correlates with hostility.** Two of the surveyed projects disable +pytest's doctest plugin — one in `pytest_configure`, one by telling users to pass +`-p no:doctest`. A `pytest11` plugin loads into sessions belonging to people who +never asked for it, and the plugin it disables is the one whose checker, failure +repr and `doctest_namespace` fixture it wants to keep. diff --git a/notes/analyses/10-cpython-doctest.md b/notes/analyses/10-cpython-doctest.md new file mode 100644 index 0000000..8d942d6 --- /dev/null +++ b/notes/analyses/10-cpython-doctest.md @@ -0,0 +1,151 @@ +# CPython `doctest` + +Pinned at [`v3.14.2`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py). + +## Classification + +A four-stage pipeline of unannotated classes, whose documented extension surface +is six classes plus three injection slots and four reporting hooks. The per-example +loop — the thing every extender eventually wants — is not among them. + +## Core data structures + +```text +Example source, want, exc_msg, lineno, indent, options + | lineno is 0-based, relative to the start of the containing string + v +DocTest examples, globs, name, filename, lineno, docstring + | globs is COPIED by __init__; __lt__ compares name as TEXT + v +TestResults namedtuple(failed, attempted), with `skipped` as an EXTRA attribute +``` + +Three properties of these are load-bearing for anything built on top: + +**`DocTest.__init__` copies the globs mapping** +([`:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565)). +Passing a shared dict through the constructor has no effect whatsoever. A shared +mapping must be assigned to `test.globs` after construction, and the runner must +be given `clear_globs=False` or it empties the mapping in its `finally`. + +**`__lt__` compares names as text** +([`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596)). +Any name carrying a position as text sorts `[10]` before `[1]`. This fails +silently: every test passes, in the wrong order. + +**`TestResults` carries `skipped` outside the tuple** +([`:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), +with a `repr` that falls back to the plain namedtuple form when it is zero. +Promoting it to a third field would break every `failures, tries = runner.run(...)` +unpack in the ecosystem, including `doctest._test()` itself. + +## Data flow + +```text +source string + | DocTestParser.parse -> list[str | Example], alternating, + | reconstructing the input exactly + | DocTestParser.get_doctest -> DocTest + v +DocTestFinder.find(obj) -> list[DocTest] + | recurses into __test__, tracks a seen-map by id() + v +DocTestRunner.run(test, compileflags, out, clear_globs) + | saves sys.stdout, pdb.set_trace, linecache.getlines, + | sys.displayhook, _colorize.can_colorize; pops PYTHON_COLORS + | and FORCE_COLOR from os.environ; restores all in finally + | + +--> __run(test, compileflags, out) <- name-mangled + for each example: + merge test-level and example-level optionflags + SKIP? -> continue BEFORE report_start; still counts as attempted + compile(source, "" % (test.name, n), + "single", flags, dont_inherit=True) + exec in test.globs + OutputChecker.check_output(want, got, flags) + report_success / report_failure / report_unexpected_exception + __record_outcome(...) +``` + +The `parse()` contract is not incidental. It must return alternating `str` and +`Example` reconstructing the input, because `script_from_examples()` walks the +`str` pieces to build the prose comments in a debugging script. `Example.indent` +is computed against the original string — after `expandtabs`, before dedent — so a +post-dedent indent shifts every reported column. + +## Extension seams + +| Seam | Kind | Documented | +|---|---|---| +| `DocTestParser` subclass, injected as `parser=` | nominal | yes | +| `DocTestFinder` subclass, injected as `test_finder=` | nominal | yes | +| `OutputChecker.check_output` / `output_difference`, injected as `checker=` | nominal | yes | +| `report_start`, `report_success`, `report_failure`, `report_unexpected_exception` ([`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)) | subclass hook | yes | +| `register_optionflag` / `OPTIONFLAGS_BY_NAME` ([`:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153)) | process-global registry | yes | +| `setUp` / `tearDown` on `DocTestSuite` / `DocFileSuite` | callable param | yes | +| `__run` ([`:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344)) | **name-mangled** | no | +| `__record_outcome` ([`:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485)) | **name-mangled** | no | +| `__patched_linecache_getlines` ([`:1501`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501)) | **name-mangled** | no | +| `_load_testfile` ([`:245`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245)), `_EXAMPLE_RE` ([`:618`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618)) | private | no | + +The name-mangled three are the interesting entry. Mangling rewrites the *call +site* at compile time, so `self._DocTestRunner__run(...)` inside `run()` is an +ordinary attribute lookup that resolves through the subclass's MRO. Defining +`_DocTestRunner__run` in a subclass therefore takes over the loop — verified on +3.14.2 with `run()` untouched. It is not an override point by *design*, but it is +one by *mechanism*, and that distinction is what lets a downstream own the loop +without cloning a code object or patching a module global. + +`register_optionflag` is the only genuinely cross-library extension point in the +module. Its ints are `1 << len(OPTIONFLAGS_BY_NAME)`, so they are +registration-order dependent, typeshed hard-codes the builtin values, and an +unregistered flag name makes a page fail to **parse** rather than to run. + +## Configuration + +There is none, in the modern sense. Behaviour is set by optionflags, which arrive +from three places with a fixed precedence: the runner's constructor, the +`DocTest`'s per-example `options` dict, and the inline `# doctest: +FLAG` comment +parsed out of the example source. `set_unittest_reportflags` mutates a module +global. `doctest.master` accumulates results across invocations — the +documentation calls it advanced tomfoolery. + +## What it cannot do + +- **Run a multi-statement body.** `"single"` mode rejects it, and `"exec"` mode + suppresses expression echo, which empties every `want`. This one fact is the + origin of every downstream monkeypatch of `doctest.compile`. +- **Report a per-example result as a value.** Outcomes exist only as counters and + as text pushed through `out`. pytest works around this by repurposing `out` from + a write-callable into a *list*; Sphinx works around it by not producing machine- + readable results at all. +- **Share a globs mapping across `DocTest`s** without the caller assigning + `test.globs` post-construction and passing `clear_globs=False`. +- **Be reentrant or thread-safe.** `run()` mutates interpreter globals for its + duration ([`:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +- **Report a skip as an outcome.** `SKIP` short-circuits before `report_start`. + There is no `report_skip` at v3.14.2; it appears in later prereleases, so a + downstream loop must probe rather than assume. + +## Anchors + +- [`TestResults`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) · + [`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) · + [`_load_testfile`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245) +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) · + [`DocTest.__lt__`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) +- [`DocTestParser`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L609) · + [`_EXAMPLE_RE`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618) · + [`DocTestFinder`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) +- [`report_*` hooks](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) · + [`__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) · + [`compile(..., "single", ...)`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) +- [`__record_outcome`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485) · + [`__patched_linecache_getlines`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501) · + [`run()` global-state save/restore](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) +- [`OutputChecker`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) · + [`DebugRunner`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1874) · + [`testfile`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2091) · + [`DocTestSuite`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) · + [`DocFileSuite`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) +- Typed contract: [`typeshed stdlib/doctest.pyi`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) diff --git a/notes/analyses/11-pytest-doctest.md b/notes/analyses/11-pytest-doctest.md new file mode 100644 index 0000000..5f616fa --- /dev/null +++ b/notes/analyses/11-pytest-doctest.md @@ -0,0 +1,127 @@ +# `_pytest.doctest` + +Pinned at [`9.1.1`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py). + +## Classification + +A deliberately thin adapter. It owns two collectors and one item, and delegates +every piece of domain knowledge to stdlib `DocTest`/`Example`/`DocTestFailure`. It +overrides exactly the seams it needs and invents no parallel abstraction. It is +the reference implementation of how to integrate with `doctest` rather than +replace it, and the model this project's pytest layer should resemble. + +## Core data structures + +```text +DoctestItem(Item) dtest: DocTest, runner: DocTestRunner, fixture_request + | obj = None (class attribute) +DoctestTextfile(Module) obj = None; one DocTest for the whole file +DoctestModule(Module) one DocTest per docstring; calls parsefactories +MultipleDoctestFailures carries a list; the workaround for stdlib having no + per-example result value +ReprFailDoctest (ReprFileLocation, lines) pairs +``` + +`obj = None` as a *class* attribute +([`:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421)) +is what keeps `Module` from trying to import a `.txt`/`.rst` file. Subclassing +`Module` rather than `File` is what makes `scope="module"` fixtures resolve +against the page — a page collector *is* the module scope. `parsefactories` +([`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556)) +is what makes autouse fixtures from a visible `conftest.py` apply at all. + +## Data flow + +```text +pytest_collect_file(file_path, parent) [:126] + | .py -> DoctestModule (when --doctest-modules) + | else -> DoctestTextfile (when _is_doctest) + v +Collector.collect() -> DoctestItem.from_parent(...) per non-empty DocTest + | an EMPTY DocTest is never yielded [:451] + v +DoctestItem.setup() [:288] + | fixture request is filled, then: self.dtest.globs.update(globs) + | -> the mapping must be MUTABLE and must survive collection + v +DoctestItem.runtest() [:295] + | _check_all_skipped(self.dtest) -> outcomes.skip if every example is SKIP + | self.runner.run(self.dtest, out=failures) + | ^^^ `out` is a LIST, not a write-callable. + | clear_globs defaults to True. + | raise MultipleDoctestFailures(failures) + v +DoctestItem.repr_failure(excinfo) [:317] + for failure in failures: [:337] + lineno = test.lineno + example.lineno + 1 [:344] +``` + +Two details in that flow decide a great deal for anything built on it. + +**`out` is repurposed as a list.** The most important consumer of stdlib's runner +deliberately violates typeshed's `_Out = Callable[[str], object]`, marked with a +`# type: ignore[arg-type]`, so that `report_failure` can append rather than write. +Any claim that a "typed vanilla core" can narrow `out` honestly has to reckon with +the fact that the ecosystem's largest caller does not. + +**`repr_failure` reads each failure's own `test`.** Locations are computed +per failure inside the loop, not once per item. That is what makes N `DocTest`s +under one item report N correct locations with no override — the fact ADR 0001 is +built on. + +## Extension seams + +| Seam | Kind | +|---|---| +| `--doctest-modules`, `--doctest-glob`, `--doctest-continue-on-failure`, `--doctest-report`, `doctest_optionflags`, `doctest_encoding` | ini/CLI, declared by the always-loaded plugin | +| `doctest_namespace` session fixture ([`:721`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721)) | fixture | +| `_get_flag_lookup` ([`:385`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385)) | private; lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES`, `NUMBER` | +| `_get_checker` ([`:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662)) | private; returns the checker implementing those flags | +| `_get_continue_on_failure` ([`:410`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L410)), `_get_report_choice` ([`:703`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L703)) | private | +| `PytestDoctestRunner` ([`:181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181)) | **unreachable** — defined inside `_init_runner_class()` ([`:178`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178)) | + +That last row matters more than it looks. `PytestDoctestRunner` is where the +`OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` conversion, and +the `continue_on_failure` buffering live. Because it is defined inside a function +and never bound at module scope, a downstream cannot import or subclass it. Any +design that assumes those behaviours "come for free" by subclassing `DoctestItem` +is wrong: the item supplies the *plumbing*, but the runner supplies the +*behaviour*, and only the plumbing is reachable. + +## Configuration + +`pytest_addoption` in this module declares six settings that the plugin reads back +through helpers. A third-party plugin may **read** them but must not re-declare +them — re-adding an existing option raises at option-parsing time. Conversely, +suppressing the built-in plugin before `pytest_configure` (`-p no:doctest`) +removes the options entirely, and any downstream read of them then fails. + +## What it cannot do + +- **Collect a page with directives.** It has no document model at all; a `.rst` + file is one string handed to `DocTestParser`. +- **Share state across items.** `runtest()` runs with `clear_globs=True`. +- **Decline a path it has claimed.** `_is_doctest` + ([`:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152)) + returns `True` for any `.txt`/`.rst` **initial path before consulting + `--doctest-glob`**, and `pytest_collect_file` is not `firstresult`, so a + third-party collector claiming the same path gets its items collected + *alongside* — not instead of — the built-in's. + +## Anchors + +- [`pytest_collect_file`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L126) · + [`_is_doctest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) · + [`_is_setup_py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L141) · + [`_is_main_py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L155) +- [`_init_runner_class`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178) · + [`PytestDoctestRunner`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181) · + [`MultipleDoctestFailures`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) +- [`DoctestItem`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L251) · + [`setup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) · + [`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) · + [`repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) +- [`get_optionflags`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L401) · + [`_check_all_skipped`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) · + [`DoctestTextfile`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420) · + [`DoctestModule`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md new file mode 100644 index 0000000..2f7cf74 --- /dev/null +++ b/notes/analyses/12-pytest-xdist.md @@ -0,0 +1,136 @@ +# pytest-xdist + +Pinned at [`v3.8.0`](https://github.com/pytest-dev/pytest-xdist/tree/v3.8.0). + +## Classification + +A controller/worker distribution layer over execnet. All scheduling is integer +indices into a per-worker collection list, and the controller **never collects**. +That single asymmetry is the source of every constraint xdist imposes on a plugin +that wants to keep related tests together. + +## Core data structures + +```text +controller worker (one per process) + NodeManager -> specs: list[str] session collects normally + Scheduling implementation reports node ids back as STRINGS + node2collection: dict[node, list[str]] + node2pending: dict[node, list[int]] <- integer indices, not ids + collection: list[str] <- the agreed id list +``` + +The controller's entire model of the suite is a list of node-id strings that +arrived from a worker. It has no items, no marks, no fixtures and no knowledge of +what any test does. A plugin that needs "these tests share state" therefore cannot +tell the controller so directly — it can only encode the fact *into the node id* +or infer it from string shape. + +## Data flow + +```text +pytest_cmdline_main xdist promotes -n N into --dist load (tryfirst) + | +pytest_sessionstart -> NodeManager.setup_nodes + | pytest_xdist_setupnodes(config, specs) <- specs already expanded + v +each worker collects independently + | + +-> pytest_xdist_node_collection_finished(node, ids) + | + v +Scheduling.add_node_collection(node, ids) + | every worker's list must be IDENTICAL, in the same ORDER + | mismatch -> log "**Different tests collected, aborting run**" + | and assign nothing. Zero tests execute. + v +Scheduling.schedule() -> send integer index batches to workers + | + v (on worker crash) + only the UNCOMPLETED items of the crashed work unit are re-sent + to a FRESH worker with FRESH process state +``` + +The abort path is the constraint that matters most. It is not an exception and it +is not loud in the usual sense: the scheduler logs a line, assigns nothing, and +the session ends having run nothing. Any collection-time decision that is not a +pure function of (files on disk, argv, ini) — a timestamp, a PID, a hostname, a +dict iteration order, an evaluated `:skipif:` that depends on the environment — +produces this. + +The crash path is the second. Because only uncompleted items of a work unit are +retried, a group whose blocks 1-2 ran before the crash has blocks 3..N re-run +against an empty process, producing a `NameError` cascade attributed to the wrong +cause. Worker restarts are on by default. + +## Extension seams + +| Seam | Kind | +|---|---| +| `pytest_xdist_make_scheduler(config, log)` | hook — substitute a `Scheduling` implementation | +| `pytest_xdist_node_collection_finished(node, ids)` | hook — observe the agreed id list | +| `pytest_xdist_setupnodes(config, specs)` | hook — receives the already-expanded spec list; never raises | +| `pytest_xdist_auto_num_workers(config)` | hook | +| `LoadScopeScheduling._split_scope(nodeid) -> str` ([`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284)) | subclass hook — **the only affinity primitive in the codebase** | +| `@pytest.mark.xdist_group(name)` | marker, honoured only under `--dist loadgroup` | + +`_split_scope` is worth stating plainly: it is a pure function from a node-id +string to a scope string, and both shipped grouping modes are two-line overrides +of it — [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) +returns the file part, [`loadgroup.py:24`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) +returns the `@`-suffix. `load` and `worksteal` have **no scope concept at any +layer**: `load` slices `pending[:num]` and `worksteal` steals a raw suffix. + +So under a user-typed `--dist load`, a plugin has exactly three options: refuse +the run, substitute the scheduler, or make the group not need protecting. There +is no "declare affinity and let the chosen scheduler honour it" API. + +The `xdist_group` marker is narrower than it appears. It is applied +[worker-side](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254), +and only when the worker's own literal `--dist` string is `loadgroup`; it works by +appending `@` to `item._nodeid`. A controller-side scheduler substitution +never reaches a worker, so no `@` suffix is written and every item becomes its own +scope — strictly worse than plain `load`. Node ids copied from a `loadgroup` run +also do not select under `-n0`. + +## Configuration + +`-n`, `--dist`, `--tx`, `--maxprocesses`, `--max-worker-restart`. Worker counts +come from +[`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37), +which builds a *list*: + +```python +xspeclist.extend([xspec[i + 1 :]] * num) +``` + +List multiplication by a negative number yields an empty list, so a negative +multiplier contributes **zero** specs. A re-implementation that sums the integer +instead contributes a negative number, and `--tx -1*popen --tx 2*popen` then +counts 1 where xdist counts 2 — a divergence whose failure direction is +permissive. + +`parse_tx_spec_config` raises `pytest.UsageError` when a run names no environment, +so it cannot be called defensively. `pytest_xdist_setupnodes(config, specs)` is +the safe source of the same information: it receives the already-expanded list, +fires during `pytest_sessionstart` — strictly before `pytest_xdist_make_scheduler` +— and never raises. + +## What it cannot do + +- **Ship a Python object between processes.** Only execnet-serializable builtins + cross. A live `globs` mapping cannot be shared, which is the whole reason a + shared doctest namespace is a distribution problem. +- **Tell the controller what an item is.** The controller sees strings. +- **Preserve process state across a worker restart** for the uncompleted tail of a + work unit. + +## Anchors + +- [`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`loadfile`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) · + [`loadgroup`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) +- [`load.schedule` abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) · + [`loadscope.schedule` abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) +- [`xdist_group` node-id append](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) diff --git a/notes/analyses/13-pytest-asyncio.md b/notes/analyses/13-pytest-asyncio.md new file mode 100644 index 0000000..ecdd322 --- /dev/null +++ b/notes/analyses/13-pytest-asyncio.md @@ -0,0 +1,112 @@ +# pytest-asyncio + +Pinned at [`v1.4.0`](https://github.com/pytest-dev/pytest-asyncio/tree/v1.4.0). + +Read here as an **idiom exemplar**, not for async semantics. It is a mature, +widely-installed `pytest11` plugin that solves the same shape of problem this +project has: a per-item resource with a configurable lifetime, an opt-in mode, and +a default it needed to change without breaking anyone. + +## Classification + +A hook-driven behaviour plugin. It contributes no collector of its own for the +common case; it normalizes configuration into one question, answers it once, and +then injects behaviour through pytest's existing machinery rather than owning the +item class. + +## Core data structures + +```text +Mode(str, enum.Enum) AUTO | STRICT [:82] +PytestAsyncioSpecs its own hookspec namespace [:90] + pytest_asyncio_loop_factories(config, item) -> Mapping | None firstresult +_ScopeName reuses pytest's scope vocabulary verbatim +``` + +`Mode` inheriting `str` is a small but deliberate choice: the ini value, the CLI +value and the internal enum are the same object, so no conversion layer exists to +drift. + +Declaring its own `HookspecMarker("pytest")` namespace is the interesting one. A +third party extends pytest-asyncio by implementing a hook, not by subclassing +anything and not by mutating a registry — which sidesteps both the nominal-typing +trap and the process-global-registry trap. + +## Data flow + +```text +pytest_addoption --asyncio-mode + asyncio_mode ini [:108] + | asyncio_default_fixture_loop_scope [:137] + | asyncio_default_test_loop_scope [:143] + | every one declared with default=None + v +pytest_configure validate scopes; addinivalue_line for the marker [:295] + | an unset default is DETECTED, not silently assumed + v +_get_asyncio_mode(config) -> Mode, resolved once [:222] + | + v +in AUTO mode: item.add_marker("asyncio") + | => the rest of the plugin has exactly ONE query path + v +fixture/loop resolution by scope, then pyfunc call wrapping +``` + +## Extension seams + +| Seam | Kind | +|---|---| +| `asyncio_mode` ini + `--asyncio-mode` CLI | configuration | +| `@pytest.mark.asyncio` | marker, registered via `addinivalue_line` | +| `asyncio_default_fixture_loop_scope`, `asyncio_default_test_loop_scope` | configuration, reusing pytest's scope names | +| `pytest_asyncio_loop_factories` | its own `firstresult` hookspec | +| `@pytest_asyncio.fixture(loop_scope=...)` | decorator, stamping `_loop_scope` on the function | + +## What is worth stealing + +**The `default=None` sentinel.** Every option is declared with `default=None` +([`:114`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L114), +[`:122`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L122), +[`:140`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L140)) +rather than with its effective default. The plugin can therefore distinguish "the +user chose the current default" from "the user has not chosen", which is what +makes a future default change *announceable* — it warns only the second group. +`pytest_configure` uses exactly this to warn about an unset +`asyncio_default_fixture_loop_scope` +([`:296-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L296-L301)). + +This project has the same problem coming: ADR 0001's vocabulary change renames +`namespace_scope` to `share`, and any future move of the block-vs-document default +needs the same mechanism. + +**Normalize, then query once.** In `AUTO` mode the plugin literally adds the +marker it would otherwise have to special-case, so downstream code has one +question with one answer shape. The alternative — branching on mode at every read +site — is what produces the "two components disagree about the current setting" +class of bug. + +**Reuse the host's vocabulary rather than inventing a parallel one.** Loop scope +uses pytest's own `function`/`class`/`module`/`package`/`session` ladder and its +scope names verbatim. It does not invent a third word for lifetime. Compare the +current `namespace_scope`/`namespace_items` naming, which collides with two pytest +concepts at once. + +**Session-wide errors raise `pytest.UsageError`; per-item errors raise +`ValueError`.** The severity matches the blast radius. A misspelled session +setting should stop the session; a bad per-item value should fail that item. + +## What it cannot tell us + +Its resource — an event loop — is cheap to create, has no cross-process identity +problem, and never needs to be scheduled onto a particular worker. It therefore +has nothing to say about the distribution question that dominates this project's +design, and its scope model should not be copied on that axis. + +## Anchors + +- [`Mode`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L82) · + [`PytestAsyncioSpecs`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) +- [`pytest_addoption`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108) · + [`_get_asyncio_mode`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222) · + [`pytest_configure`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L295) +- [`_make_asyncio_fixture_function`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L210) diff --git a/notes/analyses/14-asyncio.md b/notes/analyses/14-asyncio.md new file mode 100644 index 0000000..a2a679c --- /dev/null +++ b/notes/analyses/14-asyncio.md @@ -0,0 +1,109 @@ +# `asyncio` — the standard library's own pluggable architecture + +Pinned at [`v3.14.2`](https://github.com/python/cpython/tree/v3.14.2/Lib/asyncio). + +`asyncio` has nothing to do with doctests. It is here because it is the standard +library's worked example of a *deliberately* pluggable subsystem, written by +roughly the same community and shipped in the same tree as `doctest`. Setting the +two side by side answers a question the other notes cannot: when CPython wants an +extension point, what does it build — and why does `doctest` have almost none? + +## Classification + +A layered subsystem with four distinct seam kinds, none of which `doctest` uses: +an abstract base class defining the contract, duck-typed callback interfaces, a +policy indirection for selecting an implementation, and a context-manager runner +that owns lifecycle. + +## Core data structures + +```text +Handle / TimerHandle a scheduled callback [events.py:34, :141] +AbstractEventLoop the CONTRACT, ~90 methods [events.py:254] +BaseEventLoop(AbstractEventLoop) the shared implementation [base_events.py:417] +Future a result slot with callbacks [futures.py:31] +Task(Future) a coroutine driven by a loop [tasks.py:56] +Runner context manager owning a loop [runners.py:21] +BaseProtocol / Protocol / BufferedProtocol / DatagramProtocol / SubprocessProtocol + what YOU implement [protocols.py:9, :66, :109, :162, :177] +BaseTransport / ReadTransport / WriteTransport / Transport / ... + what the LOOP implements [transports.py:9, :46, :72, :148] +``` + +## The four seam kinds + +**1. An abstract base as a published contract.** `AbstractEventLoop` +([`events.py:254`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254)) +names every method an event loop must provide, separately from +`BaseEventLoop`, which implements most of them. A third party writing uvloop +implements the *contract*, not a subclass of the shipped implementation. Compare +`doctest`, where the contract and the implementation are the same class, so +`DocTestFinder(parser=...)` demands the class rather than the shape. + +**2. Paired duck-typed roles.** `Protocol` is what the user writes; `Transport` is +what the loop provides. Neither is registered anywhere, neither is checked with +`isinstance`, and the split is by *direction of the call*: the transport is called +by you, the protocol is called by the loop. This is the cheapest possible +extension mechanism — two documented method vocabularies — and it has carried +third-party HTTP, TLS and subprocess stacks for a decade. + +**3. A policy indirection, and its retirement.** `get_event_loop_policy` and +`set_event_loop_policy` sit at +[`events.py:804`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) +and [`:817`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817), +now delegating to private `_get_event_loop_policy` / `_set_event_loop_policy` +([`:798`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L798), +[`:808`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L808)) — +the public spellings are on the way out. That is the most instructive thing in +this file. A process-global, mutable indirection for "which implementation should +this program use" was shipped, was widely misused, and is being replaced by +passing the choice explicitly: `asyncio.run(main, loop_factory=...)` +([`runners.py:169`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169)). + +The lesson transfers directly. A process-global mutable registry is the seam you +regret. docutils' directive table is the same shape and has the same problems — +see [`16-docutils-myst.md`](16-docutils-myst.md). + +**4. A runner that owns lifecycle.** `Runner` +([`runners.py:21`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21)) +is a context manager that creates the loop, runs the work, cancels stragglers +([`:207`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207)) +and shuts down cleanly. Global state that must be restored lives in one `finally` +owned by one object. + +`doctest.DocTestRunner.run` does exactly this for `sys.stdout`, `pdb.set_trace`, +`linecache.getlines`, `sys.displayhook` and `PYTHON_COLORS` +([`doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +It is the one place `doctest` and `asyncio` agree on architecture, and it is +precisely why ADR 0001 takes over `__run` but leaves `run()` alone: the lifecycle +owner should keep owning the lifecycle. + +## Cross-cutting: what `doctest` would look like with `asyncio`'s seams + +| `asyncio` | `doctest` equivalent | Present? | +|---|---|---| +| `AbstractEventLoop` as a separate contract | an ABC or `Protocol` for finder/parser/checker | no — the class *is* the contract | +| `Protocol`/`Transport` duck-typed roles | `OutputChecker` is close: a documented method vocabulary, injected | partly | +| policy indirection | none | no | +| `Runner` owning lifecycle | `DocTestRunner.run`'s save/restore | yes | +| explicit `loop_factory=` replacing global policy | `parser=`, `checker=`, `test_finder=` injection | yes, and it is the healthy part | + +The gap is the first row, and it is the concrete reason +`DocutilsDocTestFinder` cannot be handed to `DocTestSuite(test_finder=...)` today +despite exposing a compatible `find()`. ADR 0001's answer — declare `Protocol`s +*and* subclass the stdlib classes nominally — is the cheap way to have both, and +it costs nothing at runtime. + +## Anchors + +- [`AbstractEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) · + [`BaseEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/base_events.py#L417) +- [`get_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) · + [`set_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) +- [`BaseProtocol`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L9) · + [`Transport`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L148) +- [`Runner`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21) · + [`run(main, *, loop_factory=None)`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169) · + [`_cancel_all_tasks`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207) +- [`Future`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/futures.py#L31) · + [`Task`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/tasks.py#L56) diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md new file mode 100644 index 0000000..7be7bbc --- /dev/null +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -0,0 +1,127 @@ +# `sphinx.ext.doctest` + +Pinned at [`v9.1.0`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py). + +## Classification + +A semantic fork. It invents its own document-level model — groups, phases, five +directives — and then converts all of it back into stdlib `doctest.DocTest` +objects for execution. It is the source of every author-facing spelling this +project supports, and the reason those spellings must be honoured exactly. + +It is also builder-coupled: nothing in the pipeline is callable without a built +Sphinx app, and it produces no machine-readable results at all. + +## Core data structures + +```text +TestCode code, type, filename, lineno, options [:235] + `type` in {testsetup, testcleanup, doctest, testcode, testoutput} +TestGroup name, setup: list, tests: list, cleanup: list [:200] + add_code(code, prepend=False) [:207] +DocTestBuilder(Builder) [:292] + self.type: "single" | "exec" <- mutable, read by a + process-global compile patch +SphinxDocTestRunner(doctest.DocTestRunner) [:257] +``` + +`TestGroup.tests` holds heterogeneous entries — `[code]` for a bare block, +`[code, output]` for a paired testcode/testoutput. `add_code` is where three +silent losses live: an orphan `testoutput` is discarded, a second `testoutput` +*replaces* the first, and a `testcode`'s own options are dropped in favour of its +output block's. + +## Data flow + +```text +directive run() [:66] + | parse group names from the optional argument + | trim `# doctest:` flags out of the RENDERED code, keep the original + | in node["test"] + | nodetype = nodes.comment when name in {testsetup, testcleanup} + | or "hide" in options [:92-93] + | stamp node["testnodetype"], ["groups"], ["options"], ["skipif"] + v +doctree + | +DocTestBuilder.test_doc(docname, doctree) [:428] + | for node in doctree.findall(condition): + | if self.skipped(node): continue <- GATED BLOCK IS DROPPED [:443-444] + | code = TestCode(...) + | "*" in groups -> add to every group + | else groups[name].add_code(code) + v +per group: ns = {} + | three runners: setup / test / cleanup, sharing one _fakeout + | test.globs = ns (assigned AFTER construction) + | runner.run(test, out=..., clear_globs=False) + | self.type flipped to "exec" for setup, cleanup, testcode [:548, :608] + | to "single" for ordinary doctests [:580] + v +six integer counters + text streamed to outdir/output.txt +``` + +## Extension seams + +| Seam | Kind | +|---|---| +| `TestDirective` subclassing, with `option_spec` ([`:66`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66)) | subclass | +| The node attribute stamp — `testnodetype`, `groups`, `options`, `skipif`, `test` | implicit protocol | +| `doctest_global_setup`, `doctest_global_cleanup`, `doctest_test_doctest_blocks`, `doctest_default_flags` | Sphinx confvals | +| `is_allowed_version(spec, version)` ([`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45)) | function | + +The node attribute stamp is the important one, and it is undocumented. It is the +only decoupled interface in the module: any directive that emits a +`literal_block` or `comment` carrying `testnodetype` participates. It is also +what makes a *third-party* collector — this project's — able to read a page +Sphinx's own directives produced, and vice versa. Reading attributes off the node +rather than trusting one's own directive classes is the only defence against +`Sphinx.add_directive` overriding a registration unconditionally. + +## Semantics this project must match exactly + +| Rule | Anchor | +|---|---| +| `testsetup`, `testcleanup` and `:hide:` render as `nodes.comment` | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | +| `:options:` is accepted only on `doctest` and `testoutput` | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111) | +| A gated block is dropped during collection — no outcome, id or count | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | +| `*` means every group the document declares; `default` means no argument was given | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) onward | +| Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200-L226) | +| `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | +| `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) | + +The last two are where this project deliberately diverges. The argument order was +a real defect in the local helper. The `compile` rebinding is unavailable to a +library that loads into every pytest session, which is the entire origin of +ADR 0001's decision to own the per-example loop instead. + +The gated-block rule is the one semantic this project rejects on purpose: Sphinx's +drop destroys the node id, the count and the `-rs` line, and pytest users +reasonably expect a `SKIPPED` outcome with a reason. + +## What it cannot do + +- **Run outside a Sphinx build.** `DocTestBuilder` binds an `env`, a `config`, an + `outdir`, a `sys.path` mutation and an open file handle. +- **Produce results a caller can inspect.** Six ints and text to a file. No failure + can be mapped back to its node without re-parsing prose. +- **Distinguish two blocks in one group.** Every block in a group shares + `DocTest.name`, which is why `SphinxDocTestRunner` overrides a private stdlib + method to swallow the resulting `IndexError` + ([`:257`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257)). + +## Anchors + +- [`is_allowed_version`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) · + [`TestDirective`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66) · + [`comment nodetype rule`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) +- [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) · + [`add_code`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) · + [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) +- [`SphinxDocTestRunner`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257) · + [`DocTestBuilder`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L292) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) · + [`skipped-node drop`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) · + [`type = "exec"` for testcode](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) +- [User-facing contract](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/doc/usage/extensions/doctest.rst) diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md new file mode 100644 index 0000000..e1fb1d3 --- /dev/null +++ b/notes/analyses/16-docutils-myst.md @@ -0,0 +1,130 @@ +# docutils and MyST-Parser + +docutils pinned at `docutils-0.22.2` (canonical repository is on SourceForge; the +GitHub copies are third-party mirrors, so anchors here name file and symbol rather +than a permalink). MyST-Parser pinned at +[`v5.1.0`](https://github.com/executablebooks/MyST-Parser/tree/v5.1.0). + +## Classification + +The parsing floor. Two front-ends producing one node model, with two different +line-number conventions and one shared, process-global, unscoped extension +registry. + +## Core data structures + +```text +docutils.nodes.Element attributes: dict[str, Any] + literal_block a rendered code block + comment what testsetup/testcleanup/:hide: become + doctest_block a bare >>> block in reStructuredText + .line int | None + .source the file the text lives in (None when unset) + .rawsource the pre-render source, when the node kept it + +docutils.parsers.rst.Parser + .state_classes an INSTANCE attribute, therefore substitutable + per parse with no process-global mutation + +myst_parser.parsers.docutils_.Parser(RstParser) [v5.1.0:235] + .settings_spec = (..., create_myst_settings_spec(), *RstParser.settings_spec) + [v5.1.0:241-245] +``` + +## The two line conventions + +| Front-end | Construct | `.line` reports | +|---|---|---| +| reStructuredText | top-level `doctest_block` | its **last** line | +| reStructuredText | any block nested in a directive, list item or block quote | `None`, with `.source` also `None` | +| MyST | fenced block | its **first** line | +| either | a block reached through `.. include::` | numbered against the **included** file | + +All four are real and all four have to be normalized by the front-end that knows +which it is. A collector that assumes one convention mis-anchors the other's +blocks; a collector that reads `.line` as a number crashes on the nested case. +This is why `Block.line` in ADR 0001 is nullable and `Block.path` is separate from +the collected document. + +## The directive registry + +`docutils.parsers.rst.directives` keeps one module-level dict consulted by both +the reStructuredText parser and MyST's `run_directive`. It has no per-document +scoping, no versioning, and no ownership. + +Three failure modes follow, and all three have been observed: + +1. **It can be rebound, not merely mutated.** Sphinx's `docutils_namespace()` + restores a snapshot by rebinding the module attribute, so any registration made + inside that context is discarded *and the dict's object identity changes*. A + registration guard that caches a boolean is wrong; membership must be + re-checked against the live dict. +2. **Registrations are overwritten silently.** `Sphinx.add_directive` overrides an + existing name unconditionally, with only a warning. `sphinx.ext.doctest` loaded + in the same interpreter therefore replaces a forked directive class with one + that has different option handling — including the reversed + `is_allowed_version` argument order. +3. **A missing registration is silent.** An unregistered directive parses to a + docutils error node, the page still renders, and the collector finds zero + tests. This is the GH-48 failure shape, and it is the reason ADR 0004 treats + diagnostics as data. + +The defence is not to win the registry. It is to read the *node attributes* — +`testnodetype`, `groups`, `options`, `skipif`, `test`, `hide` — which are +byte-compatible with what `sphinx.ext.doctest` stamps, so a page survives either +class having produced it. + +The registry is a smaller instance of the pattern `asyncio` is currently retiring; +see [`14-asyncio.md`](14-asyncio.md). + +## MyST configuration + +`myst_parser.parsers.docutils_.Parser` subclasses `RstParser` and composes its own +settings spec from `create_myst_settings_spec()` +([`v5.1.0:208`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208), +[`:241-245`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245)). +Driving MyST through that `Parser` is what makes the `myst_*` docutils settings — +including `myst_enable_extensions` and `myst_fence_as_directive` — reachable, and +what makes front-matter configuration merge. + +Constructing an `MdParserConfig` by hand and calling `md_parser.render()` against +a bare `make_document()` skips all of it. Colon-fence directives do not exist, a +plain ```` ```python ```` fence is only picked up by a prompt sniff, and the +line-length guard and MyST transforms never run. Those omissions become a decision +rather than an accident once the front-end owns its own configuration. + +`myst_fence_as_directive` is also the answer to a real user request: it maps a +bare language fence onto a directive name, so a project that prefers not to write +`{testcode}` can still have its ```` ```python ```` blocks collected. + +## Reporter behaviour + +Default settings send reporter output to stderr and raise `SystemMessage` at +`halt_level`, aborting mid-parse. Both are configurable: raising `halt_level` past +the abort threshold and attaching an observer with `Reporter.attach_observer` +turns messages into values. `Reporter.system_message` notifies observers for any +level above `DEBUG` independently of `report_level`, so observation and display are +separable — which is what makes ADR 0004's code-keyed suppression possible without +losing the underlying record. + +## What it cannot do + +- **Scope a directive registration** to one parse, one document or one thread. +- **Report a line for every node.** See the table above. +- **Type its own attribute channel.** `Element.attributes` is `dict[str, Any]`, and + typeshed's stub for `get(key, failobj: _T) -> _T` is actively wrong — it claims + `_T` even when the key is present holding something else. One narrowing accessor + at the parse boundary is cheaper and safer than a coercion at every read site. + +## Anchors + +- MyST: [`docutils_.Parser`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L235) · + [`create_myst_settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208) · + [`settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) · + [`MdParserConfig`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) +- Sphinx's registry snapshot/rebind: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/util/docutils.py) · + unconditional override: [`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/application.py) +- docutils: `docutils/parsers/rst/directives/__init__.py` (`_directives`), + `docutils/parsers/rst/states.py` (`state_classes`, `doctest_block` line + assignment), `docutils/utils/__init__.py` (`Reporter.attach_observer`, + `system_message`), at `docutils-0.22.2`. diff --git a/notes/analyses/17-prior-art.md b/notes/analyses/17-prior-art.md new file mode 100644 index 0000000..c9da655 --- /dev/null +++ b/notes/analyses/17-prior-art.md @@ -0,0 +1,167 @@ +# Prior art: Sybil, xdoctest, pytest-examples, typeshed + +Four projects that solved some part of this problem differently. Each is read for +one specific question, and each answers it — two of them by counterexample. + +--- + +## Sybil + +Pinned at [`10.0.1`](https://github.com/simplistix/sybil/tree/10.0.1). + +**The bet:** a document is a flat sequence of non-overlapping character spans, not +a parse tree, and everything else — markup format, language, assertion semantics, +test runner — is a plugin over that primitive. It parses nothing itself: no +docutils, no myst-parser, no CommonMark implementation, and zero runtime +dependencies. Every format is a regex `Lexer` producing `Region(start, end, +lexemes)` spans over raw text. + +**Data model.** `Region` is a half-open span with three payload slots — `lexemes`, +`parsed`, `evaluator` — and lives in two undocumented-by-type states: lexed +(lexemes only) and parsed (evaluator attached). `Document` is text plus path plus +regions plus **one `namespace: dict`**. `Example` joins document and region at run +time and holds a *reference* to that namespace. `Sybil` itself is pure +configuration. + +**What is genuinely good.** `Document.add` bisect-inserts and **raises +`ValueError` on any overlap**. That single invariant gives a total order for free +and converts the silent class of parser bug — a block dropped or collected twice — +into a loud error at collection time. It is the best structural decision in the +codebase. + +Its public testing helpers (`check_lexer`, `check_parser`, `check_sybil`) are +documented *and* used by the project's own runnable documentation, so the +extension guide is under test. That is rare and worth copying. + +**The fatal flaw.** One mutable namespace per document, and +[one independently selectable pytest item per region](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py). +`pytest -k` on an example whose predecessor bound a name raises `NameError`. The +documentation never mentions this; there is no discussion of `xdist`, parallelism +or deselection anywhere in it. This is axis 1 of +[`00-taxonomy.md`](00-taxonomy.md), in the unacknowledged column. + +**The second flaw.** [Node ids are positional](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) — +`line:{line},column:{column}`. Adding a paragraph above an example renames every +downstream test, breaking `--lf`, `--nf`, deselect files, xfail lists and CI flake +history. For a *documentation* test runner, prose above examples is the thing that +changes most often. + +**What it cannot do.** Groups. A regex cannot see a directive's options, so Sybil +has no group concept at all and directs users to clear the namespace instead. This +is the clearest available argument for paying the docutils dependency: if +`:skipif:`, `:options:` and group names are part of the product, the document +model has to be a parse tree. + +**Verdict on the invariant.** The non-overlap check does not survive contact with +docutils: two `.. include::` directives naming the same file legitimately produce +blocks over identical source spans. Adopt the *idea* — detect double collection — +as a diagnostic carrying both provenances, not as an exception. + +--- + +## xdoctest + +Pinned at [`v1.3.2`](https://github.com/Erotemic/xdoctest/tree/v1.3.2). + +**The bet:** a doctest is Python source that happens to live in a docstring, so +parse it with `ast`/`tokenize` rather than a line regex, and abandon stdlib +compatibility to fix the design. + +**What is genuinely good.** + +- `ast`-based parsing really is better than `_EXAMPLE_RE` for Python. +- Directives are **structured objects** rather than an int bitmask, and + [`REQUIRES`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) + carries a *set of unmet requirements* — so a skip knows it skipped because + `module:torch` was absent. A bool discards exactly the information the reader + wants, and this is the single most transferable idea in the survey. +- Per-part synthetic filenames plus a filename-to-block map, so an exception raised + inside a function that an *earlier* block defined is attributed to the defining + block. Directly applicable to grouped namespaces. + +**The counterexample it provides.** It is now building stdlib compatibility back, +in modules that did not exist at the tagged release. Getting stdlib semantics out +of its own intake seam requires setting `REQUIRE_WANT`, +`deferred_output_matching=False` **and** `compile_mode='single'` — three knobs to +undo one abandoned default, paid years later. This is the empirical answer to +"should the core be vanilla?" + +**Two things not to copy.** Its got/want defaults are permissive — +`ELLIPSIS`, `NORMALIZE_WHITESPACE` and `NORMALIZE_REPR` all default true — which +silently changes the meaning of tests users wrote for stdlib. And unknown +directives and parse errors `warnings.warn` and vanish, so a typo weakens a test +instead of failing it. A test that reports green while checking nothing is worse +than no test. + +It also unregisters pytest's doctest plugin outright, which is hostile in a +`pytest11` package. + +--- + +## pytest-examples + +Pinned at [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18). + +**The bet:** invert the contract — the author does not write expected output, the +runner writes it. A block is a module exec'd once with `print` captured, and the +output is rendered back into the source file. + +**What is genuinely good.** + +- **Emitting the canonical form instead of parsing it collapses check-mode and + update-mode into one code path.** That is a real structural insight. +- **Absolute byte offsets plus an invertible dedent scalar** are exactly what a + data model needs to rewrite source. Read-only tools discard the indent; keeping + it is the difference between "we could add `--update-examples` later" and "we + would have to redesign the data model first". Two int fields. +- **It composes with pytest by contributing no collector at all.** Examples are + `parametrize` params, so marks, `-k`, fixtures and xfail all work unmodified. + This is the cheapest correct integration in the entire survey. + +**What not to copy.** It hard-codes a stack depth as a magic integer — the depth of +another library's internal call stack — and forges frames through +`ctypes.pythonapi.PyFrame_New`. Its write-back splices at collection-time offsets +with no staleness check, so editing a file while it runs corrupts the file. And +its update mode is a session-scoped in-process two-phase commit, so `-x` or a crash +writes nothing and says nothing — the worst outcome for a tool whose selling point +is rewriting your files. + +**Rule extracted.** Any write-back must content-hash the region at collection and +refuse on mismatch. An unconditional splice at stale offsets is data loss, not a +race. + +--- + +## typeshed + +Pinned at [`8c7256c`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) +(no tags on this repository; commit reachable from trunk). + +**Read for:** what a typed `doctest` actually costs, and where the type system +gives up. + +The stub annotates every public name and then hands back `Any` at exactly the +three extensible points: `globs: dict[str, Any]` (correct for values, since they +are user objects), `**options: Any` on the three suite builders (incorrect — the +accepted keys are exactly known), and `optionflags: int` everywhere (so +`optionflags=4096` type-checks). + +Two facts matter for anything claiming to be a "typed vanilla core": + +**The de-facto contract is the stub, not the source.** Downstream projects run +mypy against it. Narrowing `parse()` to `list[Example]`, dropping the `bool` +overload on `find`, or typing `out` as `Callable[[str], None]` stops type-checking +for existing typed callers whose code runs fine. + +**Its largest consumer violates it deliberately.** `_pytest.doctest` repurposes +`out` from a write-callable into a *list*, with a `# type: ignore[arg-type]`. Any +honest typing of that parameter has to accommodate the ecosystem's actual usage. + +The stub also has a false negative worth knowing: `DocTestRunner.test: DocTest` is +declared unconditionally, while runtime assigns it only inside `run()`. A stub that +type-checks a crash is worse than no stub for that attribute. + +**Rule extracted.** Keep the runtime objects structurally identical to stdlib's and +put precision in a parallel layer — `TypedDict` over the node-attribute channel, +`Protocol`s for the seams, `Literal` for closed vocabularies. Do not narrow a +signature typeshed publishes wider. diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md new file mode 100644 index 0000000..3b0a730 --- /dev/null +++ b/notes/analyses/20-data-structures.md @@ -0,0 +1,86 @@ +# Cross-cutting: the data structures, lined up + +Every system in these notes ends up representing the same four things: a *unit of +source*, a *unit of execution*, a *unit of shared state*, and a *unit of result*. +The disagreements are entirely about which of those four are the same object. + +## The four units + +| System | source unit | execution unit | shared-state unit | result unit | +|---|---|---|---|---| +| CPython `doctest` | `Example` | `DocTest` | `DocTest.globs` | `TestResults` (2 ints + an attribute) | +| `_pytest.doctest` | `Example` | `DocTest` = one `Item` | `DocTest.globs`, wiped per item | `MultipleDoctestFailures` → `ReprFailDoctest` | +| `sphinx.ext.doctest` | `TestCode` | `TestGroup` | `ns` assigned post-construction | six ints + text to a file | +| Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | +| xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | +| pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | +| ADR 0001 | `Block` | `DocTest` per block | one `globs` per **group**, on the `Item` | stdlib's, per block | + +Reading across the "shared-state unit" column against the "execution unit" column +is the whole design problem. Sphinx and Sybil both put the shared state at a +coarser granularity than the execution unit; only Sphinx also puts the *pytest +item* at that same coarser granularity, which is why Sphinx never hits Sybil's +`-k` failure. + +## Field-by-field: what a source unit carries + +| Field | `Example` | `TestCode` | `Region` | `CodeExample` | `Block` (ADR 0001) | +|---|---|---|---|---|---| +| source text | `source` | `code` | via `lexemes` | `source` | `source` | +| expected output | `want` | paired separately | — | written, not read | `want` | +| line | `lineno` (0-based, string-relative) | `lineno` | computed from span | `start_line` | `line` (nullable) | +| byte offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | +| dedent scalar | `indent` | — | `Lexeme.offset` | `indent` | — (deferred) | +| kind | — | `type` | inferred from evaluator | `prefix_tags()` | `kind` | +| group | — | via `TestGroup` | — | — | `groups` | +| options | `options` | `options` | — | — | `options` | +| gate | — | `skipif` on the node | — | — | `skipif` (unevaluated) | +| file | on the `DocTest` | `filename` | on the `Document` | `path` | `path` | +| compile mode | — | on the *builder*, mutable | — | always exec | on the `Example` subclass | + +Three observations. + +**Only pytest-examples carries byte offsets and an invertible dedent.** Those two +fields are the entire difference between a read-only tool and one that can rewrite +expected output later. ADR 0001 defers them, which is a decision to be revisited +rather than a decision made. + +**Compile mode is on the wrong object everywhere except ADR 0001.** Sphinx keeps +it as mutable builder state read through a process-global patch; stdlib hard-codes +it in the loop. Putting it on the example data is what lets it survive +`copy.copy`, merging and any reordering, without a flag some runner has to be +holding at the right moment. + +**Nobody but ADR 0001 makes the line nullable.** Every other system either always +has a line (because it computed the span itself) or fabricates one. With a real +doctree there are constructs that genuinely have no recoverable line, and pytest +has a branch for exactly that — `EXAMPLE LOCATION UNKNOWN` — which is unreachable +unless the model can express it. + +## What a result unit carries + +None of these systems produce a per-example result *value*: + +- stdlib returns `TestResults(failed, attempted)` with `skipped` bolted on as an + instance attribute, and pushes everything else through `out` as text. +- pytest works around that by repurposing `out` into a list so `report_*` can + append, then rebuilds a location by slicing `test.docstring`. +- Sphinx works around it by not producing machine-readable results at all. + +This absence is why a merged `DocTest` needs a synthetic page with blank-line +padding: the only channel for a location is `(test.lineno, test.docstring, +example.lineno)`, so a group holding many blocks must fabricate a docstring in +which those arithmetic relations still hold. + +Giving each block its own `DocTest` removes the need for the fabrication rather +than improving it — the three fields are then already true. + +## Anchors + +- [`Example` / `DocTest` / `TestResults`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) +- [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) · + [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) +- [`MultipleDoctestFailures`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) · + [`repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) +- [Sybil `Region`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · + [Sybil `Document`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/document.py) diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md new file mode 100644 index 0000000..0264452 --- /dev/null +++ b/notes/analyses/21-data-flows.md @@ -0,0 +1,113 @@ +# Cross-cutting: the data flows, lined up + +Four pipelines that all end in the same `exec()`, drawn to the same scale so the +divergence points are visible. + +## The pipelines + +```text +CPython doctest + string ─► DocTestParser.parse ─► DocTestFinder.find ─► DocTestRunner.run + └─► __run ─► exec + +pytest --doctest-glob + path ─► pytest_collect_file ─► DoctestTextfile.collect ─► DoctestItem + (one DocTest for the whole file) ├─ setup(): globs.update(fixtures) + ├─ runtest(): runner.run(clear_globs=True) + └─ repr_failure(): per-failure location + +sphinx.ext.doctest + doctree ─► test_doc ─► condition filter ─► skipped? DROP + │ + ├─► TestCode ─► TestGroup{setup, tests, cleanup} + │ + └─► per group: ns={}; test.globs=ns (post-construction) + 3 runners (setup/test/cleanup), clear_globs=False + self.type flipped single/exec around each run + └─► doctest.compile (PROCESS-GLOBAL PATCH) ─► exec + +ADR 0001 + path ─► markup.parse_file ─► (Blocks, Diagnostics) + │ + ├─► project() ─► GroupTest{group, tests[], globs} + │ pure: no docutils, no pytest, no filesystem, + │ no user code — :skipif: passes through unevaluated + │ + └─► Document(pytest.Module) ─► DocutilsItem (one per GROUP) + ├─ setup(): globs cleared, then fixtures injected + ├─ runtest(): run_group() runs each block's DocTest + │ in phase order, clear_globs=False + │ :skipif: evaluated HERE + │ DocutilsRunner.__run ─► exec (mode from data) + └─ repr_failure(): inherited; reads each block's own DocTest +``` + +## Where they diverge + +**At the gate.** Sphinx evaluates `:skipif:` during collection and *drops* the +node. pytest cannot express that — an item must exist to have an outcome — so +`doctest_docutils` marks the block `SKIP` instead. ADR 0001 moves the evaluation +to `runtest()`, which additionally buys collection purity: with no user code +running at collection, worker collections cannot diverge and `--collect-only` +cannot have side effects. + +**At the globs assignment.** Every system that shares state has to assign +`test.globs` *after* `DocTest.__init__`, because the constructor +[copies](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565). +Sphinx does this explicitly. pytest does not share at all, and its +[`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) +runs with `clear_globs` defaulting to `True` — so any design that inherits +`runtest()` unchanged and expects sharing gets its mapping emptied after the first +block. That is a trap with no diagnostic; the symptom is a `NameError` in block +two. + +**At the compile call.** stdlib hard-codes `"single"`. Sphinx flips a mutable +builder attribute and reads it through a process-global rebinding of +`doctest.compile` that is never restored. `doctest_docutils` today clones the +mangled loop's code object to get a private version of that rebinding. ADR 0001 +carries the mode on the example data and reads it in a loop it owns — the only one +of the four that neither mutates process state nor copies a code object. + +**At the location.** stdlib computes `test.lineno + example.lineno + 1`. Sphinx +gives every block in a group the same `DocTest.name` and pays for it by overriding +a private method to swallow an `IndexError`. `doctest_docutils` today fabricates a +synthetic page so the arithmetic stays true across merged blocks. ADR 0001 gives +each block its own `DocTest`, so the arithmetic is true without fabrication. + +## The order-of-operations facts + +Three orderings are load-bearing and each has a failure mode with no error +message. + +**Fixtures are injected in `setup()`, into the mapping, in place.** +`DoctestItem.setup()` does `self.dtest.globs.update(globs)` +([`:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293)). +A design that clears and rebinds the mapping around that call either discards the +injected names — so `getfixture` raises `NameError` — or reuses the previous +attempt's mutations. Under `--reruns`, the second is a false green: an expectation +true only on attempt two reports as a pass. + +**Phase order and page order are different orders.** A group hands its blocks over +as setup, tests, cleanup; a reader meets them in whatever order the page writes +them. Any layout that anchors reported lines on the *run* order reports examples +against whichever block came first in that sequence — and can point past the end of +the file. Sphinx sidesteps this by not reporting useful lines at all. + +**Collection order must be numeric, never lexicographic.** `DocTest.__lt__` +compares names as text +([`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596)), +and names carry positions as text, so any accidental `sorted()` runs `page.md[10]` +before `page.md[1]`. Every test still passes, in the wrong sequence — and for a +group sharing state, the wrong sequence is the bug. + +## Anchors + +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) · + [`__lt__`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) · + [`__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) · + [`compile(..., "single", ...)`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) +- [`DoctestItem.setup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) · + [`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) · + [gated-node drop](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md new file mode 100644 index 0000000..eb39d65 --- /dev/null +++ b/notes/analyses/22-extension-seams.md @@ -0,0 +1,106 @@ +# Cross-cutting: extension seams + +Five mechanisms appear across these systems. They are not equally good, and the +ranking is not a matter of taste — each has an observed failure mode. + +## The five mechanisms + +| Mechanism | Where | Failure mode | +|---|---|---| +| **Nominal subclassing** — the API demands the class, not the shape | stdlib `doctest`, `sphinx.ext.doctest` directives | A structurally compatible object is rejected. `DocutilsDocTestFinder` exposes a compatible `find()` and still cannot be passed to `DocTestSuite(test_finder=...)` | +| **Callable aliases** — `Evaluator = Callable[[Example], str \| None]` | Sybil | Types nothing. You cannot express "this lexer emits `source` and `arguments`", so a mismatched pairing raises `KeyError` at run time. Sybil's own source comments say the payload "could likely be a `TypedDict`" | +| **Named registry** — a module dict plus a `register_*` function | `doctest.register_optionflag`, docutils directives, xdoctest's two facades | Process-global mutable state. Order-dependent, unscoped, and silently overwritable | +| **`Protocol`** — structural typing | xdoctest's `StdlibExampleLike` | None inherent; but a `Protocol` alone does not satisfy a nominal consumer | +| **Hookspec** — the host's own plugin protocol | pytest, pytest-asyncio's `PytestAsyncioSpecs` | Requires a host with a plugin system; not available to a library core | + +## Why `register_optionflag` works and the docutils registry does not + +Both are process-global mutable dicts. One is fine and one is a recurring bug +source, and the difference is instructive. + +[`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) +is **append-only and idempotent**: registering a name that exists returns the +existing bit. It cannot be overwritten, so two libraries registering `NUMBER` agree +rather than fight. Its only sharp edge is ordering — ints are +`1 << len(OPTIONFLAGS_BY_NAME)` — which matters because typeshed hard-codes the +builtin values, so a flag registered *before* the builtins would change every +stdlib constant. Registering at import of the core rather than from a plugin hook +is what keeps that ordering stable, and it is also required because an +unregistered name makes a page fail to **parse**. + +The docutils directive table is **overwrite-by-default and rebindable**. Sphinx's +`docutils_namespace()` restores a snapshot by rebinding the module attribute, so +the dict's *identity* changes and a cached boolean guard is wrong. +`Sphinx.add_directive` overwrites unconditionally with only a warning. The result +is that a directive class you registered may not be the one that ran. + +**The rule extracted:** a global registry is acceptable when registration is +append-only and idempotent, and a liability when it is last-writer-wins. Where the +registry is someone else's and last-writer-wins, do not depend on having won — +depend on the *data* both writers produce. That is why ADR 0001 reads +`BlockAttributes` off the node rather than trusting its own directive classes to +have run: the attribute set is byte-compatible with what `sphinx.ext.doctest` +stamps, so either winner is fine. + +`asyncio` reached the same conclusion about global indirection from the other +direction and is retiring `set_event_loop_policy` in favour of an explicit +`loop_factory=` argument — see [`14-asyncio.md`](14-asyncio.md). + +## The nominal/structural trap, and the cheap way out + +Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: DocTestParser += ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. A `Protocol` gives a +third party structural typing; a nominal subclass keeps you passable to the +stdlib's own builders. + +Doing both costs nothing: + +```python +class Frontend(t.Protocol): + suffixes: t.ClassVar[frozenset[str]] + + def parse( + self, text: str, path: pathlib.Path, *, settings: Settings + ) -> ParseResult: ... + + +class DocutilsDocTestParser( + doctest.DocTestParser +): # nominal, for DocFileSuite(parser=...) + ... +``` + +Sybil, xdoctest and stdlib each took one half. Taking both is the only reason this +is worth stating as a rule. + +## What deserves a seam, and what does not + +The project rule is that a new public API waits until a caller outside the module +needs it. Applied to the candidates that came up: + +| Candidate | Verdict | +|---|---| +| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Its `node_types` field is what keeps the `nodes.comment` requirement reviewable rather than buried in a `findall` call | +| Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()` — adding one there requires subclassing two classes | +| `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | +| Compile-policy callable | **No.** One implementation, and a second would have to reproduce the `` filename shape or break `linecache` | +| A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | +| A registry *object* with builders, digests and manifests | **No.** Two module-level dicts and two `register_*` functions do the same work | +| Entry-point plugin discovery | **No.** Not until a caller outside the package exists. It also breaks xdist's collection-purity requirement, since discovery is an import side effect rather than a function of (files, argv, ini) | + +That last row is worth keeping in mind generally: **any registry populated by +import side effects is in tension with +[`12-pytest-xdist.md`](12-pytest-xdist.md)'s requirement** that every worker +collect identical ids in identical order. A registry whose contents depend on +which conftest happened to be imported is not a pure function of the inputs. + +## Anchors + +- [`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) · + [`report_*` hooks](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) · + [`OutputChecker`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`pytest_xdist_setupnodes` consumers](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) +- [`PytestAsyncioSpecs`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) +- [`AbstractEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) · + [`set_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md new file mode 100644 index 0000000..524fbd3 --- /dev/null +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -0,0 +1,127 @@ +# Cross-cutting: namespace scope and test identity + +The one axis where a wrong answer is silent. Every other design choice in these +notes produces inconvenience — a renamed test, a conversion layer, an extra knob. +This one produces a `NameError` in a test the user believed they could select, or a +green run that should have been red. + +## The two questions, which are not the same question + +1. **Which blocks share a `globs` mapping?** (scope) +2. **Which blocks can be selected, reported and scheduled independently?** + (identity) + +Everything downstream — `-k`, `--lf`, `--deselect`, `-x`, `--reruns`, every +`--dist` mode, JUnit rows, flake history — depends on the second. Everything a +narrative page needs depends on the first. + +## The product space + +| | one node id | N node ids | +|---|---|---| +| **one `DocTest`** | `sphinx.ext.doctest`; `doctest_docutils` `merged` | incoherent | +| **N `DocTest`s** | *unoccupied until ADR 0001* | Sybil; `doctest_docutils` `per-block` | + +The bottom-right cell is where the silent failure lives, and two shipped projects +are in it. + +Sybil is there **unacknowledged**: one `Document.namespace` shared by reference, +one pytest item per region. `pytest -k` on an example whose predecessor bound a +name raises `NameError`, and nothing in its documentation says so. + +`doctest_docutils`'s `per-block` mode is there **acknowledged and guarded** — the +guards are the xdist scheduler substitution, the scheduler refusal, and the +run-twice refusal. Those guards are the reason `_worker_count`, `_shared_page`, +`_is_page` and `_splitting_scheduler` exist at all. + +The bottom-left cell gives per-block reporting *and* an unsplittable sharing unit, +and it needs no guards, because there is nothing to split. + +## Why the guards are expensive + +The constraints come from [`12-pytest-xdist.md`](12-pytest-xdist.md) and are all +structural, not incidental: + +- A live mapping is a Python object; only execnet-serializable builtins cross a + worker boundary. +- The controller never collects, so it cannot ask "which items share state" — it + sees node-id strings and nothing else. Any protection must be *inferred from + string shape* or *encoded into the id*. +- The only affinity primitive is + [`_split_scope(nodeid) -> str`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284). + `load` and `worksteal` have no scope concept at any layer, so under a user-typed + `--dist load` the options are refuse, substitute, or do not need protecting. +- `xdist_group` is applied + [worker-side](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) + and only when the worker's own `--dist` is literally `loadgroup`. A + controller-side substitution never reaches a worker. +- A worker crash re-runs only the *uncompleted* items of a work unit, on a fresh + process. Blocks 3..N then run against an empty mapping. Restarts are on by + default. +- A retry (`--reruns`, `--count`) re-runs a block against globals it already + mutated, so an expectation true only on attempt two reports **PASS**. + +Every one of these evaporates when the item is the sharing unit. + +## Test identity: never positional + +Two of the surveyed projects derive node ids from position — Sybil's +[`line:{line},column:{column}`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) +and pytest-examples' `path:start-end`. + +For a *documentation* test runner this is indefensible, because prose above +examples is the thing that changes most often. Adding a sentence renames every +downstream test, which breaks `--lf`, `--nf`, checked-in deselect files, xfail +lists and CI flake history. pytest-examples compounds it: the same string is the +dedupe key for its write-back, so an identity collision becomes file corruption. + +The rule: **author-declared name first, stable ordinal as fallback, and the +fallback shape invariant across configuration.** The test to apply is concrete — +adding a sentence to a page must rename zero tests. + +Two corollaries: + +- `DocTest.name` must be machine-independent. Embedding an absolute path makes a + checked-in `--deselect` resolve only on the machine that produced it, and puts a + home directory in JUnit XML. +- Column is not worth carrying. It adds churn and disambiguates nothing once names + exist. + +## What a node id does *not* promise + +Worth stating plainly, because it is the honest limit of the recommended design: +**no surveyed implementation makes a node id a promise of independent +runnability.** Selecting block two of a stateful page raises `NameError` under +Sybil, under `per-block`, and under any scheme that hands out per-block ids over +shared state. + +The choice is therefore not between "selectable blocks" and "unselectable blocks". +It is between an id that *claims* to be selectable and is not, and an id whose +granularity honestly matches what can be run alone. `merged` and ADR 0001 both +choose the latter; they differ only in whether the reporting granularity has to +match the selection granularity, and ADR 0001's answer is that it does not. + +## Fixture lifetime falls out of this + +A page collected as a `pytest.Module` **is** the module scope, so +`@pytest.fixture(scope="module")` already has page lifetime — no shim required. +Sybil reaches the same outcome through a `getparent` override that returns the +file collector when pytest asks for `Module`. + +The corollary is a trap for any design that shares state across items without +sharing the item: fixtures do not follow. A block that stashes a fixture-derived +object under a name keeps answering after that fixture has been finalized, because +the name outlives the object's lifetime. Making the item the sharing unit aligns +the two — the mapping and the fixtures have the same lifetime because they have the +same owner. + +## Anchors + +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) +- [`runtest` with `clear_globs=True`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) · + [`setup` globs update](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`xdist_group` append](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) · + [collection-mismatch abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) +- [Sybil `identify`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) · + [Sybil pytest integration](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py) diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md new file mode 100644 index 0000000..d627b6f --- /dev/null +++ b/notes/analyses/90-bibliography.md @@ -0,0 +1,162 @@ +# Bibliography + +Every external anchor cited by `docs/adrs/` and these notes, in one place. All +links are pinned to a tag, or — where a project publishes no tags — to a commit +reachable from trunk. Line anchors are only meaningful on a pinned ref and are not +used anywhere else. + +## CPython — `v3.14.2` + +[`python/cpython @ v3.14.2`](https://github.com/python/cpython/tree/v3.14.2) + +### `Lib/doctest.py` + +| Symbol | Anchor | Cited for | +|---|---|---| +| `TestResults` | [`:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) | 2-field namedtuple; `skipped` is an extra attribute | +| `register_optionflag` | [`:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) | the one append-only, idempotent cross-library registry | +| `_load_testfile` | [`:245`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245) | private, reached by `doctest_docutils` today | +| `DocTest.__init__` | [`:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | **copies** the globs mapping | +| `DocTest.__lt__` | [`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | compares names as text | +| `DocTestParser` | [`:609`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L609) | the injectable parser | +| `_EXAMPLE_RE` | [`:618`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618) | private, used for prompt sniffing | +| `DocTestFinder` | [`:844`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) | nominal type demanded by `DocTestSuite` | +| `report_*` hooks | [`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | the four supported in-loop seams; no `report_skip` here | +| `__run` | [`:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | name-mangled loop; overridable by mechanism | +| `compile(..., "single", ...)` | [`:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | the hard-coded mode `{testcode}` cannot use | +| `__record_outcome` | [`:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485) | arity and accumulator differ across supported versions | +| `__patched_linecache_getlines` | [`:1501`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501) | parses the `` filename shape back | +| `run()` save/restore | [`:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) | global interpreter state; not reentrant | +| `summarize` | [`:1590`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1590) | reads the accumulator the owned loop must write | +| `OutputChecker` | [`:1690`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) | the documented checker seam | +| `DebugRunner` | [`:1874`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1874) | `report_*` overriding as the sanctioned loop control | +| `testfile` | [`:2091`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2091) | the API `testdocutils` mirrors | +| `DocTestSuite` | [`:2467`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) | demands a nominal `DocTestFinder` | +| `DocFileSuite` | [`:2570`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) | demands a nominal `DocTestParser` | + +Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blob/v3.14.2/Doc/library/doctest.rst). + +### `Lib/asyncio/` + +| Symbol | Anchor | +|---|---| +| `AbstractEventLoop` | [`events.py:254`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) | +| `get_event_loop_policy` / `set_event_loop_policy` | [`events.py:804`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) · [`:817`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) | +| `BaseEventLoop` | [`base_events.py:417`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/base_events.py#L417) | +| `BaseProtocol` / `Protocol` | [`protocols.py:9`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L9) · [`:66`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L66) | +| `BaseTransport` / `Transport` | [`transports.py:9`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L9) · [`:148`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L148) | +| `Runner` / `run` / `_cancel_all_tasks` | [`runners.py:21`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21) · [`:169`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169) · [`:207`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207) | +| `Future` / `Task` | [`futures.py:31`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/futures.py#L31) · [`tasks.py:56`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/tasks.py#L56) | + +## pytest — `9.1.1` + +[`pytest-dev/pytest @ 9.1.1`](https://github.com/pytest-dev/pytest/tree/9.1.1) · +[`src/_pytest/doctest.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `pytest_collect_file` | [`:126`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L126) | not `firstresult` | +| `_is_setup_py` / `_is_main_py` | [`:141`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L141) · [`:155`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L155) | privates imported today | +| `_is_doctest` | [`:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | claims initpaths **before** `--doctest-glob` | +| `MultipleDoctestFailures` | [`:172`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) | the missing per-example result value, worked around | +| `_init_runner_class` / `PytestDoctestRunner` | [`:178`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178) · [`:181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181) | **unreachable by name** | +| `DoctestItem` | [`:251`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L251) | the subclassed item | +| `setup` | [`:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | `globs.update(...)` in place | +| `runtest` | [`:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | `clear_globs` defaults to `True` | +| `repr_failure` | [`:317-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) | reads each failure's own `test` | +| `_get_flag_lookup` | [`:385`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385) | lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES`, `NUMBER` | +| `get_optionflags` | [`:401`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L401) | read, not re-declared | +| `_get_continue_on_failure` | [`:410`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L410) | private helper imported today | +| `DoctestTextfile` | [`:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | `obj = None` as a class attribute | +| `_check_all_skipped` | [`:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | fires only once the item is running | +| `DoctestModule` / `parsefactories` | [`:500`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) · [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | autouse fixtures from conftest | +| `_get_checker` | [`:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662) | the checker that would have to be reimplemented | +| `_get_report_choice` | [`:703`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L703) | private helper | +| `doctest_namespace` | [`:721`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) | the fixture that survives plugin blocking today | + +## pytest-xdist — `v3.8.0` + +[`pytest-dev/pytest-xdist @ v3.8.0`](https://github.com/pytest-dev/pytest-xdist/tree/v3.8.0) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `parse_tx_spec_config` | [`workermanage.py:26-37`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) | list `extend`, so a negative multiplier contributes zero | +| `LoadScopeScheduling._split_scope` | [`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) | the only affinity primitive | +| `LoadFileScheduling._split_scope` | [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) | two-line override | +| `LoadGroupScheduling._split_scope` | [`loadgroup.py:24`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) | two-line override | +| collection-mismatch abort | [`load.py:259`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) · [`loadscope.py:359`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) | logs and runs zero tests | +| `xdist_group` node-id append | [`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) | worker-side, `loadgroup` only | + +## pytest-asyncio — `v1.4.0` + +[`pytest-dev/pytest-asyncio @ v1.4.0`](https://github.com/pytest-dev/pytest-asyncio/tree/v1.4.0) · +[`pytest_asyncio/plugin.py`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `Mode` | [`:82`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L82) | `str` enum so ini, CLI and internal value are one object | +| `PytestAsyncioSpecs` | [`:90`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) | its own hookspec namespace | +| `pytest_addoption` | [`:108`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108) | every option `default=None` | +| `_make_asyncio_fixture_function` | [`:210`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L210) | stamping scope on the function | +| `_get_asyncio_mode` | [`:222`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222) | resolve once, query once | +| `pytest_configure` | [`:295-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L295-L301) | detecting an unset default via the sentinel | + +## Sphinx — `v9.1.0` + +[`sphinx-doc/sphinx @ v9.1.0`](https://github.com/sphinx-doc/sphinx/tree/v9.1.0) · +[`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `is_allowed_version(spec, version)` | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | specifier first — the reverse of the local helper | +| `TestDirective` | [`:66`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66) | the directive base and its option handling | +| comment nodetype rule | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | `testsetup`/`testcleanup`/`:hide:` become `nodes.comment` | +| `:options:` gating | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111) | accepted only on `doctest` and `testoutput` | +| `TestGroup` / `add_code` | [`:200`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) · [`:207`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) | phase ordering; three silent-loss cases | +| `TestCode` | [`:235`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) | the parsed unit | +| `SphinxDocTestRunner` | [`:257`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257) | overrides a private method to swallow an `IndexError` | +| `DocTestBuilder` | [`:292`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L292) | builder coupling | +| `doctest.compile` rebinding | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) | process-global, never restored | +| `test_doc` | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) | group resolution and `*` | +| gated-node drop | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | no outcome, id or count | +| `type = "exec"` for testcode | [`:548`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) | the mode flip | + +Documentation: [`doc/usage/extensions/doctest.rst`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/doc/usage/extensions/doctest.rst). +Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/util/docutils.py), +[`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/application.py). + +## MyST-Parser — `v5.1.0` + +[`executablebooks/MyST-Parser @ v5.1.0`](https://github.com/executablebooks/MyST-Parser/tree/v5.1.0) + +| Symbol | Anchor | +|---|---| +| `create_myst_settings_spec` | [`parsers/docutils_.py:208`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208) | +| `Parser(RstParser)` | [`parsers/docutils_.py:235`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L235) | +| `settings_spec` | [`parsers/docutils_.py:241-245`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) | +| `MdParserConfig` (`myst_enable_extensions`, `myst_fence_as_directive`) | [`config/main.py`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) | + +## docutils — `docutils-0.22.2` + +The canonical repository is on +[SourceForge](https://sourceforge.net/p/docutils/code/); the GitHub copies are +third-party mirrors and are not linked here. Anchors name file and symbol at the +tagged release: + +| File | Symbol | Cited for | +|---|---|---| +| `docutils/parsers/rst/directives/__init__.py` | `_directives` | the process-global, rebindable registry | +| `docutils/parsers/rst/states.py` | `state_classes`, `doctest_block` line assignment | per-instance substitutability; last-line convention | +| `docutils/utils/__init__.py` | `Reporter.attach_observer`, `system_message` | observation separable from display | +| `docutils/nodes.py` | `Element.attributes`, `literal_block`, `comment`, `doctest_block` | the untyped attribute channel | + +Typed surface: [`typeshed stubs/docutils`](https://github.com/python/typeshed/tree/8c7256c/stubs/docutils). + +## Prior art + +| Project | Ref | Key anchors | +|---|---|---| +| Sybil | [`10.0.1`](https://github.com/simplistix/sybil/tree/10.0.1) | [`sybil.py:155-157`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (positional ids) · [`document.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/document.py) (one namespace, non-overlap invariant) · [`integration/pytest.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py) (one item per region) · [`region.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · [`testing.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/testing.py) (public extension-test helpers) | +| xdoctest | [`v1.3.2`](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | [`directive.py:58`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) (`REQUIRES` carries its reason) · [`plugin.py`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/plugin.py) (unregisters pytest's doctest plugin) | +| pytest-examples | [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | [`find_examples.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/find_examples.py) · [`run_code.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/run_code.py) · [`modify_files.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/modify_files.py) (byte offsets, invertible dedent, unguarded splice) | +| typeshed | [`8c7256c`](https://github.com/python/typeshed/tree/8c7256c) | [`stdlib/doctest.pyi`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | diff --git a/notes/analyses/README.md b/notes/analyses/README.md new file mode 100644 index 0000000..4114e78 --- /dev/null +++ b/notes/analyses/README.md @@ -0,0 +1,69 @@ +# Doctest ecosystem structural analyses + +Structural analysis of how the systems `doctest_docutils` sits between are built — +their core data structures, data flows, extension seams, and configuration models. +These are research notes. They inform the ADRs in `docs/adrs/` and decide nothing +themselves. + +The question they exist to answer: a doctest engine that must be vanilla-compatible +at the core, pluggable, usable as a pytest plugin, usable with docutils and +myst-parser, *and* speak all three communities' idioms is standing on three +upstreams with three separate extension models and three overlapping vocabularies. +What exactly does each of them require, and where do they contradict each other? + +## Method + +1. **Portable citations first.** Every external reference is a deep link to a + specific file **pinned at a git tag** — never `main`, `master`, `HEAD` or a bare + SHA. Line anchors are only used on a pinned ref, because they are meaningless + without one. These links are the reproducible source surface for the analysis. +2. **Source review second.** Confirm and deepen against checked-out source with + `rg`/`fd`. Local notes may inform drafting; tracked notes must be readable and + verifiable without a workstation path. +3. **Execute the load-bearing claims.** Anything an ADR rests on is run, not read. + Where a note says "verified", a snippet was executed and its output recorded. + +## Pinned versions + +| Project | Repo | Ref | +|---|---|---| +| CPython (`doctest`, `asyncio`) | `python/cpython` | `v3.14.2` | +| pytest | `pytest-dev/pytest` | `9.1.1` | +| pytest-xdist | `pytest-dev/pytest-xdist` | `v3.8.0` | +| pytest-asyncio | `pytest-dev/pytest-asyncio` | `v1.4.0` | +| Sphinx | `sphinx-doc/sphinx` | `v9.1.0` | +| MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` | +| Sybil | `simplistix/sybil` | `10.0.1` | +| xdoctest | `Erotemic/xdoctest` | `v1.3.2` | +| typeshed | `python/typeshed` | `8c7256c` (no tags; commit reachable from trunk) | +| docutils | SourceForge (the GitHub clones are third-party mirrors) | `docutils-0.22.2` | + +## Files + +- [`00-taxonomy.md`](00-taxonomy.md) — the design axes, as a classification matrix. +- Per-system structural docs: [`10-cpython-doctest.md`](10-cpython-doctest.md), + [`11-pytest-doctest.md`](11-pytest-doctest.md), + [`12-pytest-xdist.md`](12-pytest-xdist.md), + [`13-pytest-asyncio.md`](13-pytest-asyncio.md), + [`14-asyncio.md`](14-asyncio.md), + [`15-sphinx-ext-doctest.md`](15-sphinx-ext-doctest.md), + [`16-docutils-myst.md`](16-docutils-myst.md), + [`17-prior-art.md`](17-prior-art.md). +- Cross-cutting: [`20-data-structures.md`](20-data-structures.md), + [`21-data-flows.md`](21-data-flows.md), + [`22-extension-seams.md`](22-extension-seams.md), + [`23-namespace-scope-and-test-identity.md`](23-namespace-scope-and-test-identity.md). +- [`90-bibliography.md`](90-bibliography.md) — every pinned anchor cited by the + ADRs, in one place. + +Each per-system doc follows the same section order — classification · core data +structures · data flow · extension seams · configuration · what it cannot do · +anchors — so the systems are directly comparable, and the cross-cutting docs can +line them up column by column. + +`14-asyncio.md` is included even though `asyncio` has nothing to do with doctests. +It is the stdlib's own worked example of a pluggable architecture built out of +protocols, an abstract base, a policy indirection and a runner, by roughly the same +people and in roughly the same era as `doctest`'s extension model. Reading the two +side by side is the cheapest available answer to "what does the standard library +consider a good seam, and why does `doctest` have so few of them?" From 0657c59187762b1530c7538c4f0b0e603b12d255 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:21:45 -0500 Subject: [PATCH 04/24] docs(adrs[0001]): Correct the record against verified evidence why: Review found the record described PR #87's unmerged design as the status quo, and several claims did not survive checking against source. Every corrected claim below was re-verified by reading a pinned tag or by executing it. what: - Rewrite Context: trunk collects one DocTest per page; the groups, merge, skip lifting, exec runner and xdist scheduler are PR #87's proposal, named as such - Drop "per-block SKIPPED" from what the shape buys free, and add an outcome contract: TestReport.outcome is one scalar per item, so a mixed group either erases the skip or over-reports the whole group. Record subtests as the only sanctioned alternative and why it is not adopted - Credit sphinx.ext.doctest with the execution shape; the pytest identity is what is novel, not N DocTests per namespace - Replace the bare DocTest tuple with PlannedBlock and GroupPlan, so run_group can order phases, evaluate the gate and guarantee cleanup - Replace the compile-mode literal with a private ExecutionProfile, since PR #59's top-level await is a second policy a mode string cannot express - Move docutils node vocabulary out of the stdlib-only leaf, and move settings below the layers that read it - Add the item lifecycle contract, since half-reusing DoctestItem reintroduces the clear_globs wipe - Correct Sphinx: :options: on testcode is an unknown-option error, not a silent discard; :pyversion: is the silent one; cleanup does not run after setup failure, so always-cleanup is a divergence - Correct the parsefactories claim: conftest autouse fixtures arrive via FixtureManager.pytest_plugin_registered - Correct the nominal-subclassing claim: stdlib accepts a duck-typed parser or finder; typeshed is what demands the class - Narrow the xdist sentence: identical collection still binds - Repin Sphinx anchors to v8.2.3, the version this project resolves --- docs/adrs/0001-typed-vanilla-doctest-core.md | 370 +++++++++++++------ 1 file changed, 262 insertions(+), 108 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 51a4c9f..23f86f5 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -7,66 +7,77 @@ Date: 2026-08-02 ## Context +### What ships today + `doctest_docutils` re-implements the *finding* half of {func}`doctest.testfile` -over a docutils or MyST doctree, keeps CPython's *running* half, and adds a -grouping layer — namespace, phase, merge, lift — that exists in neither parent. +over a docutils or MyST doctree and keeps CPython's *running* half. `pytest_doctest_docutils` wraps that in a pytest plugin. -The two modules work. They are also the join point of three separate -vocabularies and three separate extension models, and the seams between them -were never named. Four structural costs follow. +Two facts about the shipped code set the problem. + +**A page is one test.** `DocutilsDocTestFinder` produces a single +{class}`doctest.DocTest` per document. A page cannot build state across the prose +that explains it, because there is no unit smaller than the page and no unit that +groups blocks within one. + +**The plugin blocks the plugin whose internals it imports.** +`pytest_configure` calls `config.pluginmanager.set_blocked("doctest")`, and the +same module then imports that plugin's private helpers. This survives only +because `_pytest/fixtures.py` has no `pytest_plugin_unregistered` handler, so the +already-parsed `doctest_namespace` fixture outlives unregistration. + +### What a shared namespace costs + +[PR #87](https://github.com/git-pull/gp-libs/pull/87) is an open, unmerged +attempt at the first problem: it adds Sphinx-style groups, a merge step, phase +ordering, skip lifting, an exec-mode runner and an xdist scheduler. **None of it +has shipped in any release, and none of it is on trunk.** It is described here as +a design under review, not as the status quo, because what it had to build to +work is the evidence this record turns on. + +Three costs are worth naming, because a clean-room design must either pay them +again or explain why it does not. -**The finder is one method.** `DocutilsDocTestFinder._find` performs format -dispatch, node filtering, group resolution, wildcard expansion, name-collision -detection, `:skipif:` evaluation, `testoutput` pairing, test construction, -merging, splitting and naming, in one body with two nested closures, with -configuration threaded through as positional parameters. There is no boundary -between *document → blocks*, *blocks → tests*, and *tests → items*, so a change -to any one of them is a change to all three. +**Merging blocks fights line-number fidelity.** A merged group is one `DocTest` +with one `docstring` and one `lineno`, and both doctest's `%03d` gutter and +pytest's `repr_failure` reconstruct locations by slicing that single string. So +the blocks must be laid out on a synthetic page with blank-line padding and a +clamp, and a wholly-skipped block must be lifted back out to report at all. -**It reaches through name-mangled CPython privates.** Sphinx's prompt-free -`{testcode}` form needs `compile()` in `"exec"` mode, but the per-example loop +**Prompt-free `{testcode}` needs a CPython private.** The per-example loop [`DocTestRunner.__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) hard-codes `"single"` -([`Lib/doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400)). -`sphinx.ext.doctest` solves this by rebinding `doctest.compile` process-wide and -never restoring it -([`sphinx/ext/doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310)) — +([`Lib/doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400)), +and `sphinx.ext.doctest` gets around that by rebinding `doctest.compile` +process-wide and never restoring it +([`sphinx/ext/doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310)) — unavailable to a library that loads into every pytest session that installed it. -`_exec_mode_run` therefore clones the mangled method's code object into a fresh -{class}`types.FunctionType` whose globals map `compile` to a local helper, guarded -by probing `co_freevars` and `co_names` and degrading with a logged error. That -mechanism also carries a latent defect: its globals are a snapshot of -`vars(doctest)` taken at import, so a later rebind of a module-level name in -`doctest` is invisible to the clone while remaining visible to the stock runner, -and two runners in one process disagree. - -**It blocks the plugin whose internals it imports.** `pytest_configure` calls -`config.pluginmanager.set_blocked("doctest")`, and the same module then imports -that plugin's private helpers — `_is_setup_py`, `_is_main_py`, `_get_checker`, -`_get_continue_on_failure` — and reads four ini and CLI options the blocked -plugin declared. This survives only because `_pytest/fixtures.py` has no -`pytest_plugin_unregistered` handler, so the already-parsed `doctest_namespace` -fixture outlives unregistration. - -**It forks pytest-xdist to protect state xdist cannot see.** Because a live -`globs` mapping is a Python object and only execnet-serializable builtins cross a -worker boundary, a shared namespace must either be merged into one -{class}`doctest.DocTest` or kept on one worker. Keeping it there means a -scheduler, and the only affinity primitive in all of xdist is +PR #87's answer clones the mangled method's code object into a fresh +{class}`types.FunctionType` whose globals map `compile` to a local helper. That +carries a latent defect: those globals are a snapshot of `vars(doctest)` taken at +import, so a later rebind of a module-level name in `doctest` is invisible to the +clone while remaining visible to the stock runner, and two runners in one process +disagree. + +**A live shared mapping forces an xdist fork.** Only execnet-serializable +builtins cross a worker boundary, so a shared namespace must either be merged +into one `DocTest` or kept on one worker. Keeping it there means a scheduler, and +the only affinity primitive in all of xdist is [`LoadScopeScheduling._split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284), a pure function on node-id strings. The controller never collects; it learns the -suite only as node ids arriving from workers. So `_shared_page` and `_is_page` -re-derive "these ids share state" from strings, and `_worker_count` re-implements +suite only as node ids arriving from workers. So PR #87 re-derives "these ids +share state" from strings, and re-implements [`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26) including its quirks. -Underneath all four is one conflation. **The granularity of test identity and the -granularity of shared state are different axes, and the current design couples -them.** `namespace_items = merged` gives one `DocTest` and one node id per group; -`per-block` gives N `DocTest`s, N node ids, and one live mapping — a node id that -raises `NameError` when selected alone. Every surveyed project makes the same -mistake or a worse one (see [](#prior-art)). +### The conflation underneath + +All three costs follow from one thing. **The granularity of test identity and the +granularity of shared state are different axes, and every surveyed design couples +them.** PR #87's two settings are the clearest illustration: `merged` gives one +`DocTest` and one node id per group; `per-block` gives N `DocTest`s, N node ids +and one live mapping — a node id that raises `NameError` when selected alone. +Sybil ships the second shape without acknowledging it (see [](#prior-art)). ## Decision @@ -76,11 +87,27 @@ One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block `DocTest`s run in phase order against one live `globs` mapping that never leaves the item. -This occupies the cell no surveyed project occupies: N `DocTest`s, one node id. -Per-block failure locations, per-block gutters, per-block `SKIPPED`, per-block -"location unknown" — while `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode -are structurally incapable of splitting the shared state, because there is only -one item to schedule. +The execution shape is not new: `sphinx.ext.doctest` already runs several +`DocTest`s against one shared group namespace. What is new is giving that shape a +**pytest identity**. Sphinx produces no selectable, reportable unit for a group — +every block in it shares one `DocTest.name`, which is why `SphinxDocTestRunner` +overrides a private stdlib method to swallow the resulting `IndexError`. Mapping +the group onto one {class}`pytest.Item` is the contribution. + +What that buys, for free and with no override of `repr_failure` or `reportinfo`: +per-block failure locations, per-block gutters, and per-block "location unknown". +Meanwhile `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode are structurally +incapable of splitting the shared state, because there is only one item to +schedule. + +**A per-block `SKIPPED` outcome is not among them.** +{class}`pytest.TestReport`'s `outcome` is one scalar per item, so a group holding +one all-`SKIP` block and one passing block reports `PASSED` with the skip erased. +Signalling the skip instead flips the *whole* group to `SKIPPED`. Surfacing it +per block requires pytest's builtin-but-experimental `subtests` plugin, appears +in the terminal gutter and `-rs` only at `verbosity_subtests >= 1`, and never +becomes a separate ``, `--lf` entry or rerunnable unit. See +[](#the-outcome-contract). ### The three facts this rests on @@ -109,9 +136,9 @@ and `lineno` for free. A block reached through `.. include::` reports the takes pytest's honest `EXAMPLE LOCATION UNKNOWN` branch **without poisoning its siblings**. No override of `repr_failure` or `reportinfo` is required. -This is what retires `_merge_blocks`: the synthetic merged page, its blank-line -padding and its `max(..., len(lines))` clamp exist only to reconstruct locations -from a single spliced docstring. +This is what makes merging unnecessary: the synthetic page, its blank-line +padding and its clamp exist only to reconstruct locations from a single spliced +docstring, and there is no spliced docstring here. **2. `_DocTestRunner__run` is an ordinary attribute override.** Name mangling rewrites the *call site* at compile time, so the `self.__run(...)` lookup inside @@ -148,43 +175,109 @@ variables, all in its own `finally` ([`Lib/doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). That contract is not reproduced; it is inherited. -**3. Making the item the sharing unit dissolves distribution.** A live mapping -never crosses a process boundary, so there is nothing for xdist to split, under -any `--dist` mode. No affinity primitive, no scheduler substitution, no +**3. Making the item the sharing unit dissolves the *affinity* problem.** A live +mapping never crosses a process boundary, so there is nothing for xdist to split, +under any `--dist` mode. No affinity primitive, no scheduler substitution, no `parse_tx_spec_config` fork, no node-id string sniffing. +The identical-collection requirement is untouched by this and still binds at any +granularity. It is satisfied separately, by collection being a pure function of +(bytes, argv, ini) — which is why `:skipif:` is carried through collection +unevaluated. + +(the-outcome-contract)= + +### The outcome contract + +One item means one item outcome. That is a real cost of this design and it is +stated here rather than discovered later. + +| Signal | Granularity | Notes | +|---|---|---| +| Failure location, `want`/`got`, gutter | **per block** | `repr_failure` iterates failures and reads each one's own `DocTest` | +| `EXAMPLE LOCATION UNKNOWN` | **per block** | a block with `lineno=None` does not affect its siblings | +| `passed` / `failed` / `skipped` | **per item** | `TestReport.outcome` is one scalar | +| JUnit `` | **per item** | node reporters are keyed by node id | +| `--lf`, `-k`, `--deselect`, rerun unit | **per item** | | + +The skip case is the sharp edge, and it cuts both ways. If the item swallows a +block's skip, a group with one all-`SKIP` block and one passing block reports +`PASSED` and the skip leaves no record — no count, no `-rs` line, no JUnit +``. If the item raises instead, the whole group reports `SKIPPED` even +though a sibling passed. pytest's own doctest plugin takes the second horn only +when *every* example is skipped, via `_check_all_skipped`. + +This design takes the same position: **skip the item when every runnable block is +skipped; otherwise report partial skips as typed block detail, not as a pytest +outcome.** No extra reports are synthesized. + +`subtests` — a builtin pytest plugin since 9.0, exporting `pytest.Subtests` and +`pytest.SubtestReport` — can emit per-block outcomes, and is the only sanctioned +mechanism that can. It is not adopted here: pytest documents it as experimental, +its output is invisible at default verbosity, and it produces no separate JUnit +entry, so it would buy terminal detail at the cost of depending on an unstable +surface. Revisit if it stabilizes. + ### Layers Dependencies flow one way, from the leaf toward the hosts. No layer may import a layer above it. ```text +settings one frozen Settings, resolved once <- leaf; everything reads it + | blocks inert data: Block, BlockKind, Phase, Diagnostic, Example | (stdlib imports only) markup/ _rst, _myst, _text, _python -> (blocks, diagnostics) | -project grouping, pairing, phase order, naming -> GroupTest +project grouping, pairing, phase order, naming -> GroupPlan | runner owns the per-example loop; phase sequencing | -settings one frozen Settings, resolved once - | pytest_doctest_docutils collection, items, globs lifetime, reporting ``` +`settings` sits at the bottom, not beside the host, because `Frontend.parse` and +`project()` both take a `Settings`. A layer every other layer reads is a leaf. + | Layer | Owns | Must not know | |---|---|---| -| `blocks` | `Block`, `BlockKind` registry, `Phase`, `Diagnostic`, `BlockAttributes`, `Example(doctest.Example)` carrying compile mode | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | -| `markup/` | Text → `(blocks, diagnostics)`. Line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration | Groups as a runtime concept, `DocTest`, pytest, pairing | +| `settings` | One frozen `Settings` resolved once, with `None` sentinels at the resolve boundary so a future default change is announceable | pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | +| `blocks` | `Block`, `BlockKind` registry, `Phase`, `Diagnostic`, `PlannedBlock`, `GroupPlan`, `ExecutionProfile`, `Example(doctest.Example)` | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | +| `markup/` | Text → `(blocks, diagnostics)`. **The whole docutils vocabulary**: which node classes each kind may arrive as, the `BlockAttributes` stamp, line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration, per-kind `option_spec` | Groups as a runtime concept, `DocTest`, pytest, pairing | | `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | -| `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | -| `settings` | One frozen `Settings` resolved once, with `None` sentinels at the resolve boundary so a future default change is announceable | pytest's `Config`, argparse, ini format, Sphinx's `app` | -| `pytest_doctest_docutils` | Options, `Document(pytest.Module)`, `DocutilsItem`, group `globs` lifetime, built-in-plugin dedup, surfacing diagnostics | docutils node classes, MyST configuration, grouping rules | +| `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing, run-time `:skipif:`, the profile's context manager, and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | +| `pytest_doctest_docutils` | Options, `Document(pytest.Module)`, `DocutilsItem`, group `globs` lifetime, the outcome contract, built-in-plugin composition, surfacing diagnostics | docutils node classes, MyST configuration, grouping rules | `pytest_doctest_docutils._compat` is the only module that imports `_pytest.doctest`, behind a pinned support matrix. See {doc}`0006-pytest-private-api-compatibility`. +### Item lifecycle + +The custom item is load-bearing, and half-reusing {class}`pytest.DoctestItem` +reintroduces the exact bug this design exists to avoid. The contract, stated so +an implementer cannot get it wrong by omission: + +1. **Collection** builds one `GroupPlan` per (document, group) and one item per + plan. An empty plan yields no item. +2. **`setup()`** clears the group mapping in place, then calls `super().setup()` + so fixtures inject into that same object. Clearing in place rather than + rebinding is what keeps `item.globs is plan.globs` true for every block, and + what stops attempt two of a `--reruns` run from reading attempt one's + mutations. +3. **`runtest()`** is overridden. It must not delegate to + `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` + defaulting to `True` — that would empty the shared mapping after the first + block. It calls `run_group()`, which runs each `PlannedBlock` in phase order + with `clear_globs=False`, evaluates `:skipif:` against the group mapping, and + wraps the body in a `try`/`finally` so cleanup runs whether or not the body + raised. +4. **Outcome** follows [](#the-outcome-contract): skip the item only when every + runnable block is skipped. +5. **`repr_failure`** is inherited unchanged. It already reads each failure's own + `DocTest`. + ### Vocabulary Goal (e) — speaking doctest's, pytest's *and* Sphinx's idioms — is mostly a @@ -197,8 +290,8 @@ referents. Each term below is decided once and used only that way. | namespace | — | [`doctest_namespace`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) means *injected names* | — | **Not** used for the sharing unit; pytest owns the word | | group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | | scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The block-vs-document axis becomes `share` | -| test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | -| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | pytest's meaning, doctest's mechanism. Sphinx's drop is deliberately rejected | +| test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | +| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L443-L444) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | | directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | | optionflags | an int bitmask; `register_optionflag` | `doctest_optionflags` ini | `:options:` plus `doctest_default_flags` | Keep verbatim. `register_optionflag` is the one genuinely cross-library extension point | | setup / cleanup | `setUp`/`tearDown` on the suite builders | fixtures | `testsetup`/`testcleanup` directives | Author-facing names stay Sphinx's; `phase` is the internal ordering axis; a fixture is never "setup" | @@ -229,17 +322,21 @@ class Block(t.NamedTuple): class BlockKind(t.NamedTuple): name: str phase: Phase - mode: t.Literal["single", "exec"] + profile: ExecutionProfile # how a body of this kind is compiled and run pairs_with: str | None grouped: bool - node_types: tuple[str, ...] # docutils tagnames; "comment" for testsetup/:hide: - option_spec: t.Mapping[str, t.Callable[[str], object]] -class GroupTest(t.NamedTuple): +class PlannedBlock(t.NamedTuple): + phase: Phase + skipif: str | None # UNEVALUATED; gated in run_group(), not at collection + test: doctest.DocTest # one block, its own filename/lineno/docstring + + +class GroupPlan(t.NamedTuple): group: str - tests: tuple[doctest.DocTest, ...] # one per block, in phase order - globs: dict[str, t.Any] # the live mapping, shared by every test above + blocks: tuple[PlannedBlock, ...] # in phase order + globs: dict[str, t.Any] # the live mapping, shared by every block above ``` `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block @@ -248,26 +345,51 @@ source=None` from docutils, and an `.. include::`-ed block numbers against the *included* file. Both propagate to `DocTest.lineno=None` and pytest's honest "location unknown", rather than to a fabricated number. -`BlockKind.node_types` is likewise load-bearing: `testsetup`, `testcleanup` and -any `:hide:` block are emitted as {class}`docutils.nodes.comment`, not -`literal_block` -([`sphinx/ext/doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93)). -A walker restricted to `literal_block` silently loses every one of them while the -page still renders. +`PlannedBlock` exists because `run_group()` owns phase sequencing, run-time +`:skipif:` evaluation and a cleanup `finally` — and cannot do any of the three +from a bare tuple of `DocTest`s. A `DocTest` carries no phase and no gate, so the +plan has to. Tagging each entry also makes the ordering self-describing rather +than a convention a comment asserts. + +`ExecutionProfile` is a **private** protocol carrying compile mode, extra compile +flags, and an optional per-group context manager. It is not a `Literal["single", +"exec"]`, because a second execution policy already exists in this repository: +[PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await`, +which needs `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` and an event-loop lifetime that a +mode string cannot express. The profile's context manager is what `run_group()` +enters, so that lifetime is served without overriding `run()` and stdlib's +save-and-restore `finally` stays inherited. It stays private until a caller +outside the package needs it; PR #59 is the in-repo caller that justifies it +existing at all. + +**Which docutils node classes a kind may arrive as is a front-end concern, not a +`BlockKind` field.** `testsetup`, `testcleanup` and any `:hide:` block are +emitted as {class}`docutils.nodes.comment`, not `literal_block` +([`sphinx/ext/doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93)), +and a walker restricted to `literal_block` silently loses every one of them while +the page still renders. That requirement is real, but it belongs to `markup/` +alongside the rest of the node vocabulary — putting it on `BlockKind` would drag +docutils into a layer declared stdlib-only. ### Typing The runtime objects are stdlib's, unconditionally. Precision lives in a parallel layer that never changes what is constructed. -- **`Protocol` for the seams, nominal subclassing for the concrete classes.** - `Frontend` is a `Protocol` so a third party can supply one structurally; the - shipped parser also subclasses {class}`doctest.DocTestParser` so it stays - passable to `doctest.testfile(parser=...)` and - {func}`doctest.DocFileSuite`. Typeshed's signatures demand the class, not the - shape — `DocTestFinder.__init__(parser: DocTestParser = ...)` — so a compatible - `find()` alone is not enough, which is why `DocutilsDocTestFinder` cannot be - passed to `DocTestSuite(test_finder=...)` today. +- **`Protocol` for the seams, nominal subclassing for type-checkability.** + `Frontend` is a `Protocol` so a third party can supply one structurally. The + shipped parser *also* subclasses {class}`doctest.DocTestParser` — not because + the interpreter requires it (stdlib `doctest` performs no `isinstance` check on + `parser` or `test_finder`; a duck-typed object works at runtime) but because + typeshed's signatures name the class, so a checker rejects what the interpreter + accepts. The subclass buys type-checkability, not passability. Passability is a + matter of matching the call signature: a finder whose `find()` takes a string + first cannot be handed to `DocTestSuite`, which passes a module — and + subclassing does not fix that. + + One genuine nominal edge does exist: `DocTestSuite` sorts its results, and + `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom + finder must return real `DocTest` objects. - **`TypedDict` at the docutils boundary.** `BlockAttributes` types what a directive stamps on a node, with one narrowing accessor that validates once. This is where `Any` currently enters: `str(node.get("test") or ...)` and @@ -312,7 +434,8 @@ that owns the loop must probe for it rather than assume it. | `DoctestItem.setup()` does `self.dtest.globs.update(globs)`, so the mapping must be mutable and survive collection → setup → run | [`doctest.py:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | | `runtest()` calls `run(self.dtest, out=failures)` with `clear_globs` defaulting to `True` — which would empty a shared mapping after the first block | [`doctest.py:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | | `PytestDoctestRunner` is defined *inside* `_init_runner_class()` and is not importable, so the `OutcomeException` re-raise, `BdbQuit` → `outcomes.exit` and `continue_on_failure` handling do **not** come for free | [`doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181) | -| A page must be a `pytest.Module` with `obj = None`, and the collector must call `parsefactories` | [`doctest.py:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421), [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | +| A page must be a `pytest.Module` with `obj = None` as a **class** attribute, or the `Module` machinery tries to import the `.rst`/`.md` file | [`doctest.py:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | +| Conftest autouse fixtures reach page items through `FixtureManager.pytest_plugin_registered`, **not** through a collector calling `parsefactories` — that call is `DoctestModule`-only, for fixtures defined in the collected `.py` itself | [`fixtures.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/fixtures.py), [`doctest.py:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | | `_is_doctest` claims any `.txt`/`.rst` **initial path before consulting `--doctest-glob`**, so `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob | [`doctest.py:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | | An empty `DocTest` must not be yielded as an item | [`doctest.py:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | @@ -325,18 +448,33 @@ that owns the loop must probe for it rather than assume it. | `xdist_group` is honoured only when the *worker's own* `--dist` is `loadgroup`, and works by appending `@name` to `item._nodeid` | [`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) | | `parse_tx_spec_config` builds a list, so a negative multiplier contributes zero specs — `xspeclist.extend([spec] * num)`, not a sum | [`workermanage.py:26-37`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) | -Making the item the sharing unit removes the need to satisfy the first three at -all. The fourth is why `_worker_count` is deleted rather than fixed. +Making the item the sharing unit removes the need to satisfy the second and third +at all. **The first still binds at any granularity** — identical collection is +required whether a page yields one item or fifty — and is satisfied instead by +collection being a pure function of (bytes, argv, ini). The fourth is why a +worker-count fork is not worth carrying: `pytest_xdist_setupnodes(config, specs)` +hands over the already-expanded spec list and never raises. -### Sphinx (v9.1.0) +There is a fifth hazard the single-item shape also removes, worth naming because +it has no guard otherwise: a worker crash re-runs only the *uncompleted* items of +a work unit on a fresh process, so blocks 3..N of a shared group would run +against an empty mapping. Worker restarts are on by default. + +### Sphinx (v8.2.3, the version this project resolves) | Constraint | Anchor | |---|---| -| `testsetup`, `testcleanup` and `:hide:` blocks are emitted as `nodes.comment` | [`doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | -| A `:skipif:`-gated node is dropped during collection, with no outcome, id or count | [`doctest.py:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | -| `:options:` is accepted only on `doctest` and `testoutput`; a `testcode`'s own options are discarded in favour of its output block's | [`doctest.py:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111), [`:207`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) | -| `is_allowed_version(spec, version)` takes the specifier **first** — the reverse of this project's local helper | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | -| `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310), [`:548`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) | +| `testsetup`, `testcleanup` and `:hide:` blocks are emitted as `nodes.comment` | [`doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | +| A `:skipif:`-gated node is dropped during collection, with no outcome, id or count | [`doctest.py:449-450`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | +| `:options:` is **not in `TestcodeDirective.option_spec`**, so writing it on a `testcode` is an unknown-option error that drops the block — a loud rejection, not a silent discard | [`doctest.py:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180), [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111) | +| `:pyversion:` **is** in `TestcodeDirective.option_spec` and is silently ignored there — the real silent loss on a testcode | [`doctest.py:177`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L177) | +| Cleanup does **not** run when setup fails: the group returns early | [`doctest.py:554-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L554-L556) | +| `is_allowed_version(spec, version)` takes the specifier **first** | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | +| `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310), [`:549`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L549) | + +Sphinx v9.1.0 changes the default group an unargumented block joins. Any +group-naming behaviour matched against "Sphinx" has to say which Sphinx, and +{doc}`0005-line-recovery-for-nested-blocks` proposes moving this floor. ## Tensions @@ -371,25 +509,41 @@ evaluates no user Python, so it is a pure function of (bytes, argv, ini). *Price `--collect-only` no longer shows which blocks will skip. **Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards -an orphan `testoutput`, silently overwrites a duplicate one, and silently drops a -`testcode`'s own `:options:`. *Position:* keep the behaviour, add a diagnostic -with a stable code. *Price:* a page warns under pytest and is silent under -`sphinx-build`; the results still match. +an orphan `testoutput`, silently discards a `testoutput` following a `doctest` +block, silently overwrites a duplicate `testoutput`, and silently ignores +`:pyversion:` on a `testcode`. *Position:* keep the behaviour, add a diagnostic +with a stable code for each of the four. *Price:* a page warns under pytest and +is silent under `sphinx-build`; the results still match. + +**Guaranteed cleanup versus Sphinx's setup-failure short-circuit.** When setup +fails, Sphinx returns before the cleanup phase, skipping page `testcleanup` +blocks and `doctest_global_cleanup` alike. *Position:* run cleanup +unconditionally in a `try`/`finally`, because a page that spawns a server in +setup and fails mid-way should not leak it. *Price:* a page whose setup fails +leaves different residue under pytest than under `sphinx-build`, and that is a +deliberate divergence rather than an oversight. **Owning the loop versus tracking CPython.** *Position:* own it, because constraint 4 makes it the only way to control compile mode without a process-global patch or a code-object clone. *Price:* a version shim and a conformance harness. See {doc}`0002-runner-conformance-across-cpython`. -## What this deletes +## What this avoids + +Only one item here exists on trunk today: `set_blocked("doctest")` and its +unblock path, which {doc}`0006-pytest-private-api-compatibility` replaces. The +rest are machinery [PR #87](https://github.com/git-pull/gp-libs/pull/87) has to +build to make a shared namespace work, and which this shape never needs: -`_find`'s monolith · `_merge_blocks` and its padding and clamp · -`_split_skipped_blocks` and `_lifted_name` · `_exec_mode_run`, `_compile_source` -and `_ExecSource` · `_worker_count` · `_shared_page` and `_is_page` · -`_splitting_scheduler` and both xdist hooks · the `_init_runner_class` fork · -`set_blocked("doctest")` and its unblock path. +- the merge step, its blank-line padding and its `max()` clamp +- skip lifting, which pulls a wholly-skipped block back out of a running group +- the code-object clone with its stale `vars(doctest)` snapshot +- the worker-count fork of `parse_tx_spec_config` +- the node-id string sniffing that infers "these ids share state" +- the scheduler substitution and both xdist hooks -**The result is not smaller.** It lands roughly flat against the current source. +**The result is not smaller.** It lands roughly flat against a finder that does +the same job. Under this project's rule that every function carries a NumPy docstring with a working doctest, splitting one long method into five functions costs prose it did not previously pay. The value is fewer hazards, not fewer lines: a code-object From 47c76cc4976a41754a3160d244687353c08a6da5 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:22:47 -0500 Subject: [PATCH 05/24] docs(adrs[0003]): Recast per-block items as a rejected design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: The record proposed deprecating doctest_docutils_namespace_items, but neither that setting nor its scope twin has shipped — both live on PR #87, in no release and on no tag. Deprecating an unshipped setting fails the Published-Release Test, and there is no downstream to warn. what: - Retitle and reslug: the question is whether the shape should ship, not how to retire it - State plainly that nothing shipped, so there is no migration path, no warning and no downstream grep - Give the shape its due first: merging costs node ids, fixture lifetime and gutter locality, which is what per-block answers - Then give the four reasons against, each with its guard: an id that NameErrors when selected, a mapping that cannot cross a worker, a mapping that cannot survive a rerun, and a crash tail that has no guard at all - Note that the guards are the cost of the shape, not incidental - Record the honest limit of the alternative: no per-block outcome and no per-block node id --- docs/adrs/0001-typed-vanilla-doctest-core.md | 2 +- docs/adrs/0003-rejecting-per-block-items.md | 81 +++++++++++++++++++ ...0003-retiring-per-block-namespace-items.md | 54 ------------- docs/adrs/index.md | 2 +- 4 files changed, 83 insertions(+), 56 deletions(-) create mode 100644 docs/adrs/0003-rejecting-per-block-items.md delete mode 100644 docs/adrs/0003-retiring-per-block-namespace-items.md diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 23f86f5..6feb84f 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -688,7 +688,7 @@ the narrow default set in {doc}`0004-diagnostics-as-data`. This ADR fixes the architecture. Five decisions it defers get their own records: {doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven -equivalent), {doc}`0003-retiring-per-block-namespace-items` (the deprecation +equivalent), {doc}`0003-rejecting-per-block-items` (the deprecation path), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed), {doc}`0005-line-recovery-for-nested-blocks` (the optional last step), and {doc}`0006-pytest-private-api-compatibility` (the quarantine and its matrix). diff --git a/docs/adrs/0003-rejecting-per-block-items.md b/docs/adrs/0003-rejecting-per-block-items.md new file mode 100644 index 0000000..f3c5ebc --- /dev/null +++ b/docs/adrs/0003-rejecting-per-block-items.md @@ -0,0 +1,81 @@ +(adr-0003-rejecting-per-block-items)= + +# ADR 0003: Rejecting per-block items over a shared mapping + +Status: Draft +Date: 2026-08-02 + +## Context + +[PR #87](https://github.com/git-pull/gp-libs/pull/87) proposes two settings that +together choose how a page's blocks are collected. One of them, +`doctest_docutils_namespace_items = per-block`, keeps a node id for every block +of a shared page and hands those blocks one live `globs` mapping rather than +merging them into a single test. + +**Neither setting has shipped.** Both live on an open branch, in no release and +on no tag. There is nothing to deprecate, and this record does not propose a +deprecation — it records why the shape should not ship. + +## The shape, and why it is attractive + +`per-block` answers a real complaint about merging. Merging a group into one +`DocTest` collapses N node ids into one, merges fixture lifetime across the whole +group, and makes the failure gutter span the page. Keeping one id per block fixes +all three, and on a large documentation tree the difference is the bulk of the +suite's visible granularity. + +## Why it should not ship + +**A node id that cannot be selected is not a node id.** Selecting block three of +a stateful page raises `NameError`, because the blocks that bound the names it +reads did not run. The id promises an addressable unit and does not deliver one. + +**A live mapping cannot cross a process.** Only execnet-serializable builtins +reach an xdist worker, so the shape needs a scheduler that keeps a page whole — +and the only affinity primitive in xdist is +[`_split_scope(nodeid) -> str`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284). +Under a user-typed `--dist load` there is no scheduler to influence, so the +options collapse to refusing the run. + +**A live mapping cannot survive an item running twice.** A retry re-runs a block +against globals it already mutated, so an expectation true only on the second +attempt reports as a pass. Guarding that means refusing reruns. + +**A worker crash re-runs only the uncompleted tail** of a work unit, on a fresh +process — so blocks 3..N of a shared group run against an empty mapping, and +worker restarts are on by default. This one has no guard at all. + +Those four are why the branch also carries a worker-count fork, a page-inference +heuristic over node-id strings, a scheduler substitution, a scheduler refusal and +a rerun refusal. The guards are the cost of the shape, not incidental. + +## Decision + +Do not ship per-block items over a shared mapping, under this or any spelling. + +{doc}`0001-typed-vanilla-doctest-core` reaches the same granularity goal from the +other side: one item per group, holding one `DocTest` per block. That gives +per-block failure locations, gutters and "location unknown" without a shared +mapping ever becoming schedulable, so none of the four guards is needed. + +What it does not give is a per-block *outcome* or a per-block *node id*. That +limit is honest and is recorded in {doc}`0001-typed-vanilla-doctest-core`'s +outcome contract, rather than papered over with an id that raises when used. + +## Consequences + +Because nothing shipped, there is no migration path to write, no deprecation +warning to add and no downstream grep to run. If PR #87 lands before this +architecture does, this record converts into exactly those things — and that is +the argument for settling the two in order rather than in parallel. + +## Open + +- Whether a human-facing block *label* — in failure text and the report header, + never as a node id — is worth adding later, so a reader can find the failing + block without the design promising `-k` isolation. Not in a first version. +- Whether `--doctest-docutils-namespace-scope` should be renamed to + `--doctest-docutils-share` before or after this architecture lands. + {doc}`0001-typed-vanilla-doctest-core` settles the vocabulary; the rename is + independently schedulable and, since neither spelling has shipped, cheap. diff --git a/docs/adrs/0003-retiring-per-block-namespace-items.md b/docs/adrs/0003-retiring-per-block-namespace-items.md deleted file mode 100644 index e9c7787..0000000 --- a/docs/adrs/0003-retiring-per-block-namespace-items.md +++ /dev/null @@ -1,54 +0,0 @@ -(adr-0003-retiring-per-block-namespace-items)= - -# ADR 0003: Retiring `namespace_items = per-block` - -Status: Draft -Date: 2026-08-02 - -## Context - -`doctest_docutils_namespace_items` selects between `merged` (the default: a group -collects as one test) and `per-block` (each block keeps its own node id while the -blocks share one live `globs` mapping). - -{doc}`0001-typed-vanilla-doctest-core` makes the setting unnecessary by -decoupling the two axes it was invented to trade between: one pytest item per -group, with one {class}`doctest.DocTest` per block inside it. That gives -`per-block`'s per-block failure locations, gutters and `SKIPPED` reporting -without `per-block`'s shared live mapping. - -It also removes `per-block`'s cost. A live mapping is a Python object, so it -neither crosses a worker process nor survives an item running twice. Guarding it -is the sole reason `_worker_count`, `_shared_page`, `_is_page`, -`_splitting_scheduler` and both xdist hooks exist. - -The setting's remaining distinguishing feature is a node id per block — and those -ids raise `NameError` when selected alone, because selecting one block does not -run the blocks that bound the names it reads. A node id that cannot be selected -is not a node id. - -## Question - -`per-block` is a shipped CLI option with an ini twin and its own how-to sections. -How does it get retired without breaking a user who set it? - -## Direction - -Announce it as a breaking change with the migration path stated plainly, and keep -the option accepting `per-block` for one minor release, mapped to `merged` with a -{class}`pytest.PytestDeprecationWarning`. - -Before landing, confirm no downstream consumer sets it. -`--doctest-docutils-modules` and `--no-doctest-docutils-modules` keep their exact -current spelling and `dest`, because downstream projects carry them in `addopts`. - -## Open - -- Whether the deprecation shim warns once per session or once per page. -- Whether `--doctest-docutils-namespace-scope` is renamed to `--doctest-docutils-share` - in the same release or a later one. {doc}`0001-typed-vanilla-doctest-core` - decides the vocabulary — `scope` is reserved for pytest's fixture-lifetime - ladder — but the rename is independently schedulable. -- Whether the `pytest11` entry point rename (from `sphinx`, which is what - `-p no:sphinx` disables today) ships in the same release as this removal or its - own. diff --git a/docs/adrs/index.md b/docs/adrs/index.md index ed763f1..02af2f3 100644 --- a/docs/adrs/index.md +++ b/docs/adrs/index.md @@ -31,7 +31,7 @@ the anchor lands on unrelated code while still resolving. 0001-typed-vanilla-doctest-core 0002-runner-conformance-across-cpython -0003-retiring-per-block-namespace-items +0003-rejecting-per-block-items 0004-diagnostics-as-data 0005-line-recovery-for-nested-blocks 0006-pytest-private-api-compatibility From 4d55e83c02db33cb5195d599a68cee32412e091b Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:23:19 -0500 Subject: [PATCH 06/24] docs(adrs[0005]): Raise the docutils floor instead why: The proposed mechanism does not work and was not needed. Nested state machines build from nested_sm_kwargs, so substituting a parser instance's state_classes never reaches the constructs with missing lines; and RSTState.nested_sm_cache is a shared class attribute, so the substitution is not scoped to one parse either. Meanwhile docutils 0.22 fixed the defect upstream. what: - Replace the substitution mechanism, the feature probe and the fallback design with a floor bump to docutils >=0.22 - Record why the mechanism failed, so it is not proposed again - Name the real cost: docutils >=0.22 requires Sphinx >=9.1 - Version-qualify the line conventions, and state that an unqualified claim about docutils line numbers is a bug in the claim - Keep the nullable line and the normalization layer, which serve .. include:: attribution regardless of version --- .../0005-line-recovery-for-nested-blocks.md | 89 ++++++++++++------- 1 file changed, 57 insertions(+), 32 deletions(-) diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index f24616b..79325b1 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -1,59 +1,84 @@ (adr-0005-line-recovery-for-nested-blocks)= -# ADR 0005: Exact line recovery for nested blocks +# ADR 0005: Line recovery for nested blocks Status: Draft Date: 2026-08-02 ## Context -docutils does not report a usable line for every node, and what it does report -differs by front-end. +docutils does not report a usable line for every node, and what it reports +differs by front-end and by version. -A bare `>>>` block nested in a `.. note::`, a list item, a block quote or a -`{tab}` directive reports `line=None, source=None`. A top-level reStructuredText -`doctest_block` reports its **last** line; a MyST fence reports its **first**. An +At **docutils 0.21.2**, which this project currently pins, a bare `>>>` block +nested in a `.. note::`, a list item, a block quote or a `{tab}` directive +reports `line=None, source=None`. A top-level reStructuredText `doctest_block` +reports its **last** line. A MyST fence reports its **first** line. An `.. include::`-ed block numbers against the *included* file. -{doc}`0001-typed-vanilla-doctest-core` handles all three honestly rather than -approximately. `Block.line` is nullable, the per-front-end meaning is normalized +{doc}`0001-typed-vanilla-doctest-core` handles all of this honestly rather than +approximately: `Block.line` is nullable, the per-front-end meaning is normalized inside the front-end that knows it, `Block.path` carries the file the text actually lives in, and a block with no recoverable line propagates `DocTest.lineno=None` into pytest's `EXAMPLE LOCATION UNKNOWN` branch. That is correct but not maximal. A nested block's failure says the location is -unknown when the information exists in the parser and was discarded. +unknown when the parser knew it and threw it away. -## Question +## The mechanism this record originally proposed does not work -Can the exact line be recovered for a nested block without a fragile dependency? +The idea was to substitute `docutils.parsers.rst.Parser.state_classes` per +parser instance, on the reasoning that `state_classes` is an instance attribute +and therefore scoped to one parse. -The mechanism identified is per-instance substitution of -`docutils.parsers.rst.Parser.state_classes`. `state_classes` is an instance -attribute, so substitution is fully scoped to one parse with no process-global -mutation — unlike the directive registry, which has no such scoping. +It fails on two counts, both checked: -It also depends on undocumented docutils structure, and a refactor upstream would -break it as `None` spans rather than as an exception. +**It does not reach nested blocks.** A nested parse builds its state machine from +`nested_sm_kwargs`, so a substitution applied only to the top-level +`state_classes` never reaches the constructs that need it — which are exactly the +constructs with the missing lines. -## Direction +**It is not scoped.** `RSTState.nested_sm_cache` is a shared *class* attribute, +so substituted classes leak into subsequent parses that did not ask for them. +The claim that substitution is "fully scoped to one parse with no process-global +mutation" is wrong. -Land it last, as a strictly optional improvement behind a feature probe. +## Decision -When the expected structure is absent, fall back to `node.line` normalization and -emit `line=None`. Degrading to an honest disclaimer is acceptable; degrading to a -fabricated number is not, and that is the whole reason this is separable from -{doc}`0001-typed-vanilla-doctest-core` rather than part of it. +**Raise the docutils floor instead.** -Pin with tests asserting exact `(path, line)` for a bare block nested in each of -the four constructs, plus a test that forces the probe to fail and asserts the -fallback yields `lineno=None`. +docutils 0.22 fixed the underlying defect upstream: a nested block reports a real +line, and top-level and nested blocks agree on reporting the **first** line +rather than the last. Every case this record was invented to work around is +resolved by a floor bump, with no probe, no substitution and no fallback path. + +The cost is that docutils ≥ 0.22 requires Sphinx ≥ 9.1, so this is a coordinated +dependency move rather than a one-line pin change. + +## Consequences + +The nullable `Block.line` stays. It is not a workaround for this defect; it is +the honest representation of a front-end that may legitimately not know, and +`.. include::` attribution still needs `Block.path` regardless of version. + +The line-convention normalization in `markup/` gets *simpler* at the new floor — +both reStructuredText and MyST report the first line — but the normalization +layer stays, because the conventions still differ below the floor and a front-end +is the right place to know which it is dealing with. + +Every line-convention claim elsewhere in these records is version-qualified. +A statement about "docutils" that does not name a version is a bug in the +statement. ## Open -- Whether the same technique gives MyST nested blocks anything, or whether - markdown-it's token stream already carries enough. -- Whether `.. include::` line attribution needs the same treatment or is already - correct once `Block.path` is respected. -- Whether the probe result is reported anywhere, so a user can tell which mode - produced a given report. +- Whether to raise the floor now or support both, since 0.21.2 is what the + project pins today. Supporting both means keeping the normalization branch and + documenting two behaviours for the same page. +- Whether the Sphinx ≥ 9.1 move belongs in this record or its own. It changes the + default group an unargumented block joins, which is a semantics change beyond + line numbers. +- Whether tests should pin exact `(path, line)` for a bare block nested in a + `.. note::`, a list item, a block quote and a `{tab}` directive. They should — + they are the regression net for the floor, and this repository already has + `{tab}` coverage from the GH-48 regression. From 97b55f8f4d33493c3c3984e0287e34c695841a64 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:24:23 -0500 Subject: [PATCH 07/24] docs(adrs[0006]): Filter the collector instead of deselecting items why: Deselecting in pytest_collection_modifyitems runs too late. DoctestTextfile.collect() reads and parses the page inside collect(), so by then the built-in has already produced an item, or already reported a collection error that deselection cannot retract. what: - Filter the built-in's collector out of the pytest_collect_file result in a hookimpl wrapper, before it parses anything - Record that narrowing --doctest-glob cannot help either, because _is_doctest claims .rst initial paths before consulting the glob - Move the compatibility failure from plugin registration to the collection of an affected document, matching 0001 and 0002 on not taking down a session whose tests never touch a doctest - Replace the settled open question with the one the filter raises: scoping it to paths this plugin actually claims --- .../0006-pytest-private-api-compatibility.md | 40 ++++++++++++++----- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index 943e9d9..35c6503 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -39,6 +39,14 @@ configuration, and `pytest_collect_file` is not `firstresult` — the directory collector yields the results of *every* implementation for a path. Declining the path is therefore not sufficient; the duplicate has to be removed. +**Narrowing `--doctest-glob` cannot help**, because `_is_doctest` returns `True` +for an `.rst` initial path *before* it consults the glob at all. + +**And removing the duplicate late is too late.** `DoctestTextfile.collect()` +reads and parses the page inside `collect()`, so by the time +`pytest_collection_modifyitems` runs, the built-in has already produced an item — +or already reported a collection error, which deselection cannot retract. + ## Question What private surface is depended on, and how does a pytest release that changes @@ -56,13 +64,22 @@ reimplemented rather than inherited. ## Direction Quarantine every private import in one module, `pytest_doctest_docutils._compat`, -with a pinned support matrix and an import-time probe that raises a named error at -plugin registration — naming the pytest version and the missing symbol — rather -than failing somewhere in the middle of collection. - -Claim `.rst` and `.md` paths and deselect the built-in plugin's duplicate items in -a `pytest_collection_modifyitems` hookwrapper, firing `pytest_deselected` so the -reported counts stay consistent. +with a pinned support matrix. + +**Filter the built-in's collector out of the `pytest_collect_file` result, in a +`@pytest.hookimpl(wrapper=True)`.** The directory collector consumes the +multicall result directly, and returning a modified result from a hook wrapper is +documented and supported. Filtering there removes the duplicate *before* the +built-in collector parses anything, so neither the duplicate item nor its +collection error is ever produced — which late deselection cannot achieve. + +**Fail on an unsupported pytest only when an affected document is collected**, +not at plugin registration. A `pytest11` plugin that raises at import takes down +sessions whose majority of tests never touch a doctest, and +{doc}`0001-typed-vanilla-doctest-core` and +{doc}`0002-runner-conformance-across-cpython` both reject session-wide startup +failures for the same reason. The error names the pytest version and the missing +symbol, and it names the document that triggered it. CI carries a job pinned to the minimum supported pytest and one tracking its prerelease. @@ -72,10 +89,11 @@ prerelease. - Whether requiring the built-in plugin should be stated as a hard dependency. `-p no:doctest` already fails today with a raw `ValueError` about an unknown option, so this is not a regression — but the message should become actionable. -- Whether deselecting is preferable to narrowing the built-in's globs. Deselecting - is explicit and reportable; narrowing is quieter but mutates another plugin's - configuration. - Whether the probe should accept a *newer* pytest it has not been tested against, - or refuse it. Refusing is safer and more annoying. + or refuse it. Refusing is safer and more annoying; for a private-API quarantine + with a small matrix, safer probably wins. +- What the filtering wrapper should do when the built-in's collector is the *only* + one for a path — that is the ordinary `--doctest-glob` case this plugin has no + business touching, so the filter must be scoped to paths it actually claims. - Whether any of these helpers can be promoted upstream, which would delete the quarantine entirely. From e0759eea536a98eeebc574d7bfbabd94cbdbc74f Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:24:23 -0500 Subject: [PATCH 08/24] docs(adrs[0002]): Assert the result triple, not TestResults equality why: TestResults is a two-field namedtuple carrying skipped off-tuple, so == compares two of three values and a skip-count regression passes the harness silently. attempted is also incremented before the SKIP check, so a skip that wrongly executes moves neither counter. what: - Assert (failed, attempted, skipped) explicitly, and say why the tuple comparison is insufficient - Add an exec-mode case to the matrix, and note it has no stock counterpart since a stock runner rejects a multi-statement body --- .../0002-runner-conformance-across-cpython.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md index bd7882e..f7183d9 100644 --- a/docs/adrs/0002-runner-conformance-across-cpython.md +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -50,9 +50,22 @@ gating the build step that lands the runner. A fixed case matrix — pass, fail, unexpected exception, `SyntaxError`, all examples skipped, partially skipped, `FAIL_FAST`, `REPORT_ONLY_FIRST_FAILURE`, -`IGNORE_EXCEPTION_DETAIL` — is run through both this runner and a stock -{class}`doctest.DocTestRunner`, asserting equality of `TestResults`, the captured -`report_*` text, `summarize()` output, and the accumulator contents. +`IGNORE_EXCEPTION_DETAIL`, and an exec-mode body — is run through both this +runner and a stock {class}`doctest.DocTestRunner`, asserting the captured +`report_*` text, `summarize()` output, the accumulator contents, and the result +as `(failed, attempted, skipped)`. + +**Assert the triple, not `TestResults` equality.** `TestResults` is a two-field +namedtuple carrying `skipped` off-tuple, so `==` compares only two of the three +values and a skip-count regression passes silently. `attempted` is also +incremented *before* the `SKIP` check, so a skip that wrongly executes moves +neither counter — it is invisible to both the tuple and to `summarize()` at zero +failures, and only the `report_*` text distinguishes it. + +The exec-mode case has no stock counterpart to compare against: a stock runner +raises `SyntaxError: multiple statements found` on a multi-statement body. It is +asserted against a recorded expectation instead, and that asymmetry is the point +— it is the one behaviour this runner exists to add. Version handling is by capability probe, never by version comparison, so a backport, a vendored interpreter or a fork behaves correctly rather than by From 3a6f4c18594bf46575be5f3d6ed789c24e230d22 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:28:29 -0500 Subject: [PATCH 09/24] notes(analyses): Correct the claims review falsified why: Review checked these notes against source and found several assertions wrong. Each correction below was re-verified by reading a pinned tag or by executing it. what: - Repin to what the project resolves: docutils 0.21.2, Sphinx 8.2.3, and myst-parser split across two Python floors. Repin every Sphinx anchor, including the gated-drop line, which moved between tags - Sphinx runs one DocTest per BLOCK against one shared group namespace. Move it into the N-DocTests row of the product space; its "one node id" is really no id, since every block shares DocTest.name - Sphinx rejects :options: on a testcode with an unknown-option error rather than discarding it. The silent loss there is :pyversion:. Add the doctest-then-testoutput case, and the setup-failure short-circuit that skips cleanup - parsefactories collects fixtures from the .py being collected; conftest autouse fixtures arrive via pytest_plugin_registered - stdlib doctest accepts a duck-typed parser or finder. Typeshed is what names the class, so a nominal subclass buys type-checkability, not passability - A regex can parse directive options; Sybil does. Justify the docutils dependency on host fidelity instead - Version-qualify the line conventions, since docutils 0.22 changed them - Correct the pytest-asyncio note: it does own an item class, it does convert str to enum, and the None sentinel is used on three of six options rather than all - Record subtests as pytest's only sanctioned sub-item mechanism --- notes/analyses/00-taxonomy.md | 10 ++- notes/analyses/11-pytest-doctest.md | 13 ++- notes/analyses/13-pytest-asyncio.md | 58 ++++++++---- notes/analyses/15-sphinx-ext-doctest.md | 90 ++++++++++++------- notes/analyses/16-docutils-myst.md | 15 ++-- notes/analyses/17-prior-art.md | 15 ++-- notes/analyses/20-data-structures.md | 4 +- notes/analyses/21-data-flows.md | 6 +- notes/analyses/22-extension-seams.md | 13 ++- .../23-namespace-scope-and-test-identity.md | 10 ++- notes/analyses/90-bibliography.md | 47 +++++----- notes/analyses/README.md | 6 +- 12 files changed, 180 insertions(+), 107 deletions(-) diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md index 0f6bf22..dc53189 100644 --- a/notes/analyses/00-taxonomy.md +++ b/notes/analyses/00-taxonomy.md @@ -67,10 +67,12 @@ abandoning the stdlib object model, and it is now building the bridge back. The cost of divergence is paid years later, in knobs that exist only to restore the default that was abandoned. -**Axis 4 is decided by axis 6.** A regex cannot see `:skipif:` or a group name, -which is why Sybil has no group concept at all and tells users to clear the -namespace instead. If directive options are part of the product, the document -model must be a parse tree. +**Axis 4 is decided by host fidelity, not by lexing power.** A regex *can* parse +directive arguments and options — Sybil's directive lexers do it. What a regex +cannot give you is the same tree Sphinx renders from, and that is what makes a +page behave identically under `sphinx-build` and under pytest. Sybil having no +group concept, and telling users to clear the namespace instead, is a design +choice rather than a limit of its lexer. **Axis 7 correlates with hostility.** Two of the surveyed projects disable pytest's doctest plugin — one in `pytest_configure`, one by telling users to pass diff --git a/notes/analyses/11-pytest-doctest.md b/notes/analyses/11-pytest-doctest.md index 5f616fa..b365dac 100644 --- a/notes/analyses/11-pytest-doctest.md +++ b/notes/analyses/11-pytest-doctest.md @@ -16,7 +16,8 @@ replace it, and the model this project's pytest layer should resemble. DoctestItem(Item) dtest: DocTest, runner: DocTestRunner, fixture_request | obj = None (class attribute) DoctestTextfile(Module) obj = None; one DocTest for the whole file -DoctestModule(Module) one DocTest per docstring; calls parsefactories +DoctestModule(Module) one DocTest per docstring; parsefactories for + fixtures defined in the collected .py itself MultipleDoctestFailures carries a list; the workaround for stdlib having no per-example result value ReprFailDoctest (ReprFileLocation, lines) pairs @@ -26,9 +27,15 @@ ReprFailDoctest (ReprFileLocation, lines) pairs ([`:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421)) is what keeps `Module` from trying to import a `.txt`/`.rst` file. Subclassing `Module` rather than `File` is what makes `scope="module"` fixtures resolve -against the page — a page collector *is* the module scope. `parsefactories` +against the page — a page collector *is* the module scope. + +`parsefactories` ([`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556)) -is what makes autouse fixtures from a visible `conftest.py` apply at all. +is `DoctestModule`-only, and it collects fixtures defined *in the `.py` being +collected*. It is **not** what makes conftest autouse fixtures apply — those are +registered through `FixtureManager.pytest_plugin_registered` when the conftest is +loaded, independently of any collector. A page collector needs no `parsefactories` +call to see them. ## Data flow diff --git a/notes/analyses/13-pytest-asyncio.md b/notes/analyses/13-pytest-asyncio.md index ecdd322..3ba4400 100644 --- a/notes/analyses/13-pytest-asyncio.md +++ b/notes/analyses/13-pytest-asyncio.md @@ -9,10 +9,18 @@ a default it needed to change without breaking anyone. ## Classification -A hook-driven behaviour plugin. It contributes no collector of its own for the -common case; it normalizes configuration into one question, answers it once, and -then injects behaviour through pytest's existing machinery rather than owning the -item class. +A hook-driven behaviour plugin that **does** own its item class. It defines +`PytestAsyncioFunction(Function)` +([`:506`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L506)) +with four concrete subclasses — `Coroutine`, `AsyncGenerator`, +`AsyncStaticMethod`, `AsyncHypothesisTest` — and a `pytest_pycollect_makeitem` +hookwrapper +([`:689-723`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L689-L723)) +that substitutes them for every collected async `Function`. + +That substitution-by-hookwrapper is itself the pattern worth noting: it swaps the +item class without owning collection, so pytest still decides *what* is a test +and the plugin only decides *how* it runs. ## Core data structures @@ -23,9 +31,13 @@ PytestAsyncioSpecs its own hookspec namespace [:90] _ScopeName reuses pytest's scope vocabulary verbatim ``` -`Mode` inheriting `str` is a small but deliberate choice: the ini value, the CLI -value and the internal enum are the same object, so no conversion layer exists to -drift. +`Mode` inherits `str`, but a conversion layer still exists and is exactly where +drift would occur: `_get_asyncio_mode` +([`:222-232`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222-L232)) +reads the CLI value, falls back to the ini value, and calls `Mode(val)` inside a +`try`, translating a `ValueError` into a {exc}`pytest.UsageError` that lists the +valid modes. The lesson is not "no conversion" — it is that the conversion +happens **once**, in one named function, with a good error. Declaring its own `HookspecMarker("pytest")` namespace is the interesting one. A third party extends pytest-asyncio by implementing a hook, not by subclassing @@ -47,7 +59,7 @@ _get_asyncio_mode(config) -> Mode, resolved once [:222] | v in AUTO mode: item.add_marker("asyncio") - | => the rest of the plugin has exactly ONE query path + | => marker presence becomes the single question downstream asks v fixture/loop resolution by scope, then pyfunc call wrapping ``` @@ -64,14 +76,16 @@ fixture/loop resolution by scope, then pyfunc call wrapping ## What is worth stealing -**The `default=None` sentinel.** Every option is declared with `default=None` -([`:114`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L114), -[`:122`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L122), -[`:140`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L140)) -rather than with its effective default. The plugin can therefore distinguish "the -user chose the current default" from "the user has not chosen", which is what -makes a future default change *announceable* — it warns only the second group. -`pytest_configure` uses exactly this to warn about an unset +**The `default=None` sentinel, used selectively.** Of the six options +`pytest_addoption` declares +([`:108-147`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108-L147)), +three carry `None` and three carry their effective default — so this is a +technique applied where it earns its keep, not a blanket rule. + +It earns its keep on the options whose default the project intends to move. A +`None` lets the plugin distinguish "the user chose the current default" from "the +user has not chosen", which is what makes a future change *announceable* — only +the second group is warned. `pytest_configure` does exactly that for an unset `asyncio_default_fixture_loop_scope` ([`:296-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L296-L301)). @@ -91,9 +105,15 @@ scope names verbatim. It does not invent a third word for lifetime. Compare the current `namespace_scope`/`namespace_items` naming, which collides with two pytest concepts at once. -**Session-wide errors raise `pytest.UsageError`; per-item errors raise -`ValueError`.** The severity matches the blast radius. A misspelled session -setting should stop the session; a bad per-item value should fail that item. +**Session-wide errors raise {exc}`pytest.UsageError`.** An invalid `asyncio_mode` +and an invalid loop-scope ini value both raise it, and both are reached from +session-level config in `pytest_configure`. A misspelled session setting stops the +session, which is the right blast radius for a value that would otherwise +mis-apply to every item. + +The mirror rule — per-item errors raising something narrower — is *not* something +this plugin demonstrates cleanly, so do not cite it as precedent. Marker parsing +is one function with one blast radius. The session half is the transferable part. ## What it cannot tell us diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md index 7be7bbc..3590dd7 100644 --- a/notes/analyses/15-sphinx-ext-doctest.md +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -1,6 +1,8 @@ # `sphinx.ext.doctest` -Pinned at [`v9.1.0`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py). +Pinned at [`v8.2.3`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py), +the version this project resolves. v9.1.0 changes the default group an +unargumented block joins; nothing else below differs between the two. ## Classification @@ -26,10 +28,20 @@ SphinxDocTestRunner(doctest.DocTestRunner) [:257] ``` `TestGroup.tests` holds heterogeneous entries — `[code]` for a bare block, -`[code, output]` for a paired testcode/testoutput. `add_code` is where three -silent losses live: an orphan `testoutput` is discarded, a second `testoutput` -*replaces* the first, and a `testcode`'s own options are dropped in favour of its -output block's. +`[code, output]` for a paired testcode/testoutput. `add_code` is where the silent +losses live: an orphan `testoutput` is discarded; a `testoutput` following a +`doctest` block is discarded, because a doctest entry has length 1 and fails the +`len(latest_test) == 2` guard; and a second `testoutput` *replaces* the first. + +A fourth silent loss is in the directive rather than `add_code`: `:pyversion:` is +in `TestcodeDirective.option_spec` +([`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180)) +and is then ignored, because the version gate only runs for `doctest` and +`testoutput`. + +`:options:` on a `testcode` is **not** a silent loss. It is absent from +`TestcodeDirective.option_spec` entirely, so writing it is an unknown-option +error that drops the block — loud, and visible in the build output. ## Data flow @@ -46,17 +58,20 @@ doctree | DocTestBuilder.test_doc(docname, doctree) [:428] | for node in doctree.findall(condition): - | if self.skipped(node): continue <- GATED BLOCK IS DROPPED [:443-444] + | if self.skipped(node): continue <- GATED BLOCK IS DROPPED [:449-450] | code = TestCode(...) | "*" in groups -> add to every group | else groups[name].add_code(code) v per group: ns = {} | three runners: setup / test / cleanup, sharing one _fakeout - | test.globs = ns (assigned AFTER construction) + | ONE doctest.DocTest PER BLOCK, each with test.globs = ns + | (assigned AFTER construction, since __init__ copies) | runner.run(test, out=..., clear_globs=False) - | self.type flipped to "exec" for setup, cleanup, testcode [:548, :608] + | self.type flipped to "exec" for setup, cleanup, testcode [:549, :608] | to "single" for ordinary doctests [:580] + | + | if setup fails -> RETURN. Cleanup does not run. [:554-556] v six integer counters + text streamed to outdir/output.txt ``` @@ -65,10 +80,10 @@ six integer counters + text streamed to outdir/output.txt | Seam | Kind | |---|---| -| `TestDirective` subclassing, with `option_spec` ([`:66`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66)) | subclass | +| `TestDirective` subclassing, with `option_spec` ([`:66`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66)) | subclass | | The node attribute stamp — `testnodetype`, `groups`, `options`, `skipif`, `test` | implicit protocol | | `doctest_global_setup`, `doctest_global_cleanup`, `doctest_test_doctest_blocks`, `doctest_default_flags` | Sphinx confvals | -| `is_allowed_version(spec, version)` ([`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45)) | function | +| `is_allowed_version(spec, version)` ([`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45)) | function | The node attribute stamp is the important one, and it is undocumented. It is the only decoupled interface in the module: any directive that emits a @@ -82,22 +97,29 @@ rather than trusting one's own directive classes is the only defence against | Rule | Anchor | |---|---| -| `testsetup`, `testcleanup` and `:hide:` render as `nodes.comment` | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | -| `:options:` is accepted only on `doctest` and `testoutput` | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111) | -| A gated block is dropped during collection — no outcome, id or count | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | -| `*` means every group the document declares; `default` means no argument was given | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) onward | -| Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200-L226) | -| `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | -| `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) | +| `testsetup`, `testcleanup` and `:hide:` render as `nodes.comment` | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | +| `:options:` is accepted only on `doctest` and `testoutput`; on a `testcode` it is an unknown-option error, not a discard | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111), [`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180) | +| Cleanup does **not** run when setup fails — the group returns early | [`:554-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L554-L556) | +| A gated block is dropped during collection — no outcome, id or count | [`:449-450`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | +| `*` means every group the document declares; `default` means no argument was given | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) onward | +| Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200-L226) | +| `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | +| `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) | The last two are where this project deliberately diverges. The argument order was a real defect in the local helper. The `compile` rebinding is unavailable to a library that loads into every pytest session, which is the entire origin of ADR 0001's decision to own the per-example loop instead. -The gated-block rule is the one semantic this project rejects on purpose: Sphinx's -drop destroys the node id, the count and the `-rs` line, and pytest users -reasonably expect a `SKIPPED` outcome with a reason. +Two more are rejected on purpose. Sphinx's gated-block drop destroys the node id, +the count and the `-rs` line, where pytest users reasonably expect a `SKIPPED` +outcome with a reason. And the setup-failure short-circuit leaves a page's +`testcleanup` unrun, which for a page that spawns a server in setup means a leak. + +**One thing Sphinx already does that is worth stating plainly:** it runs one +`DocTest` *per block* against one shared group namespace. That execution shape is +not novel to ADR 0001. What Sphinx lacks is any selectable, reportable identity +for those blocks — they all share one `DocTest.name`, which is the defect below. ## What it cannot do @@ -108,20 +130,20 @@ reasonably expect a `SKIPPED` outcome with a reason. - **Distinguish two blocks in one group.** Every block in a group shares `DocTest.name`, which is why `SphinxDocTestRunner` overrides a private stdlib method to swallow the resulting `IndexError` - ([`:257`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257)). + ([`:257`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257)). ## Anchors -- [`is_allowed_version`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) · - [`TestDirective`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66) · - [`comment nodetype rule`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) -- [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) · - [`add_code`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) · - [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) -- [`SphinxDocTestRunner`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257) · - [`DocTestBuilder`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L292) · - [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) -- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) · - [`skipped-node drop`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) · - [`type = "exec"` for testcode](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) -- [User-facing contract](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/doc/usage/extensions/doctest.rst) +- [`is_allowed_version`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) · + [`TestDirective`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66) · + [`comment nodetype rule`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) +- [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) · + [`add_code`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L207) · + [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) +- [`SphinxDocTestRunner`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257) · + [`DocTestBuilder`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L292) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) · + [`skipped-node drop`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) · + [`type = "exec"` for testcode](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L548) +- [User-facing contract](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/doc/usage/extensions/doctest.rst) diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md index e1fb1d3..9b524c4 100644 --- a/notes/analyses/16-docutils-myst.md +++ b/notes/analyses/16-docutils-myst.md @@ -1,6 +1,6 @@ # docutils and MyST-Parser -docutils pinned at `docutils-0.22.2` (canonical repository is on SourceForge; the +docutils pinned at `docutils-0.21.2` (canonical repository is on SourceForge; the GitHub copies are third-party mirrors, so anchors here name file and symbol rather than a permalink). MyST-Parser pinned at [`v5.1.0`](https://github.com/executablebooks/MyST-Parser/tree/v5.1.0). @@ -33,7 +33,12 @@ myst_parser.parsers.docutils_.Parser(RstParser) [v5.1.0:235] ## The two line conventions -| Front-end | Construct | `.line` reports | +**These are docutils 0.21.2 behaviours.** docutils 0.22 fixed the nested case +upstream and made top-level and nested blocks agree on the **first** line, so any +claim here that does not name a version is a bug in the claim. ADR 0005 +(`docs/adrs/0005-line-recovery-for-nested-blocks.md`) covers the floor question. + +| Front-end | Construct | `.line` reports (0.21.2) | |---|---|---| | reStructuredText | top-level `doctest_block` | its **last** line | | reStructuredText | any block nested in a directive, list item or block quote | `None`, with `.source` also `None` | @@ -122,9 +127,9 @@ losing the underlying record. [`create_myst_settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208) · [`settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) · [`MdParserConfig`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) -- Sphinx's registry snapshot/rebind: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/util/docutils.py) · - unconditional override: [`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/application.py) +- Sphinx's registry snapshot/rebind: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/util/docutils.py) · + unconditional override: [`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/application.py) - docutils: `docutils/parsers/rst/directives/__init__.py` (`_directives`), `docutils/parsers/rst/states.py` (`state_classes`, `doctest_block` line assignment), `docutils/utils/__init__.py` (`Reporter.attach_observer`, - `system_message`), at `docutils-0.22.2`. + `system_message`), at `docutils-0.21.2`. diff --git a/notes/analyses/17-prior-art.md b/notes/analyses/17-prior-art.md index c9da655..4933274 100644 --- a/notes/analyses/17-prior-art.md +++ b/notes/analyses/17-prior-art.md @@ -46,11 +46,16 @@ downstream test, breaking `--lf`, `--nf`, deselect files, xfail lists and CI fla history. For a *documentation* test runner, prose above examples is the thing that changes most often. -**What it cannot do.** Groups. A regex cannot see a directive's options, so Sybil -has no group concept at all and directs users to clear the namespace instead. This -is the clearest available argument for paying the docutils dependency: if -`:skipif:`, `:options:` and group names are part of the product, the document -model has to be a parse tree. +**What it does not do.** Groups. Sybil has no group concept and directs users to +clear the namespace instead — but that is a *design choice*, not a lexing limit: +its directive lexers do parse directive arguments and options. + +So the argument for paying the docutils dependency is not "a regex cannot see +`:skipif:`". It is **host fidelity**: collecting from the same doctree Sphinx +renders means a page behaves the same under `sphinx-build` and under pytest, and +this project already depends on docutils regardless. Nesting and source +attribution are not the argument either — Sybil locates a nested block exactly, +where docutils 0.21.2 reports `None`. **Verdict on the invariant.** The non-overlap check does not survive contact with docutils: two `.. include::` directives naming the same file legitimately produce diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index 3b0a730..e38c83b 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -78,8 +78,8 @@ than improving it — the three fields are then already true. ## Anchors - [`Example` / `DocTest` / `TestResults`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) -- [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) · - [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) +- [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) · + [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) - [`MultipleDoctestFailures`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) · [`repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) - [Sybil `Region`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md index 0264452..ad371e8 100644 --- a/notes/analyses/21-data-flows.md +++ b/notes/analyses/21-data-flows.md @@ -108,6 +108,6 @@ group sharing state, the wrong sequence is the bug. [`compile(..., "single", ...)`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) - [`DoctestItem.setup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) · [`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) -- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) · - [gated-node drop](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) · - [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) · + [gated-node drop](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index eb39d65..013c20c 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -7,7 +7,7 @@ ranking is not a matter of taste — each has an observed failure mode. | Mechanism | Where | Failure mode | |---|---|---| -| **Nominal subclassing** — the API demands the class, not the shape | stdlib `doctest`, `sphinx.ext.doctest` directives | A structurally compatible object is rejected. `DocutilsDocTestFinder` exposes a compatible `find()` and still cannot be passed to `DocTestSuite(test_finder=...)` | +| **Nominal subclassing** — the *type checker* demands the class, the interpreter does not | stdlib `doctest` (via typeshed), `sphinx.ext.doctest` directives | Typed callers are rejected for objects that work fine at runtime. stdlib performs no `isinstance` check on `parser` or `test_finder`; the pressure comes entirely from `doctest.pyi`. One genuine runtime edge: `DocTestSuite` sorts, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom finder must return real `DocTest`s | | **Callable aliases** — `Evaluator = Callable[[Example], str \| None]` | Sybil | Types nothing. You cannot express "this lexer emits `source` and `arguments`", so a mismatched pairing raises `KeyError` at run time. Sybil's own source comments say the payload "could likely be a `TypedDict`" | | **Named registry** — a module dict plus a `register_*` function | `doctest.register_optionflag`, docutils directives, xdoctest's two facades | Process-global mutable state. Order-dependent, unscoped, and silently overwritable | | **`Protocol`** — structural typing | xdoctest's `StdlibExampleLike` | None inherent; but a `Protocol` alone does not satisfy a nominal consumer | @@ -49,9 +49,14 @@ direction and is retiring `set_event_loop_policy` in favour of an explicit ## The nominal/structural trap, and the cheap way out Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: DocTestParser -= ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. A `Protocol` gives a -third party structural typing; a nominal subclass keeps you passable to the -stdlib's own builders. += ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. The interpreter does +not — a duck-typed parser or finder runs fine. So a `Protocol` gives a third party +structural typing, and a nominal subclass keeps you *type-checkable* against the +stub. + +The subclass does not buy passability. That is a matter of matching the call +signature: a finder whose `find()` takes a string first cannot be handed to +`DocTestSuite`, which passes a module — and subclassing does not change that. Doing both costs nothing: diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index 524fbd3..70bcb75 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -19,8 +19,14 @@ narrative page needs depends on the first. | | one node id | N node ids | |---|---|---| -| **one `DocTest`** | `sphinx.ext.doctest`; `doctest_docutils` `merged` | incoherent | -| **N `DocTest`s** | *unoccupied until ADR 0001* | Sybil; `doctest_docutils` `per-block` | +| **one `DocTest`** | PR #87's `merged` | incoherent | +| **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids at all; ADR 0001 gives the shape a pytest identity | Sybil; PR #87's `per-block` | + +Sphinx belongs in the bottom row: it already builds one `DocTest` per block +against one shared group namespace. Its "one node id" is really *no* id — every +block in a group shares one `DocTest.name`, which is why `SphinxDocTestRunner` +overrides a private stdlib method to swallow the resulting `IndexError`. So the +execution shape is well-trodden; the contribution is making it addressable. The bottom-right cell is where the silent failure lives, and two shipped projects are in it. diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md index d627b6f..5f6daed 100644 --- a/notes/analyses/90-bibliography.md +++ b/notes/analyses/90-bibliography.md @@ -20,7 +20,7 @@ used anywhere else. | `DocTest.__lt__` | [`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | compares names as text | | `DocTestParser` | [`:609`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L609) | the injectable parser | | `_EXAMPLE_RE` | [`:618`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618) | private, used for prompt sniffing | -| `DocTestFinder` | [`:844`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) | nominal type demanded by `DocTestSuite` | +| `DocTestFinder` | [`:844`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) | the type typeshed names; accepted structurally at runtime | | `report_*` hooks | [`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | the four supported in-loop seams; no `report_skip` here | | `__run` | [`:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | name-mangled loop; overridable by mechanism | | `compile(..., "single", ...)` | [`:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | the hard-coded mode `{testcode}` cannot use | @@ -31,8 +31,8 @@ used anywhere else. | `OutputChecker` | [`:1690`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) | the documented checker seam | | `DebugRunner` | [`:1874`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1874) | `report_*` overriding as the sanctioned loop control | | `testfile` | [`:2091`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2091) | the API `testdocutils` mirrors | -| `DocTestSuite` | [`:2467`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) | demands a nominal `DocTestFinder` | -| `DocFileSuite` | [`:2570`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) | demands a nominal `DocTestParser` | +| `DocTestSuite` | [`:2467`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) | no `isinstance` on `test_finder`; sorts, so results must be real `DocTest`s | +| `DocFileSuite` | [`:2570`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) | no `isinstance` on `parser` | Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blob/v3.14.2/Doc/library/doctest.rst). @@ -69,7 +69,8 @@ Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blo | `_get_continue_on_failure` | [`:410`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L410) | private helper imported today | | `DoctestTextfile` | [`:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | `obj = None` as a class attribute | | `_check_all_skipped` | [`:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | fires only once the item is running | -| `DoctestModule` / `parsefactories` | [`:500`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) · [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | autouse fixtures from conftest | +| `DoctestModule` / `parsefactories` | [`:500`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) · [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | fixtures defined in the collected `.py`; **not** conftest autouse, which arrives via `FixtureManager.pytest_plugin_registered` | +| `subtests` | [`src/_pytest/subtests.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/subtests.py) | builtin since 9.0; the only sanctioned sub-item outcome mechanism, and experimental | | `_get_checker` | [`:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662) | the checker that would have to be reimplemented | | `_get_report_choice` | [`:703`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L703) | private helper | | `doctest_namespace` | [`:721`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) | the fixture that survives plugin blocking today | @@ -103,27 +104,27 @@ Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blo ## Sphinx — `v9.1.0` -[`sphinx-doc/sphinx @ v9.1.0`](https://github.com/sphinx-doc/sphinx/tree/v9.1.0) · -[`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py) +[`sphinx-doc/sphinx @ v9.1.0`](https://github.com/sphinx-doc/sphinx/tree/v8.2.3) · +[`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py) | Symbol | Anchor | Cited for | |---|---|---| -| `is_allowed_version(spec, version)` | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | specifier first — the reverse of the local helper | -| `TestDirective` | [`:66`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66) | the directive base and its option handling | -| comment nodetype rule | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | `testsetup`/`testcleanup`/`:hide:` become `nodes.comment` | -| `:options:` gating | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111) | accepted only on `doctest` and `testoutput` | -| `TestGroup` / `add_code` | [`:200`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) · [`:207`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) | phase ordering; three silent-loss cases | -| `TestCode` | [`:235`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) | the parsed unit | -| `SphinxDocTestRunner` | [`:257`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257) | overrides a private method to swallow an `IndexError` | -| `DocTestBuilder` | [`:292`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L292) | builder coupling | -| `doctest.compile` rebinding | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) | process-global, never restored | -| `test_doc` | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) | group resolution and `*` | -| gated-node drop | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | no outcome, id or count | -| `type = "exec"` for testcode | [`:548`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) | the mode flip | - -Documentation: [`doc/usage/extensions/doctest.rst`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/doc/usage/extensions/doctest.rst). -Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/util/docutils.py), -[`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/application.py). +| `is_allowed_version(spec, version)` | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | specifier first — the reverse of the local helper | +| `TestDirective` | [`:66`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66) | the directive base and its option handling | +| comment nodetype rule | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | `testsetup`/`testcleanup`/`:hide:` become `nodes.comment` | +| `:options:` gating | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111) | accepted only on `doctest` and `testoutput` | +| `TestGroup` / `add_code` | [`:200`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) · [`:207`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L207) | phase ordering; three silent-loss cases | +| `TestCode` | [`:235`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) | the parsed unit | +| `SphinxDocTestRunner` | [`:257`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257) | overrides a private method to swallow an `IndexError` | +| `DocTestBuilder` | [`:292`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L292) | builder coupling | +| `doctest.compile` rebinding | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) | process-global, never restored | +| `test_doc` | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) | group resolution and `*` | +| gated-node drop | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | no outcome, id or count | +| `type = "exec"` for testcode | [`:548`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L548) | the mode flip | + +Documentation: [`doc/usage/extensions/doctest.rst`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/doc/usage/extensions/doctest.rst). +Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/util/docutils.py), +[`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/application.py). ## MyST-Parser — `v5.1.0` @@ -136,7 +137,7 @@ Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sp | `settings_spec` | [`parsers/docutils_.py:241-245`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) | | `MdParserConfig` (`myst_enable_extensions`, `myst_fence_as_directive`) | [`config/main.py`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) | -## docutils — `docutils-0.22.2` +## docutils — `docutils-0.21.2` (the version this project pins) The canonical repository is on [SourceForge](https://sourceforge.net/p/docutils/code/); the GitHub copies are diff --git a/notes/analyses/README.md b/notes/analyses/README.md index 4114e78..74df866 100644 --- a/notes/analyses/README.md +++ b/notes/analyses/README.md @@ -31,12 +31,12 @@ What exactly does each of them require, and where do they contradict each other? | pytest | `pytest-dev/pytest` | `9.1.1` | | pytest-xdist | `pytest-dev/pytest-xdist` | `v3.8.0` | | pytest-asyncio | `pytest-dev/pytest-asyncio` | `v1.4.0` | -| Sphinx | `sphinx-doc/sphinx` | `v9.1.0` | -| MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` | +| Sphinx | `sphinx-doc/sphinx` | `v8.2.3` — what this project resolves. `v9.1.0` is cited only where a difference is called out | +| MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` on Python ≥ 3.11; `v4.0.1` below | | Sybil | `simplistix/sybil` | `10.0.1` | | xdoctest | `Erotemic/xdoctest` | `v1.3.2` | | typeshed | `python/typeshed` | `8c7256c` (no tags; commit reachable from trunk) | -| docutils | SourceForge (the GitHub clones are third-party mirrors) | `docutils-0.22.2` | +| docutils | SourceForge (the GitHub clones are third-party mirrors) | `docutils-0.21.2` | ## Files From 80ec3111a5d4432b13700cd2b0b6efa8dbb26084 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:28:51 -0500 Subject: [PATCH 10/24] docs(adrs[0001]): Repin the gated-drop anchor to v8.2.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: The skipif-drop line moved between Sphinx tags, and the vocabulary table still pointed at the v9.1.0 offset under a v8.2.3 path — an anchor that resolves onto the wrong lines. --- docs/adrs/0001-typed-vanilla-doctest-core.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 6feb84f..a28a10b 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -291,7 +291,7 @@ referents. Each term below is decided once and used only that way. | group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | | scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The block-vs-document axis becomes `share` | | test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | -| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L443-L444) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | +| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | | directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | | optionflags | an int bitmask; `register_optionflag` | `doctest_optionflags` ini | `:options:` plus `doctest_default_flags` | Keep verbatim. `register_optionflag` is the one genuinely cross-library extension point | | setup / cleanup | `setUp`/`tearDown` on the suite builders | fixtures | `testsetup`/`testcleanup` directives | Author-facing names stay Sphinx's; `phase` is the internal ordering axis; a fixture is never "setup" | From 9b4a04d287a0b630975220cae610ddd7fd369b87 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 14:56:15 -0500 Subject: [PATCH 11/24] docs(adrs[0001]): Fix the baseline, the Example model and purity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: A second review round found the corrected baseline was still wrong, and three model claims did not survive execution. Trunk's _find appends one DocTest per matched node named page.md[k] and the collector yields one item per test, so released gp-libs already has per-block identity with isolated copied globals — it lacks only a sharing unit. Example.__eq__ gates on exact type identity, so a bare subclass compares unequal to a stock Example in both directions. what: - Restate the baseline from trunk's source: per-block identity ships today, and it is what this design preserves rather than invents - Restate the invariant as scheduling identity versus diagnostic identity, which is what the decoupling actually is - Keep the Example subclass but override __eq__ with an isinstance check and rebind __hash__, pinned by a doctest asserting symmetry both ways; record the stock-Example fallback - Correct Example.source: stdlib-normalized executable body, not the author's verbatim text, which is what Block.source holds - Drop the purity claim. Includes read transitive files, directives execute during parsing, the registry is process-global and MyST plugins alter the tree; the contract is determinism over source closure, normalized settings and a frozen registry - Split GroupPlan from GroupRun so planning stays immutable and the live mapping belongs to an attempt - Add BlockResult and GroupResult as the channel for partial-skip detail, with its visibility and its accepted loss stated - Record that a paired want depends on a run-time gate, that wildcard membership needs a distinct DocTest per group, and that the gate's namespace diverges from Sphinx's fresh context - Complete the item lifecycle: seed restore, cleanup failure precedence, and the runner behaviours that must be reimplemented --- docs/adrs/0001-typed-vanilla-doctest-core.md | 184 +++++++++++++++---- 1 file changed, 151 insertions(+), 33 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index a28a10b..c306597 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -13,12 +13,18 @@ Date: 2026-08-02 over a docutils or MyST doctree and keeps CPython's *running* half. `pytest_doctest_docutils` wraps that in a pytest plugin. -Two facts about the shipped code set the problem. - -**A page is one test.** `DocutilsDocTestFinder` produces a single -{class}`doctest.DocTest` per document. A page cannot build state across the prose -that explains it, because there is no unit smaller than the page and no unit that -groups blocks within one. +**Released gp-libs already has per-block identity, and no sharing unit at all.** +`DocutilsDocTestFinder._find` walks the doctree and appends one +{class}`doctest.DocTest` per matched node, named `page.md[k]` where `k` is the +document-order index. The collector yields one {class}`pytest.DoctestItem` per +test. Each test is built by handing `globs` to +{meth}`doctest.DocTestParser.get_doctest`, and `DocTest.__init__` **copies** the +mapping — so every block runs against its own isolated namespace. + +That is the real starting point, and it frames the problem precisely: the +granularity this design wants to *preserve* is already shipped. What is missing +is any unit coarser than a block — no groups, no phases, no way for a narrative +page to build state across the prose that explains it. **The plugin blocks the plugin whose internals it imports.** `pytest_configure` calls `config.pluginmanager.set_blocked("doctest")`, and the @@ -81,7 +87,12 @@ Sybil ships the second shape without acknowledging it (see [](#prior-art)). ## Decision -**The pytest item is the group. The {class}`doctest.DocTest` is the block.** +**One pytest item owns one shared-state group; inside it, each source block +remains a real, independent {class}`doctest.DocTest`.** + +The decoupling is not "group versus block". It is **scheduling identity versus +diagnostic identity**: pytest schedules the group, while each `DocTest` keeps its +own source location, examples and failure gutter. One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block `DocTest`s run in phase order against one live `globs` mapping that never leaves @@ -211,6 +222,17 @@ This design takes the same position: **skip the item when every runnable block i skipped; otherwise report partial skips as typed block detail, not as a pytest outcome.** No extra reports are synthesized. +"Typed block detail" needs a channel, or implementers will re-invent skip lifting +or write to stderr. The channel is a `GroupResult` — one `BlockResult` per block, +each carrying phase, outcome, gate reason and location — attached to the item and +rendered in two places: the failure longrepr when the group fails, and a +terminal-summary line at `-rs`. It is **not** visible at default verbosity, and +it never becomes a JUnit `` entry. + +That is a real product loss relative to lifting a gated block into its own item, +and it is accepted deliberately: **a gated block inside a mixed group gives up its +selectable skip row.** The information survives; the addressable unit does not. + `subtests` — a builtin pytest plugin since 9.0, exporting `pytest.Subtests` and `pytest.SubtestReport` — can emit per-block outcomes, and is the only sanctioned mechanism that can. It is not adopted here: pytest documents it as experimental, @@ -261,18 +283,26 @@ an implementer cannot get it wrong by omission: 1. **Collection** builds one `GroupPlan` per (document, group) and one item per plan. An empty plan yields no item. -2. **`setup()`** clears the group mapping in place, then calls `super().setup()` - so fixtures inject into that same object. Clearing in place rather than - rebinding is what keeps `item.globs is plan.globs` true for every block, and - what stops attempt two of a `--reruns` run from reading attempt one's - mutations. +2. **`setup()`** starts an attempt. In order: clear the live mapping **in place**; + restore the plan's `seed`, `extraglobs` and `__name__`; then call + `super().setup()` so fixtures inject into that same object. Clearing in place + rather than rebinding is what keeps `item.globs is run.globs` true for every + block, and what stops attempt two of a `--reruns` run from reading attempt + one's mutations. Every block's `DocTest.globs` is assigned this object *after* + construction, because `DocTest.__init__` copies. 3. **`runtest()`** is overridden. It must not delegate to `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` defaulting to `True` — that would empty the shared mapping after the first block. It calls `run_group()`, which runs each `PlannedBlock` in phase order - with `clear_globs=False`, evaluates `:skipif:` against the group mapping, and - wraps the body in a `try`/`finally` so cleanup runs whether or not the body - raised. + with `clear_globs=False`, evaluates `:skipif:` against the live mapping, + finalizes each paired `want` from its gated `ExpectedOutput`, and wraps the + body in a `try`/`finally` so cleanup runs whether or not the body raised. When + cleanup *also* fails, the body's failure is the one raised; cleanup's is + recorded in the `GroupResult`. + + The `OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` + conversion and `continue_on_failure` are reimplemented here, because + `PytestDoctestRunner` is nested inside a factory and cannot be imported. 4. **Outcome** follows [](#the-outcome-contract): skip the item only when every runnable block is skipped. 5. **`repr_failure`** is inherited unchanged. It already reads each failure's own @@ -335,8 +365,27 @@ class PlannedBlock(t.NamedTuple): class GroupPlan(t.NamedTuple): group: str - blocks: tuple[PlannedBlock, ...] # in phase order - globs: dict[str, t.Any] # the live mapping, shared by every block above + blocks: tuple[PlannedBlock, ...] # in phase order; IMMUTABLE + seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt + + +class GroupRun: + """One execution attempt. Owns the live mapping; a plan never does.""" + + plan: GroupPlan + globs: dict[str, t.Any] # cleared and reseeded per attempt + + +class BlockResult(t.NamedTuple): + block: PlannedBlock + outcome: t.Literal["passed", "failed", "skipped"] + reason: str | None # the gate expression, when skipped + + +class GroupResult(t.NamedTuple): + group: str + blocks: tuple[BlockResult, ...] + failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] ``` `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block @@ -351,6 +400,25 @@ from a bare tuple of `DocTest`s. A `DocTest` carries no phase and no gate, so th plan has to. Tagging each entry also makes the ordering self-describing rather than a convention a comment asserts. +**A `PlannedBlock.test` cannot always be fully preconstructed.** Sphinx accepts +`:skipif:` on a `testoutput`, and when that output is gated away its `testcode` +still runs — expecting *empty* output. So the `want` of a paired block is a +function of a gate evaluated at run time, and the plan must carry the paired +output as data (`ExpectedOutput`, itself gated) with the `DocTest` finalized in +`run_group()`. Freezing `want` at projection time silently runs the wrong +assertion. + +**A wildcard block needs a distinct `DocTest` per group it joins.** `DocTest.globs` +is a plain mutable attribute, so one object shared across two `GroupPlan`s would +have the second group's assignment win and both groups would execute against one +mapping. `*` membership therefore materializes a separate `DocTest` per group. + +**The gate's evaluation namespace is a deliberate divergence.** Sphinx evaluates +each `:skipif:` in a fresh context seeded with `doctest_global_setup`; this design +evaluates it against the live group mapping, after fixture injection and after +earlier blocks have run. That is more useful — a gate can consult a fixture — and +it is not what `sphinx-build` does. Recorded rather than hidden. + `ExecutionProfile` is a **private** protocol carrying compile mode, extra compile flags, and an optional per-group context manager. It is not a `Literal["single", "exec"]`, because a second execution policy already exists in this repository: @@ -450,8 +518,9 @@ that owns the loop must probe for it rather than assume it. Making the item the sharing unit removes the need to satisfy the second and third at all. **The first still binds at any granularity** — identical collection is -required whether a page yields one item or fifty — and is satisfied instead by -collection being a pure function of (bytes, argv, ini). The fourth is why a +required whether a page yields one item or fifty — and is approached instead +through determinism over source closure, normalized settings and a frozen +registry, not through a purity claim collection cannot make. The fourth is why a worker-count fork is not worth carrying: `pytest_xdist_setupnodes(config, specs)` hands over the already-expanded spec list and never raises. @@ -482,15 +551,50 @@ Each is a genuine conflict where satisfying one goal costs another. "Both" is no an answer; the position taken and its price are recorded. **A vanilla `DocTest` cannot carry the front-end's metadata.** It has exactly -`(examples, globs, name, filename, lineno, docstring)`. *Position:* metadata that -the runner needs rides on a nominal {class}`doctest.Example` subclass — -`doctest.Example` has no `__slots__`, so attributes survive -{func}`copy.copy`, {mod}`pickle`, and a third party's naive +`(examples, globs, name, filename, lineno, docstring)`. *Position:* metadata the +runner needs at run time rides on a {class}`doctest.Example` subclass — +`doctest.Example` has no `__slots__`, so attributes survive {func}`copy.copy`, +{mod}`pickle`, and a third party's naive `DocTest(examples, globs, name, filename, lineno, docstring)` rebuild, because that rebuild reuses the same `Example` objects. Metadata the runner does *not* need — groups, wildcards, pairing — never touches a stdlib object and dies in the -projection layer. *Price:* one subclass to explain, and a rule that nothing may -smuggle through `DocTest.name`. +projection layer. + +*Price:* the subclass must restore equality, because +{meth}`doctest.Example.__eq__` gates on **exact type identity** +([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)) — +a bare subclass compares unequal to a stock `Example` with identical fields, in +both directions, while hashing the same. So the subclass overrides `__eq__` with +an {func}`isinstance` check and re-binds `__hash__` explicitly; Python's +reflected-operand rule then makes equality symmetric again: + +```{doctest} +>>> import doctest +>>> class Tagged(doctest.Example): +... def __eq__(self, other): +... if not isinstance(other, doctest.Example): +... return NotImplemented +... return (self.source, self.want, self.lineno, self.indent, +... self.options, self.exc_msg) == ( +... other.source, other.want, other.lineno, other.indent, +... other.options, other.exc_msg) +... __hash__ = doctest.Example.__hash__ +>>> plain, tagged = doctest.Example("1\n", "1\n"), Tagged("1\n", "1\n") +>>> tagged == plain, plain == tagged, tagged in [plain] +(True, True, True) + +Without the override, a bare subclass is unequal both ways: + +>>> class Bare(doctest.Example): pass +>>> bare = Bare("1\n", "1\n") +>>> bare == plain, plain == bare +(False, False) +``` + +The alternative — setting the attribute on a *stock* `Example` with no subclass +at all — is equality-safe by construction and equally durable. It is rejected +only because it forfeits the typed surface; if the subclass ever proves +troublesome, that is the fallback. **Node-id granularity versus shared state.** *Position:* decouple them — N `DocTest`s under one node id. *Price:* selecting a group runs all its blocks; @@ -504,9 +608,20 @@ block gains an item relative to `sphinx-build -b doctest`, and `--collect-only` must not evaluate the gate, which is why `:skipif:` is carried through collection unevaluated and run in `runtest()`. -**Collection purity versus `--collect-only` fidelity.** *Position:* collection -evaluates no user Python, so it is a pure function of (bytes, argv, ini). *Price:* -`--collect-only` no longer shows which blocks will skip. +**Collection determinism versus `--collect-only` fidelity.** *Position:* +collection evaluates no *author-supplied* Python — `:skipif:` is carried through +unevaluated and run in `runtest()`. *Price:* `--collect-only` no longer shows +which blocks will skip. + +Collection is **not** a pure function of (bytes, argv, ini), and claiming so +would be wrong on four counts: `.. include::` reads transitive files, docutils +directive implementations execute during parsing, the directive registry is +process-global, and MyST plugins change the tree. The defensible contract is +determinism over **complete source closure + normalized settings + frozen +registry**. Deferring the author's gate removes the largest divergence risk; it +does not make xdist divergence structurally impossible, and the +{doc}`registry freeze <0006-pytest-private-api-compatibility>` is what closes the +rest. **Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards an orphan `testoutput`, silently discards a `testoutput` following a `doctest` @@ -640,8 +755,8 @@ conformance test in CI. fabricated line, and does not affect its siblings. - Every `--dist` mode works, because there is no shared state to split. - No CPython code-object clone, and no process-global rebinding of anything. -- Collection is a pure function of (bytes, argv, ini), so worker divergence is - structurally impossible and `--collect-only` runs no user code. +- Collection runs no author-supplied Python, so `--collect-only` has no side + effects and the largest source of worker divergence is removed. - Grouping is one pure function with no docutils, pytest or filesystem dependency, and is testable without any of them. - A new block kind is a registration, not an edit to a method branching on string @@ -696,9 +811,12 @@ path), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed) ## Final position The core produces real {class}`doctest.DocTest` objects holding real -{class}`doctest.Example` objects, and `Example.source` is the author's verbatim -text. Everything else — groups, phases, pairing, diagnostics, distribution — is a -layer above that fact, and no layer reaches around another. +{class}`doctest.Example` objects. `Example.source` is the stdlib-normalized +executable body — prompts and indentation stripped, trailing newline added, the +stripped column recorded in `Example.indent` — not a synthesized wrapper and not +the author's verbatim text, which is what `Block.source` holds. Everything else — +groups, phases, pairing, diagnostics, distribution — is a layer above that fact, +and no layer reaches around another. The unit that shares a `globs` mapping is the unit pytest schedules. That is the one invariant every other property in this document follows from, and it is not From 098017cc04c650f44c6b16d216e3c020dd33d456 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 14:58:59 -0500 Subject: [PATCH 12/24] docs(adrs[0002]): Correct the SyntaxError claim and scope the matrix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: A stock DocTestRunner does not raise on a multi-statement body — the compile call sits inside the loop's own try, so it records an unexpected exception and returns failed=1. Only DebugRunner propagates. The harness also needs report_* events, since a skip that wrongly executes moves neither counter. what: - State that stock records rather than raises, and assert the exec-mode case as a pair: stock fails, this runner passes - Add report_* hook events and repeated runs to the matrix - Exclude cross-block FAIL_FAST and cleanup aggregation, which belong to run_group's lifecycle tests rather than the per-example loop --- .../0002-runner-conformance-across-cpython.md | 29 ++++++++++++++++--- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md index f7183d9..90bc45c 100644 --- a/docs/adrs/0002-runner-conformance-across-cpython.md +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -62,10 +62,31 @@ incremented *before* the `SKIP` check, so a skip that wrongly executes moves neither counter — it is invisible to both the tuple and to `summarize()` at zero failures, and only the `report_*` text distinguishes it. -The exec-mode case has no stock counterpart to compare against: a stock runner -raises `SyntaxError: multiple statements found` on a multi-statement body. It is -asserted against a recorded expectation instead, and that asymmetry is the point -— it is the one behaviour this runner exists to add. +The exec-mode case is the one the two runners are *meant* to disagree on, and it +still compares against stock. `compile()` raises on a multi-statement body, but +that call sits inside the loop's own `try` +([`Lib/doctest.py:1398-1408`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1398-L1408)), +so a stock {class}`doctest.DocTestRunner` catches the `SyntaxError` and records +it as an unexpected exception rather than propagating it: one +`report_unexpected_exception` call, `TestResults(failed=1, attempted=1)`, and +`_stats` at `(1, 1, 0)`. Only {class}`doctest.DebugRunner` — and pytest's runner +beneath it — converts that into a raise, as +{exc}`doctest.UnexpectedException`. So the case is asserted as a pair: stock +records the failure, this runner records a pass. A regression that silently +reverts to `"single"` mode shows up as the two converging. + +**What else belongs in the matrix, and what does not.** Add `report_*` hook +events — the only channel that distinguishes a skip which wrongly *executed*, +since `attempted` increments before the `SKIP` check and neither counter moves — +and repeated runs of one test, which exercise accumulator arithmetic across +calls. + +Cross-block `FAIL_FAST` and cleanup aggregation stay out. Both are properties of +`run_group()` rather than of the per-example loop, so a stock runner offers +nothing to compare them against; they belong to +{doc}`0001-typed-vanilla-doctest-core`'s item-lifecycle tests. A +{exc}`pytest.skip` raised inside an example and a debugger exit are likewise +pytest-layer concerns, testable only through a pytest session. Version handling is by capability probe, never by version comparison, so a backport, a vendored interpreter or a fork behaves correctly rather than by From fd26150c080dd9b75600c48dd26897b9797fd459 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 14:58:59 -0500 Subject: [PATCH 13/24] docs(adrs[0004]): Replace the mechanism that does not exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: Both mechanism assumptions were wrong. A docutils system_message carries a level and text and nothing semantically stable, so keying suppression on a stable code is impossible for exactly the messages this record wants to suppress. And attaching an observer is additive — the message still reaches the warning stream. what: - Say codes exist only for diagnostics this project emits, and that docutils-originated messages must be classified instead - Give the full three-part recipe: halt_level above 4, report_level 5 or warning_stream disabled, then the observer - Open the classifier question, with the two candidate answers and the two docutils dialects a text table would have to handle - Replace the promotion rule: registered-name is inverted for a typo and never fires for a swallowed foreign container. Promote on body content matching the example regex, with near-miss as an additive rule, and record that this is reST-only - Mark the direction not yet implementable, rather than implying it is --- docs/adrs/0004-diagnostics-as-data.md | 47 ++++++++++++++++++++++----- 1 file changed, 39 insertions(+), 8 deletions(-) diff --git a/docs/adrs/0004-diagnostics-as-data.md b/docs/adrs/0004-diagnostics-as-data.md index 0eb82a1..aceb8cb 100644 --- a/docs/adrs/0004-diagnostics-as-data.md +++ b/docs/adrs/0004-diagnostics-as-data.md @@ -19,9 +19,24 @@ tests and the session exits green. A page that checks nothing reports the same way as a page that passes. {doc}`0001-typed-vanilla-doctest-core` gives the front-end layer a second return -value for this: `Diagnostic(level, code, message, path, line)`, produced by -setting `halt_level` past the abort threshold and attaching a reporter observer. -Suppression and promotion key on the stable `code`, never on message text. +value for this: `Diagnostic(level, code, message, path, line)`. + +**Two mechanism assumptions in the first draft were wrong, and the fix is not +cosmetic.** + +*There is no stable code to key on.* A docutils `system_message` carries a level +and text, and nothing semantically stable. So codes exist only for diagnostics +**this project emits**; docutils-originated messages arrive code-less and have to +be *classified* before they can be suppressed or promoted. The classifier is the +open question below, and it cannot be "key on the code", because for these +messages there is none. + +*An observer does not silence the stream.* Attaching one is additive: the message +still reaches the warning stream. Turning reporter output into values needs three +settings together — `halt_level` above 4 (both to avoid the mid-parse abort and +because a halting message bypasses observer notification entirely), +`report_level` at 5 or `warning_stream` disabled to stop the write, and then the +observer. ## Question @@ -40,8 +55,10 @@ surface. ## Direction -Suppress by code, narrowly, and only for the two classes a bare-docutils parse -cannot judge: unknown roles and unknown directives. +Suppress narrowly, and only for the two classes a bare-docutils parse cannot +judge: unknown roles and unknown directives. "By code" is the intent; the +classifier that assigns a code to a docutils message is unsettled, so this +direction is not yet implementable. Every diagnostic raised by this project's own layers defaults to visible, and `level="error"` from those layers fails collection with the file and line named. @@ -56,8 +73,22 @@ global on/off switch. - Whether diagnostics surface as {class}`pytest.PytestWarning` subclasses, giving `-W error::` control for free, or as a dedicated report section. -- Whether an unknown-directive diagnostic should be promoted when the directive - name is one this project registers — that case is not Sphinx supplying it, it - is a registration that did not happen, which is the GH-48 failure mode. +- **What classifies a code-less docutils message.** The options are an owned, + version-pinned message-text table with a test that fails on upstream rewording + (and which must handle two dialects — reST's `Unknown directive type "x".` at + ERROR/3 versus MyST's `Unknown directive type: 'x'` at WARNING/2), or + pre-empting at the source by overriding the directive-dispatch path so the + unknown case never becomes a reporter message at all. This is the decision + ADR 0004 cannot ship without. +- **What promotes an unknown-directive message back to visible.** The first + draft proposed "when the name is one this project registers", but that is + inverted for the typo case — a misspelled `.. doctset::` is precisely *not* a + registered name — and never fires for a foreign container whose body was + swallowed unparsed. The rule that covers both is **body content**: promote when + the swallowed body matches `DocTestParser._EXAMPLE_RE`. Near-miss-to-a-registered + -name is a useful additional rule for the typo case, where the body check does + not help. +- The body-content rule is reST-only as stated, because myst-parser discards the + fence body. The Markdown equivalent is unsolved. - Whether the CLI (`python -m doctest_docutils`) and the pytest plugin share one formatter or two. From 4d9596c0c78cf1c7d5e72b17c59375bf3b67f0b3 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 14:58:59 -0500 Subject: [PATCH 14/24] docs(adrs[0005,0006]): Settle the matrices these records depend on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: 0005 said docutils >=0.22 requires Sphinx >=9.1. Sphinx 9.0 already permits it — but requires Python 3.11, and gp-sphinx's sphinx<9 cap makes the requirement unsatisfiable regardless. 0006 proposed wrapper=True, which is gated on pluggy >=1.2 rather than pytest 8, and pytest 7 permits a pluggy that raises at plugin import. what: - 0005: state the move as three parts, with the gp-sphinx cap named as the binding constraint and upstream of this repo; add the resolved matrix per interpreter; target >=0.22,<0.23 rather than an open-ended floor; correct "pins" to "resolves"; make dropping Python 3.10 the blocking open question - 0006: use old-style hookwrapper with force_result, which needs no floor and was verified on pytest 7 and 9; name the minimum supported pytest; record that DoctestItem is public and the filtered collector is not --- .../0005-line-recovery-for-nested-blocks.md | 51 +++++++++++++++---- .../0006-pytest-private-api-compatibility.md | 29 ++++++++--- 2 files changed, 62 insertions(+), 18 deletions(-) diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index 79325b1..99cd1f8 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -10,7 +10,8 @@ Date: 2026-08-02 docutils does not report a usable line for every node, and what it reports differs by front-end and by version. -At **docutils 0.21.2**, which this project currently pins, a bare `>>>` block +At **docutils 0.21.2** — which this project does not pin but does resolve, via +its Sphinx and myst-parser constraints — a bare `>>>` block nested in a `.. note::`, a list item, a block quote or a `{tab}` directive reports `line=None, source=None`. A top-level reStructuredText `doctest_block` reports its **last** line. A MyST fence reports its **first** line. An @@ -43,17 +44,39 @@ so substituted classes leak into subsequent parses that did not ask for them. The claim that substitution is "fully scoped to one parse with no process-global mutation" is wrong. -## Decision +## Direction -**Raise the docutils floor instead.** +**Raise the docutils floor instead** — but that is a support-matrix decision, not +a pin change, and it is not this record's to make alone. docutils 0.22 fixed the underlying defect upstream: a nested block reports a real line, and top-level and nested blocks agree on reporting the **first** line rather than the last. Every case this record was invented to work around is -resolved by a floor bump, with no probe, no substitution and no fallback path. +resolved by the floor, with no probe, no substitution and no fallback path. -The cost is that docutils ≥ 0.22 requires Sphinx ≥ 9.1, so this is a coordinated -dependency move rather than a one-line pin change. +Getting there is three moves, and the second is upstream of this repository: + +1. **Raise `requires-python` to `>= 3.11`.** Sphinx 9.0 is the first release that + permits docutils 0.22, and it declares `requires-python >= 3.11`. Dropping 3.10 + also touches the classifiers, the mypy and ruff target versions, and the CI + matrix. +2. **Ship a `gp-sphinx` release that widens its `sphinx < 9` cap.** This is the + binding constraint today, and it is not in this repository. With the cap in + place, a resolver asked for `docutils >= 0.22` reports the requirements + unsatisfiable. +3. **Then** declare `docutils >= 0.22, < 0.23`. An open-ended floor breaks the + moment a resolver reaches 0.23. + +Resolved versions per interpreter, with `docutils >= 0.22` requested: + +| Python | docutils | myst-parser | Sphinx | +|---|---|---|---| +| 3.10 | 0.23 | 0.13.6 | 3.5.3 — a degenerate backtrack, not viable | +| 3.11 | 0.22.4 | 5.1.0 | 9.0.4 | +| 3.12–3.14 | 0.22.4 | 5.1.0 | 9.1.0 | + +Today's lock resolves Sphinx 8.1.3 on Python 3.10 and 8.2.3 elsewhere, and +neither permits docutils 0.22. ## Consequences @@ -72,12 +95,18 @@ statement. ## Open -- Whether to raise the floor now or support both, since 0.21.2 is what the - project pins today. Supporting both means keeping the normalization branch and +- **The blocking question: does this project drop Python 3.10?** Everything else + here is downstream of that, and it is a support-matrix decision that outlives + this record. Until it is settled, "raise the floor" is a direction, not a + decision — which is why this record's status stays `Draft`. +- Whether to raise the floor at all or support both, since docutils 0.21.2 is + what resolves today. Supporting both means keeping the normalization branch and documenting two behaviours for the same page. -- Whether the Sphinx ≥ 9.1 move belongs in this record or its own. It changes the - default group an unargumented block joins, which is a semantics change beyond - line numbers. +- Sequencing with the `gp-sphinx` cap. That release has to land first, and this + repository does not control it. +- Whether the Sphinx move belongs in this record or its own. Sphinx 9.1 changes + the default group an unargumented block joins, which is a semantics change + beyond line numbers. - Whether tests should pin exact `(path, line)` for a bare block nested in a `.. note::`, a list item, a block quote and a `{tab}` directive. They should — they are the regression net for the floor, and this repository already has diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index 35c6503..8cae260 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -53,8 +53,11 @@ What private surface is depended on, and how does a pytest release that changes it fail? The current surface is `_get_checker`, `get_optionflags`, -`_get_continue_on_failure`, `_get_report_choice`, `MultipleDoctestFailures`, and -a `pytest.DoctestItem` subclass. `_init_runner_class` is explicitly *not* usable: +`_get_continue_on_failure`, `_get_report_choice` and `MultipleDoctestFailures`. +Not everything in the quarantine is equally risky: {class}`pytest.DoctestItem` is +**public** — exported from `pytest` — so subclassing it is ordinary API use. The +collector class filtered out of the multicall result is private, and that filter +is the part that needs a version matrix. `_init_runner_class` is explicitly *not* usable: `PytestDoctestRunner` is defined inside it ([`_pytest/doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181)) and is unreachable by name, which is why the `OutcomeException` re-raise, @@ -67,11 +70,23 @@ Quarantine every private import in one module, `pytest_doctest_docutils._compat` with a pinned support matrix. **Filter the built-in's collector out of the `pytest_collect_file` result, in a -`@pytest.hookimpl(wrapper=True)`.** The directory collector consumes the -multicall result directly, and returning a modified result from a hook wrapper is -documented and supported. Filtering there removes the duplicate *before* the -built-in collector parses anything, so neither the duplicate item nor its -collection error is ever produced — which late deselection cannot achieve. +hook wrapper.** The directory collector consumes the multicall result directly, +and returning a modified result from a wrapper is documented and supported. +Filtering there removes the duplicate *before* the built-in collector parses +anything, so neither the duplicate item nor its collection error is ever produced +— which late deselection cannot achieve. + +**Use the old-style `hookwrapper=True` with `outcome.force_result()`.** New-style +`wrapper=True` is gated on **pluggy ≥ 1.2**, not on pytest 8 — it works fine +under pytest 7 with a new enough pluggy. But pytest 7 declares only +`pluggy>=0.12,<2.0`, so a resolver may legally install pluggy 1.0 or 1.1, where +`wrapper=True` raises `TypeError` *while importing the plugin* — a session-wide +abort, which this record and {doc}`0001-typed-vanilla-doctest-core` both forbid. +Old style needs no floor at all and was verified working on both pytest 7 and 9. + +Whichever spelling is used, **name the minimum supported pytest**. The CI matrix +floor is 7 and the package declares no pytest dependency, so today the support +statement exists only in the workflow file. **Fail on an unsupported pytest only when an affected document is collected**, not at plugin registration. A `pytest11` plugin that raises at import takes down From 16d1a182c176fa07d87ab959131cf53ec057f0fa Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 15:01:29 -0500 Subject: [PATCH 15/24] notes(analyses): Correct the drift a second review found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: Several notes still described PR #87 as shipped, carried the superseded GroupTest model, or repeated claims later verification falsified. The true baseline is now known from trunk's source: released gp-libs appends one DocTest per matched node named page.md[k] and gives each its own copied globs. what: - Split the taxonomy rows: released gp-libs is one block, one item, isolated globals; PR #87 is a separate proposed row - Move Sphinx into the N-DocTests row and drop "unoccupied", since Sphinx already executes that shape without addressable ids - Rename the identity rule to never-source-coordinate-derived, and say an ordinal among extracted blocks satisfies it — which is what the released finder already uses - Replace GroupTest with GroupPlan in the data-flow diagram, and attribute the clone and the synthetic page to PR #87 - Separate Sphinx's three units in 20: runner call per block, shared state per group, result as process-wide counters - Drop BlockKind.node_types from the seam list and replace the rejected compile-policy callable with the private ExecutionProfile, matching ADR 0001 - Record that :pyversion: is declared on both testcode and testoutput and honoured on neither - Correct the reporter section: an observer is additive and does not silence the stream, and a system_message carries no stable code - Record why the state_classes substitution is not parse-scoped - Fix the Sphinx heading and link that disagreed on version --- notes/analyses/00-taxonomy.md | 22 ++++++++++------ notes/analyses/15-sphinx-ext-doctest.md | 10 ++++--- notes/analyses/16-docutils-myst.md | 26 ++++++++++++++----- notes/analyses/20-data-structures.md | 13 +++++++--- notes/analyses/21-data-flows.md | 6 ++--- notes/analyses/22-extension-seams.md | 4 +-- .../23-namespace-scope-and-test-identity.md | 10 +++++-- notes/analyses/90-bibliography.md | 4 +-- 8 files changed, 63 insertions(+), 32 deletions(-) diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md index dc53189..dbc6d12 100644 --- a/notes/analyses/00-taxonomy.md +++ b/notes/analyses/00-taxonomy.md @@ -9,7 +9,7 @@ a different philosophy. | # | Axis | Positions | |---|---|---| | 1 | **Sharing unit vs. selection unit** | same object · different objects, acknowledged · different objects, unacknowledged | -| 2 | **Test identity** | author-declared name · symbol-derived · positional (line/column or byte range) | +| 2 | **Test identity** | author-declared name · symbol-derived · ordinal among extracted blocks · source-coordinate-derived (line/column or byte range) | | 3 | **Runtime object model** | stdlib `DocTest`/`Example` · own model with a bridge · own model, no bridge | | 4 | **Document model** | real parse tree · flat character spans · regex over text · none | | 5 | **Option representation** | `int` bitmask · structured state · enum | @@ -28,7 +28,8 @@ a different philosophy. | Sybil | **different, unacknowledged** | positional `line:N,column:N` | stdlib `Example`, one-line `DocTest` fork | flat character spans | | xdoctest | same (one `DocTest` per docstring) | `Callname:N` | own, with a late bridge back | none for `.rst`/`.txt` | | pytest-examples | none — no implicit sharing | positional `path:start-end` | none | regex over fences | -| `doctest_docutils` today | configurable; `per-block` is different-and-guarded | group name, or `page.md[k]` | stdlib | real doctree | +| `doctest_docutils` released | same object (one block, one item, isolated copied globals) | `page.md[k]` ordinal | stdlib | real doctree | +| PR #87 (proposed) | configurable; `per-block` is different-and-guarded | group name, or `page.md[k]` | stdlib | real doctree | | ADR 0001 | **different, decoupled by construction** | group name, or `page.md[k]` | stdlib | real doctree | | System | 5 options | 6 extension | 7 vs. pytest doctest | 8 strictness | 9 direction | @@ -39,7 +40,7 @@ a different philosophy. | Sybil | `int` | callable aliases | replace by instruction (`-p no:doctest`) | strict | read-only | | xdoctest | structured `TypedDict` + bridge | two registries, else fork | unregisters it | **permissive** | read-only | | pytest-examples | n/a | none | no collector — composes trivially | no want at all | **write-back** | -| `doctest_docutils` today | `int` + a forked lookup | directive subclassing | blocks it, imports its privates | strict | read-only | +| `doctest_docutils` released | `int` | directive subclassing | blocks it, imports its privates | strict | read-only | | ADR 0001 | `int` + registry | `Protocol` + nominal + `BlockKind` registry | **compose; require it** | strict | read-only | ## What the matrix shows @@ -55,12 +56,17 @@ product space is four cells: | | one node id | N node ids | |---|---|---| -| **one `DocTest`** | Sphinx, `doctest_docutils` `merged` | — (incoherent) | -| **N `DocTest`s** | *unoccupied until ADR 0001* | Sybil, `doctest_docutils` `per-block` | +| **one `DocTest`** | PR #87's `merged` | — (incoherent) | +| **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids; ADR 0001 adds the pytest identity | Sybil, PR #87's `per-block`, released `doctest_docutils` (no sharing) | -The bottom-right cell is where the silent failure lives. The bottom-left cell — -per-block `DocTest`s under one item — gives per-block reporting *and* an -unsplittable sharing unit, and no surveyed project occupies it. +The bottom-right cell is where the silent failure lives — but only when the +blocks share state. Released `doctest_docutils` sits there safely because its +blocks share nothing: each gets its own copied `globs`. + +The bottom-left cell is where the design goes. Sphinx already *executes* that +shape — N `DocTest`s over one group namespace — but produces no addressable unit +for any of them, since every block shares one `DocTest.name`. Giving that shape a +pytest identity is the contribution; the execution shape is not new. **Axis 3 has an empirical answer.** xdoctest is the controlled experiment for abandoning the stdlib object model, and it is now building the bridge back. The diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md index 3590dd7..538366b 100644 --- a/notes/analyses/15-sphinx-ext-doctest.md +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -33,11 +33,13 @@ losses live: an orphan `testoutput` is discarded; a `testoutput` following a `doctest` block is discarded, because a doctest entry has length 1 and fails the `len(latest_test) == 2` guard; and a second `testoutput` *replaces* the first. -A fourth silent loss is in the directive rather than `add_code`: `:pyversion:` is -in `TestcodeDirective.option_spec` +A fourth silent loss is in the directives rather than `add_code`: `:pyversion:` is +declared on **both** `testcode` ([`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180)) -and is then ignored, because the version gate only runs for `doctest` and -`testoutput`. +and `testoutput` +([`:184-190`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L184-L190)), +and honoured on neither — the version gate runs only for `doctest`. An author who +writes it on either gets no error and no gate. `:options:` on a `testcode` is **not** a silent loss. It is absent from `TestcodeDirective.option_spec` entirely, so writing it is an unknown-option diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md index 9b524c4..f1b002c 100644 --- a/notes/analyses/16-docutils-myst.md +++ b/notes/analyses/16-docutils-myst.md @@ -24,7 +24,7 @@ docutils.nodes.Element attributes: dict[str, Any] docutils.parsers.rst.Parser .state_classes an INSTANCE attribute, therefore substitutable - per parse with no process-global mutation + per parse — but NOT scoped in practice: see below myst_parser.parsers.docutils_.Parser(RstParser) [v5.1.0:235] .settings_spec = (..., create_myst_settings_spec(), *RstParser.settings_spec) @@ -105,16 +105,28 @@ bare language fence onto a directive name, so a project that prefers not to writ ## Reporter behaviour Default settings send reporter output to stderr and raise `SystemMessage` at -`halt_level`, aborting mid-parse. Both are configurable: raising `halt_level` past -the abort threshold and attaching an observer with `Reporter.attach_observer` -turns messages into values. `Reporter.system_message` notifies observers for any -level above `DEBUG` independently of `report_level`, so observation and display are -separable — which is what makes ADR 0004's code-keyed suppression possible without -losing the underlying record. +`halt_level`, aborting mid-parse. + +Turning messages into values takes **three** settings, not one. `attach_observer` +is *additive*: the observer receives the message and the warning stream still gets +written. So all of `halt_level` above 4 (both to avoid the abort and because a +halting message bypasses observer notification entirely), `report_level` at 5 or +`warning_stream` disabled to stop the write, and the observer itself. + +A `system_message` carries a level and text and **nothing semantically stable** — +no code. So a downstream that wants to suppress or promote by category has to +*classify* the message, and cannot key on an attribute docutils does not provide. +That is the open problem in ADR 0004. ## What it cannot do - **Scope a directive registration** to one parse, one document or one thread. +- **Scope a `state_classes` substitution either.** `state_classes` is an instance + attribute, which makes substitution *look* parse-local — but a nested parse + builds its machine from `nested_sm_kwargs`, so a top-level substitution never + reaches a nested block, and `RSTState.nested_sm_cache` is a shared **class** + attribute that leaks substituted classes into later parses. This is why ADR 0005 + abandoned the mechanism. - **Report a line for every node.** See the table above. - **Type its own attribute channel.** `Element.attributes` is `dict[str, Any]`, and typeshed's stub for `get(key, failobj: _T) -> _T` is actively wrong — it claims diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index e38c83b..dea809d 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -10,7 +10,7 @@ The disagreements are entirely about which of those four are the same object. |---|---|---|---|---| | CPython `doctest` | `Example` | `DocTest` | `DocTest.globs` | `TestResults` (2 ints + an attribute) | | `_pytest.doctest` | `Example` | `DocTest` = one `Item` | `DocTest.globs`, wiped per item | `MultipleDoctestFailures` → `ReprFailDoctest` | -| `sphinx.ext.doctest` | `TestCode` | `TestGroup` | `ns` assigned post-construction | six ints + text to a file | +| `sphinx.ext.doctest` | `TestCode` | one `DocTest` **per block** (`TestGroup` is the batching unit, not the execution unit) | `ns`, assigned post-construction to every block's `DocTest` | six ints + text to a file | | Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | | xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | | pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | @@ -18,9 +18,14 @@ The disagreements are entirely about which of those four are the same object. Reading across the "shared-state unit" column against the "execution unit" column is the whole design problem. Sphinx and Sybil both put the shared state at a -coarser granularity than the execution unit; only Sphinx also puts the *pytest -item* at that same coarser granularity, which is why Sphinx never hits Sybil's -`-k` failure. +coarser granularity than the execution unit. Sphinx gets away with it by having +no selectable unit at all — it is a builder, not a test runner, so there is +nothing for a `-k` to split. Sybil does not: it hands out one pytest item per +span over one shared mapping, which is the failure. + +Three units are easy to conflate for Sphinx specifically, so keep them apart: the +**runner call** is per block, the **shared state** is per group, and the **result** +is six process-wide integers. ## Field-by-field: what a source unit carries diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md index ad371e8..f6b1f40 100644 --- a/notes/analyses/21-data-flows.md +++ b/notes/analyses/21-data-flows.md @@ -29,7 +29,7 @@ sphinx.ext.doctest ADR 0001 path ─► markup.parse_file ─► (Blocks, Diagnostics) │ - ├─► project() ─► GroupTest{group, tests[], globs} + ├─► project() ─► GroupPlan{group, blocks[], seed} │ pure: no docutils, no pytest, no filesystem, │ no user code — :skipif: passes through unevaluated │ @@ -63,14 +63,14 @@ two. **At the compile call.** stdlib hard-codes `"single"`. Sphinx flips a mutable builder attribute and reads it through a process-global rebinding of -`doctest.compile` that is never restored. `doctest_docutils` today clones the +`doctest.compile` that is never restored. PR #87 proposes cloning the mangled loop's code object to get a private version of that rebinding. ADR 0001 carries the mode on the example data and reads it in a loop it owns — the only one of the four that neither mutates process state nor copies a code object. **At the location.** stdlib computes `test.lineno + example.lineno + 1`. Sphinx gives every block in a group the same `DocTest.name` and pays for it by overriding -a private method to swallow an `IndexError`. `doctest_docutils` today fabricates a +a private method to swallow an `IndexError`. PR #87 fabricates a synthetic page so the arithmetic stays true across merged blocks. ADR 0001 gives each block its own `DocTest`, so the arithmetic is true without fabrication. diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index 013c20c..05de168 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -85,10 +85,10 @@ needs it. Applied to the candidates that came up: | Candidate | Verdict | |---|---| -| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Its `node_types` field is what keeps the `nodes.comment` requirement reviewable rather than buried in a `findall` call | +| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind`, so the leaf keeps its stdlib-only contract | | Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()` — adding one there requires subclassing two classes | | `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | -| Compile-policy callable | **No.** One implementation, and a second would have to reproduce the `` filename shape or break `linecache` | +| `ExecutionProfile` (private) | **Yes, private.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. Stays private until an out-of-tree caller exists | | A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | | A registry *object* with builders, digests and manifests | **No.** Two module-level dicts and two `register_*` functions do the same work | | Entry-point plugin discovery | **No.** Not until a caller outside the package exists. It also breaks xdist's collection-purity requirement, since discovery is an import side effect rather than a function of (files, argv, ini) | diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index 70bcb75..e284608 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -69,12 +69,18 @@ structural, not incidental: Every one of these evaporates when the item is the sharing unit. -## Test identity: never positional +## Test identity: never source-coordinate-derived -Two of the surveyed projects derive node ids from position — Sybil's +Two of the surveyed projects derive node ids from **source coordinates** — +Sybil's [`line:{line},column:{column}`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) and pytest-examples' `path:start-end`. +An **ordinal among the extracted blocks** is a different thing and is fine. It is +not a source coordinate: `page.md[3]` is unchanged by a paragraph inserted above +it, which is exactly the edit that renames a `line:N` id. Released +`doctest_docutils` names its tests that way already, and the rule preserves it. + For a *documentation* test runner this is indefensible, because prose above examples is the thing that changes most often. Adding a sentence renames every downstream test, which breaks `--lf`, `--nf`, checked-in deselect files, xfail diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md index 5f6daed..07bcc0a 100644 --- a/notes/analyses/90-bibliography.md +++ b/notes/analyses/90-bibliography.md @@ -102,9 +102,9 @@ Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blo | `_get_asyncio_mode` | [`:222`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222) | resolve once, query once | | `pytest_configure` | [`:295-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L295-L301) | detecting an unset default via the sentinel | -## Sphinx — `v9.1.0` +## Sphinx — `v8.2.3` -[`sphinx-doc/sphinx @ v9.1.0`](https://github.com/sphinx-doc/sphinx/tree/v8.2.3) · +[`sphinx-doc/sphinx @ v8.2.3`](https://github.com/sphinx-doc/sphinx/tree/v8.2.3) · [`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py) | Symbol | Anchor | Cited for | From 6ca61ebfe7d6c43b4e3597f74ee4d78442a207fe Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 17:17:18 -0500 Subject: [PATCH 16/24] docs(adrs[0001]): Take metadata off Example and design the seams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: A third review found the surrounding model incomplete or self-contradictory even though the invariant holds. The Example subclass was the worst of it: restoring equality with an isinstance check makes two UNRELATED subclasses compare equal, and equal to any third party's bare subclass. Verified by execution on 3.10 through 3.15. what: - Move execution policy off doctest.Example onto ProjectedBlock. The runner establishes an active execution request before delegating to stdlib run() and clears it in a finally, so stock Example objects stay stock and nothing in the kernel is subclassed for metadata - Replace the type set with the lifecycle it actually has: ParsedBlock/ParsedOutput (inert) -> ProjectedBlock (stock DocTest, phase, profile, gate, gateable ExpectedOutput) -> GroupPlan (immutable) -> RunContext (live globs) -> BlockResult/GroupResult - Drop want from the parsed layer: neither of its owners is the parsed block - Give BlockKind a profile NAME so a public type stops holding a private one - Add an error outcome and primary/secondary failures, with the precedence rule stated and pytest classification left to the adapter - Design Settings as three facets with a stated precedence chain, and resolve the frozen registry alongside them rather than pointing at ADR 0006, which defines no freeze - Record that entry-point discovery is not what breaks xdist determinism; nondeterministic contribution is - Split DocumentParser from extract_blocks so a Sphinx extension can pass its own resolved doctree, and say why that is not a builder - Move Python object discovery out of markup/: it takes an object, not text, and is DocTestFinder-shaped - Replace the share axis with ungrouped = default | block, which is the question actually being answered - Correct __lt__: it compares (name, filename, lineno, id), with name leading — the hazard holds, the old wording did not - Correct the Sphinx precedent: setup blocks are combined into one simulated DocTest, likewise cleanup; only test blocks are per-block --- docs/adrs/0001-typed-vanilla-doctest-core.md | 236 ++++++++++++++----- 1 file changed, 177 insertions(+), 59 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index c306597..8c83a3e 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -98,12 +98,19 @@ One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block `DocTest`s run in phase order against one live `globs` mapping that never leaves the item. -The execution shape is not new: `sphinx.ext.doctest` already runs several -`DocTest`s against one shared group namespace. What is new is giving that shape a -**pytest identity**. Sphinx produces no selectable, reportable unit for a group — -every block in it shares one `DocTest.name`, which is why `SphinxDocTestRunner` -overrides a private stdlib method to swallow the resulting `IndexError`. Mapping -the group onto one {class}`pytest.Item` is the contribution. +The execution shape is partly precedented. `sphinx.ext.doctest` runs several +`DocTest`s against one shared group namespace — but only for the *test* phase. +All of a group's `testsetup` blocks are combined into a **single** simulated +`DocTest` named `f"{group.name} (setup code)"`, and likewise cleanup; only +`group.tests` is one `DocTest` per block +([`sphinx/ext/doctest.py:525-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L525-L556)). + +So this design is per-block in all three phases where Sphinx is per-block in one, +and — more importantly — Sphinx produces no selectable, reportable unit for any +of them: every ordinary test block in a group shares one `DocTest.name`, which is +why `SphinxDocTestRunner` overrides a private stdlib method to swallow the +resulting `IndexError`. Mapping the group onto one {class}`pytest.Item` while +each block keeps its own identity is the contribution. What that buys, for free and with no override of `repr_failure` or `reportinfo`: per-block failure locations, per-block gutters, and per-block "location unknown". @@ -250,7 +257,8 @@ settings one frozen Settings, resolved once <- leaf; everything rea | blocks inert data: Block, BlockKind, Phase, Diagnostic, Example | (stdlib imports only) -markup/ _rst, _myst, _text, _python -> (blocks, diagnostics) +markup/ DocumentParser: _rst, _myst -> doctree + | extract_blocks(doctree) -> (parsed blocks, diagnostics) | project grouping, pairing, phase order, naming -> GroupPlan | @@ -264,8 +272,8 @@ pytest_doctest_docutils collection, items, globs lifetime, reporting | Layer | Owns | Must not know | |---|---|---| -| `settings` | One frozen `Settings` resolved once, with `None` sentinels at the resolve boundary so a future default change is announceable | pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | -| `blocks` | `Block`, `BlockKind` registry, `Phase`, `Diagnostic`, `PlannedBlock`, `GroupPlan`, `ExecutionProfile`, `Example(doctest.Example)` | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | +| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings` — plus the **frozen registry**, resolved together exactly once per session. `None` sentinels at the resolve boundary make a future default change announceable | pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | +| `blocks` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan`, `RunContext`, `ExecutionProfile`, the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | | `markup/` | Text → `(blocks, diagnostics)`. **The whole docutils vocabulary**: which node classes each kind may arrive as, the `BlockAttributes` stamp, line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration, per-kind `option_spec` | Groups as a runtime concept, `DocTest`, pytest, pairing | | `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | | `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing, run-time `:skipif:`, the profile's context manager, and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | @@ -275,6 +283,41 @@ pytest_doctest_docutils collection, items, globs lifetime, reporting `_pytest.doctest`, behind a pinned support matrix. See {doc}`0006-pytest-private-api-compatibility`. +### Settings and the frozen registry + +Settings are **faceted**, not one object every layer imports — a single shared +`Settings` becomes a dependency magnet, and the facets have genuinely different +consumers: + +| Facet | Owns | Read by | +|---|---|---| +| `ParseSettings` | encoding, MyST extensions, front-matter policy, diagnostic promotion and suppression | `markup/` | +| `ProjectionSettings` | `ungrouped` (`"default"` or `"block"`), wildcard resolution, name minting | `project` | +| `RunSettings` | optionflags, `continue_on_failure`, checker selection, report style | `runner` | + +Precedence is fixed and stated once: **per-block directive option → per-document +front matter → host configuration (pytest ini/CLI, Sphinx `conf.py`, argparse) → +built-in default.** Each adapter translates its host's configuration into the +facets exactly once, at session start; no layer reads a host config object. + +The **registry** is resolved and frozen alongside them, not as a separate object +with its own lifecycle. It holds the contributed block kinds, front ends and +execution profiles. "Frozen" means an immutable mapping built once before +collection begins; registering afterwards is an error, not a silent late +addition. + +This is what {ref}`the collection contract ` means by +*frozen registry*. Two consequences worth stating: + +- **Entry-point discovery is not inherently unsafe** under pytest-xdist. Workers + are fresh processes that re-run plugin loading with the same argv, so + entry-point resolution is identical in the controller and every worker. What + breaks determinism is *nondeterministic* contribution — a conftest that + registers conditionally on the environment — not discovery itself. +- **Nothing here is public in v1.** Contribution happens through the host's own + extension point (a pytest hook, Sphinx's `setup(app)`), and the registry type + stays private until a caller outside the package needs it. + ### Item lifecycle The custom item is load-bearing, and half-reusing {class}`pytest.DoctestItem` @@ -319,7 +362,7 @@ referents. Each term below is decided once and used only that way. | `globs` | the dict examples exec in; [`DocTest.__init__` stores a **copy**](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | — | assigned to `test.globs` after construction, run with `clear_globs=False` | Keep `globs` for the mapping | | namespace | — | [`doctest_namespace`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) means *injected names* | — | **Not** used for the sharing unit; pytest owns the word | | group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | -| scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The block-vs-document axis becomes `share` | +| scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The real question is what an *unlabelled* block joins, so the setting is `ungrouped = "default" | "block"`, not a `share` axis | | test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | | skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | | directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | @@ -336,10 +379,12 @@ class Phase(enum.IntEnum): CLEANUP = 2 -class Block(t.NamedTuple): +# --- parsed: inert, produced by extraction, owns no semantics ------------- + + +class ParsedBlock(t.NamedTuple): kind: str # registered BlockKind name - source: str # dedented body, verbatim - want: str | None # from a paired testoutput + source: str # dedented body, verbatim author text path: pathlib.Path # the file the text lives in, not the collected document line: int | None # None when docutils could not recover one position: int # 0-based document pre-order index; the naming key @@ -349,45 +394,91 @@ class Block(t.NamedTuple): hidden: bool +class ParsedOutput(t.NamedTuple): + """A `testoutput` body. Not a block: it never runs.""" + + text: str + path: pathlib.Path + line: int | None + position: int + groups: tuple[str, ...] + options: t.Mapping[int, bool] + skipif: str | None # a gated output means its testcode expects nothing + + class BlockKind(t.NamedTuple): name: str phase: Phase - profile: ExecutionProfile # how a body of this kind is compiled and run + profile_name: str # resolved against the frozen registry, not held here pairs_with: str | None grouped: bool -class PlannedBlock(t.NamedTuple): +# --- projected: one per block, with everything the runner needs ------------ + + +class ExpectedOutput(t.NamedTuple): + text: str + options: t.Mapping[int, bool] + skipif: str | None # when truthy at run time, `want` becomes "" + + +class ProjectedBlock(t.NamedTuple): phase: Phase - skipif: str | None # UNEVALUATED; gated in run_group(), not at collection - test: doctest.DocTest # one block, its own filename/lineno/docstring + test: doctest.DocTest # stock objects; its own filename/lineno/docstring + profile: ExecutionProfile # uniform for the block, resolved at projection + skipif: str | None # UNEVALUATED; gated in run_group() + expected: ExpectedOutput | None # paired testoutput, itself gateable class GroupPlan(t.NamedTuple): group: str - blocks: tuple[PlannedBlock, ...] # in phase order; IMMUTABLE + blocks: tuple[ProjectedBlock, ...] # in phase order; IMMUTABLE seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt -class GroupRun: +# --- run: one attempt ------------------------------------------------------ + + +class RunContext: """One execution attempt. Owns the live mapping; a plan never does.""" plan: GroupPlan - globs: dict[str, t.Any] # cleared and reseeded per attempt + globs: dict[str, t.Any] # cleared in place and reseeded per attempt class BlockResult(t.NamedTuple): - block: PlannedBlock - outcome: t.Literal["passed", "failed", "skipped"] - reason: str | None # the gate expression, when skipped + block: ProjectedBlock + outcome: t.Literal["passed", "failed", "skipped", "error"] + reason: str | None # gate expression when skipped; message when error + error: BaseException | None # gate or profile failure, not a doctest one class GroupResult(t.NamedTuple): group: str blocks: tuple[BlockResult, ...] failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] + primary: BaseException | None # what runtest() re-raises + secondary: tuple[BaseException, ...] # e.g. a cleanup failure after a body failure ``` +`ParsedBlock` carries no `want`, because neither owner of a `want` is the parsed +block: for a prompt-form block it is *inside* `source` and +{class}`doctest.DocTestParser` extracts it at projection, and for a +`testcode`/`testoutput` pair it is a separate `ParsedOutput`. Conflating the two +was what made "projection owns pairing" untrue. + +`BlockKind` names a profile rather than holding one, so a public type never +contains a private one. The name resolves against the frozen registry. + +`BlockResult.outcome` needs `"error"` because a gate that raises, or a profile +that fails to establish, is neither a pass, a fail, nor a skip. +`GroupResult.primary` and `.secondary` make the precedence explicit: **a body +failure outranks a cleanup failure**, a control-flow exception +({exc}`pytest.skip`, `xfail`, exit) outranks both, and the loser is recorded +rather than dropped. Classifying which exceptions are control-flow is the pytest +adapter's job, not the core's. + `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, source=None` from docutils, and an `.. include::`-ed block numbers against the @@ -444,8 +535,44 @@ docutils into a layer declared stdlib-only. The runtime objects are stdlib's, unconditionally. Precision lives in a parallel layer that never changes what is constructed. +- **Parsing and extraction are two seams, not one.** A single + `Frontend.parse(text, path)` cannot serve Sphinx, because a Sphinx extension + already *has* a doctree and a raw re-parse is not the same tree. So: + + ```python + class DocumentParser(t.Protocol): + """Text -> doctree. Plural implementations: _rst, _myst, third-party.""" + + suffixes: t.ClassVar[frozenset[str]] + + def parse( + self, text: str, path: pathlib.Path, *, settings: ParseSettings + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: ... + + + def extract_blocks( + doctree: nodes.document, *, settings: ParseSettings + ) -> ParseResult: ... + ``` + + `extract_blocks` is deliberately a plain function, not a `Protocol`: exactly + one extractor is the *point* of the split, and a second implementation would + reintroduce the standalone-versus-Sphinx divergence it exists to prevent. + Standalone reST and MyST use both halves; a Sphinx extension calls only the + extractor, on the doctree it already resolved. + + This is not a Sphinx builder. A builder owns discovery, an `env`, an `outdir` + and a reporting format; an extractor is a pure function from a doctree to + blocks, and {doc}`0001-typed-vanilla-doctest-core`'s rejection of a builder + stands. + +- **Python object discovery is not a front end.** Finding doctests in a module's + docstrings takes an *object*, not `(text, path)`, and stdlib already has the + right shape for it. It is a {class}`doctest.DocTestFinder`-shaped adapter, and + putting a `_python` module in `markup/` was a category error. + - **`Protocol` for the seams, nominal subclassing for type-checkability.** - `Frontend` is a `Protocol` so a third party can supply one structurally. The + `DocumentParser` is a `Protocol` so a third party can supply one structurally. The shipped parser *also* subclasses {class}`doctest.DocTestParser` — not because the interpreter requires it (stdlib `doctest` performs no `isinstance` check on `parser` or `test_finder`; a duck-typed object works at runtime) but because @@ -483,7 +610,7 @@ cited. The full derivation is in `notes/analyses/`. |---|---| | A failure's file line is `test.lineno + example.lineno + 1`; `Example.lineno` is 0-based within the containing string | [`doctest.py:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | | `DocTest.__init__` **copies** the globs mapping, so a shared mapping must be assigned after construction and run with `clear_globs=False` | [`doctest.py:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | -| Never sort collected tests by name: `__lt__` compares names as text, and a name carries its position as text, so `page.md[10]` sorts before `page.md[1]` | [`doctest.py:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | +| Never sort collected tests. `__lt__` compares `(name, filename, lineno, id(self))`; `name` leads, so a name carrying its position as text sorts `page.md[10]` before `page.md[1]` however correct `lineno` is. `filename` and `lineno` only break ties among equal names | [`doctest.py:596-603`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596-L603) | | The per-example loop is name-mangled; the supported in-loop seams are the four `report_*` methods and the injected checker | [`doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | | `run()` mutates global interpreter state for its duration and restores in `finally`; it is neither reentrant nor thread-safe | [`doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) | | Each example compiles under `""` in `"single"` mode with `dont_inherit=True`; `"exec"` suppresses expression echo, emptying every `want` | [`doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | @@ -560,41 +687,31 @@ that rebuild reuses the same `Example` objects. Metadata the runner does *not* need — groups, wildcards, pairing — never touches a stdlib object and dies in the projection layer. -*Price:* the subclass must restore equality, because -{meth}`doctest.Example.__eq__` gates on **exact type identity** -([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)) — -a bare subclass compares unequal to a stock `Example` with identical fields, in -both directions, while hashing the same. So the subclass overrides `__eq__` with -an {func}`isinstance` check and re-binds `__hash__` explicitly; Python's -reflected-operand rule then makes equality symmetric again: +*Price:* **none of it goes on {class}`doctest.Example`.** An earlier draft put +compile mode on an `Example` subclass, which fails twice. +{meth}`doctest.Example.__eq__` gates on exact type identity +([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)), +so a bare subclass is unequal to a stock `Example` with identical fields in both +directions while hashing the same. Restoring equality with an {func}`isinstance` +override then over-corrects: two *unrelated* subclasses compare equal to each +other, and to any third party's bare subclass. ```{doctest} >>> import doctest ->>> class Tagged(doctest.Example): -... def __eq__(self, other): -... if not isinstance(other, doctest.Example): -... return NotImplemented -... return (self.source, self.want, self.lineno, self.indent, -... self.options, self.exc_msg) == ( -... other.source, other.want, other.lineno, other.indent, -... other.options, other.exc_msg) -... __hash__ = doctest.Example.__hash__ ->>> plain, tagged = doctest.Example("1\n", "1\n"), Tagged("1\n", "1\n") ->>> tagged == plain, plain == tagged, tagged in [plain] -(True, True, True) - -Without the override, a bare subclass is unequal both ways: - ->>> class Bare(doctest.Example): pass ->>> bare = Bare("1\n", "1\n") ->>> bare == plain, plain == bare -(False, False) +>>> def tagged(name): +... ns = {"__eq__": lambda s, o: isinstance(o, doctest.Example) +... and s.source == o.source, "__hash__": doctest.Example.__hash__} +... return type(name, (doctest.Example,), ns) +>>> Exec, Await = tagged("Exec"), tagged("Await") +>>> Exec("1\n", "1\n") == Await("1\n", "1\n") +True ``` -The alternative — setting the attribute on a *stock* `Example` with no subclass -at all — is equality-safe by construction and equally durable. It is rejected -only because it forfeits the typed surface; if the subclass ever proves -troublesome, that is the fallback. +A block's execution policy is **uniform across its examples**, so it belongs on +`ProjectedBlock`, not on the examples. The runner establishes an *active +execution request* immediately before delegating to stdlib `run()` and clears it +in a `finally`; `_DocTestRunner__run` reads it. Stock `Example` objects stay +stock, and nothing in the compatibility kernel is subclassed for metadata at all. **Node-id granularity versus shared state.** *Position:* decouple them — N `DocTest`s under one node id. *Price:* selecting a group runs all its blocks; @@ -613,15 +730,16 @@ collection evaluates no *author-supplied* Python — `:skipif:` is carried throu unevaluated and run in `runtest()`. *Price:* `--collect-only` no longer shows which blocks will skip. +(the-collection-contract)= + Collection is **not** a pure function of (bytes, argv, ini), and claiming so would be wrong on four counts: `.. include::` reads transitive files, docutils directive implementations execute during parsing, the directive registry is process-global, and MyST plugins change the tree. The defensible contract is determinism over **complete source closure + normalized settings + frozen -registry**. Deferring the author's gate removes the largest divergence risk; it -does not make xdist divergence structurally impossible, and the -{doc}`registry freeze <0006-pytest-private-api-compatibility>` is what closes the -rest. +registry** — all three defined above. Deferring the author's gate removes the +largest divergence risk; it does not make xdist divergence structurally +impossible. **Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards an orphan `testoutput`, silently discards a `testoutput` following a `doctest` From 8444bfa41d014dbf845ea8dbd3b1f4dc8ca4cbcc Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 17:19:08 -0500 Subject: [PATCH 17/24] notes(analyses): Correct a third round of drift why: Review found remaining inaccuracies, and the ADR 0001 model changed under the notes. Each correction below was verified against a pinned tag or by execution. what: - __lt__ compares (name, filename, lineno, id), with name leading. The hazard survives, and it is LIVE: find() sorts blocks named page.md[k], so an eleven-block page runs its eleventh block second - parse() covers the NORMALIZED input, not the input exactly: tabs are expanded, common indent stripped, and a comment-only example is dropped outright - pytest_xdist_make_scheduler is a broader affinity seam than _split_scope; the honest claim is scoped to the shipped schedulers - myst_fence_as_directive runs a fence through the directive of the SAME name; python does not become testcode without an alias - pytest-asyncio's _get_asyncio_mode is called from several sites, so the lesson is one conversion site, not one resolution - pytest-examples stores Python string indices, and one indent scalar does not invert a dedent in general - Reconcile with the corrected ADR model: ParsedBlock/ParsedOutput, compile mode on the projected block, ungrouped instead of share - Replace note 22's reason for deferring entry points: discovery is identical across xdist workers, so nondeterministic contribution is the hazard, not discovery. Record that the freeze is real even though the registry object is not --- notes/analyses/10-cpython-doctest.md | 13 ++++++++----- notes/analyses/12-pytest-xdist.md | 10 +++++++++- notes/analyses/13-pytest-asyncio.md | 15 ++++++++------- notes/analyses/16-docutils-myst.md | 7 ++++--- notes/analyses/17-prior-art.md | 10 ++++++---- notes/analyses/20-data-structures.md | 24 +++++++++++++----------- notes/analyses/22-extension-seams.md | 16 ++++++++-------- 7 files changed, 56 insertions(+), 39 deletions(-) diff --git a/notes/analyses/10-cpython-doctest.md b/notes/analyses/10-cpython-doctest.md index 8d942d6..6fcb4f8 100644 --- a/notes/analyses/10-cpython-doctest.md +++ b/notes/analyses/10-cpython-doctest.md @@ -28,10 +28,13 @@ Passing a shared dict through the constructor has no effect whatsoever. A shared mapping must be assigned to `test.globs` after construction, and the runner must be given `clear_globs=False` or it empties the mapping in its `finally`. -**`__lt__` compares names as text** -([`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596)). -Any name carrying a position as text sorts `[10]` before `[1]`. This fails -silently: every test passes, in the wrong order. +**`__lt__` compares `(name, filename, lineno, id(self))`** +([`:596-603`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596-L603)). +`name` leads, so any name carrying a position as text sorts `[10]` before `[1]` +however correct `lineno` is; the later terms only break ties among equal names. +This fails silently: every test passes, in the wrong order. Released gp-libs hits +it — `find()` calls `tests.sort()` over blocks named `page.md[k]`, so an +eleven-block page runs its eleventh block second. **`TestResults` carries `skipped` outside the tuple** ([`:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), @@ -44,7 +47,7 @@ unpack in the ecosystem, including `doctest._test()` itself. ```text source string | DocTestParser.parse -> list[str | Example], alternating, - | reconstructing the input exactly + | covering the NORMALIZED input | DocTestParser.get_doctest -> DocTest v DocTestFinder.find(obj) -> list[DocTest] diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md index 2f7cf74..7fc3852 100644 --- a/notes/analyses/12-pytest-xdist.md +++ b/notes/analyses/12-pytest-xdist.md @@ -71,9 +71,17 @@ cause. Worker restarts are on by default. | `pytest_xdist_node_collection_finished(node, ids)` | hook — observe the agreed id list | | `pytest_xdist_setupnodes(config, specs)` | hook — receives the already-expanded spec list; never raises | | `pytest_xdist_auto_num_workers(config)` | hook | -| `LoadScopeScheduling._split_scope(nodeid) -> str` ([`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284)) | subclass hook — **the only affinity primitive in the codebase** | +| `LoadScopeScheduling._split_scope(nodeid) -> str` ([`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284)) | subclass hook — the only affinity primitive *inside the shipped schedulers* | | `@pytest.mark.xdist_group(name)` | marker, honoured only under `--dist loadgroup` | +`pytest_xdist_make_scheduler` is the broader seam: a plugin may substitute an +entire `Scheduling` implementation, which is strictly more control than +`_split_scope` alone. That does not rescue shared state, because the substitution +happens controller-side and the controller only ever sees node-id strings — but +"the only affinity seam in the codebase" overstates it, and the honest claim is +narrower: *within the shipped schedulers*, `_split_scope` is the only affinity +primitive. + `_split_scope` is worth stating plainly: it is a pure function from a node-id string to a scope string, and both shipped grouping modes are two-line overrides of it — [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) diff --git a/notes/analyses/13-pytest-asyncio.md b/notes/analyses/13-pytest-asyncio.md index 3ba4400..fdb76ea 100644 --- a/notes/analyses/13-pytest-asyncio.md +++ b/notes/analyses/13-pytest-asyncio.md @@ -36,8 +36,9 @@ drift would occur: `_get_asyncio_mode` ([`:222-232`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222-L232)) reads the CLI value, falls back to the ini value, and calls `Mode(val)` inside a `try`, translating a `ValueError` into a {exc}`pytest.UsageError` that lists the -valid modes. The lesson is not "no conversion" — it is that the conversion -happens **once**, in one named function, with a good error. +valid modes. It is *called* from several sites, so the value is not literally +resolved once per session — but the conversion and its error message live in one +named function, and that is the transferable part. Declaring its own `HookspecMarker("pytest")` namespace is the interesting one. A third party extends pytest-asyncio by implementing a hook, not by subclassing @@ -89,9 +90,9 @@ the second group is warned. `pytest_configure` does exactly that for an unset `asyncio_default_fixture_loop_scope` ([`:296-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L296-L301)). -This project has the same problem coming: ADR 0001's vocabulary change renames -`namespace_scope` to `share`, and any future move of the block-vs-document default -needs the same mechanism. +This project has the same problem coming: ADR 0001 settles the vocabulary as +`ungrouped = "default" | "block"`, and any future move of that default needs the +same mechanism. **Normalize, then query once.** In `AUTO` mode the plugin literally adds the marker it would otherwise have to special-case, so downstream code has one @@ -101,8 +102,8 @@ class of bug. **Reuse the host's vocabulary rather than inventing a parallel one.** Loop scope uses pytest's own `function`/`class`/`module`/`package`/`session` ladder and its -scope names verbatim. It does not invent a third word for lifetime. Compare the -current `namespace_scope`/`namespace_items` naming, which collides with two pytest +scope names verbatim. It does not invent a third word for lifetime. Compare +PR #87's `namespace_scope`/`namespace_items`, which collides with two pytest concepts at once. **Session-wide errors raise {exc}`pytest.UsageError`.** An invalid `asyncio_mode` diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md index f1b002c..8021966 100644 --- a/notes/analyses/16-docutils-myst.md +++ b/notes/analyses/16-docutils-myst.md @@ -98,9 +98,10 @@ plain ```` ```python ```` fence is only picked up by a prompt sniff, and the line-length guard and MyST transforms never run. Those omissions become a decision rather than an accident once the front-end owns its own configuration. -`myst_fence_as_directive` is also the answer to a real user request: it maps a -bare language fence onto a directive name, so a project that prefers not to write -`{testcode}` can still have its ```` ```python ```` blocks collected. +`myst_fence_as_directive` is narrower than it sounds. It runs the fence through +the directive of the **same name**, so a ```` ```python ```` fence looks for a +directive called `python`. It does not rename `python` to `testcode`; a project +wanting that must register a `python` directive or alias itself. ## Reporter behaviour diff --git a/notes/analyses/17-prior-art.md b/notes/analyses/17-prior-art.md index 4933274..ef55ecf 100644 --- a/notes/analyses/17-prior-art.md +++ b/notes/analyses/17-prior-art.md @@ -115,10 +115,12 @@ output is rendered back into the source file. - **Emitting the canonical form instead of parsing it collapses check-mode and update-mode into one code path.** That is a real structural insight. -- **Absolute byte offsets plus an invertible dedent scalar** are exactly what a - data model needs to rewrite source. Read-only tools discard the indent; keeping - it is the difference between "we could add `--update-examples` later" and "we - would have to redesign the data model first". Two int fields. +- **Absolute source offsets plus a recorded indent** are most of what a data model + needs to rewrite source. They are Python *string* indices, not byte offsets, and + a single indent scalar does not invert a dedent in general — a block whose lines + carry differing leading whitespace, or a tab, does not round-trip. Keeping the + offsets is still the difference between "we could add `--update-examples` later" + and "we would have to redesign the data model first". - **It composes with pytest by contributing no collector at all.** Examples are `parametrize` params, so marks, `-k`, fixtures and xfail all work unmodified. This is the cheapest correct integration in the entire survey. diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index dea809d..d8bdeb0 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -14,7 +14,7 @@ The disagreements are entirely about which of those four are the same object. | Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | | xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | | pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | -| ADR 0001 | `Block` | `DocTest` per block | one `globs` per **group**, on the `Item` | stdlib's, per block | +| ADR 0001 | `ParsedBlock` | `DocTest` per block | one `globs` per **group**, on the `Item` | `BlockResult`/`GroupResult`, projected to stdlib's | Reading across the "shared-state unit" column against the "execution unit" column is the whole design problem. Sphinx and Sybil both put the shared state at a @@ -29,10 +29,10 @@ is six process-wide integers. ## Field-by-field: what a source unit carries -| Field | `Example` | `TestCode` | `Region` | `CodeExample` | `Block` (ADR 0001) | +| Field | `Example` | `TestCode` | `Region` | `CodeExample` | `ParsedBlock` (ADR 0001) | |---|---|---|---|---|---| | source text | `source` | `code` | via `lexemes` | `source` | `source` | -| expected output | `want` | paired separately | — | written, not read | `want` | +| expected output | `want` | paired separately | — | written, not read | a separate `ParsedOutput` | | line | `lineno` (0-based, string-relative) | `lineno` | computed from span | `start_line` | `line` (nullable) | | byte offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | | dedent scalar | `indent` | — | `Lexeme.offset` | `indent` | — (deferred) | @@ -41,20 +41,22 @@ is six process-wide integers. | options | `options` | `options` | — | — | `options` | | gate | — | `skipif` on the node | — | — | `skipif` (unevaluated) | | file | on the `DocTest` | `filename` | on the `Document` | `path` | `path` | -| compile mode | — | on the *builder*, mutable | — | always exec | on the `Example` subclass | +| compile mode | — | on the *builder*, mutable | — | always exec | on `ProjectedBlock` | Three observations. -**Only pytest-examples carries byte offsets and an invertible dedent.** Those two -fields are the entire difference between a read-only tool and one that can rewrite -expected output later. ADR 0001 defers them, which is a decision to be revisited -rather than a decision made. +**Only pytest-examples carries source offsets and a recorded dedent.** They are +Python string indices rather than byte offsets, and one indent scalar does not +generally invert a dedent — but carrying them at all is the difference between a +read-only tool and one that can rewrite expected output later. ADR 0001 defers +them, which is a decision to be revisited rather than a decision made. **Compile mode is on the wrong object everywhere except ADR 0001.** Sphinx keeps it as mutable builder state read through a process-global patch; stdlib hard-codes -it in the loop. Putting it on the example data is what lets it survive -`copy.copy`, merging and any reordering, without a flag some runner has to be -holding at the right moment. +it in the loop. ADR 0001 puts it on the projected *block*, which is where it +actually belongs — a block's execution policy is uniform across its examples, so +putting it on an `Example` subclass would both over-specify and drag the +compatibility kernel into carrying metadata. **Nobody but ADR 0001 makes the line nullable.** Every other system either always has a line (because it computed the span itself) or fabricates one. With a real diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index 05de168..57a1bc7 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -90,14 +90,14 @@ needs it. Applied to the candidates that came up: | `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | | `ExecutionProfile` (private) | **Yes, private.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. Stays private until an out-of-tree caller exists | | A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | -| A registry *object* with builders, digests and manifests | **No.** Two module-level dicts and two `register_*` functions do the same work | -| Entry-point plugin discovery | **No.** Not until a caller outside the package exists. It also breaks xdist's collection-purity requirement, since discovery is an import side effect rather than a function of (files, argv, ini) | - -That last row is worth keeping in mind generally: **any registry populated by -import side effects is in tension with -[`12-pytest-xdist.md`](12-pytest-xdist.md)'s requirement** that every worker -collect identical ids in identical order. A registry whose contents depend on -which conftest happened to be imported is not a pure function of the inputs. +| A registry *object* with builders, digests and manifests | **No object, but there is a freeze.** The contributed block kinds, front ends and profiles are resolved into an immutable mapping alongside the frozen `Settings` facets, before collection begins; registering after that is an error. That is what ADR 0001's collection contract means by *frozen registry* — not a class with its own lifecycle | +| Entry-point plugin discovery | **Not in v1**, for want of a caller outside the package — *not* because it breaks xdist. Workers are fresh processes that re-run plugin loading with the same argv, so entry-point resolution is identical everywhere. What breaks determinism is nondeterministic *contribution*, such as a conftest registering conditionally on the environment | + +The distinction those two rows turn on is worth stating once: **discovery is not +the hazard, nondeterminism is.** A registry populated identically in every process +satisfies [`12-pytest-xdist.md`](12-pytest-xdist.md)'s identical-collection +requirement no matter how it was populated; a registry whose contents depend on +which conftest happened to run, or on the environment, does not. ## Anchors From 58385406fd4386e2bb7bac1386377fe2219d2551 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 18:08:40 -0500 Subject: [PATCH 18/24] docs(adrs[0001]): Make the plan a recipe and name the missing contracts why: A fourth review found the surrounding contracts did not compose. GroupPlan was called immutable while holding a mutable DocTest whose globs a run reassigns, so reruns would operate on last attempt's state. DocumentParser was said to subclass doctest.DocTestParser, whose parse(string, name) signature it cannot satisfy. And the record promised a pluggable core while declaring nothing public. what: - Hold ingredients, not a DocTest: ProjectedBlock is a recipe and RunContext materializes fresh stock Example and DocTest objects per attempt, so a plan can never carry run state - A gated testoutput makes its output ABSENT, not empty-with-options, which is what Sphinx does - Replace the nullable result record with a discriminated union, so a passed result carrying an exception is unrepresentable - Give exception precedence explicitly: control-flow, then body, then cleanup, with the loser recorded - Split ExecutionProfile (immutable factory) from ExecutionRuntime (per attempt, one per profile a group uses), so a group may mix prompt, exec and async blocks and an async runtime owns one loop - Run ordinary prompt blocks on CPython's untouched loop and reserve the owned __run for extended profiles, making the common lane compatible by construction - Give the parser three lanes, since DocumentParser cannot be a DocTestParser: strings, markup, and DocTestFinder-shaped objects - Define self.dtest as a synthetic zero-example group DocTest whose globs is the live mapping, and keep the darwin capture guard - Add the report-attribute channel, since the controller sees serialized reports rather than items - Give settings lifetimes (session, document, block) rather than one precedence sentence; move encoding to the loader and report style to the host; make wildcard and naming invariants, not knobs - Publish a contributor protocol while keeping the registry private, and record the conftest-timing and heterogeneous-worker hazards --- docs/adrs/0001-typed-vanilla-doctest-core.md | 224 ++++++++++++++----- 1 file changed, 165 insertions(+), 59 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 8c83a3e..7e5a67e 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -285,38 +285,49 @@ pytest_doctest_docutils collection, items, globs lifetime, reporting ### Settings and the frozen registry -Settings are **faceted**, not one object every layer imports — a single shared -`Settings` becomes a dependency magnet, and the facets have genuinely different -consumers: +Settings have **lifetimes**, not just precedence. "Resolve exactly once at session +start" cannot be true of document front matter, which does not exist until that +document is parsed. Three scopes: -| Facet | Owns | Read by | +| Scope | Owns | Resolved | |---|---|---| -| `ParseSettings` | encoding, MyST extensions, front-matter policy, diagnostic promotion and suppression | `markup/` | -| `ProjectionSettings` | `ungrouped` (`"default"` or `"block"`), wildcard resolution, name minting | `project` | -| `RunSettings` | optionflags, `continue_on_failure`, checker selection, report style | `runner` | - -Precedence is fixed and stated once: **per-block directive option → per-document -front matter → host configuration (pytest ini/CLI, Sphinx `conf.py`, argparse) → -built-in default.** Each adapter translates its host's configuration into the -facets exactly once, at session start; no layer reads a host config object. - -The **registry** is resolved and frozen alongside them, not as a separate object -with its own lifecycle. It holds the contributed block kinds, front ends and -execution profiles. "Frozen" means an immutable mapping built once before -collection begins; registering afterwards is an error, not a silent late -addition. - -This is what {ref}`the collection contract ` means by -*frozen registry*. Two consequences worth stating: - -- **Entry-point discovery is not inherently unsafe** under pytest-xdist. Workers - are fresh processes that re-run plugin loading with the same argv, so - entry-point resolution is identical in the controller and every worker. What - breaks determinism is *nondeterministic* contribution — a conftest that - registers conditionally on the environment — not discovery itself. -- **Nothing here is public in v1.** Contribution happens through the host's own - extension point (a pytest hook, Sphinx's `setup(app)`), and the registry type - stays private until a caller outside the package needs it. +| `SessionSettings` | defaults plus normalized host configuration; the frozen registry | once, at session start | +| `DocumentSettings` | the front-matter overlay a page is permitted to set | per document, after parsing | +| block / example policy | directive options, then inline `# doctest:` flags | per block, at projection and run | + +Not every field shares one ladder, so the precedence is stated per axis. For +option flags it follows Sphinx: **runner defaults → directive or output +`:options:` → inline flags.** + +Two fields move out of the core entirely. **Encoding** belongs to the source +loader, because `DocumentParser` already receives `str`. **Report style** belongs +to the host adapter. And wildcard resolution and name minting are *invariants*, +not user-configurable knobs — exposing them would let a project produce node ids +no other project can read. + +The **registry** is resolved and frozen alongside `SessionSettings`. "Frozen" +means an immutable mapping built once; registering afterwards is an error. + +**Contribution is public; the registry is not.** The stated goal is an +extendable, pluggable core, so a small host-neutral contributor protocol ships in +v1 — what may be contributed (front ends, block kinds, execution profiles, +checkers), when registration closes, how duplicate names are resolved, and what +ordering guarantees exist. Its *implementation* stays private. pytest hooks, +Sphinx's `setup(app)`, direct library calls and — optionally — entry points all +feed that one surface. + +Two hazards to name rather than assume away: + +- **Nested `conftest.py` files load during collection**, so "freeze before + collection" is too simple. Either the boundary moves later, or conftest-level + contribution is explicitly excluded. That is an open question in + {doc}`0006-pytest-private-api-compatibility`. +- **Entry-point discovery is not identical across workers in general.** xdist + supports SSH and socket workers and heterogeneous environments, so "same argv, + same plugins" holds only for a homogeneous local run. Either state a + homogeneous-plugin requirement or compare a deterministic registry manifest + across workers. What breaks determinism is *nondeterministic contribution*, not + discovery as such — but heterogeneity makes that distinction load-bearing. ### Item lifecycle @@ -324,6 +335,14 @@ The custom item is load-bearing, and half-reusing {class}`pytest.DoctestItem` reintroduces the exact bug this design exists to avoid. The contract, stated so an implementer cannot get it wrong by omission: +0. **The carrier.** {class}`pytest.DoctestItem` reads `self.dtest` in + `setup()`, `reportinfo()` and `_check_all_skipped()`, so the subclass must + define it even though a group holds many tests. `self.dtest` is a synthetic + **zero-example** `DocTest` for the group, and its `globs` **is** the canonical + live mapping — the same object every `RunnableBlock`'s test is given. That + makes the inherited `setup()` inject fixtures into exactly the right place + with no override of the injection itself. + 1. **Collection** builds one `GroupPlan` per (document, group) and one item per plan. An empty plan yields no item. 2. **`setup()`** starts an attempt. In order: clear the live mapping **in place**; @@ -348,8 +367,21 @@ an implementer cannot get it wrong by omission: `PytestDoctestRunner` is nested inside a factory and cannot be imported. 4. **Outcome** follows [](#the-outcome-contract): skip the item only when every runnable block is skipped. -5. **`repr_failure`** is inherited unchanged. It already reads each failure's own - `DocTest`. + `runtest()` must also keep `_disable_output_capturing_for_darwin()`, which + the inherited implementation calls before running and which has nothing to do + with grouping. + +5. **`repr_failure`** is inherited unchanged for *failures* — it already reads + each failure's own `DocTest`. It does **not** render `GroupResult`, and + claiming it does would conflate two channels. Partial-skip detail goes to a + report section and the terminal summary; see [](#the-outcome-contract). + +6. **Reporting across processes.** The controller never sees the item — it + receives serialized `TestReport` dictionaries. So `pytest_runtest_makereport` + copies a **versioned, JSON-safe** block summary onto the report; pytest + serializes arbitrary report attributes and xdist reconstructs them + controller-side. The rich `GroupResult`, and every exception in it, stays + worker-local. ### Vocabulary @@ -424,20 +456,33 @@ class ExpectedOutput(t.NamedTuple): class ProjectedBlock(t.NamedTuple): + """A RECIPE. Holds no `DocTest`, because a `DocTest` is mutable.""" + phase: Phase - test: doctest.DocTest # stock objects; its own filename/lineno/docstring - profile: ExecutionProfile # uniform for the block, resolved at projection + name: str # the minted test name + source: str # normalized executable body + filename: str + lineno: int | None + profile_name: str # resolved against the frozen registry per attempt skipif: str | None # UNEVALUATED; gated in run_group() expected: ExpectedOutput | None # paired testoutput, itself gateable class GroupPlan(t.NamedTuple): group: str - blocks: tuple[ProjectedBlock, ...] # in phase order; IMMUTABLE + blocks: tuple[ProjectedBlock, ...] # in phase order; genuinely immutable seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt -# --- run: one attempt ------------------------------------------------------ +# --- run: one attempt materializes fresh stdlib objects -------------------- + + +class RunnableBlock(t.NamedTuple): + """Materialized per attempt. Its `DocTest` is never retained by a plan.""" + + recipe: ProjectedBlock + test: doctest.DocTest # fresh stock objects, built this attempt + runtime: ExecutionRuntime # from the profile factory, this attempt class RunContext: @@ -445,21 +490,39 @@ class RunContext: plan: GroupPlan globs: dict[str, t.Any] # cleared in place and reseeded per attempt + runtimes: dict[str, ExecutionRuntime] # one per profile used by the group + + +# --- results: a discriminated union, so invalid states cannot be built ----- + + +class Passed(t.NamedTuple): + block: ProjectedBlock + + +class Failed(t.NamedTuple): + block: ProjectedBlock + failure: doctest.DocTestFailure | doctest.UnexpectedException + + +class Skipped(t.NamedTuple): + block: ProjectedBlock + reason: str # the gate expression -class BlockResult(t.NamedTuple): +class Errored(t.NamedTuple): block: ProjectedBlock - outcome: t.Literal["passed", "failed", "skipped", "error"] - reason: str | None # gate expression when skipped; message when error - error: BaseException | None # gate or profile failure, not a doctest one + error: BaseException # a gate that raised, or a runtime that would not start + + +BlockResult: t.TypeAlias = Passed | Failed | Skipped | Errored class GroupResult(t.NamedTuple): group: str blocks: tuple[BlockResult, ...] - failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] primary: BaseException | None # what runtest() re-raises - secondary: tuple[BaseException, ...] # e.g. a cleanup failure after a body failure + secondary: tuple[BaseException, ...] # e.g. cleanup failing after the body ``` `ParsedBlock` carries no `want`, because neither owner of a `want` is the parsed @@ -471,13 +534,27 @@ was what made "projection owns pairing" untrue. `BlockKind` names a profile rather than holding one, so a public type never contains a private one. The name resolves against the frozen registry. -`BlockResult.outcome` needs `"error"` because a gate that raises, or a profile -that fails to establish, is neither a pass, a fail, nor a skip. -`GroupResult.primary` and `.secondary` make the precedence explicit: **a body -failure outranks a cleanup failure**, a control-flow exception -({exc}`pytest.skip`, `xfail`, exit) outranks both, and the loser is recorded -rather than dropped. Classifying which exceptions are control-flow is the pytest -adapter's job, not the core's. +**A plan holds no `DocTest`.** {class}`doctest.DocTest` is mutable — the design +assigns `globs` to it after construction, and a run mutates that mapping — so a +plan retaining one would not be a recipe, it would be last attempt's state. Under +`--reruns` that is the false-green this design exists to prevent. `ProjectedBlock` +therefore carries the *ingredients*, and `RunContext` materializes fresh stock +`Example` and `DocTest` objects for every attempt. + +The same applies to a gated `testoutput`: when its gate is truthy the output is +**absent**, not empty. Its text *and* its output-specific options both disappear, +which is what Sphinx does, and which a pre-built `want=""` with retained options +would get wrong. + +**Results are a discriminated union**, not one record with nullable fields, so +"passed with an exception attached" is unrepresentable rather than merely +unlikely. `Errored` exists because a gate that raises, or a runtime that will not +start, is none of pass, fail or skip. + +`GroupResult.primary` and `.secondary` make precedence explicit: **a control-flow +exception outranks a body failure, which outranks a cleanup failure**, and the +loser is recorded rather than dropped. Classifying which exceptions are +control-flow is the pytest adapter's job, not the core's. `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, @@ -510,16 +587,27 @@ evaluates it against the live group mapping, after fixture injection and after earlier blocks have run. That is more useful — a gate can consult a fixture — and it is not what `sphinx-build` does. Recorded rather than hidden. -`ExecutionProfile` is a **private** protocol carrying compile mode, extra compile -flags, and an optional per-group context manager. It is not a `Literal["single", -"exec"]`, because a second execution policy already exists in this repository: -[PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await`, -which needs `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` and an event-loop lifetime that a -mode string cannot express. The profile's context manager is what `run_group()` -enters, so that lifetime is served without overriding `run()` and stdlib's -save-and-restore `finally` stays inherited. It stays private until a caller -outside the package needs it; PR #59 is the in-repo caller that justifies it -existing at all. +**`ExecutionProfile` is an immutable factory; `ExecutionRuntime` is per attempt.** +A group can mix prompt, `exec` and async blocks, so there is no single per-group +profile. The profile is chosen per block and names a factory; `RunContext` creates +one runtime *per distinct profile* the group uses, and blocks sharing a profile +share its runtime. An async runtime therefore owns one event loop for the whole +group, which is what lets awaited state cross block boundaries. + +It is not a `Literal["single", "exec"]`, because a second execution policy already +exists in this repository: [PR #59](https://github.com/git-pull/gp-libs/pull/59) +adds top-level `await`, which needs `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` and an +event-loop lifetime a mode string cannot express. The runtime's context manager is +what `run_group()` enters, so that lifetime is served without overriding `run()` +and stdlib's save-and-restore `finally` stays inherited. + +**The ordinary lane does not use the owned loop at all.** For prompt-form blocks — +the overwhelming majority — the runner is a plain reporter subclass over CPython's +*untouched* per-example loop. `_DocTestRunner__run` is invoked only for extended +profiles: `exec` bodies, top-level await, and whatever comes next. Ordinary +doctests are then compatible **by construction** rather than by differential +testing, and {doc}`0002-runner-conformance-across-cpython`'s harness shrinks to +guarding the extended lane. **Which docutils node classes a kind may arrive as is a front-end concern, not a `BlockKind` field.** `testsetup`, `testcleanup` and any `:hide:` block are @@ -561,6 +649,24 @@ layer that never changes what is constructed. Standalone reST and MyST use both halves; a Sphinx extension calls only the extractor, on the doctree it already resolved. + **`DocumentParser` is not a {class}`doctest.DocTestParser`.** The two + signatures are incompatible — stdlib's is `parse(self, string, name='')` + returning alternating `str` and `Example`, and `get_doctest` depends on exactly + that + ([`Lib/doctest.py:657`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L657), + [`:696`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L696)). + So there are three lanes, not one: + + | Lane | Contract | + |---|---| + | plain text and strings | the exact `DocTestParser` contract, unmodified | + | reST / MyST | `DocumentParser` → doctree → `extract_blocks` | + | Python objects | a `DocTestFinder`-shaped adapter | + + Anything promising `DocFileSuite(parser=...)` compatibility is a separate + stdlib-shaped façade over the first lane, not the markup lane wearing a + stdlib name. + This is not a Sphinx builder. A builder owns discovery, an `env`, an `outdir` and a reporting format; an extractor is a pure function from a doctree to blocks, and {doc}`0001-typed-vanilla-doctest-core`'s rejection of a builder From 0d52d92eaedd3099ba476aff558c93f54e805dd7 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 18:10:32 -0500 Subject: [PATCH 19/24] docs,notes: Narrow 0002, reframe 0005, and fix a fourth round of drift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: ADR 0001 now runs ordinary prompt blocks on CPython's untouched loop, which shrinks what 0002 must prove and makes 0005 orthogonal to the architecture. Review also found the notes still generalized Sphinx's per-block execution and conflated xdist's two channels. what: - 0002: scope the harness to the extended lane; ordinary blocks are the reference rather than something to differentially prove - 0005: state plainly that the floor is support policy, not core architecture, and that no part of 0001 depends on the answer - 0005: correct the Sphinx version — the group fallback for a bare, unstamped doctest_block changed in 9.0, not 9.1, and directives always stamp groups so unargumented directives are unaffected - Notes: Sphinx is per-block for its TEST phase only; setup blocks are combined into one simulated DocTest and cleanup into another - Notes: separate xdist's scheduling channel from its reporting channel — the controller sees node-id strings when scheduling and serialized reports afterwards, and reports carry arbitrary extra attributes - Notes: a nullable line is not unique to this design; Sphinx's get_line_number returns None too. What is new is per-block propagation - Notes: narrow the affinity claim to the shipped schedulers, since pytest_xdist_make_scheduler substitutes a whole Scheduling - Notes: PR #87's per-block mode is proposed, not shipped - Notes: carry the block/runtime split into the data-flow diagram --- .../0002-runner-conformance-across-cpython.md | 7 ++++ .../0005-line-recovery-for-nested-blocks.md | 15 +++++--- notes/analyses/00-taxonomy.md | 12 +++--- notes/analyses/12-pytest-xdist.md | 18 ++++++--- notes/analyses/20-data-structures.md | 11 +++--- notes/analyses/21-data-flows.md | 5 ++- .../23-namespace-scope-and-test-identity.md | 38 ++++++++++--------- 7 files changed, 67 insertions(+), 39 deletions(-) diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md index 90bc45c..afd1889 100644 --- a/docs/adrs/0002-runner-conformance-across-cpython.md +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -48,6 +48,13 @@ continuously, without a `sys.version_info` ladder? A differential conformance harness, run in CI on every supported interpreter, gating the build step that lands the runner. +**Scoped to the extended lane.** {doc}`0001-typed-vanilla-doctest-core` runs +ordinary prompt blocks on CPython's untouched per-example loop, so those need no +differential proof — they *are* the reference. The owned `__run` is invoked only +for `exec` bodies, top-level await and future profiles, and that is what this +harness guards. It is a smaller obligation than an unconditionally owned loop, +and it is the reason owning the loop is affordable at all. + A fixed case matrix — pass, fail, unexpected exception, `SyntaxError`, all examples skipped, partially skipped, `FAIL_FAST`, `REPORT_ONLY_FIRST_FAILURE`, `IGNORE_EXCEPTION_DETAIL`, and an exec-mode body — is run through both this diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index 99cd1f8..7a43251 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -46,8 +46,11 @@ mutation" is wrong. ## Direction -**Raise the docutils floor instead** — but that is a support-matrix decision, not -a pin change, and it is not this record's to make alone. +**Raise the docutils floor instead** — but that is **support policy, not core +architecture**. Nothing in {doc}`0001-typed-vanilla-doctest-core` depends on the +answer: a nullable line is the honest representation either way, and the floor +only decides how often it is `None`. This record can stay open indefinitely +without blocking the design. docutils 0.22 fixed the underlying defect upstream: a nested block reports a real line, and top-level and nested blocks agree on reporting the **first** line @@ -104,9 +107,11 @@ statement. documenting two behaviours for the same page. - Sequencing with the `gp-sphinx` cap. That release has to land first, and this repository does not control it. -- Whether the Sphinx move belongs in this record or its own. Sphinx 9.1 changes - the default group an unargumented block joins, which is a semantics change - beyond line numbers. +- Whether the Sphinx move belongs in this record or its own. Sphinx **9.0** + changed the fallback group for a bare, unstamped `doctest_block` from + `['default']` to `[doctest_test_doctest_blocks]`; directives always stamp + `groups`, so unargumented *directives* are unaffected. 9.x also differs in + fail-fast and result propagation. All of that is semantics beyond line numbers. - Whether tests should pin exact `(path, line)` for a bare block nested in a `.. note::`, a list item, a block quote and a `{tab}` directive. They should — they are the regression net for the floor, and this repository already has diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md index dbc6d12..233184a 100644 --- a/notes/analyses/00-taxonomy.md +++ b/notes/analyses/00-taxonomy.md @@ -24,7 +24,7 @@ a different philosophy. |---|---|---|---|---| | CPython `doctest` | same (one `DocTest` per docstring) | dotted symbol path | *is* the model | none — line regex over a string | | `_pytest.doctest` | same (one item per `DocTest`) | `path::module.qualname` | stdlib, unchanged | none — delegates | -| `sphinx.ext.doctest` | same (one group, one runner pass) | group name, shared by every block | stdlib, constructed per group | real doctree | +| `sphinx.ext.doctest` | same (one group, no selectable unit) | group name, shared by every test block | stdlib; per test block, combined for setup and cleanup | real doctree | | Sybil | **different, unacknowledged** | positional `line:N,column:N` | stdlib `Example`, one-line `DocTest` fork | flat character spans | | xdoctest | same (one `DocTest` per docstring) | `Callname:N` | own, with a late bridge back | none for `.rst`/`.txt` | | pytest-examples | none — no implicit sharing | positional `path:start-end` | none | regex over fences | @@ -57,16 +57,18 @@ product space is four cells: | | one node id | N node ids | |---|---|---| | **one `DocTest`** | PR #87's `merged` | — (incoherent) | -| **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids; ADR 0001 adds the pytest identity | Sybil, PR #87's `per-block`, released `doctest_docutils` (no sharing) | +| **N `DocTest`s** | `sphinx.ext.doctest` for its *test* phase only, and with *no* ids; ADR 0001 adds the pytest identity | Sybil, PR #87's `per-block`, released `doctest_docutils` (no sharing) | The bottom-right cell is where the silent failure lives — but only when the blocks share state. Released `doctest_docutils` sits there safely because its blocks share nothing: each gets its own copied `globs`. The bottom-left cell is where the design goes. Sphinx already *executes* that -shape — N `DocTest`s over one group namespace — but produces no addressable unit -for any of them, since every block shares one `DocTest.name`. Giving that shape a -pytest identity is the contribution; the execution shape is not new. +shape — but only for ordinary test blocks: all of a group's setup blocks are +combined into one simulated `DocTest`, and likewise cleanup. It also produces no +addressable unit for any of them, since every test block shares one +`DocTest.name`. So the design is per-block in three phases where Sphinx is +per-block in one, and the pytest identity is new either way. **Axis 3 has an empirical answer.** xdoctest is the controlled experiment for abandoning the stdlib object model, and it is now building the bridge back. The diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md index 7fc3852..40e82ea 100644 --- a/notes/analyses/12-pytest-xdist.md +++ b/notes/analyses/12-pytest-xdist.md @@ -20,11 +20,19 @@ controller worker (one per process) collection: list[str] <- the agreed id list ``` -The controller's entire model of the suite is a list of node-id strings that -arrived from a worker. It has no items, no marks, no fixtures and no knowledge of -what any test does. A plugin that needs "these tests share state" therefore cannot -tell the controller so directly — it can only encode the fact *into the node id* -or infer it from string shape. +The controller's entire model of the suite **during scheduling** is a list of +node-id strings that arrived from a worker. It has no items, no marks, no fixtures +and no knowledge of what any test does. A plugin that needs "these tests share +state" therefore cannot tell the controller so directly — it can only encode the +fact *into the node id* or infer it from string shape. + +**Reporting is a separate channel with a different shape.** After execution the +controller receives serialized `TestReport` dictionaries, and pytest serializes +arbitrary extra attributes on a report, which xdist reconstructs controller-side. +So a worker *can* ship structured per-block detail to the controller — as +JSON-safe data on the report, never as an object hanging off the item. Confusing +the two channels is what makes a controller-side summary look impossible when it +is not. ## Data flow diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index d8bdeb0..3aae0e9 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -58,11 +58,12 @@ actually belongs — a block's execution policy is uniform across its examples, putting it on an `Example` subclass would both over-specify and drag the compatibility kernel into carrying metadata. -**Nobody but ADR 0001 makes the line nullable.** Every other system either always -has a line (because it computed the span itself) or fabricates one. With a real -doctree there are constructs that genuinely have no recoverable line, and pytest -has a branch for exactly that — `EXAMPLE LOCATION UNKNOWN` — which is unreachable -unless the model can express it. +**A nullable line is not unique to ADR 0001.** Sphinx's own `get_line_number` +returns `None` — its docstring says "get the real line number or admit we don't +know" — for a block whose source is a stripped docstring. What ADR 0001 adds is +not the nullability but the *propagation*: `lineno=None` reaches pytest's +`EXAMPLE LOCATION UNKNOWN` branch per block, without a sibling's known line +masking it. ## What a result unit carries diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md index f6b1f40..e2ff665 100644 --- a/notes/analyses/21-data-flows.md +++ b/notes/analyses/21-data-flows.md @@ -64,8 +64,9 @@ two. **At the compile call.** stdlib hard-codes `"single"`. Sphinx flips a mutable builder attribute and reads it through a process-global rebinding of `doctest.compile` that is never restored. PR #87 proposes cloning the -mangled loop's code object to get a private version of that rebinding. ADR 0001 -carries the mode on the example data and reads it in a loop it owns — the only one +mangled loop's code object to get a private version of that rebinding. ADR 0001 puts the policy on the projected *block*, materializes a runtime per +profile, and reads it in a loop it owns — and only for extended profiles, since +ordinary prompt blocks run on CPython's untouched loop — the only one of the four that neither mutates process state nor copies a code object. **At the location.** stdlib computes `test.lineno + example.lineno + 1`. Sphinx diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index e284608..e457d20 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -22,11 +22,13 @@ narrative page needs depends on the first. | **one `DocTest`** | PR #87's `merged` | incoherent | | **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids at all; ADR 0001 gives the shape a pytest identity | Sybil; PR #87's `per-block` | -Sphinx belongs in the bottom row: it already builds one `DocTest` per block -against one shared group namespace. Its "one node id" is really *no* id — every -block in a group shares one `DocTest.name`, which is why `SphinxDocTestRunner` -overrides a private stdlib method to swallow the resulting `IndexError`. So the -execution shape is well-trodden; the contribution is making it addressable. +Sphinx belongs in the bottom row, with a qualification: it builds one `DocTest` +per *ordinary test* block against one shared group namespace, while combining all +setup blocks into one simulated `DocTest` and all cleanup into another. Its "one +node id" is really *no* id — every test block in a group shares one +`DocTest.name`, which is why `SphinxDocTestRunner` overrides a private stdlib +method to swallow the resulting `IndexError`. So the execution shape is partly +well-trodden; the contribution is making it addressable. The bottom-right cell is where the silent failure lives, and two shipped projects are in it. @@ -35,9 +37,9 @@ Sybil is there **unacknowledged**: one `Document.namespace` shared by reference, one pytest item per region. `pytest -k` on an example whose predecessor bound a name raises `NameError`, and nothing in its documentation says so. -`doctest_docutils`'s `per-block` mode is there **acknowledged and guarded** — the -guards are the xdist scheduler substitution, the scheduler refusal, and the -run-twice refusal. Those guards are the reason `_worker_count`, `_shared_page`, +PR #87's proposed `per-block` mode would sit there **acknowledged and guarded** — +the guards being an xdist scheduler substitution, a scheduler refusal and a +run-twice refusal. It has not shipped; {doc}`ADR 0003 <../../docs/adrs/0003-rejecting-per-block-items>` rejects the shape. Those guards are the reason `_worker_count`, `_shared_page`, `_is_page` and `_splitting_scheduler` exist at all. The bottom-left cell gives per-block reporting *and* an unsplittable sharing unit, @@ -53,10 +55,12 @@ structural, not incidental: - The controller never collects, so it cannot ask "which items share state" — it sees node-id strings and nothing else. Any protection must be *inferred from string shape* or *encoded into the id*. -- The only affinity primitive is +- The only affinity primitive *inside the shipped schedulers* is [`_split_scope(nodeid) -> str`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284). - `load` and `worksteal` have no scope concept at any layer, so under a user-typed - `--dist load` the options are refuse, substitute, or do not need protecting. + A plugin may substitute a whole `Scheduling` via `pytest_xdist_make_scheduler`, + but that still reasons only over node-id strings. `load` and `worksteal` have no + scope concept at any layer, so under a user-typed `--dist load` the options are + refuse, substitute, or do not need protecting. - `xdist_group` is applied [worker-side](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) and only when the worker's own `--dist` is literally `loadgroup`. A @@ -76,12 +80,7 @@ Sybil's [`line:{line},column:{column}`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) and pytest-examples' `path:start-end`. -An **ordinal among the extracted blocks** is a different thing and is fine. It is -not a source coordinate: `page.md[3]` is unchanged by a paragraph inserted above -it, which is exactly the edit that renames a `line:N` id. Released -`doctest_docutils` names its tests that way already, and the rule preserves it. - -For a *documentation* test runner this is indefensible, because prose above +For a *documentation* test runner that is indefensible, because prose above examples is the thing that changes most often. Adding a sentence renames every downstream test, which breaks `--lf`, `--nf`, checked-in deselect files, xfail lists and CI flake history. pytest-examples compounds it: the same string is the @@ -91,6 +90,11 @@ The rule: **author-declared name first, stable ordinal as fallback, and the fallback shape invariant across configuration.** The test to apply is concrete — adding a sentence to a page must rename zero tests. +An **ordinal among the extracted blocks** is not a source coordinate and is fine: +`page.md[3]` is unchanged by a paragraph inserted above it, which is exactly the +edit that renames a `line:N` id. Released `doctest_docutils` names its tests that +way already, and the rule preserves it. + Two corollaries: - `DocTest.name` must be machine-independent. Embedding an absolute path makes a From 08d02b25f848d1af71ac89e3169dd13d7f8b553c Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 19:33:06 -0500 Subject: [PATCH 20/24] docs(adrs[0001]): Make the recipe reconstructible and the results honest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: Making the plan a recipe last round dropped information the recipe must carry. One prompt block yields SEVERAL doctest.Example objects — three for a block ending in a traceback — each with its own source, want, exc_msg, lineno, indent and options, and pytest's failure renderer needs the docstring besides. A single source string and one lineno cannot rebuild that. The result types lost outcomes for the same reason: continue_on_failure produces several failures from one block, and a PASSING block can report attempted=2 skipped=1. what: - Add ExampleRecipe and carry examples plus docstring on ProjectedBlock, so materialization reproduces get_doctest exactly rather than approximating it. Pinned by a doctest asserting the three-example case - Make Failed.failures plural, give Passed and Failed counts, and type SkipReason, since a skip may come from :skipif:, an inline flag, :pyversion: or a profile declining - Replace the single precedence ladder with a phase-aware table: process aborts always win; a cleanup failure — including a pytest.skip raised in cleanup — never outranks a body failure - Enter profile runtimes through an ExitStack so partial startup unwinds in reverse - Rename position to source_ordinal and define it as the ordinal among runnable candidates before gating, filtering and group expansion, so inserting prose renames nothing - Call GroupPlan structurally immutable and say the seed is shallow-copied per attempt, matching doctest's namespace semantics - Retire the last PlannedBlock and plan-held-DocTest language, and restate wildcard membership as per-group materialization --- docs/adrs/0001-typed-vanilla-doctest-core.md | 129 +++++++++++++++---- 1 file changed, 105 insertions(+), 24 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 7e5a67e..633e4c5 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -350,12 +350,14 @@ an implementer cannot get it wrong by omission: `super().setup()` so fixtures inject into that same object. Clearing in place rather than rebinding is what keeps `item.globs is run.globs` true for every block, and what stops attempt two of a `--reruns` run from reading attempt - one's mutations. Every block's `DocTest.globs` is assigned this object *after* - construction, because `DocTest.__init__` copies. + one's mutations. Each `RunnableBlock` is materialized against this object — + its `DocTest.globs` assigned *after* construction, because `DocTest.__init__` + copies. 3. **`runtest()`** is overridden. It must not delegate to `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` defaulting to `True` — that would empty the shared mapping after the first - block. It calls `run_group()`, which runs each `PlannedBlock` in phase order + block. It calls `run_group()`, which materializes and runs each + `RunnableBlock` in phase order with `clear_globs=False`, evaluates `:skipif:` against the live mapping, finalizes each paired `want` from its gated `ExpectedOutput`, and wraps the body in a `try`/`finally` so cleanup runs whether or not the body raised. When @@ -419,7 +421,8 @@ class ParsedBlock(t.NamedTuple): source: str # dedented body, verbatim author text path: pathlib.Path # the file the text lives in, not the collected document line: int | None # None when docutils could not recover one - position: int # 0-based document pre-order index; the naming key + source_ordinal: int # ordinal among runnable candidates, BEFORE gating, + # filtering and group expansion; the naming key groups: tuple[str, ...] # declared verbatim; () and ("*",) unresolved here options: t.Mapping[int, bool] # plain int keys, exactly as doctest produces skipif: str | None # UNEVALUATED @@ -432,7 +435,7 @@ class ParsedOutput(t.NamedTuple): text: str path: pathlib.Path line: int | None - position: int + source_ordinal: int groups: tuple[str, ...] options: t.Mapping[int, bool] skipif: str | None # a gated output means its testcode expects nothing @@ -455,14 +458,27 @@ class ExpectedOutput(t.NamedTuple): skipif: str | None # when truthy at run time, `want` becomes "" +class ExampleRecipe(t.NamedTuple): + """Everything needed to rebuild one stock `doctest.Example`.""" + + source: str + want: str + exc_msg: str | None + lineno: int # 0-based, relative to the block's docstring + indent: int + options: t.Mapping[int, bool] + + class ProjectedBlock(t.NamedTuple): """A RECIPE. Holds no `DocTest`, because a `DocTest` is mutable.""" phase: Phase name: str # the minted test name - source: str # normalized executable body + examples: tuple[ExampleRecipe, ...] # a prompt block yields SEVERAL + docstring: str # what pytest's failure renderer slices filename: str - lineno: int | None + lineno: int | None # the block's own line; examples are relative to it + options: t.Mapping[int, bool] # block-level directive :options: profile_name: str # resolved against the frozen registry per attempt skipif: str | None # UNEVALUATED; gated in run_group() expected: ExpectedOutput | None # paired testoutput, itself gateable @@ -470,7 +486,7 @@ class ProjectedBlock(t.NamedTuple): class GroupPlan(t.NamedTuple): group: str - blocks: tuple[ProjectedBlock, ...] # in phase order; genuinely immutable + blocks: tuple[ProjectedBlock, ...] # in phase order; STRUCTURALLY immutable seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt @@ -496,18 +512,31 @@ class RunContext: # --- results: a discriminated union, so invalid states cannot be built ----- +class Counts(t.NamedTuple): + attempted: int + skipped: int # a PASSING block can still carry skipped examples + + +class SkipReason(t.NamedTuple): + kind: t.Literal["skipif", "inline-flag", "pyversion", "profile"] + detail: str # the gate expression, the flag, the specifier + + class Passed(t.NamedTuple): block: ProjectedBlock + counts: Counts class Failed(t.NamedTuple): block: ProjectedBlock - failure: doctest.DocTestFailure | doctest.UnexpectedException + counts: Counts + # PLURAL: continue_on_failure yields several from one block + failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] class Skipped(t.NamedTuple): block: ProjectedBlock - reason: str # the gate expression + reason: SkipReason class Errored(t.NamedTuple): @@ -541,20 +570,70 @@ plan retaining one would not be a recipe, it would be last attempt's state. Unde therefore carries the *ingredients*, and `RunContext` materializes fresh stock `Example` and `DocTest` objects for every attempt. +**The ingredients are per example, not per block.** One prompt block routinely +yields several {class}`doctest.Example` objects, each with its own `source`, +`want`, `exc_msg`, `lineno`, `indent` and `options` — three, for a block whose +last statement raises: + +```{doctest} +>>> import doctest +>>> src = ">>> x = 1\n>>> x + 1\n2\n>>> int('z')\nTraceback (most recent call last):\nValueError: bad\n" +>>> test = doctest.DocTestParser().get_doctest(src, {}, "blk", "p.md", 0) +>>> len(test.examples) +3 +>>> [(e.lineno, e.want.strip()) for e in test.examples] +[(0, ''), (1, '2'), (3, 'Traceback (most recent call last):...')] +``` + +A single `source` and one `lineno` cannot represent that, and `docstring` is +separately required by pytest's known-location failure renderer. Hence +`ExampleRecipe` and `ProjectedBlock.docstring`: the recipe reproduces exactly what +{meth}`doctest.DocTestParser.get_doctest` produced, rather than approximating it. + The same applies to a gated `testoutput`: when its gate is truthy the output is **absent**, not empty. Its text *and* its output-specific options both disappear, which is what Sphinx does, and which a pre-built `want=""` with retained options would get wrong. +`GroupPlan` is **structurally** immutable, not deeply so. Its tuples cannot be +rebound, but `seed` is a `Mapping` whose *values* are arbitrary user objects. Each +attempt shallow-copies it into the live mapping, which is exactly what +`DocTest.__init__` does with `globs` — matching doctest's own namespace semantics +rather than inventing a deeper guarantee the ecosystem does not provide. + **Results are a discriminated union**, not one record with nullable fields, so "passed with an exception attached" is unrepresentable rather than merely unlikely. `Errored` exists because a gate that raises, or a runtime that will not start, is none of pass, fail or skip. -`GroupResult.primary` and `.secondary` make precedence explicit: **a control-flow -exception outranks a body failure, which outranks a cleanup failure**, and the -loser is recorded rather than dropped. Classifying which exceptions are -control-flow is the pytest adapter's job, not the core's. +Three details the first draft got wrong, each checked by execution: + +- **`Failed.failures` is plural.** Under `continue_on_failure` one block reports + several failures; a singular field silently keeps the first. +- **`Passed` carries counts.** A block can pass *and* have skipped examples — + `failed=0 attempted=2 skipped=1` — and a result type without counts loses the + skip entirely, which is the same information ADR 0001's outcome contract + promises to surface. +- **`SkipReason` is typed.** A skip originates from `:skipif:`, an inline + `# doctest: +SKIP`, `:pyversion:`, or a profile declining to run — and "the gate + expression" describes only the first. + +**Exception precedence is phase-aware, not a single ladder.** Grouping +{exc}`KeyboardInterrupt`, a debugger quit, {exc}`pytest.skip` and `xfail` into one +"control-flow" tier is unsafe: a `pytest.skip()` raised *in cleanup* must not +erase a real body failure. + +| Tier | Examples | Rule | +|---|---|---| +| 1. process abort | {exc}`KeyboardInterrupt`, `SystemExit`, `bdb.BdbQuit` | always wins, from any phase | +| 2. pytest outcome from the **body** | `skip`, `xfail`, `exit` | wins over doctest failures | +| 3. doctest failure from the **body** | `DocTestFailure`, `UnexpectedException` | | +| 4. runtime startup failure | a profile that would not start | fails the block as `Errored` | +| 5. cleanup failure, of any kind | including a `pytest.skip()` in cleanup | never outranks 2–4; always recorded in `secondary` | + +Profile runtimes are entered through {class}`contextlib.ExitStack`, so a partial +startup unwinds deterministically in reverse. Classifying which exceptions are +pytest outcomes is the pytest adapter's job; the core knows only the phase. `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, @@ -562,13 +641,13 @@ source=None` from docutils, and an `.. include::`-ed block numbers against the *included* file. Both propagate to `DocTest.lineno=None` and pytest's honest "location unknown", rather than to a fabricated number. -`PlannedBlock` exists because `run_group()` owns phase sequencing, run-time -`:skipif:` evaluation and a cleanup `finally` — and cannot do any of the three -from a bare tuple of `DocTest`s. A `DocTest` carries no phase and no gate, so the -plan has to. Tagging each entry also makes the ordering self-describing rather -than a convention a comment asserts. +`ProjectedBlock` carries phase and gate because `run_group()` owns phase +sequencing, run-time `:skipif:` evaluation and a cleanup `finally` — and cannot do +any of the three from a bare tuple of `DocTest`s. A `DocTest` carries no phase and +no gate, so the recipe has to. Tagging each entry also makes the ordering +self-describing rather than a convention a comment asserts. -**A `PlannedBlock.test` cannot always be fully preconstructed.** Sphinx accepts +**A paired `want` is not known until run time.** Sphinx accepts `:skipif:` on a `testoutput`, and when that output is gated away its `testcode` still runs — expecting *empty* output. So the `want` of a paired block is a function of a gate evaluated at run time, and the plan must carry the paired @@ -576,10 +655,12 @@ output as data (`ExpectedOutput`, itself gated) with the `DocTest` finalized in `run_group()`. Freezing `want` at projection time silently runs the wrong assertion. -**A wildcard block needs a distinct `DocTest` per group it joins.** `DocTest.globs` -is a plain mutable attribute, so one object shared across two `GroupPlan`s would -have the second group's assignment win and both groups would execute against one -mapping. `*` membership therefore materializes a separate `DocTest` per group. +**A wildcard block is materialized separately per group it joins.** Under the +recipe model this costs nothing — the same `ProjectedBlock` appears in each +group's plan and each `RunContext` builds its own `DocTest` from it, with its own +minted name. What would be wrong is *sharing* a materialized `DocTest` across two +groups: `DocTest.globs` is a plain mutable attribute, so the second group's +assignment would win and both would execute against one mapping. **The gate's evaluation namespace is a deliberate divergence.** Sphinx evaluates each `:skipif:` in a fresh context seeded with `doctest_global_setup`; this design From f2bd2b0edc1c9fbefdd5c7e45105d86938ed7f2f Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 19:45:41 -0500 Subject: [PATCH 21/24] docs,notes: Close the registry, compatibility and diagnostics gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: The remaining review items were the contracts named but never written: a registry that ADR 0001 depended on and ADR 0006 did not define, a compatibility promise stated as one phrase covering surfaces of very different strength, and an unknown-directive default that could swallow a body holding doctests. what: - 0006: define the freeze lifecycle — Contributor and Registrar signatures, build_registry, duplicate names as an error unless replace=True, deterministic contribution order, and a freeze point per host - 0006: nested conftests may not contribute, since they load after any freeze that precedes collection; their fixtures and hooks are unaffected - 0006: workers compare a registry manifest rather than node ids, since two workers can collect identical ids while resolving one profile name to different code — which is what makes SSH and socket workers safe rather than assumed homogeneous - 0001: decompose "vanilla-compatible" into a matrix, and narrow the Sphinx promise to consuming a resolved doctree, since no execution lifecycle or result channel is specified - 0001: name the registry as a fifth input collection is not pure over - 0001: correct the layer diagram, which still said the runner always owns the example loop - 0004: split unknown roles from unknown body-owning directives. A role cannot swallow a code block; a container can, so it is a collection error by default and a project registers legitimate containers explicitly - Notes: the execution profile is contributable, not private, matching 0001's public contributor protocol; entry-point discovery is optional behind it rather than out of scope - Notes: narrow the node-id claim to per-block ids over shared MUTABLE state, since ids over isolated state do keep the promise --- docs/adrs/0001-typed-vanilla-doctest-core.md | 53 ++++++++++++++---- docs/adrs/0004-diagnostics-as-data.md | 39 ++++++++----- .../0006-pytest-private-api-compatibility.md | 56 +++++++++++++++++++ notes/analyses/22-extension-seams.md | 4 +- .../23-namespace-scope-and-test-identity.md | 9 +-- 5 files changed, 129 insertions(+), 32 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 633e4c5..5c5c36a 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -262,7 +262,8 @@ markup/ DocumentParser: _rst, _myst -> doctree | project grouping, pairing, phase order, naming -> GroupPlan | -runner owns the per-example loop; phase sequencing +runner stock loop for prompt blocks; owned loop for extended + | profiles only. Phase sequencing, gates, cleanup finally | pytest_doctest_docutils collection, items, globs lifetime, reporting ``` @@ -316,18 +317,18 @@ ordering guarantees exist. Its *implementation* stays private. pytest hooks, Sphinx's `setup(app)`, direct library calls and — optionally — entry points all feed that one surface. -Two hazards to name rather than assume away: +Two hazards decide the lifecycle, and +{doc}`0006-pytest-private-api-compatibility` settles both: -- **Nested `conftest.py` files load during collection**, so "freeze before - collection" is too simple. Either the boundary moves later, or conftest-level - contribution is explicitly excluded. That is an open question in - {doc}`0006-pytest-private-api-compatibility`. +- **Nested `conftest.py` files load during collection**, after any freeze that + precedes it. They therefore may not contribute block kinds, parsers, profiles or + checkers; attempting to is an error naming the file. Their fixtures and ordinary + hooks are unaffected. - **Entry-point discovery is not identical across workers in general.** xdist supports SSH and socket workers and heterogeneous environments, so "same argv, - same plugins" holds only for a homogeneous local run. Either state a - homogeneous-plugin requirement or compare a deterministic registry manifest - across workers. What breaks determinism is *nondeterministic contribution*, not - discovery as such — but heterogeneity makes that distinction load-bearing. + same plugins" holds only for a homogeneous local run. Workers therefore compare + a deterministic registry **manifest**, not node ids — two workers can collect + identical ids while resolving one profile name to different code. ### Item lifecycle @@ -786,6 +787,32 @@ layer that never changes what is constructed. `tests`, and the marker file does not exist, so every consumer sees the package as untyped. This is a packaging defect independent of the rest of this ADR. +### What "vanilla-compatible" promises + +The phrase is worth decomposing, because it covers several different promises of +several different strengths. + +| Surface | Promise | +|---|---| +| `Example` / `DocTest` runtime types | **Exact.** Stock instances, never subclassed for metadata | +| plain-text parsing | **Exact.** The stdlib lane uses `DocTestParser` unmodified | +| option flags and checkers | **Exact.** `register_optionflag` and the stdlib checker contract, with pytest's `ALLOW_UNICODE`/`ALLOW_BYTES`/`NUMBER` reachable | +| prompt-block execution | **Exact.** CPython's own per-example loop, unmodified | +| `DocTestFinder`-shaped Python-object discovery | **Shaped**, as a separate adapter | +| `DocFileSuite` / `DocTestSuite` | **Façade only**, over the stdlib lane. A group plan cannot be expressed through an API that returns one `DocTest` per parser call | +| `{testcode}`, async, groups, phases, Sphinx gates | **Deliberate extension.** No stdlib equivalent to be compatible with | +| pytest collection, fixtures, reporting | pytest's own contracts, composed with rather than replaced | +| Sphinx **node** vocabulary | **Exact.** The `BlockAttributes` stamp is byte-compatible | +| Sphinx **execution** | **Not promised.** See below | + +**The Sphinx promise is narrow, and this record narrows it deliberately.** What is +offered is an *extractor over a Sphinx-resolved doctree* — a pure function from a +doctree to blocks, callable from an extension. A full Sphinx host would need an +execution lifecycle and a result channel that neither this record nor +{doc}`0006-pytest-private-api-compatibility` defines, and inventing one would be +the builder that [](#alternatives-rejected) turns down. Until someone specifies +that adapter, the record promises doctree consumption and nothing more. + ## Constraints The design is pinned by facts about three upstreams. Each was verified at the tag @@ -920,9 +947,11 @@ which blocks will skip. (the-collection-contract)= Collection is **not** a pure function of (bytes, argv, ini), and claiming so -would be wrong on four counts: `.. include::` reads transitive files, docutils +would be wrong on five counts: `.. include::` reads transitive files, docutils directive implementations execute during parsing, the directive registry is -process-global, and MyST plugins change the tree. The defensible contract is +process-global, MyST plugins change the tree, and the frozen registry is itself +an input — assembled from installed plugins and conftests, neither of which is +argv or ini. The defensible contract is determinism over **complete source closure + normalized settings + frozen registry** — all three defined above. Deferring the author's gate removes the largest divergence risk; it does not make xdist divergence structurally diff --git a/docs/adrs/0004-diagnostics-as-data.md b/docs/adrs/0004-diagnostics-as-data.md index aceb8cb..45ed820 100644 --- a/docs/adrs/0004-diagnostics-as-data.md +++ b/docs/adrs/0004-diagnostics-as-data.md @@ -55,10 +55,25 @@ surface. ## Direction -Suppress narrowly, and only for the two classes a bare-docutils parse cannot -judge: unknown roles and unknown directives. "By code" is the intent; the -classifier that assigns a code to a docutils message is unsettled, so this -direction is not yet implementable. +**Treat unknown roles and unknown directives differently.** They are not the same +risk, and the first draft's symmetric treatment was the mistake. + +An **unknown role** is inline markup. It cannot swallow a code block, so a +bare-docutils parse seeing `:mod:` in a Sphinx project is noise and is suppressed +by default. + +An **unknown body-owning directive** is a collection error. It swallows its body +unparsed, so a page whose doctests live inside one collects zero tests and exits +green — which is the failure diagnostics-as-data exists to prevent. Suppressing it +by default trades a loud, correct error for a silent wrong answer. A project with +legitimate foreign containers registers them as known vocabulary; that is an +explicit act, not a default. + +For a Sphinx project the question does not arise: the extractor consumes an +already-resolved doctree, in which every registered directive has run. + +"By code" remains the intent for everything else; the classifier that assigns a +code to a docutils message is unsettled, which is why this record stays `Draft`. Every diagnostic raised by this project's own layers defaults to visible, and `level="error"` from those layers fails collection with the file and line named. @@ -80,15 +95,11 @@ global on/off switch. pre-empting at the source by overriding the directive-dispatch path so the unknown case never becomes a reporter message at all. This is the decision ADR 0004 cannot ship without. -- **What promotes an unknown-directive message back to visible.** The first - draft proposed "when the name is one this project registers", but that is - inverted for the typo case — a misspelled `.. doctset::` is precisely *not* a - registered name — and never fires for a foreign container whose body was - swallowed unparsed. The rule that covers both is **body content**: promote when - the swallowed body matches `DocTestParser._EXAMPLE_RE`. Near-miss-to-a-registered - -name is a useful additional rule for the typo case, where the body check does - not help. -- The body-content rule is reST-only as stated, because myst-parser discards the - fence body. The Markdown equivalent is unsolved. +- **How a project registers a legitimate foreign container**, so that an unknown + body-owning directive it genuinely does not care about stops erroring. This is + the escape hatch the default requires, and it needs a spelling. +- Whether a near-miss to a registered name (`.. doctset::` for `.. doctest::`) + earns a distinct, more helpful message than the generic unknown-directive + error. Cheap, and the typo is the common case. - Whether the CLI (`python -m doctest_docutils`) and the pytest plugin share one formatter or two. diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index 8cae260..e72b56d 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -99,6 +99,62 @@ symbol, and it names the document that triggered it. CI carries a job pinned to the minimum supported pytest and one tracking its prerelease. +### The registry freeze lifecycle + +{doc}`0001-typed-vanilla-doctest-core` depends on a frozen registry and leaves the +lifecycle here. This is it. + +```python +class Contributor(t.Protocol): + """What a plugin implements to extend the core.""" + + def contribute(self, registrar: Registrar) -> None: ... + + +class Registrar(t.Protocol): + def add_block_kind(self, kind: BlockKind, *, replace: bool = False) -> None: ... + def add_document_parser( + self, parser: DocumentParser, *, replace: bool = False + ) -> None: ... + def add_execution_profile( + self, profile: ExecutionProfile, *, replace: bool = False + ) -> None: ... + def add_output_checker( + self, name: str, factory: CheckerFactory, *, replace: bool = False + ) -> None: ... + + +def build_registry(contributors: t.Sequence[Contributor]) -> Registry: ... +``` + +**Duplicate names are an error unless `replace=True`.** Silent last-writer-wins is +what makes the docutils directive table a recurring bug source, and this registry +does not repeat it. Contribution order is: built-ins, then installed plugins in +`pluginmanager` registration order, then explicit contributors — deterministic +under a given plugin set. + +**Freeze points, per host:** + +| Host | Contributors accepted | Frozen at | +|---|---|---| +| direct API | whatever the caller passes to `build_registry` | on return | +| pytest | built-ins, installed plugins, explicit contributors, and **initial/root conftests only** | `pytest_sessionstart` | +| Sphinx | extensions, via `setup(app)` | after extension setup, before reading | + +**Nested conftests may not contribute.** They load during collection, after the +freeze, and a registry that grows while collection runs cannot be the same in +every worker. Registration from a nested conftest is an error naming the file — +not a silent late addition. Fixtures and ordinary pytest hooks in nested conftests +are unaffected; this is only about block kinds, parsers, profiles and checkers. + +**Workers compare a manifest, not node ids.** Matching collection is not +sufficient: two workers can collect identical ids while resolving the same profile +name to different code. So the controller ships a deterministic registry +manifest — sorted `(kind, name, provider, version)` tuples — through worker +configuration, and a worker whose manifest differs fails the session with both +manifests in the message. This is what makes heterogeneous xdist (SSH, socket, +mixed environments) safe rather than assumed-homogeneous. + ## Open - Whether requiring the built-in plugin should be stated as a hard dependency. diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index 57a1bc7..ecf02c1 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -88,10 +88,10 @@ needs it. Applied to the candidates that came up: | `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind`, so the leaf keeps its stdlib-only contract | | Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()` — adding one there requires subclassing two classes | | `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | -| `ExecutionProfile` (private) | **Yes, private.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. Stays private until an out-of-tree caller exists | +| `ExecutionProfile` | **Yes, and contributable.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. The *type* is part of the public contributor protocol, since a pluggable core that cannot accept a new execution policy is not pluggable; the registry holding them stays private | | A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | | A registry *object* with builders, digests and manifests | **No object, but there is a freeze.** The contributed block kinds, front ends and profiles are resolved into an immutable mapping alongside the frozen `Settings` facets, before collection begins; registering after that is an error. That is what ADR 0001's collection contract means by *frozen registry* — not a class with its own lifecycle | -| Entry-point plugin discovery | **Not in v1**, for want of a caller outside the package — *not* because it breaks xdist. Workers are fresh processes that re-run plugin loading with the same argv, so entry-point resolution is identical everywhere. What breaks determinism is nondeterministic *contribution*, such as a conftest registering conditionally on the environment | +| Entry-point plugin discovery | **Optional, behind the contributor protocol.** Not the hazard it was once called: what breaks determinism is nondeterministic *contribution*, not discovery. But "identical across workers" holds only for a homogeneous local run — xdist supports SSH and socket workers — so a heterogeneous session needs a registry manifest comparison or an explicit homogeneous-plugin requirement | The distinction those two rows turn on is worth stating once: **discovery is not the hazard, nondeterminism is.** A registry populated identically in every process diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index e457d20..0167fb6 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -106,10 +106,11 @@ Two corollaries: ## What a node id does *not* promise Worth stating plainly, because it is the honest limit of the recommended design: -**no surveyed implementation makes a node id a promise of independent -runnability.** Selecting block two of a stateful page raises `NameError` under -Sybil, under `per-block`, and under any scheme that hands out per-block ids over -shared state. +**a per-block node id over shared mutable state cannot truthfully promise +independent execution.** Selecting block two of a stateful page raises +`NameError` under Sybil, under PR #87's `per-block`, and under any scheme of that +shape. Ids over *isolated* state — released `doctest_docutils`, or a page whose +blocks declare no group — promise independence and keep the promise. The choice is therefore not between "selectable blocks" and "unselectable blocks". It is between an id that *claims* to be selectable and is not, and an id whose From cb780d33bc926e2f74f982a6bae59793685bac34 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 20:55:24 -0500 Subject: [PATCH 22/24] doctest_core(docs[model]): Complete contracts why: The proposed core mixed settings with discovered capabilities and left source ordering, host outcomes, checker rendering, and result projection underspecified. what: - Separate contracts, settings, models, and registry snapshots - Define stable source identity and complete gate/result records - Specify pytest failure projection and phase-aware exceptions --- docs/adrs/0001-typed-vanilla-doctest-core.md | 294 +++++++++++-------- 1 file changed, 177 insertions(+), 117 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 5c5c36a..1925caa 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -112,8 +112,9 @@ why `SphinxDocTestRunner` overrides a private stdlib method to swallow the resulting `IndexError`. Mapping the group onto one {class}`pytest.Item` while each block keeps its own identity is the contribution. -What that buys, for free and with no override of `repr_failure` or `reportinfo`: -per-block failure locations, per-block gutters, and per-block "location unknown". +With the default checker that buys, for free and with no override of +`repr_failure` or `reportinfo`: per-block failure locations, per-block gutters, +and per-block "location unknown". Meanwhile `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode are structurally incapable of splitting the shared state, because there is only one item to schedule. @@ -152,7 +153,8 @@ With one `DocTest` per block, every failure therefore carries its own `filename` and `lineno` for free. A block reached through `.. include::` reports the *included* file. A block docutils could not locate carries `lineno=None` and takes pytest's honest `EXAMPLE LOCATION UNKNOWN` branch **without poisoning its -siblings**. No override of `repr_failure` or `reportinfo` is required. +siblings**. The default-checker path requires no override of `repr_failure` or +`reportinfo`. This is what makes merging unnecessary: the synthetic page, its blank-line padding and its clamp exist only to reconstruct locations from a single spliced @@ -199,9 +201,9 @@ under any `--dist` mode. No affinity primitive, no scheduler substitution, no `parse_tx_spec_config` fork, no node-id string sniffing. The identical-collection requirement is untouched by this and still binds at any -granularity. It is satisfied separately, by collection being a pure function of -(bytes, argv, ini) — which is why `:skipif:` is carried through collection -unevaluated. +granularity. It is approached separately through deterministic projection over +the complete source closure, normalized settings and a frozen registry — which +is why `:skipif:` is carried through collection unevaluated. (the-outcome-contract)= @@ -212,7 +214,7 @@ stated here rather than discovered later. | Signal | Granularity | Notes | |---|---|---| -| Failure location, `want`/`got`, gutter | **per block** | `repr_failure` iterates failures and reads each one's own `DocTest` | +| Failure location, `want`/`got`, gutter | **per block** | default `repr_failure` iterates failures and reads each one's own `DocTest`; custom checkers use the same locations in the narrow rendering branch | | `EXAMPLE LOCATION UNKNOWN` | **per block** | a block with `lineno=None` does not affect its siblings | | `passed` / `failed` / `skipped` | **per item** | `TestReport.outcome` is one scalar | | JUnit `` | **per item** | node reporters are keyed by node id | @@ -225,9 +227,10 @@ block's skip, a group with one all-`SKIP` block and one passing block reports though a sibling passed. pytest's own doctest plugin takes the second horn only when *every* example is skipped, via `_check_all_skipped`. -This design takes the same position: **skip the item when every runnable block is -skipped; otherwise report partial skips as typed block detail, not as a pytest -outcome.** No extra reports are synthesized. +This design takes the same position over the test phase: **skip the item when +every `Phase.TEST` block is skipped; otherwise report partial skips as typed block +detail, not as a pytest outcome.** Setup and cleanup are infrastructure and do +not contribute a passed or skipped test. No extra reports are synthesized. "Typed block detail" needs a channel, or implementers will re-invent skip lifting or write to stderr. The channel is a `GroupResult` — one `BlockResult` per block, @@ -249,32 +252,29 @@ surface. Revisit if it stabilizes. ### Layers -Dependencies flow one way, from the leaf toward the hosts. No layer may import a -layer above it. +Dependencies flow from the hosts toward small foundations. No foundational +layer imports a host, and configuration never owns discovered capabilities. ```text -settings one frozen Settings, resolved once <- leaf; everything reads it - | -blocks inert data: Block, BlockKind, Phase, Diagnostic, Example - | (stdlib imports only) -markup/ DocumentParser: _rst, _myst -> doctree - | extract_blocks(doctree) -> (parsed blocks, diagnostics) - | -project grouping, pairing, phase order, naming -> GroupPlan - | -runner stock loop for prompt blocks; owned loop for extended - | profiles only. Phase sequencing, gates, cleanup finally - | -pytest_doctest_docutils collection, items, globs lifetime, reporting +contracts settings model + \ | / + +--------- registry --------+ + | + markup + | + project + | + runner + | + direct / pytest / Sphinx hosts ``` -`settings` sits at the bottom, not beside the host, because `Frontend.parse` and -`project()` both take a `Settings`. A layer every other layer reads is a leaf. - | Layer | Owns | Must not know | |---|---|---| -| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings` — plus the **frozen registry**, resolved together exactly once per session. `None` sentinels at the resolve boundary make a future default change announceable | pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | -| `blocks` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan`, `RunContext`, `ExecutionProfile`, the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | +| `contracts` | Public protocols and immutable contribution records: `DocumentParser`, `ExecutionProfile`, `ExecutionRuntime`, `CheckerFactory`, `Contributor` and `Registrar` | Sphinx, pytest, xdist and host lifecycle objects. Only stdlib and public parser types cross this boundary | +| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings`. `None` sentinels at the resolve boundary make a future default change announceable | registries, pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | +| `model` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan`, `RunContext` and the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only | +| `registry` | A private mutable builder and the public immutable `RegistrySnapshot` consumed by every later layer | host lifecycle objects after the snapshot is frozen | | `markup/` | Text → `(blocks, diagnostics)`. **The whole docutils vocabulary**: which node classes each kind may arrive as, the `BlockAttributes` stamp, line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration, per-kind `option_spec` | Groups as a runtime concept, `DocTest`, pytest, pairing | | `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | | `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing, run-time `:skipif:`, the profile's context manager, and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | @@ -292,7 +292,7 @@ document is parsed. Three scopes: | Scope | Owns | Resolved | |---|---|---| -| `SessionSettings` | defaults plus normalized host configuration; the frozen registry | once, at session start | +| `SessionSettings` | defaults plus normalized host configuration | once, before collection or direct execution | | `DocumentSettings` | the front-matter overlay a page is permitted to set | per document, after parsing | | block / example policy | directive options, then inline `# doctest:` flags | per block, at projection and run | @@ -306,29 +306,28 @@ to the host adapter. And wildcard resolution and name minting are *invariants*, not user-configurable knobs — exposing them would let a project produce node ids no other project can read. -The **registry** is resolved and frozen alongside `SessionSettings`. "Frozen" -means an immutable mapping built once; registering afterwards is an error. - -**Contribution is public; the registry is not.** The stated goal is an -extendable, pluggable core, so a small host-neutral contributor protocol ships in -v1 — what may be contributed (front ends, block kinds, execution profiles, -checkers), when registration closes, how duplicate names are resolved, and what -ordering guarantees exist. Its *implementation* stays private. pytest hooks, -Sphinx's `setup(app)`, direct library calls and — optionally — entry points all -feed that one surface. - -Two hazards decide the lifecycle, and -{doc}`0006-pytest-private-api-compatibility` settles both: - -- **Nested `conftest.py` files load during collection**, after any freeze that - precedes it. They therefore may not contribute block kinds, parsers, profiles or - checkers; attempting to is an error naming the file. Their fixtures and ordinary - hooks are unaffected. -- **Entry-point discovery is not identical across workers in general.** xdist - supports SSH and socket workers and heterogeneous environments, so "same argv, - same plugins" holds only for a homogeneous local run. Workers therefore compare - a deterministic registry **manifest**, not node ids — two workers can collect - identical ids while resolving one profile name to different code. +`ProjectionSettings.ungrouped` defaults to `"default"`. An unlabelled runnable +block therefore joins the page's `default` group unless the user explicitly asks +for block isolation. This clean-slate default follows Sphinx's author vocabulary +and makes state-sharing opt-out rather than a project-specific surprise. + +The **registry** is a separate input resolved after `SessionSettings`. "Frozen" +means the public `RegistrySnapshot` contains immutable mappings and records; the +mutable builder is private and discarded. Registering after the host freezes its +snapshot is an error. Keeping these values separate matters under xdist: settings +are normalized user input, while the snapshot is the capability set discovered in +that process. + +**Contribution and the snapshot are public; mutation is not.** The stated goal is +an extendable, pluggable core, so a small host-neutral contributor protocol ships +in v1. Front ends, block kinds, execution profiles and checkers all feed one +builder and every consumer receives the same `RegistrySnapshot`. Host-specific +registration timing is a separate decision; the core contract does not import a +pytest hook or a Sphinx application. + +The direct, pytest and Sphinx lifecycles, including the xdist consistency check, +are specified separately so this record does not mistake a host bootstrap policy +for a core dependency. ### Item lifecycle @@ -358,8 +357,8 @@ an implementer cannot get it wrong by omission: `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` defaulting to `True` — that would empty the shared mapping after the first block. It calls `run_group()`, which materializes and runs each - `RunnableBlock` in phase order - with `clear_globs=False`, evaluates `:skipif:` against the live mapping, + `RunnableBlock` in phase order with `clear_globs=False`, evaluates `:skipif:` + and `:pyversion:` against the live mapping and interpreter, finalizes each paired `want` from its gated `ExpectedOutput`, and wraps the body in a `try`/`finally` so cleanup runs whether or not the body raised. When cleanup *also* fails, the body's failure is the one raised; cleanup's is @@ -368,20 +367,29 @@ an implementer cannot get it wrong by omission: The `OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` conversion and `continue_on_failure` are reimplemented here, because `PytestDoctestRunner` is nested inside a factory and cannot be imported. -4. **Outcome** follows [](#the-outcome-contract): skip the item only when every - runnable block is skipped. +4. **Outcome** follows [](#the-outcome-contract): only `Phase.TEST` blocks + determine pass versus skip. A plan with no test block yields no item. Setup and + cleanup are infrastructure: an error there may fail or abort the item, but a + successful setup is not a passed test and a cleanup skip cannot erase the test + result. Skip the item only when every test block is skipped. `runtest()` must also keep `_disable_output_capturing_for_darwin()`, which the inherited implementation calls before running and which has nothing to do with grouping. -5. **`repr_failure`** is inherited unchanged for *failures* — it already reads - each failure's own `DocTest`. It does **not** render `GroupResult`, and - claiming it does would conflate two channels. Partial-skip detail goes to a - report section and the terminal summary; see [](#the-outcome-contract). +5. **Failure projection** flattens every `Failed.failures` tuple in block order + and raises pytest's `MultipleDoctestFailures`. With the default pytest checker, + `repr_failure` is inherited unchanged and therefore preserves pytest's exact + output. A contributed checker must supply both `check_output` and + `output_difference`; the item takes one small custom-rendering branch so the + checker that decided the failure also explains it. `repr_failure` does not + render `GroupResult`. Partial-skip detail goes to a report section and the + terminal summary; see [](#the-outcome-contract). 6. **Reporting across processes.** The controller never sees the item — it receives serialized `TestReport` dictionaries. So `pytest_runtest_makereport` - copies a **versioned, JSON-safe** block summary onto the report; pytest + copies a **versioned, JSON-safe** block summary onto the report. The summary + contains only the schema version, group name, block name, phase, outcome, + counts and structured skip reason — never exceptions or live objects. pytest serializes arbitrary report attributes and xdist reconstructs them controller-side. The rich `GroupResult`, and every exception in it, stays worker-local. @@ -414,6 +422,37 @@ class Phase(enum.IntEnum): CLEANUP = 2 +# --- contracts: public, host-neutral extension seams ---------------------- + + +Failure: t.TypeAlias = doctest.DocTestFailure | doctest.UnexpectedException + + +class RuntimeOutcome(t.NamedTuple): + results: doctest.TestResults + failures: tuple[Failure, ...] + + +class RuntimeSettings(t.NamedTuple): + optionflags: int + continue_on_failure: bool + checker: doctest.OutputChecker + + +class CheckerFactory(t.Protocol): + def __call__(self) -> doctest.OutputChecker: ... + + +class ExecutionRuntime(t.Protocol): + def run(self, test: doctest.DocTest) -> RuntimeOutcome: ... + + +class ExecutionProfile(t.Protocol): + def open( + self, settings: RuntimeSettings + ) -> contextlib.AbstractContextManager[ExecutionRuntime]: ... + + # --- parsed: inert, produced by extraction, owns no semantics ------------- @@ -422,11 +461,12 @@ class ParsedBlock(t.NamedTuple): source: str # dedented body, verbatim author text path: pathlib.Path # the file the text lives in, not the collected document line: int | None # None when docutils could not recover one - source_ordinal: int # ordinal among runnable candidates, BEFORE gating, - # filtering and group expansion; the naming key + document_order: int # position among blocks AND outputs; the pairing key + block_ordinal: int # position among runnable blocks; the identity key groups: tuple[str, ...] # declared verbatim; () and ("*",) unresolved here options: t.Mapping[int, bool] # plain int keys, exactly as doctest produces skipif: str | None # UNEVALUATED + pyversion: str | None # UNEVALUATED PEP 440 specifier hidden: bool @@ -436,10 +476,11 @@ class ParsedOutput(t.NamedTuple): text: str path: pathlib.Path line: int | None - source_ordinal: int + document_order: int # shares one sequence with ParsedBlock for pairing groups: tuple[str, ...] options: t.Mapping[int, bool] skipif: str | None # a gated output means its testcode expects nothing + pyversion: str | None class BlockKind(t.NamedTuple): @@ -457,6 +498,7 @@ class ExpectedOutput(t.NamedTuple): text: str options: t.Mapping[int, bool] skipif: str | None # when truthy at run time, `want` becomes "" + pyversion: str | None # when disallowed at run time, `want` becomes "" class ExampleRecipe(t.NamedTuple): @@ -475,6 +517,7 @@ class ProjectedBlock(t.NamedTuple): phase: Phase name: str # the minted test name + block_ordinal: int # stable among runnable blocks before filtering examples: tuple[ExampleRecipe, ...] # a prompt block yields SEVERAL docstring: str # what pytest's failure renderer slices filename: str @@ -482,6 +525,7 @@ class ProjectedBlock(t.NamedTuple): options: t.Mapping[int, bool] # block-level directive :options: profile_name: str # resolved against the frozen registry per attempt skipif: str | None # UNEVALUATED; gated in run_group() + pyversion: str | None # UNEVALUATED; gated in run_group() expected: ExpectedOutput | None # paired testoutput, itself gateable @@ -532,11 +576,12 @@ class Failed(t.NamedTuple): block: ProjectedBlock counts: Counts # PLURAL: continue_on_failure yields several from one block - failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] + failures: tuple[Failure, ...] class Skipped(t.NamedTuple): block: ProjectedBlock + counts: Counts reason: SkipReason @@ -561,6 +606,14 @@ block: for a prompt-form block it is *inside* `source` and `testcode`/`testoutput` pair it is a separate `ParsedOutput`. Conflating the two was what made "projection owns pairing" untrue. +`document_order` is one monotonic sequence shared by runnable blocks and +`ParsedOutput` records. Pairing therefore follows the source stream even when an +output sits between two runnable candidates. `block_ordinal` counts runnable +blocks only and survives gating, filtering and wildcard expansion, so adding or +removing expected output cannot rename every later test. Both `:skipif:` and +`:pyversion:` remain data until the run boundary; collection never evaluates +either gate. + `BlockKind` names a profile rather than holding one, so a public type never contains a private one. The name resolves against the frozen registry. @@ -607,7 +660,7 @@ rather than inventing a deeper guarantee the ecosystem does not provide. unlikely. `Errored` exists because a gate that raises, or a runtime that will not start, is none of pass, fail or skip. -Three details the first draft got wrong, each checked by execution: +Four result details are load-bearing: - **`Failed.failures` is plural.** Under `continue_on_failure` one block reports several failures; a singular field silently keeps the first. @@ -615,28 +668,31 @@ Three details the first draft got wrong, each checked by execution: `failed=0 attempted=2 skipped=1` — and a result type without counts loses the skip entirely, which is the same information ADR 0001's outcome contract promises to surface. +- **`Skipped` carries counts too.** A whole-block gate attempts zero examples, + while an all-`SKIP` doctest has parsed examples and reports them skipped. The + reason alone cannot distinguish those cases. - **`SkipReason` is typed.** A skip originates from `:skipif:`, an inline `# doctest: +SKIP`, `:pyversion:`, or a profile declining to run — and "the gate expression" describes only the first. **Exception precedence is phase-aware, not a single ladder.** Grouping -{exc}`KeyboardInterrupt`, a debugger quit, {exc}`pytest.skip` and `xfail` into one -"control-flow" tier is unsafe: a `pytest.skip()` raised *in cleanup* must not -erase a real body failure. +{exc}`KeyboardInterrupt`, a debugger quit, {exc}`pytest.skip`, `xfail` and +`pytest.exit` into one "control-flow" tier is unsafe. A cleanup skip must not erase +a real test failure, while a session exit must never be converted into block data. -| Tier | Examples | Rule | +| Class | Examples | Rule | |---|---|---| -| 1. process abort | {exc}`KeyboardInterrupt`, `SystemExit`, `bdb.BdbQuit` | always wins, from any phase | -| 2. pytest outcome from the **body** | `skip`, `xfail`, `exit` | wins over doctest failures | -| 3. doctest failure from the **body** | `DocTestFailure`, `UnexpectedException` | | -| 4. runtime startup failure | a profile that would not start | fails the block as `Errored` | -| 5. cleanup failure, of any kind | including a `pytest.skip()` in cleanup | never outranks 2–4; always recorded in `secondary` | +| process, debugger or session abort | {exc}`KeyboardInterrupt`, `SystemExit`, `bdb.BdbQuit`, `pytest.exit` | always propagates from every phase; cleanup still runs, but cannot replace it | +| host outcome from setup or test | `pytest.skip`, `pytest.xfail` | propagates as the host outcome after cleanup | +| doctest mismatch from test | `DocTestFailure`, `UnexpectedException` | retained in source order and projected to the host after cleanup | +| runtime error from setup or test | a gate that raises, or a profile that cannot start | recorded as `Errored`; becomes the primary failure when no abort or host outcome exists | +| cleanup exception or cleanup host outcome | any exception, including `pytest.skip` or `pytest.xfail` | recorded as `secondary` when a primary exists; otherwise becomes the item failure, never a skip or xfail | Profile runtimes are entered through {class}`contextlib.ExitStack`, so a partial startup unwinds deterministically in reverse. Classifying which exceptions are pytest outcomes is the pytest adapter's job; the core knows only the phase. -`Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block +`ParsedBlock.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, source=None` from docutils, and an `.. include::`-ed block numbers against the *included* file. Both propagate to `DocTest.lineno=None` and pytest's honest @@ -656,12 +712,12 @@ output as data (`ExpectedOutput`, itself gated) with the `DocTest` finalized in `run_group()`. Freezing `want` at projection time silently runs the wrong assertion. -**A wildcard block is materialized separately per group it joins.** Under the -recipe model this costs nothing — the same `ProjectedBlock` appears in each -group's plan and each `RunContext` builds its own `DocTest` from it, with its own -minted name. What would be wrong is *sharing* a materialized `DocTest` across two -groups: `DocTest.globs` is a plain mutable attribute, so the second group's -assignment would win and both would execute against one mapping. +**A wildcard block is projected separately per group it joins.** Projection +clones the recipe and mints a group-qualified name for each destination. Each +`RunContext` then builds its own `DocTest` from its own recipe. Reusing one +`ProjectedBlock` would make its name ambiguous; sharing one materialized +`DocTest` would be worse, because `DocTest.globs` is mutable and the second +group's assignment would win. **The gate's evaluation namespace is a deliberate divergence.** Sphinx evaluates each `:skipif:` in a fresh context seeded with `doctest_global_setup`; this design @@ -691,6 +747,14 @@ doctests are then compatible **by construction** rather than by differential testing, and {doc}`0002-runner-conformance-across-cpython`'s harness shrinks to guarding the extended lane. +**A checker owns both comparison and explanation.** The default pytest +registration constructs pytest's checker, preserving `ALLOW_UNICODE`, +`ALLOW_BYTES`, `NUMBER` and its inherited failure representation exactly. A +contributed `CheckerFactory` constructs a fresh checker for each runtime. The +same instance performs `check_output()` and `output_difference()`; using pytest's +private `_get_checker()` only at rendering time would let one checker reject the +example and another explain why. + **Which docutils node classes a kind may arrive as is a front-end concern, not a `BlockKind` field.** `testsetup`, `testcleanup` and any `:hide:` block are emitted as {class}`docutils.nodes.comment`, not `literal_block` @@ -706,7 +770,7 @@ The runtime objects are stdlib's, unconditionally. Precision lives in a parallel layer that never changes what is constructed. - **Parsing and extraction are two seams, not one.** A single - `Frontend.parse(text, path)` cannot serve Sphinx, because a Sphinx extension + `DocumentParser.parse(text, path)` cannot serve Sphinx, because a Sphinx extension already *has* a doctree and a raw re-parse is not the same tree. So: ```python @@ -759,16 +823,15 @@ layer that never changes what is constructed. right shape for it. It is a {class}`doctest.DocTestFinder`-shaped adapter, and putting a `_python` module in `markup/` was a category error. -- **`Protocol` for the seams, nominal subclassing for type-checkability.** - `DocumentParser` is a `Protocol` so a third party can supply one structurally. The - shipped parser *also* subclasses {class}`doctest.DocTestParser` — not because - the interpreter requires it (stdlib `doctest` performs no `isinstance` check on - `parser` or `test_finder`; a duck-typed object works at runtime) but because - typeshed's signatures name the class, so a checker rejects what the interpreter - accepts. The subclass buys type-checkability, not passability. Passability is a - matter of matching the call signature: a finder whose `find()` takes a string - first cannot be handed to `DocTestSuite`, which passes a module — and - subclassing does not fix that. +- **`Protocol` for markup seams; nominal classes only for stdlib façades.** + `DocumentParser` is a `Protocol` so a third party can supply one structurally. + It must not subclass {class}`doctest.DocTestParser`: their `parse()` signatures + and return types are incompatible, so the apparent typeshed accommodation is + itself an invalid override. The optional `DocFileSuite` façade instead owns a + separate nominal `DocTestParser` adapter with the exact stdlib signature. + Runtime passability still comes from matching the called method: a finder whose + `find()` takes a string first cannot be handed to `DocTestSuite`, which passes a + module, and subclassing does not fix that. One genuine nominal edge does exist: `DocTestSuite` sorts its results, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom @@ -892,17 +955,13 @@ Each is a genuine conflict where satisfying one goal costs another. "Both" is no an answer; the position taken and its price are recorded. **A vanilla `DocTest` cannot carry the front-end's metadata.** It has exactly -`(examples, globs, name, filename, lineno, docstring)`. *Position:* metadata the -runner needs at run time rides on a {class}`doctest.Example` subclass — -`doctest.Example` has no `__slots__`, so attributes survive {func}`copy.copy`, -{mod}`pickle`, and a third party's naive -`DocTest(examples, globs, name, filename, lineno, docstring)` rebuild, because -that rebuild reuses the same `Example` objects. Metadata the runner does *not* -need — groups, wildcards, pairing — never touches a stdlib object and dies in the -projection layer. - -*Price:* **none of it goes on {class}`doctest.Example`.** An earlier draft put -compile mode on an `Example` subclass, which fails twice. +`(examples, globs, name, filename, lineno, docstring)`. *Position:* all extension +metadata stays on `ProjectedBlock`, `RunContext` and the result records. Stock +`DocTest` and `Example` objects remain exact compatibility objects, not metadata +carriers. *Price:* a consumer holding only the stdlib object sees only stdlib +semantics; it must retain the core recipe to inspect groups, profiles or gates. + +Putting metadata on an `Example` subclass is rejected because {meth}`doctest.Example.__eq__` gates on exact type identity ([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)), so a bare subclass is unequal to a stock `Example` with identical fields in both @@ -931,7 +990,8 @@ stock, and nothing in the compatibility kernel is subclassed for metadata at all `DocTest`s under one node id. *Price:* selecting a group runs all its blocks; there is no id that names block three alone. That is honest: no surveyed implementation makes a node id a promise of independent runnability, and the -current `per-block` mode ships ids that raise `NameError` when selected. +proposed shared per-block shape would expose ids that raise `NameError` when +selected without their predecessors. **Sphinx's skip versus pytest's skip.** *Position:* pytest's meaning, doctest's mechanism — set `SKIP`, never drop the node. *Price:* a page carrying a gated @@ -1051,8 +1111,8 @@ the boundaries the change was supposed to clean. mode in the same user-writable namespace as `ELLIPSIS`. A `doctest_optionflags = EXEC` in any ini, or a stray `# doctest: +EXEC`, compiles in exec mode, which suppresses expression echo so every `want` compares against empty output and the -suite passes vacuously. Compile mode is an attribute on the `Example` subclass, -unreachable from user configuration. +suite passes vacuously. Execution policy is selected by +`ProjectedBlock.profile_name`, outside the user-writable optionflag namespace. **Sybil's non-overlap-raises invariant.** Two `.. include::` directives naming the same file produce blocks over identical source spans. That is a legitimate @@ -1083,8 +1143,8 @@ conformance test in CI. ### Positive -- Failure locations are correct by construction, including through `.. include::`, - with no `repr_failure` or `reportinfo` override. +- Failure locations are correct by construction, including through `.. include::`; + the default-checker path needs no `repr_failure` or `reportinfo` override. - A block docutils cannot locate degrades to an honest disclaimer instead of a fabricated line, and does not affect its siblings. - Every `--dist` mode works, because there is no shared state to split. @@ -1108,7 +1168,6 @@ conformance test in CI. runner, and a test asserts it. - The plugin now *requires* the built-in doctest plugin rather than blocking it, so `-p no:doctest` is an error rather than a degraded mode. -- Retiring `namespace_items = per-block` is a real feature removal. - The line count does not fall. ### Risks @@ -1137,8 +1196,9 @@ the narrow default set in {doc}`0004-diagnostics-as-data`. This ADR fixes the architecture. Five decisions it defers get their own records: {doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven -equivalent), {doc}`0003-rejecting-per-block-items` (the deprecation -path), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed), +equivalent), {doc}`0003-rejecting-per-block-items` (why shared per-block items +are rejected), {doc}`0004-diagnostics-as-data` (what is reported and what is +suppressed), {doc}`0005-line-recovery-for-nested-blocks` (the optional last step), and {doc}`0006-pytest-private-api-compatibility` (the quarantine and its matrix). @@ -1148,9 +1208,9 @@ The core produces real {class}`doctest.DocTest` objects holding real {class}`doctest.Example` objects. `Example.source` is the stdlib-normalized executable body — prompts and indentation stripped, trailing newline added, the stripped column recorded in `Example.indent` — not a synthesized wrapper and not -the author's verbatim text, which is what `Block.source` holds. Everything else — -groups, phases, pairing, diagnostics, distribution — is a layer above that fact, -and no layer reaches around another. +the author's verbatim text, which is what `ParsedBlock.source` holds. Everything +else — groups, phases, pairing, diagnostics, distribution — is a layer above that +fact, and no layer reaches around another. The unit that shares a `globs` mapping is the unit pytest schedules. That is the one invariant every other property in this document follows from, and it is not From 2dd1260902a7ef8b37e138b41e04599094fe5499 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 20:59:58 -0500 Subject: [PATCH 23/24] doctest_core(docs[plugins]): Define lifecycle why: Registry construction is a host-neutral extension contract, not a pytest private-API detail, and each supported host freezes contributors at a different point. what: - Add typed contributor, registrar, and snapshot contracts - Define direct, pytest, Sphinx, and xdist lifecycles - Keep mutable construction private and late registration explicit --- docs/adrs/0001-typed-vanilla-doctest-core.md | 28 +-- .../0006-pytest-private-api-compatibility.md | 58 +----- ...0007-host-plugin-registration-lifecycle.md | 194 ++++++++++++++++++ docs/adrs/index.md | 1 + 4 files changed, 213 insertions(+), 68 deletions(-) create mode 100644 docs/adrs/0007-host-plugin-registration-lifecycle.md diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 1925caa..a2b9de1 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -326,8 +326,8 @@ registration timing is a separate decision; the core contract does not import a pytest hook or a Sphinx application. The direct, pytest and Sphinx lifecycles, including the xdist consistency check, -are specified separately so this record does not mistake a host bootstrap policy -for a core dependency. +are specified in {doc}`0007-host-plugin-registration-lifecycle` so this record +does not mistake a host bootstrap policy for a core dependency. ### Item lifecycle @@ -484,7 +484,6 @@ class ParsedOutput(t.NamedTuple): class BlockKind(t.NamedTuple): - name: str phase: Phase profile_name: str # resolved against the frozen registry, not held here pairs_with: str | None @@ -615,7 +614,8 @@ removing expected output cannot rename every later test. Both `:skipif:` and either gate. `BlockKind` names a profile rather than holding one, so a public type never -contains a private one. The name resolves against the frozen registry. +contains a private implementation. The profile name and the block kind's own +registration name resolve against the frozen registry. **A plan holds no `DocTest`.** {class}`doctest.DocTest` is mutable — the design assigns `globs` to it after construction, and a run mutates that mapping — so a @@ -870,11 +870,11 @@ several different strengths. **The Sphinx promise is narrow, and this record narrows it deliberately.** What is offered is an *extractor over a Sphinx-resolved doctree* — a pure function from a -doctree to blocks, callable from an extension. A full Sphinx host would need an -execution lifecycle and a result channel that neither this record nor -{doc}`0006-pytest-private-api-compatibility` defines, and inventing one would be -the builder that [](#alternatives-rejected) turns down. Until someone specifies -that adapter, the record promises doctree consumption and nothing more. +doctree to blocks, callable from an extension. +{doc}`0007-host-plugin-registration-lifecycle` defines how Sphinx extensions +contribute capabilities, but not a Sphinx execution lifecycle or result channel. +Inventing those would be the builder that [](#alternatives-rejected) turns down. +The record promises doctree consumption and nothing more. ## Constraints @@ -1194,13 +1194,15 @@ the narrow default set in {doc}`0004-diagnostics-as-data`. ## Relationship to other ADRs -This ADR fixes the architecture. Five decisions it defers get their own records: +This ADR fixes the architecture. Six decisions it defers get their own records: {doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven equivalent), {doc}`0003-rejecting-per-block-items` (why shared per-block items are rejected), {doc}`0004-diagnostics-as-data` (what is reported and what is -suppressed), -{doc}`0005-line-recovery-for-nested-blocks` (the optional last step), and -{doc}`0006-pytest-private-api-compatibility` (the quarantine and its matrix). +suppressed), {doc}`0005-line-recovery-for-nested-blocks` (the optional last +step), {doc}`0006-pytest-private-api-compatibility` (the quarantine and its +matrix), and +{doc}`0007-host-plugin-registration-lifecycle` (host registration and freeze +points). ## Final position diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index e72b56d..175bba4 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -99,61 +99,9 @@ symbol, and it names the document that triggered it. CI carries a job pinned to the minimum supported pytest and one tracking its prerelease. -### The registry freeze lifecycle - -{doc}`0001-typed-vanilla-doctest-core` depends on a frozen registry and leaves the -lifecycle here. This is it. - -```python -class Contributor(t.Protocol): - """What a plugin implements to extend the core.""" - - def contribute(self, registrar: Registrar) -> None: ... - - -class Registrar(t.Protocol): - def add_block_kind(self, kind: BlockKind, *, replace: bool = False) -> None: ... - def add_document_parser( - self, parser: DocumentParser, *, replace: bool = False - ) -> None: ... - def add_execution_profile( - self, profile: ExecutionProfile, *, replace: bool = False - ) -> None: ... - def add_output_checker( - self, name: str, factory: CheckerFactory, *, replace: bool = False - ) -> None: ... - - -def build_registry(contributors: t.Sequence[Contributor]) -> Registry: ... -``` - -**Duplicate names are an error unless `replace=True`.** Silent last-writer-wins is -what makes the docutils directive table a recurring bug source, and this registry -does not repeat it. Contribution order is: built-ins, then installed plugins in -`pluginmanager` registration order, then explicit contributors — deterministic -under a given plugin set. - -**Freeze points, per host:** - -| Host | Contributors accepted | Frozen at | -|---|---|---| -| direct API | whatever the caller passes to `build_registry` | on return | -| pytest | built-ins, installed plugins, explicit contributors, and **initial/root conftests only** | `pytest_sessionstart` | -| Sphinx | extensions, via `setup(app)` | after extension setup, before reading | - -**Nested conftests may not contribute.** They load during collection, after the -freeze, and a registry that grows while collection runs cannot be the same in -every worker. Registration from a nested conftest is an error naming the file — -not a silent late addition. Fixtures and ordinary pytest hooks in nested conftests -are unaffected; this is only about block kinds, parsers, profiles and checkers. - -**Workers compare a manifest, not node ids.** Matching collection is not -sufficient: two workers can collect identical ids while resolving the same profile -name to different code. So the controller ships a deterministic registry -manifest — sorted `(kind, name, provider, version)` tuples — through worker -configuration, and a worker whose manifest differs fails the session with both -manifests in the message. This is what makes heterogeneous xdist (SSH, socket, -mixed environments) safe rather than assumed-homogeneous. +Registry construction is not a pytest-private-API concern. The host-neutral +contract, pytest hookspec, Sphinx adapter and xdist manifest are specified in +{doc}`0007-host-plugin-registration-lifecycle`. ## Open diff --git a/docs/adrs/0007-host-plugin-registration-lifecycle.md b/docs/adrs/0007-host-plugin-registration-lifecycle.md new file mode 100644 index 0000000..be3b515 --- /dev/null +++ b/docs/adrs/0007-host-plugin-registration-lifecycle.md @@ -0,0 +1,194 @@ +(adr-0007-host-plugin-registration-lifecycle)= + +# ADR 0007: Host plugin registration lifecycle + +Status: Draft +Date: 2026-08-02 + +## Context + +{doc}`0001-typed-vanilla-doctest-core` makes block kinds, document parsers, +execution profiles and output checkers extensible. The core needs one typed +contribution contract, but its hosts discover contributors at different times: +direct callers already have an explicit iterable, pytest loads installed plugins +and conftests in stages, and Sphinx loads extensions before it reads doctrees. + +Treating settings and registrations as one object obscures that difference. +Settings are normalized user input. Registrations are discovered capabilities, +and xdist may discover them in more than one process. The mutable construction +mechanism must not leak into parsing or execution. + +## Decision + +The public boundary consists of `Contributor`, `Registrar`, immutable registration +records and `RegistrySnapshot`. A private builder is the only mutable object. It +accepts contributions, validates them and produces a snapshot; every parser, +projector and runner receives that snapshot explicitly. + +```python +T = t.TypeVar("T") + + +class Provider(t.NamedTuple): + name: str + version: str | None + + +class Registration(t.NamedTuple, t.Generic[T]): + name: str + value: T + provider: Provider + + +class Contributor(t.Protocol): + provider: Provider + + def contribute(self, registrar: Registrar) -> None: ... + + +class Registrar(t.Protocol): + def add_block_kind( + self, name: str, kind: BlockKind, *, replace: bool = False + ) -> None: ... + + def add_document_parser( + self, name: str, parser: DocumentParser, *, replace: bool = False + ) -> None: ... + + def add_execution_profile( + self, name: str, profile: ExecutionProfile, *, replace: bool = False + ) -> None: ... + + def add_output_checker( + self, name: str, factory: CheckerFactory, *, replace: bool = False + ) -> None: ... + + +class RegistrySnapshot(t.NamedTuple): + block_kinds: t.Mapping[str, Registration[BlockKind]] + document_parsers: t.Mapping[str, Registration[DocumentParser]] + execution_profiles: t.Mapping[str, Registration[ExecutionProfile]] + output_checkers: t.Mapping[str, Registration[CheckerFactory]] + + +def build_registry( + contributors: t.Iterable[Contributor] = (), +) -> RegistrySnapshot: ... +``` + +The snapshot fields are read-only `MappingProxyType` views over private copies, +not mutable dictionaries typed as `Mapping`. The builder is discarded after +`freeze()`. While applying each `Contributor`, the builder gives it a registrar +bound to that contributor's `Provider`; registrations cannot claim a different +origin. A contributor retaining that registrar cannot retain a mutation path: +every method raises `RegistryClosedError` after the snapshot is made. + +### Names, collisions and order + +Registration names are case-sensitive ASCII identifiers matching +`[a-z][a-z0-9_.-]*`. The same name may exist in different categories. Within one +category a duplicate is an error naming the category, incumbent provider and +challenger provider unless the challenger passes `replace=True`. Replacement +retains the incumbent's insertion position, so an explicit override cannot +silently reorder parser or profile selection. + +Built-ins register first. Contributor order is then the order supplied by the +host, and calls within a contributor retain program order. The snapshot preserves +that order. Any selection rule that needs precedence uses this declared sequence; +it never sorts by an implementation object's representation or module path. + +For document parsers, overlapping suffix claims are also collisions. They are +accepted only when the challenger uses the incumbent parser's name and passes +`replace=True`; two differently named parsers cannot both win `.md` by incidental +plugin load order. + +## Host adapters + +### Direct API + +Direct callers pass contributors to `build_registry()`. The function registers +built-ins, applies the iterable once, freezes and returns `RegistrySnapshot`. +There is no entry-point scan or process-global default in the core API. + +### pytest + +The pytest adapter publishes its hookspec in `pytest_addhooks`: + +```python +class DoctestCoreHooks: + @pytest.hookspec + def pytest_doctest_core_contributors( + self, + ) -> Contributor | t.Iterable[Contributor] | None: + """Return doctest-core contributors before collection.""" +``` + +Its `pytest_configure(trylast=True)` implementation invokes the hook, flattens +its non-`None` results in pluggy's hook-call order, builds the registry and stores +the snapshot on pytest's stash. Installed plugins and initial conftests are +already registered at that point. The snapshot is therefore ready before +`pytest_sessionstart`, when xdist starts controller nodes, and before collection. + +Nested conftests load during collection and are outside this lifecycle. A +`pytest_plugin_registered` guard detects a late plugin implementing the hookspec +and raises `pytest.UsageError` naming that plugin and the closed registration +phase. Fixtures and unrelated hooks in nested conftests remain valid. + +### Sphinx + +The Sphinx adapter exposes +`add_doctest_core_contributor(app, contributor)`. Extensions call it from their +`setup(app)` function. At `config-inited`, after extension setup and before any +document is read, the adapter emits a `doctest-core-contributors` event, appends +the `Contributor` objects returned by its listeners to the queued contributors, +builds the registry and freezes the snapshot on the application. + +The adapter function is the order-independent path. An extension that connects +directly to the custom event must list the doctest-core extension before itself, +because Sphinx cannot connect a listener to an event that has not been declared. +Calling the adapter after `config-inited` raises `RegistryClosedError` with the +extension name. + +This lifecycle makes the extractor usable on Sphinx-resolved doctrees. It does +not add a builder or claim parity with `sphinx-build -b doctest` execution. + +## xdist consistency + +The controller sends a JSON-safe manifest through `workerinput` from +`pytest_configure_node`. Each worker builds its own snapshot during +`pytest_configure` and compares before collection. The manifest has a schema +version and contains: + +- a JSON-safe projection of normalized `SessionSettings` +- every registry category, name, provider and provider version in declared order +- `doctest.OPTIONFLAGS_BY_NAME`, sorted by flag name + +A mismatch aborts the session with the controller and worker manifests. This is +an extension-set consistency check, not proof that two workers are semantically +identical. Equal provider names and versions do not prove equal source code, and +the manifest does not hash included documents, directive implementations or MyST +plugins. + +Version 1 therefore supports homogeneous worker environments. Equal source +closure and equal installed provider code are preconditions, while xdist's own +identical-collection check remains authoritative for node ids. Stronger support +for deliberately heterogeneous SSH or socket workers would require content or +environment attestation and is deferred. + +## Consequences + +- Core extension authors implement one `Contributor` regardless of host. +- Settings remain serializable inputs; discovered objects remain in the registry. +- Parse and execution code cannot mutate capabilities after collection starts. +- pytest and Sphinx own timing and diagnostics in their native idioms without + leaking their lifecycle types into the core. +- Replacement is possible but visible, attributed and deterministic. +- Supporting heterogeneous xdist workers is explicitly outside the first + contract rather than implied by a weak manifest. + +## Open + +- Whether a later release should add opt-in entry-point discovery to the direct + adapter. The core function remains explicit either way. +- Whether provider code hashes are useful enough to justify the packaging and + editable-install edge cases they introduce. diff --git a/docs/adrs/index.md b/docs/adrs/index.md index 02af2f3..be6a504 100644 --- a/docs/adrs/index.md +++ b/docs/adrs/index.md @@ -35,4 +35,5 @@ the anchor lands on unrelated code while still resolving. 0004-diagnostics-as-data 0005-line-recovery-for-nested-blocks 0006-pytest-private-api-compatibility +0007-host-plugin-registration-lifecycle ``` From 7f041dc27e1c9937d6e7e1b8d7ea01dbe3c25817 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 21:05:48 -0500 Subject: [PATCH 24/24] doctest_core(docs[analysis]): Correct evidence why: Several supporting claims overstated Sphinx block granularity, xdist determinism, stdlib subclass compatibility, and source offset semantics. what: - Correct Sphinx 9.0 grouping and phase execution claims - Separate markup protocols from stdlib-shaped facades - Qualify xdist manifests and source rewrite metadata --- docs/adrs/0001-typed-vanilla-doctest-core.md | 12 ++-- docs/adrs/0003-rejecting-per-block-items.md | 4 +- .../0005-line-recovery-for-nested-blocks.md | 13 ++-- notes/analyses/10-cpython-doctest.md | 6 +- notes/analyses/12-pytest-xdist.md | 13 ++-- notes/analyses/14-asyncio.md | 8 ++- notes/analyses/15-sphinx-ext-doctest.md | 20 ++++-- notes/analyses/16-docutils-myst.md | 4 +- notes/analyses/20-data-structures.md | 12 ++-- notes/analyses/22-extension-seams.md | 64 ++++++++++--------- .../23-namespace-scope-and-test-identity.md | 10 +-- notes/analyses/90-bibliography.md | 2 +- notes/analyses/README.md | 2 +- 13 files changed, 96 insertions(+), 74 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index a2b9de1..44e813f 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -945,9 +945,13 @@ against an empty mapping. Worker restarts are on by default. | `is_allowed_version(spec, version)` takes the specifier **first** | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | | `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310), [`:549`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L549) | -Sphinx v9.1.0 changes the default group an unargumented block joins. Any -group-naming behaviour matched against "Sphinx" has to say which Sphinx, and -{doc}`0005-line-recovery-for-nested-blocks` proposes moving this floor. +Sphinx 9.0 changed the fallback only for a bare doctest node with no `groups` +attribute: it now uses `doctest_test_doctest_blocks` +([`v9.0.0:463`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L463)). +An unargumented directive still stamps `groups=["default"]` +([`v9.0.0:94-98`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L94-L98)). +Compatibility therefore distinguishes directive-produced nodes from bare +`doctest_block` nodes instead of claiming the author-facing default changed. ## Tensions @@ -1069,7 +1073,7 @@ ceiling, and the `import-linter` contract on the leaf. |---|---|---| | [Sybil 10.0.1](https://github.com/simplistix/sybil/tree/10.0.1) | A document is a flat sequence of non-overlapping character spans; every format is a regex lexer; zero runtime dependencies | Format independence at no dependency cost, and a non-overlap invariant that raises on double collection. But one mutable namespace per document with [one independently selectable item per span](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py), so `-k` on a later example raises `NameError`. [Node ids are positional](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (`line:4,column:1`), so adding a paragraph renames every downstream test. No group support at all — a regex cannot see directive options | | [xdoctest v1.3.2](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | Abandon stdlib compatibility; own the parser via `ast`/`tokenize`; make directives structured objects | `ast` parsing and structured directives are real advances — [`REQUIRES`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) carries *why* a block skipped, which a bool cannot. But it is now building compatibility back, and its permissive got/want defaults silently change tests users wrote for stdlib. It unregisters pytest's doctest plugin outright | -| [pytest-examples v0.0.18](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | Emit the canonical form rather than parse it; rewrite expected output in place | Check-mode and update-mode collapse into one path, and byte offsets plus an invertible dedent scalar are what a data model needs to rewrite source. Composes with pytest by contributing no collector at all — the cheapest correct integration in the survey | +| [pytest-examples v0.0.18](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | Emit the canonical form rather than parse it; rewrite expected output in place | Check-mode and update-mode collapse into one path. Its absolute Python string indices enable source rewriting, although one indent scalar does not invert dedent in general. It composes with pytest by contributing no collector at all — the cheapest correct integration in the survey | | [typeshed](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | Annotate the 2001 API faithfully | Hands back `Any` at exactly the three extensible points — `globs`, `**options`, `optionflags: int`. Declares `DocTestRunner.test: DocTest` unconditionally although runtime assigns it only inside `run()`, so the stub type-checks a crash | The collective lesson: **namespace scope is not test identity, markup parsing is diff --git a/docs/adrs/0003-rejecting-per-block-items.md b/docs/adrs/0003-rejecting-per-block-items.md index f3c5ebc..2a273dd 100644 --- a/docs/adrs/0003-rejecting-per-block-items.md +++ b/docs/adrs/0003-rejecting-per-block-items.md @@ -66,9 +66,7 @@ outcome contract, rather than papered over with an id that raises when used. ## Consequences Because nothing shipped, there is no migration path to write, no deprecation -warning to add and no downstream grep to run. If PR #87 lands before this -architecture does, this record converts into exactly those things — and that is -the argument for settling the two in order rather than in parallel. +warning to add and no downstream grep to run. ## Open diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index 7a43251..784f7c7 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -18,9 +18,9 @@ reports its **last** line. A MyST fence reports its **first** line. An `.. include::`-ed block numbers against the *included* file. {doc}`0001-typed-vanilla-doctest-core` handles all of this honestly rather than -approximately: `Block.line` is nullable, the per-front-end meaning is normalized -inside the front-end that knows it, `Block.path` carries the file the text -actually lives in, and a block with no recoverable line propagates +approximately: `ParsedBlock.line` is nullable, the per-front-end meaning is +normalized inside the front-end that knows it, `ParsedBlock.path` carries the +file the text actually lives in, and a block with no recoverable line propagates `DocTest.lineno=None` into pytest's `EXAMPLE LOCATION UNKNOWN` branch. That is correct but not maximal. A nested block's failure says the location is @@ -83,9 +83,10 @@ neither permits docutils 0.22. ## Consequences -The nullable `Block.line` stays. It is not a workaround for this defect; it is -the honest representation of a front-end that may legitimately not know, and -`.. include::` attribution still needs `Block.path` regardless of version. +The nullable `ParsedBlock.line` stays. It is not a workaround for this defect; +it is the honest representation of a front-end that may legitimately not know, +and `.. include::` attribution still needs `ParsedBlock.path` regardless of +version. The line-convention normalization in `markup/` gets *simpler* at the new floor — both reStructuredText and MyST report the first line — but the normalization diff --git a/notes/analyses/10-cpython-doctest.md b/notes/analyses/10-cpython-doctest.md index 6fcb4f8..aa75509 100644 --- a/notes/analyses/10-cpython-doctest.md +++ b/notes/analyses/10-cpython-doctest.md @@ -80,9 +80,9 @@ post-dedent indent shifts every reported column. | Seam | Kind | Documented | |---|---|---| -| `DocTestParser` subclass, injected as `parser=` | nominal | yes | -| `DocTestFinder` subclass, injected as `test_finder=` | nominal | yes | -| `OutputChecker.check_output` / `output_difference`, injected as `checker=` | nominal | yes | +| `parser=` object with the `DocTestParser` methods | structural at runtime, nominal in typeshed | yes | +| `test_finder=` object with the `DocTestFinder` methods | structural at runtime, nominal in typeshed | yes | +| `checker=` object with `check_output` / `output_difference` | structural at runtime, nominal in typeshed | yes | | `report_start`, `report_success`, `report_failure`, `report_unexpected_exception` ([`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)) | subclass hook | yes | | `register_optionflag` / `OPTIONFLAGS_BY_NAME` ([`:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153)) | process-global registry | yes | | `setUp` / `tearDown` on `DocTestSuite` / `DocFileSuite` | callable param | yes | diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md index 40e82ea..d2f6e7b 100644 --- a/notes/analyses/12-pytest-xdist.md +++ b/notes/analyses/12-pytest-xdist.md @@ -61,10 +61,15 @@ Scheduling.schedule() -> send integer index batches to workers The abort path is the constraint that matters most. It is not an exception and it is not loud in the usual sense: the scheduler logs a line, assigns nothing, and -the session ends having run nothing. Any collection-time decision that is not a -pure function of (files on disk, argv, ini) — a timestamp, a PID, a hostname, a -dict iteration order, an evaluated `:skipif:` that depends on the environment — -produces this. +the session ends having run nothing. Collection is not a pure function of files, +argv and ini: included files, directive implementations, MyST plugins and the +discovered registry are inputs too. A timestamp, PID, hostname, unstable iteration +order or evaluated `:skipif:` can make workers diverge when any of those values +affects identity or order. + +A registry manifest can expose differing extension sets before collection. It +cannot prove equal source closure or equal provider code, so it supplements rather +than replaces xdist's identical-node-id check. The crash path is the second. Because only uncompleted items of a work unit are retried, a group whose blocks 1-2 ran before the crash has blocks 3..N re-run diff --git a/notes/analyses/14-asyncio.md b/notes/analyses/14-asyncio.md index a2a679c..685a449 100644 --- a/notes/analyses/14-asyncio.md +++ b/notes/analyses/14-asyncio.md @@ -90,9 +90,11 @@ owner should keep owning the lifecycle. The gap is the first row, and it is the concrete reason `DocutilsDocTestFinder` cannot be handed to `DocTestSuite(test_finder=...)` today -despite exposing a compatible `find()`. ADR 0001's answer — declare `Protocol`s -*and* subclass the stdlib classes nominally — is the cheap way to have both, and -it costs nothing at runtime. +despite exposing a differently shaped `find()`. ADR 0001's answer is to keep the +contracts separate: structural `DocumentParser` implementations for markup, an +exact `DocTestParser` lane for strings, and a `DocTestFinder`-shaped adapter for +Python objects. Nominal subclassing is used only when the replacement preserves +the nominal method signature. ## Anchors diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md index 538366b..e73f415 100644 --- a/notes/analyses/15-sphinx-ext-doctest.md +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -1,8 +1,13 @@ # `sphinx.ext.doctest` Pinned at [`v8.2.3`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py), -the version this project resolves. v9.1.0 changes the default group an -unargumented block joins; nothing else below differs between the two. +the version this project resolves. Sphinx 9.0 changed only the fallback for a +bare doctest node with no `groups` attribute: it now uses +`doctest_test_doctest_blocks` +([`v9.0.0:463`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L463)). +An unargumented Sphinx directive still stamps `groups=["default"]` +([`v9.0.0:94-98`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L94-L98)), +so its group did not change. ## Classification @@ -66,16 +71,17 @@ DocTestBuilder.test_doc(docname, doctree) [:428] | else groups[name].add_code(code) v per group: ns = {} - | three runners: setup / test / cleanup, sharing one _fakeout - | ONE doctest.DocTest PER BLOCK, each with test.globs = ns - | (assigned AFTER construction, since __init__ copies) + | setup codes -> ONE simulated DocTest containing N Examples + | each ordinary or paired test -> ONE DocTest + | cleanup codes -> ONE simulated DocTest containing N Examples + | all have test.globs = ns after construction, since __init__ copies | runner.run(test, out=..., clear_globs=False) | self.type flipped to "exec" for setup, cleanup, testcode [:549, :608] | to "single" for ordinary doctests [:580] | | if setup fails -> RETURN. Cleanup does not run. [:554-556] v -six integer counters + text streamed to outdir/output.txt +six builder counters + text streamed to outdir/output.txt ``` ## Extension seams @@ -103,7 +109,7 @@ rather than trusting one's own directive classes is the only defence against | `:options:` is accepted only on `doctest` and `testoutput`; on a `testcode` it is an unknown-option error, not a discard | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111), [`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180) | | Cleanup does **not** run when setup fails — the group returns early | [`:554-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L554-L556) | | A gated block is dropped during collection — no outcome, id or count | [`:449-450`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | -| `*` means every group the document declares; `default` means no argument was given | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) onward | +| `*` means every group the document declares; an unargumented directive stamps `default` | [`:94-98`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L94-L98), [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) onward | | Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200-L226) | | `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | | `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) | diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md index 8021966..0ea4cea 100644 --- a/notes/analyses/16-docutils-myst.md +++ b/notes/analyses/16-docutils-myst.md @@ -48,8 +48,8 @@ claim here that does not name a version is a bug in the claim. ADR 0005 All four are real and all four have to be normalized by the front-end that knows which it is. A collector that assumes one convention mis-anchors the other's blocks; a collector that reads `.line` as a number crashes on the nested case. -This is why `Block.line` in ADR 0001 is nullable and `Block.path` is separate from -the collected document. +This is why `ParsedBlock.line` in ADR 0001 is nullable and `ParsedBlock.path` is +separate from the collected document. ## The directive registry diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index 3aae0e9..eeb6a6f 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -10,7 +10,7 @@ The disagreements are entirely about which of those four are the same object. |---|---|---|---|---| | CPython `doctest` | `Example` | `DocTest` | `DocTest.globs` | `TestResults` (2 ints + an attribute) | | `_pytest.doctest` | `Example` | `DocTest` = one `Item` | `DocTest.globs`, wiped per item | `MultipleDoctestFailures` → `ReprFailDoctest` | -| `sphinx.ext.doctest` | `TestCode` | one `DocTest` **per block** (`TestGroup` is the batching unit, not the execution unit) | `ns`, assigned post-construction to every block's `DocTest` | six ints + text to a file | +| `sphinx.ext.doctest` | `TestCode` | one `DocTest` per ordinary/paired test; setup and cleanup are each combined into one simulated `DocTest` | `ns`, assigned post-construction to every `DocTest` | six builder counters + text to a file | | Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | | xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | | pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | @@ -24,8 +24,8 @@ nothing for a `-k` to split. Sybil does not: it hands out one pytest item per span over one shared mapping, which is the failure. Three units are easy to conflate for Sphinx specifically, so keep them apart: the -**runner call** is per block, the **shared state** is per group, and the **result** -is six process-wide integers. +**runner call** is per ordinary test but combined per setup or cleanup phase, the +**shared state** is per group, and the **result** is six counters on the builder. ## Field-by-field: what a source unit carries @@ -34,12 +34,14 @@ is six process-wide integers. | source text | `source` | `code` | via `lexemes` | `source` | `source` | | expected output | `want` | paired separately | — | written, not read | a separate `ParsedOutput` | | line | `lineno` (0-based, string-relative) | `lineno` | computed from span | `start_line` | `line` (nullable) | -| byte offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | +| string offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | | dedent scalar | `indent` | — | `Lexeme.offset` | `indent` | — (deferred) | +| document order | list position | list position | span order | list position | `document_order`, shared with outputs | +| identity order | example index | — | positional node id | parametrization id | `block_ordinal`, runnable blocks only | | kind | — | `type` | inferred from evaluator | `prefix_tags()` | `kind` | | group | — | via `TestGroup` | — | — | `groups` | | options | `options` | `options` | — | — | `options` | -| gate | — | `skipif` on the node | — | — | `skipif` (unevaluated) | +| gate | — | `skipif` on the node | — | — | `skipif` and `pyversion` (unevaluated) | | file | on the `DocTest` | `filename` | on the `Document` | `path` | `path` | | compile mode | — | on the *builder*, mutable | — | always exec | on `ProjectedBlock` | diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index ecf02c1..e32a615 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -7,7 +7,7 @@ ranking is not a matter of taste — each has an observed failure mode. | Mechanism | Where | Failure mode | |---|---|---| -| **Nominal subclassing** — the *type checker* demands the class, the interpreter does not | stdlib `doctest` (via typeshed), `sphinx.ext.doctest` directives | Typed callers are rejected for objects that work fine at runtime. stdlib performs no `isinstance` check on `parser` or `test_finder`; the pressure comes entirely from `doctest.pyi`. One genuine runtime edge: `DocTestSuite` sorts, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom finder must return real `DocTest`s | +| **Nominal subclassing** — the *type checker* demands the class, the interpreter does not | stdlib `doctest` (via typeshed), `sphinx.ext.doctest` directives | Typed callers reject structurally valid objects, but subclassing is invalid when the replacement method has a different signature. stdlib performs no `isinstance` check on `parser` or `test_finder`; the pressure comes from `doctest.pyi`. One genuine runtime edge: `DocTestSuite` sorts, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom finder must return real `DocTest`s | | **Callable aliases** — `Evaluator = Callable[[Example], str \| None]` | Sybil | Types nothing. You cannot express "this lexer emits `source` and `arguments`", so a mismatched pairing raises `KeyError` at run time. Sybil's own source comments say the payload "could likely be a `TypedDict`" | | **Named registry** — a module dict plus a `register_*` function | `doctest.register_optionflag`, docutils directives, xdoctest's two facades | Process-global mutable state. Order-dependent, unscoped, and silently overwritable | | **`Protocol`** — structural typing | xdoctest's `StdlibExampleLike` | None inherent; but a `Protocol` alone does not satisfy a nominal consumer | @@ -46,37 +46,37 @@ stamps, so either winner is fine. direction and is retiring `set_event_loop_policy` in favour of an explicit `loop_factory=` argument — see [`14-asyncio.md`](14-asyncio.md). -## The nominal/structural trap, and the cheap way out +## The nominal/structural trap requires separate adapters -Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: DocTestParser -= ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. The interpreter does -not — a duck-typed parser or finder runs fine. So a `Protocol` gives a third party -structural typing, and a nominal subclass keeps you *type-checkable* against the -stub. +Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: +DocTestParser = ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. The +interpreter does not perform an `isinstance` check, but it still calls the exact +stdlib methods. A markup parser whose `parse(text, path, *, settings)` returns a +doctree cannot override `DocTestParser.parse(string, name)` returning alternating +strings and examples. A mypy probe rejects that override, and subclassing does not +make the runtime calls compatible. -The subclass does not buy passability. That is a matter of matching the call -signature: a finder whose `find()` takes a string first cannot be handed to -`DocTestSuite`, which passes a module — and subclassing does not change that. - -Doing both costs nothing: +The solution is two contracts, not one class wearing two names: ```python -class Frontend(t.Protocol): +class DocumentParser(t.Protocol): suffixes: t.ClassVar[frozenset[str]] def parse( - self, text: str, path: pathlib.Path, *, settings: Settings - ) -> ParseResult: ... + self, text: str, path: pathlib.Path, *, settings: ParseSettings + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: ... -class DocutilsDocTestParser( - doctest.DocTestParser -): # nominal, for DocFileSuite(parser=...) - ... +class StdlibParserFacade(doctest.DocTestParser): + def parse( + self, string: str, name: str = "" + ) -> list[str | doctest.Example]: ... ``` -Sybil, xdoctest and stdlib each took one half. Taking both is the only reason this -is worth stating as a rule. +The markup `DocumentParser` has reStructuredText and MyST implementations. Plain +strings use the exact stdlib parser lane, and Python objects use a separate +`DocTestFinder`-shaped adapter. A nominal façade is useful only where it preserves +the nominal API's signature. ## What deserves a seam, and what does not @@ -85,19 +85,21 @@ needs it. Applied to the candidates that came up: | Candidate | Verdict | |---|---| -| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind`, so the leaf keeps its stdlib-only contract | -| Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()` — adding one there requires subclassing two classes | -| `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | -| `ExecutionProfile` | **Yes, and contributable.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. The *type* is part of the public contributor protocol, since a pluggable core that cannot accept a new execution policy is not pluggable; the registry holding them stays private | +| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a record. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind` | +| Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()`. One factory supplies the checker used for both comparison and failure explanation | +| `DocumentParser` protocol | **Yes.** Two markup implementations ship initially: reStructuredText and MyST. Plain text and Python objects use the separate stdlib-shaped lanes | +| stdlib parser and finder façades | **Yes, but separate.** They preserve `DocTestParser` and `DocTestFinder` signatures instead of subclassing them with incompatible markup or object-discovery methods | +| `ExecutionProfile` | **Yes, and contributable.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. The type and its immutable registration are public; the mutable builder is not | | A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | -| A registry *object* with builders, digests and manifests | **No object, but there is a freeze.** The contributed block kinds, front ends and profiles are resolved into an immutable mapping alongside the frozen `Settings` facets, before collection begins; registering after that is an error. That is what ADR 0001's collection contract means by *frozen registry* — not a class with its own lifecycle | -| Entry-point plugin discovery | **Optional, behind the contributor protocol.** Not the hazard it was once called: what breaks determinism is nondeterministic *contribution*, not discovery. But "identical across workers" holds only for a homogeneous local run — xdist supports SSH and socket workers — so a heterogeneous session needs a registry manifest comparison or an explicit homogeneous-plugin requirement | +| `RegistrySnapshot` and a builder | **Yes, with asymmetric visibility.** The snapshot and immutable registration records are public inputs; the mutable builder is private. Settings remain a separate value. Direct, pytest and Sphinx adapters own their freeze points in ADR 0007 | +| Entry-point plugin discovery | **Optional, behind the contributor protocol.** Discovery is not itself nondeterministic, but matching manifests only compare declared extension sets. Version 1 requires homogeneous provider code and source closure; heterogeneous-worker attestation is deferred | The distinction those two rows turn on is worth stating once: **discovery is not -the hazard, nondeterminism is.** A registry populated identically in every process -satisfies [`12-pytest-xdist.md`](12-pytest-xdist.md)'s identical-collection -requirement no matter how it was populated; a registry whose contents depend on -which conftest happened to run, or on the environment, does not. +the hazard; divergent inputs are.** A registry populated with matching names and +versions can still resolve to different code, and matching registries can still +parse different included files. ADR 0007's manifest catches declared extension +drift; [`12-pytest-xdist.md`](12-pytest-xdist.md)'s identical-collection check +remains authoritative for node ids. ## Anchors diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index 0167fb6..5ce0aa1 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -30,8 +30,8 @@ node id" is really *no* id — every test block in a group shares one method to swallow the resulting `IndexError`. So the execution shape is partly well-trodden; the contribution is making it addressable. -The bottom-right cell is where the silent failure lives, and two shipped projects -are in it. +The bottom-right cell is where the silent failure lives. One surveyed project +ships it, and one open proposal implements it with guards. Sybil is there **unacknowledged**: one `Document.namespace` shared by reference, one pytest item per region. `pytest -k` on an example whose predecessor bound a @@ -39,8 +39,10 @@ name raises `NameError`, and nothing in its documentation says so. PR #87's proposed `per-block` mode would sit there **acknowledged and guarded** — the guards being an xdist scheduler substitution, a scheduler refusal and a -run-twice refusal. It has not shipped; {doc}`ADR 0003 <../../docs/adrs/0003-rejecting-per-block-items>` rejects the shape. Those guards are the reason `_worker_count`, `_shared_page`, -`_is_page` and `_splitting_scheduler` exist at all. +run-twice refusal. It has not shipped; +{doc}`ADR 0003 <../../docs/adrs/0003-rejecting-per-block-items>` rejects the +shape. Those guards are the reason `_worker_count`, `_shared_page`, `_is_page` +and `_splitting_scheduler` exist at all. The bottom-left cell gives per-block reporting *and* an unsplittable sharing unit, and it needs no guards, because there is nothing to split. diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md index 07bcc0a..e404049 100644 --- a/notes/analyses/90-bibliography.md +++ b/notes/analyses/90-bibliography.md @@ -159,5 +159,5 @@ Typed surface: [`typeshed stubs/docutils`](https://github.com/python/typeshed/tr |---|---|---| | Sybil | [`10.0.1`](https://github.com/simplistix/sybil/tree/10.0.1) | [`sybil.py:155-157`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (positional ids) · [`document.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/document.py) (one namespace, non-overlap invariant) · [`integration/pytest.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py) (one item per region) · [`region.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · [`testing.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/testing.py) (public extension-test helpers) | | xdoctest | [`v1.3.2`](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | [`directive.py:58`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) (`REQUIRES` carries its reason) · [`plugin.py`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/plugin.py) (unregisters pytest's doctest plugin) | -| pytest-examples | [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | [`find_examples.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/find_examples.py) · [`run_code.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/run_code.py) · [`modify_files.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/modify_files.py) (byte offsets, invertible dedent, unguarded splice) | +| pytest-examples | [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | [`find_examples.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/find_examples.py) · [`run_code.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/run_code.py) · [`modify_files.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/modify_files.py) (Python string offsets, recorded indent, unguarded splice) | | typeshed | [`8c7256c`](https://github.com/python/typeshed/tree/8c7256c) | [`stdlib/doctest.pyi`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | diff --git a/notes/analyses/README.md b/notes/analyses/README.md index 74df866..72b5187 100644 --- a/notes/analyses/README.md +++ b/notes/analyses/README.md @@ -31,7 +31,7 @@ What exactly does each of them require, and where do they contradict each other? | pytest | `pytest-dev/pytest` | `9.1.1` | | pytest-xdist | `pytest-dev/pytest-xdist` | `v3.8.0` | | pytest-asyncio | `pytest-dev/pytest-asyncio` | `v1.4.0` | -| Sphinx | `sphinx-doc/sphinx` | `v8.2.3` — what this project resolves. `v9.1.0` is cited only where a difference is called out | +| Sphinx | `sphinx-doc/sphinx` | `v8.2.3` — what this project resolves. `v9.0.0` is cited only for the bare-node group fallback change | | MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` on Python ≥ 3.11; `v4.0.1` below | | Sybil | `simplistix/sybil` | `10.0.1` | | xdoctest | `Erotemic/xdoctest` | `v1.3.2` |