[milindsrivastava1997]

Closes #941.

Why

The planner has no reference answer to test against, so a PR can change the plans chosen for real-world queries without anyone noticing.

What

For 14 o11y-bench PromQL queries, each planned on its own, the stage_pipeline document (default search) is committed as a snapshot. A new corpus-snapshots CI job fails when any snapshot changes. The PR must then regenerate the snapshots and explain the change under Snapshot changes.

How

• crates/devtools/tests/corpus_snapshots.rs (#[ignore], release build):
  1. For each tests/corpus_snapshots/<corpus>/corpus.json, it builds a one-query workload for each query id and runs the stage_pipeline binary with a 600 s time limit.
  2. It removes the search report's *_ms timings, and the copy of that report embedded in a selection error.
  3. It compares the pretty-printed document with <id>.json.
  4. UPDATE_SNAPSHOTS=1 rewrites the snapshots instead of comparing.
• The test also fails when the time limit ended a search (its plan would depend on machine speed), when a snapshot has no query in the manifest, or when stage_pipeline crashes.
• .github/workflows/rust.yml: the new corpus-snapshots job runs the test with --release --include-ignored. On failure it uploads the regenerated snapshots as the corpus-snapshots artifact.

Scope decisions and trial measurements are in the #941 comments.

Key code interfaces added / modified

None. This PR adds a test, test data, a CI job and docs.

Major algorithm changes

None.

Before this PR

A change to plan selection that altered, say, which sketch or pane width a dashboard query uses passed CI as long as the existing assertions held. Nobody saw which real-world plans had changed.

After this PR

The same change fails corpus-snapshots, names each changed query, and uploads the regenerated snapshots. The author commits them and explains each change under Snapshot changes.

Evidence

The check was run locally with cost_per_memory_byte_second in ResourceUnitCost::illustrative() temporarily raised from 1.25e-7 to 1.25e-5 (memory 100 times more expensive). The change was then reverted; it is not in this PR.

11 snapshot(s) differ; regenerated ones are in .../target/tmp/corpus_snapshots.
If the change is intended, run with UPDATE_SNAPSHOTS=1 and explain it under "Snapshot changes" in the PR.

o11y_bench/q02: plan changed at line 619:
  -               "guarantee": null,
  +               "guarantee": {
...

• 8 queries (q02, q18, q19, q21, q22, q23, q25, q26) selected a different physical plan.
• 3 queries (q12, q20, q27) kept their plan with different costs.
• q05, q06 and q11 record no-plan failures and were unchanged.

Determinism: the run that generated the snapshots, a second run, and a single-core run (taskset -c 0) produced identical snapshots, taking 53 s, 53 s and 102 s.

Verification

• Unit tests (crates/devtools/tests/corpus_snapshots.rs):
  • strip_timings_keeps_inputs: removes only the search report's timings and the report embedded in an error. Inputs such as step_ms and pane_retention_ms stay.
  • time_limit_detection: a search that used its whole wall-clock budget is detected; one stopped by the evaluation budget, or with no search report, is not.
  • first_difference_names_the_line: the failure message points at the first differing line.
• End-to-end tests: corpus_plans_match_snapshots, the 14 o11y-bench snapshots. It passes locally (release build, about 53 s).
• Other checks: the pre-commit hooks passed on every commit (fmt, check, clippy, cargo test --workspace). The new CI job runs for the first time on this PR.

Snapshot changes

All 14 snapshots are new: crates/devtools/tests/corpus_snapshots/o11y_bench/q*.json, about 1.3 MB.

After rebasing onto #943, #961 and #959, six were regenerated (8565f98): q02, q19, q22, q23, q25, q26. Their selected plans and costs are unchanged. Only the search report changed: about twice as many joint-price solves, and the direct-cost path now validates plans (q19, q22, q23, q25, q26) instead of always falling back.

Architectural decisions

• Default search, not --exhaustive. With --exhaustive --max-combinations 1000, 17 of the 27 single o11y-bench queries went over an 8 GB memory cap; default search planned 22 of the 27. The cost is that a snapshot records only the candidates the search evaluated.
• The binary, not an in-process call. document_of lives in the stage_pipeline binary. Running the binary needs no refactor, and each query runs in its own process.
• A separate release-build job. Debug builds would add minutes to the existing test job.
• Query text copied into corpus.json under stable ids. Editing the source corpus file cannot quietly move a snapshot to a different query.
• The snapshot is the whole viewer document. No custom text format to maintain, and a snapshot opens directly in dag-viewer. The diffs are large; the format is to be shortened once MemoGroup lands.

Limitations and follow-up

• Only 14 of the 27 o11y-bench queries are included. The others take 29–60 s and up to 5 GB, hit the time limit, or go over 8 GB; they are listed in Add planner snapshot tests to CI #941.
• No SQL corpus. netflow selects no plan, because the executor rejects its CAST(... AS TIMESTAMP) predicates. The other SQL corpora are untested.
• No multi-query workloads. Pairs of queries are deferred.
• The source facts for the corpus metrics (series_count 1000, 1 s samples) are assumed, not measured.

Human review — do not complete with an agent

• The MVP boundary is correct.
• New conceptual layers or public interfaces are necessary.
• The before/after description matches the intended product behavior.
• Human reviewer:
• Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKYtMxgJTLtPW9uAqPpiEG

[milindsrivastava1997]

FYI @zzylol