Add planner snapshot tests to CI

Problem

The planner has no correct-answer reference to test against, so a PR can change the plans chosen for real-world queries without anyone noticing.

Proposal: snapshot testing

For the real-world query corpora (crates/frontend-promql/tests/observability/, crates/frontend-sql/tests/...), record the planner's output for each workload in snapshot files committed to the repo:

the logical candidates
the physical candidates
the selected physical DAG

CI re-runs the planner and fails if the output no longer matches the committed snapshots. When a change is intended, the PR regenerates the snapshots and explains why they changed. The snapshot diff is what reviewers approve.

Scope to be settled: workload granularity, what goes into a snapshot, determinism (MILP ties and time limits), and CI runtime and memory.

Agreed scope

Corpora: o11y-bench, netflow, bgp_analytics, tpch_deequ, bgp_jan2024. The large PromQL corpora come later, as a sample.
Workloads: every query on its own. Whole-file workloads for netflow, tpch_deequ and bgp_jan2024. Hand-picked groups of 3 to 5 queries for o11y-bench and bgp_analytics.
Content: all of Stages 1 to 3, as text. Shorten the format once MemoGroup lands.
Search: --exhaustive with --max-combinations 1000. Fail if the catalog is not fully covered.
Determinism: a 600 s time limit, so the combination cap is the only budget that binds. Fail if the time limit is hit. Flag any workload that takes more than 60 s.
Harness: crates/devtools/tests/corpus_snapshots.rs, calling the planner in-process and driven by a manifest. Snapshots in tests/snapshots/<corpus>/<workload>.snap. UPDATE_SNAPSHOTS=1 regenerates them. Runs in the existing CI job, which uploads the regenerated snapshots as a build artifact when it fails.
Inputs: one fixed set for every workload: approximate accuracy (0.01 / 0.001) with guarantees enforced; 60 s step over a 1 h span; 1 s sample interval; built-in deployment capabilities and cost model; a tables.json per SQL corpus.
Approval: a "Snapshot changes" section in the PR description explaining which workloads changed and why. Convention only: no bot, no CODEOWNERS.
Failures: a lowering or planning error is recorded in the snapshot as the error kind and the first line of its message. A panic fails the test.

Open: whether this fits CI's time and memory. A bounded trial (netflow as one workload) comes first.

Scope update after trial runs

These changes replace the matching items in the agreed scope above.

Search: default search, not --exhaustive (replaces item 4). With --exhaustive --max-combinations 1000, 17 of the 27 o11y-bench queries, each planned on its own, went over an 8 GB memory cap. With default search (MILP-guided, at most 128 evaluations), 22 of 27 selected a plan. A snapshot holds the selected plan's Stage 1 and Stage 2 candidates, the pruned candidates, and the Stage 3 search report, not every candidate.

Reproducibility (adds to item 5): a run that stops at the evaluation limit or proves optimality is reproducible. A run that stops at the time limit depends on machine speed, so the harness fails on it instead of recording a snapshot.

First snapshot set (replaces items 1 and 2): o11y-bench queries 2, 5, 6, 11, 12, 18, 19, 20, 21, 22, 23, 25, 26 and 27, each planned on its own. Each one finishes in under 10 s and under 500 MB, about 60 s in total. Queries 5, 6 and 11 select no plan and are recorded as failure snapshots.

Deferred:

o11y-bench queries 1, 3, 8–10, 15 and 24: 29–37 s and up to 5 GB each.
Queries 7, 14, 16 and 17: stop at the 60 s time limit.
Query 4: no plan, up to 5.2 GB.
Query 13: goes over 8 GB.
Pairs of queries.
SQL corpora: netflow selects no plan, because the executor rejects its CAST(... AS TIMESTAMP) time predicates. The other SQL corpora are untested.

Inputs (adds to item 7): each corpus metric needs declared source facts: series_count (1000), sample_interval (1 s), metric_type, and an arrival rate. SQL sources also need rows_held.

test(devtools): add planner corpus snapshot harness

test(devtools): snapshot 14 o11y-bench queries

ci: check planner snapshots in a release-build job

docs: explain planner snapshots and require Snapshot changes in PRs

test(devtools): add planner corpus snapshot harness

test(devtools): snapshot 14 o11y-bench queries

ci: check planner snapshots in a release-build job

docs: explain planner snapshots and require Snapshot changes in PRs

test(devtools): add planner corpus snapshot harness

test(devtools): snapshot 14 o11y-bench queries

ci: check planner snapshots in a release-build job

docs: explain planner snapshots and require Snapshot changes in PRs

test(devtools): add planner corpus snapshot harness

test(devtools): snapshot 14 o11y-bench queries

ci: check planner snapshots in a release-build job

docs: explain planner snapshots and require Snapshot changes in PRs

test: snapshot planner output over real-world queries in CI (#956)

Navigation Menu

Repository navigation

Add planner snapshot tests to CI #941

Problem

Proposal: snapshot testing

Agreed scope

Scope update after trial runs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add planner snapshot tests to CI #941

Description

Problem

Proposal: snapshot testing

Activity

milindsrivastava1997 commented on Oct 27, 2026

Agreed scope

milindsrivastava1997 commented on Oct 27, 2026

Scope update after trial runs

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions