Repository navigation
Tracking: DAG viewer shows the memo after each step and every candidate DAG it stands for #1008
Description
Problem, by example
Example workload: one query, evaluated every minute, with max_error 0.01 and failure_probability 0.001:
avg by (job) (quantile_over_time(0.5, data[5m]) / quantile_over_time(0.5, data[5m]) + quantile_over_time(0.9, data[5m]))
The memo the production planner builds for it, read from the stage-pipeline document tools/dag-viewer/examples/memo-ratio-of-quantiles.json (main at d728d05c, made by stage_pipeline from tools/dag-viewer/memo-examples/promql/ratio-of-quantiles-5m.json):
Corrected 2026-10-28: production numbers from the stage-pipeline document; earlier numbers came from a test helper without the series-identity column.
| Group | Logical sub-expression | Reads |
|---|---|---|
| G0 | Scan data |
– |
| G1 | TimeRange 5m |
G0 |
| G2 | per-series quantile 0.5 (both occurrences; see below) |
G1 |
| G3 | G2 / G2 |
G2 (both operands) |
| G4 | per-series quantile 0.9 |
G1 |
| G5 | G3 + G4 |
G3, G4 |
| G6 | avg by (job) |
G5 |
| After | Groups | Implementation candidates | Full candidate DAGs |
|---|---|---|---|
| Stage 0 | 7 | – | 1 |
| Stage 1, Pass 1 | 7 | 19 (G2 and G4 have 7 each) | 49 = 7 × 7 (16 use only budget_share 1) |
| Pass 2 window composition | 7 | 259 (G2 and G4 have 127 each: 7 Pass 1, 120 Panes) |
> 1,000 (enumeration limit) |
| Pass 2 rules + concrete state groups | 354 (336 are PaneChain shared-build groups) |
1738 (G2 and G4 have 691 each: 10 Pass 1, 336 Panes, 345 SharedBuild) |
> 1,000 (enumeration limit) |
The two quantile_over_time(0.5, data[5m]) are one group, G2: share_common_sub_dags (crates/types/src/ir/cse.rs) shares a non-ASAP node only when its schema has a provable unique key, and the PromQL frontend's series-identity column ($promql_series_identity) gives the per-series quantile's output one (series identity, timestamp). A non-ASAP node without a unique key, such as a without(..) grouping, would still be one group per reader.
To review or debug the planner stage by stage, we need to see this memo after every step, and the complete candidate DAGs it stands for. That shows whether:
- the representation is wrong;
- a rule is wrong or missing;
- the selected plan really is the cheapest.
Today the DAG viewer shows one plan's three DAGs. The stage-pipeline document has no memo. Nothing converts the memo back into candidate DAGs.
Goal
The DAG viewer shows the memo after each step, lists the candidate DAGs it stands for, draws any of them, and on small workloads checks the MILP's choice against all of them.
Sub-issues (one PR each, in order)
- Memo: convert memo-group candidates into the vector of all candidate DAGs #1002 Memo → vector of candidate DAGs:
memo::enumerateandmemo::compose. - stage_pipeline: write the memo after each step, the selected choice and the enumeration #1003
stage_pipeline: write the memo after each step, the selected choice and the enumeration. - stage_pipeline: price every enumerated candidate DAG and flag any cheaper than the selected plan #1004
stage_pipeline: price every enumerated DAG and flag any cheaper than the selected plan. - DAG viewer: draw the memo after each step (groups with their candidates) #1005 DAG viewer: draw the memo after each step.
- DAG viewer: list the enumerated candidate DAGs and draw any of them #1006 DAG viewer: list the enumerated candidate DAGs and draw any of them.
- Physical candidate labels don't say which pane chain is maintained at ingestion time; distinct plans share a label #876 DAG viewer: physical candidate labels name the pane chain maintained at ingestion time.
- planner-layering.md: the memo through the stages, and the memo to candidate DAGs algorithm #1007 planner-layering.md: the memo through the stages, and the memo → candidate DAGs algorithm.
#1005 and #876 can go in parallel with #1004. #977 (stage inputs and outputs) is merged before #1007.
Related
- test: snapshot planner output over real-world queries in CI #956 (stage-pipeline snapshots in CI) is merged first. stage_pipeline: write the memo after each step, the selected choice and the enumeration #1003 then regenerates the snapshots with the memo counts.
- Reorganize by stage: one memo-based Pass 2, Stage 1 finishes the memo, deployment inputs split by stage, asap-planner as the only entry point #958 (Stage 1 reorganization) moves the concrete-state step into Stage 1. stage_pipeline: write the memo after each step, the selected choice and the enumeration #1003 names the snapshot by step, so only its position changes.
- Replaces [Feature] Sync DAG viewer for new logical and physical planning based on PR #509 and PR #511 #529, which is closed.
- In production the two
quantile_over_time(0.5, data[5m])are one group (G2). Only thelower_promqltest helper, which omits the series-identity column, keeps them as two groups.
Activity
feat(devtools): stage_pipeline writes the memo after each step, the s…
zzylol commentedon Oct 28, 2026
All sub-issues are done: #1009, #1024, #1028, #1020, #1019, #1018, #1022 and #1031 are merged, the Stage Viewer guide was rewritten in #1030, and the production-count test is in #1029.