Repository navigation
DAG viewer: draw the memo after each step (groups with their candidates) #1005
Description
Problem, by example
Example workload: one query, evaluated every minute, with max_error 0.01 and failure_probability 0.001:
avg by (job) (quantile_over_time(0.5, data[5m]) / quantile_over_time(0.5, data[5m]) + quantile_over_time(0.9, data[5m]))
The memo the production planner builds for it, read from the stage-pipeline document tools/dag-viewer/examples/memo-ratio-of-quantiles.json (main at d728d05c, made by stage_pipeline from tools/dag-viewer/memo-examples/promql/ratio-of-quantiles-5m.json):
Corrected 2026-10-28: production numbers from the stage-pipeline document; earlier numbers came from a test helper without the series-identity column.
| Group | Logical sub-expression | Reads |
|---|---|---|
| G0 | Scan data |
– |
| G1 | TimeRange 5m |
G0 |
| G2 | per-series quantile 0.5 (both occurrences; see below) |
G1 |
| G3 | G2 / G2 |
G2 (both operands) |
| G4 | per-series quantile 0.9 |
G1 |
| G5 | G3 + G4 |
G3, G4 |
| G6 | avg by (job) |
G5 |
| After | Groups | Implementation candidates | Full candidate DAGs |
|---|---|---|---|
| Stage 0 | 7 | – | 1 |
| Stage 1, Pass 1 | 7 | 19 (G2 and G4 have 7 each) | 49 = 7 × 7 (16 use only budget_share 1) |
| Pass 2 window composition | 7 | 259 (G2 and G4 have 127 each: 7 Pass 1, 120 Panes) |
> 1,000 (enumeration limit) |
| Pass 2 rules + concrete state groups | 354 (336 are PaneChain shared-build groups) |
1738 (G2 and G4 have 691 each: 10 Pass 1, 336 Panes, 345 SharedBuild) |
> 1,000 (enumeration limit) |
The two quantile_over_time(0.5, data[5m]) are one group, G2: share_common_sub_dags (crates/types/src/ir/cse.rs) shares a non-ASAP node only when its schema has a provable unique key, and the PromQL frontend's series-identity column ($promql_series_identity) gives the per-series quantile's output one (series identity, timestamp). A non-ASAP node without a unique key, such as a without(..) grouping, would still be one group per reader.
The DAG viewer can't show any of this. It draws three lanes for one plan: Stage 0, the selected Stage 1 candidate and its Stage 2 DAG. A reviewer can't see:
- that Stage 0 made 7 groups (the two
quantile 0.5share G2); - which 7 candidates Pass 1 gave each quantile group;
- which 336
Panesand 345SharedBuildcandidates Pass 2 and the concrete-state step added.
So a missing rule, a wrong rule or a missed sharing can't be found from the viewer.
Proposal
A Memo view in tools/dag-viewer, reading the document's memo section:
- Stage tabs: Stage 0, Pass 1, Pass 2, concrete states.
- Groups: each group is a cytoscape compound node with its candidates inside. An edge goes from a candidate to each child group it reads.
- Candidates: colored by
origin. A badge marks the candidates this step added. The candidates inselected_choiceare highlighted. - Details: clicking a candidate shows its own nodes in the details panel.
- Large memos: groups with more than N candidates collapse to one count per origin (
336 Panes), and expand on click. - Highlight query: the existing menu also dims, in the Memo view, the groups that query doesn't reach.
Scope and acceptance
stages.test.mjscovers the memo validation and the per-step diff.test_viewer.pychecks, on the example above, 7 groups on the Stage 0 tab and 19 candidates on the Pass 1 tab, and that the Pass 2 tab shows G2'sPanesandSharedBuildcounts.- The example above is added to
examples.json.
Depends on #1003.
Activity