Skip to content

DAG viewer: draw the memo after each step (groups with their candidates) #1005

Closed

Description

@zzylol

Problem, by example

Example workload: one query, evaluated every minute, with max_error 0.01 and failure_probability 0.001:

avg by (job) (quantile_over_time(0.5, data[5m]) / quantile_over_time(0.5, data[5m]) + quantile_over_time(0.9, data[5m]))

The memo the production planner builds for it, read from the stage-pipeline document tools/dag-viewer/examples/memo-ratio-of-quantiles.json (main at d728d05c, made by stage_pipeline from tools/dag-viewer/memo-examples/promql/ratio-of-quantiles-5m.json):

Corrected 2026-10-28: production numbers from the stage-pipeline document; earlier numbers came from a test helper without the series-identity column.

Group Logical sub-expression Reads
G0 Scan data –
G1 TimeRange 5m G0
G2 per-series quantile 0.5 (both occurrences; see below) G1
G3 G2 / G2 G2 (both operands)
G4 per-series quantile 0.9 G1
G5 G3 + G4 G3, G4
G6 avg by (job) G5
After Groups Implementation candidates Full candidate DAGs
Stage 0 7 – 1
Stage 1, Pass 1 7 19 (G2 and G4 have 7 each) 49 = 7 × 7 (16 use only budget_share 1)
Pass 2 window composition 7 259 (G2 and G4 have 127 each: 7 Pass 1, 120 Panes) > 1,000 (enumeration limit)
Pass 2 rules + concrete state groups 354 (336 are PaneChain shared-build groups) 1738 (G2 and G4 have 691 each: 10 Pass 1, 336 Panes, 345 SharedBuild) > 1,000 (enumeration limit)

The two quantile_over_time(0.5, data[5m]) are one group, G2: share_common_sub_dags (crates/types/src/ir/cse.rs) shares a non-ASAP node only when its schema has a provable unique key, and the PromQL frontend's series-identity column ($promql_series_identity) gives the per-series quantile's output one (series identity, timestamp). A non-ASAP node without a unique key, such as a without(..) grouping, would still be one group per reader.

The DAG viewer can't show any of this. It draws three lanes for one plan: Stage 0, the selected Stage 1 candidate and its Stage 2 DAG. A reviewer can't see:

  • that Stage 0 made 7 groups (the two quantile 0.5 share G2);
  • which 7 candidates Pass 1 gave each quantile group;
  • which 336 Panes and 345 SharedBuild candidates Pass 2 and the concrete-state step added.

So a missing rule, a wrong rule or a missed sharing can't be found from the viewer.

Proposal

A Memo view in tools/dag-viewer, reading the document's memo section:

  • Stage tabs: Stage 0, Pass 1, Pass 2, concrete states.
  • Groups: each group is a cytoscape compound node with its candidates inside. An edge goes from a candidate to each child group it reads.
  • Candidates: colored by origin. A badge marks the candidates this step added. The candidates in selected_choice are highlighted.
  • Details: clicking a candidate shows its own nodes in the details panel.
  • Large memos: groups with more than N candidates collapse to one count per origin (336 Panes), and expand on click.
  • Highlight query: the existing menu also dims, in the Memo view, the groups that query doesn't reach.

Scope and acceptance

  • stages.test.mjs covers the memo validation and the per-step diff.
  • test_viewer.py checks, on the example above, 7 groups on the Stage 0 tab and 19 candidates on the Pass 1 tab, and that the Pass 2 tab shows G2's Panes and SharedBuild counts.
  • The example above is added to examples.json.

Depends on #1003.

Metadata

Metadata

Assignees

No one assigned

    Labels

    dag-viewerDAG export and visualization toolingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions