Skip to content

Tracking: DAG viewer shows the memo after each step and every candidate DAG it stands for #1008

Open
8 of 10 issues completed

Description

@zzylol

Problem, by example

Example workload: one query, evaluated every minute, with max_error 0.01 and failure_probability 0.001:

avg by (job) (quantile_over_time(0.5, data[5m]) / quantile_over_time(0.5, data[5m]) + quantile_over_time(0.9, data[5m]))

The memo the production planner builds for it, read from the stage-pipeline document tools/dag-viewer/examples/memo-ratio-of-quantiles.json (main at d728d05c, made by stage_pipeline from tools/dag-viewer/memo-examples/promql/ratio-of-quantiles-5m.json):

Corrected 2026-10-28: production numbers from the stage-pipeline document; earlier numbers came from a test helper without the series-identity column.

Group Logical sub-expression Reads
G0 Scan data –
G1 TimeRange 5m G0
G2 per-series quantile 0.5 (both occurrences; see below) G1
G3 G2 / G2 G2 (both operands)
G4 per-series quantile 0.9 G1
G5 G3 + G4 G3, G4
G6 avg by (job) G5
After Groups Implementation candidates Full candidate DAGs
Stage 0 7 – 1
Stage 1, Pass 1 7 19 (G2 and G4 have 7 each) 49 = 7 × 7 (16 use only budget_share 1)
Pass 2 window composition 7 259 (G2 and G4 have 127 each: 7 Pass 1, 120 Panes) > 1,000 (enumeration limit)
Pass 2 rules + concrete state groups 354 (336 are PaneChain shared-build groups) 1738 (G2 and G4 have 691 each: 10 Pass 1, 336 Panes, 345 SharedBuild) > 1,000 (enumeration limit)

The two quantile_over_time(0.5, data[5m]) are one group, G2: share_common_sub_dags (crates/types/src/ir/cse.rs) shares a non-ASAP node only when its schema has a provable unique key, and the PromQL frontend's series-identity column ($promql_series_identity) gives the per-series quantile's output one (series identity, timestamp). A non-ASAP node without a unique key, such as a without(..) grouping, would still be one group per reader.

To review or debug the planner stage by stage, we need to see this memo after every step, and the complete candidate DAGs it stands for. That shows whether:

  • the representation is wrong;
  • a rule is wrong or missing;
  • the selected plan really is the cheapest.

Today the DAG viewer shows one plan's three DAGs. The stage-pipeline document has no memo. Nothing converts the memo back into candidate DAGs.

Goal

The DAG viewer shows the memo after each step, lists the candidate DAGs it stands for, draws any of them, and on small workloads checks the MILP's choice against all of them.

Sub-issues (one PR each, in order)

  1. Memo: convert memo-group candidates into the vector of all candidate DAGs #1002 Memo → vector of candidate DAGs: memo::enumerate and memo::compose.
  2. stage_pipeline: write the memo after each step, the selected choice and the enumeration #1003 stage_pipeline: write the memo after each step, the selected choice and the enumeration.
  3. stage_pipeline: price every enumerated candidate DAG and flag any cheaper than the selected plan #1004 stage_pipeline: price every enumerated DAG and flag any cheaper than the selected plan.
  4. DAG viewer: draw the memo after each step (groups with their candidates) #1005 DAG viewer: draw the memo after each step.
  5. DAG viewer: list the enumerated candidate DAGs and draw any of them #1006 DAG viewer: list the enumerated candidate DAGs and draw any of them.
  6. Physical candidate labels don't say which pane chain is maintained at ingestion time; distinct plans share a label #876 DAG viewer: physical candidate labels name the pane chain maintained at ingestion time.
  7. planner-layering.md: the memo through the stages, and the memo to candidate DAGs algorithm #1007 planner-layering.md: the memo through the stages, and the memo → candidate DAGs algorithm.

#1005 and #876 can go in parallel with #1004. #977 (stage inputs and outputs) is merged before #1007.

Related

Activity

  1. self-assigned this
    on Oct 28, 2026
  2. zzylol commented on Oct 28, 2026

    @zzylol
    ContributorAuthor

    All sub-issues are done: #1009, #1024, #1028, #1020, #1019, #1018, #1022 and #1031 are merged, the Stage Viewer guide was rewritten in #1030, and the production-count test is in #1029.

  3. reopened this on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions