stage_pipeline memo: write each node once per step instead of a full subtree per group and candidate #1027

Problem, by example

The committed DAG viewer example for one query,

avg by (job) (quantile_over_time(0.5, data[5m]) / quantile_over_time(0.5, data[5m]) + quantile_over_time(0.9, data[5m]))

(tools/dag-viewer/examples/memo-ratio-of-quantiles.json, made by stage_pipeline from tools/dag-viewer/memo-examples/promql/ratio-of-quantiles-5m.json, main at d728d05c) is 13,818,953 bytes. About 12.9 MB of that is the memo section. Measured by re-serializing the file's sections with 2-space indentation:

Every group and every listed candidate carries its full subtree as a FlatDag (dag: its node and everything below), and every enumerated choice carries its whole composed DAG. The same nodes are written again and again: across all steps, the listed groups and candidates hold 3,008 nodes, but only 391 of them differ (counting a node with its child indices). These 391 take about 0.4 MB as compact JSON. The Scan data and TimeRange 5m nodes, for example, appear in nearly every one of the 336 PaneChain groups' and candidates' DAGs.

The repository and the viewer page pay for this. The viewer downloads and parses 13.8 MB to show a 7-group query, and a workload with more queries grows with groups × candidates × subtree depth.

Why it happens

stage_pipeline's memo writer (crates/devtools/src/bin/stage_pipeline.rs, the memo section added with the memo snapshots) serializes each group's dag, each candidate's dag and each enumerated choice's dag independently with the existing FlatDag export. FlatDag numbers nodes per DAG, so nothing is shared across groups or candidates.

Proposal

Pick one:

1. Shared node table per step (preferred). Each memo.steps[i] gets nodes: [...], each distinct node once (operator, schema, guarantee, materialization, children as node ids in this table). A group's and a candidate's dag becomes { "roots": [node_id] } (or just root), and group_of_node keeps mapping node ids to child groups. Enumerated choices reference the last listed step's table, or a table of their own. Deduplication is by structural equality of the serialized node, children included.
2. Drop subtrees below group boundaries. A group's or candidate's dag stops at the nodes that are a child group's node (group_of_node), and the viewer draws the child group's own dag beneath it. This is smaller, but the viewer must stitch subtrees to draw one candidate whole.

Option 1 keeps every DAG reconstructible without stitching rules, and the viewer only needs a resolver from node ids to nodes.

Scope and acceptance

• In scope: the memo section's format (versioned, with the asap-stage-pipeline/v1 validator in tools/dag-viewer/stages.js updated), the writer in stage_pipeline, and the viewer's reader.
• Out of scope: the Stage 0 to Stage 3 lanes outside memo, and memo or search behavior.
• Acceptance:
  • The regenerated tools/dag-viewer/examples/memo-ratio-of-quantiles.json is under about 1 MB.
  • The viewer behaves the same: each step's groups, candidate counts by origin, each candidate's drawn DAG, the selected choice and the enumerated DAG list render as before. Existing viewer tests pass (node --test tools/dag-viewer/stages.test.mjs, test_viewer.py).
  • A stage_pipeline test rebuilds each listed group's, candidate's and choice's DAG from the node table and checks it equals what FlatDag exported before.

Status and related work

• Found after feat(devtools): stage_pipeline writes the memo after each step, the selected choice and the enumeration #1024, which added the memo section. Part of Tracking: DAG viewer shows the memo after each step and every candidate DAG it stands for #1008.
• The viewer's memo tabs (DAG viewer: draw the memo after each step (groups with their candidates) #1005, DAG viewer: list the enumerated candidate DAGs and draw any of them #1006) read this section.