Deployment interface review: InstallablePlan, materialized views, DataStore and the executor #1016

Part of #1010. Line numbers are on main at 958778400.

Example: what the deployment receives

Same query as the tracking issue (avg by (job) (quantile_over_time(0.5, data[5m]) / quantile_over_time(0.5, data[5m]) + quantile_over_time(0.9, data[5m])), every minute). After Stage 3 selects a plan, plan_installable (crates/planner/src/installable.rs:240) returns:

pub struct InstallablePlan {            // installable.rs:62
    pub wire_version: u32,              // INSTALLABLE_PLAN_WIRE_VERSION
    pub dag: PhysicalASAPDAG,           // Stage 3's timings and materialization, pane chains expanded,
                                        // each materialized view and query-time Scan with its retention_ms
    pub query_texts: Vec<String>,       // one per workload query; query i is answered by dag.roots[i]
}

For this one query, query_texts has one entry and dag.roots one root. If Stage 3 maintains the quantile's KLL panes at ingestion time, each maintained state is a MaterializedView (crates/types/src/ir/materialized_view.rs:68) that the runtime refreshes and later reads through DataStore::read_materialized_view; the raw data rows a query-time node reads come from DataStore::read_raw_data (crates/executor/src/sources/data_store.rs:19).

Functionality

• The boundary between the planner and a runtime: what a runtime installs (the DAG, its materialized views and retention), how it feeds raw data and stored state to the executor, and how the executor compiles and runs the DAG.
• The deployment's inputs to planning (cost model, accuracy model, capabilities, Deployment inputs: one bundle for cost model, accuracy model and capabilities — after #511 #525) are asap-planner facade review: orchestration and deployment input assembly #1017's; this issue is the output side and the executor.

Concepts (design doc term ↔ code)

Layers from the selected plan to a running query: InstallablePlan → MaterializedView list → compile_before → view/raw slots → Scan operator → DataStore.

Interface decided in #1047 (2026-10-29)

Decided in #1047, the stage-boundary types. Its sub-issue is #1048. Names follow #1050 within #1047's scope. Principles: a stage or pass reads only the previous step's output type and calls no other stage; the asap-planner facade runs the stages in order; each stage prunes only what its own knowledge rules out and records it as PrunedCandidate<C> { group, candidate, reason: PruneReason }; queries are entry indices. Planning stays as fast as on main: P6 and P7 report before/after stage_pipeline runs.

Review checklist

User's suspected problem: over-abstraction in InstallablePlan and the runtime interfaces

• InstallablePlan::materialized_views / raw_data_retention (installable.rs:382, :389) and PhysicalASAPDAG::materialized_views / raw_data_retention (materialized_view.rs:257, :393) compute the same thing on two types. Decide which one a runtime calls.
• installable_plan_of (installable.rs:269) rebuilds the physical DAG from Stage 3's selected logical roots (re-times, re-exports, expands pane chains, then maps Stage 3's node ids to installed ids). Decide whether Stage 3 should hand over the PhysicalASAPDAG directly so there is one export. [Refactor] One named type per stage boundary: PerComputationLogicalMemoCandidates → LogicalASAPMemoCandidates → PhysicalASAPMemoCandidates #1047 P7: the facade builds the one plan (build_selected_physical_plan) and installable_plan_of installs it.
• Traits with one implementation: DataStore (by design, runtimes implement it), ExecutorCompileCheck (only PlannerExecutorCompileCheck), AccuracyEvidenceProvider (only NoAccuracyEvidence, crates/plan-selection/src/accuracy.rs). Keep the ones a deployment implements; question the others.
• PlannerExecutorCompileCheck maps every installation error other than the listed ones to "compiles" (Err(_) => Ok(()), installable.rs:350-353); the comment says those are owned by plan_installable. Check that no plan Stage 3 accepts fails at installation (Remove ExecutorCompileCheck from plan selection #949 proposes removing the check from selection).

1. Boundaries

• Executor capabilities used in planning: the supported-summary list comes from executor/src/supported_summaries.rs:13 and is defaulted in two places (installable.rs, devtools stage_pipeline.rs). Plans whose operators have no executor kernel still count as valid (Plans whose operators have no executor kernel (KMV, IRate) count as valid in Stage 2/3 #701).
• PhysicalASAPDAG::validate accepts an unstored ingestion-time output read at query time (PhysicalASAPDAG::validate accepts an unstored ingestion-time output read at query time #747) and does not check per-operator timing (PhysicalASAPDAG::validate does not check per-operator timing #656).

2. Naming

• selection appears both as Stage 3's selection and as SummaryCoverage.selection in installable.rs; executor "source" has several meanings ([Refactor] Repo-wide naming survey: overloaded and misleading names #948).

3. Stale code

• [Refactor] asap-executor has two compilers from PhysicalASAPDAG to operators #999 reports two compilers (physical_planner::compile and physical_planner::precompute::compile). There is no physical_planner/precompute.rs on main; check whether [Refactor] asap-executor has two compilers from PhysicalASAPDAG to operators #999 can be closed.
• [Tracking] Physical DAG → deployment boundary: InstallablePlan, input slots, TableProvider, BindingDataSources #968's title and body still name TableProvider, ScanRequest, BindingDataSources; align them with DataStore and the request types on main ([Docs] Physical DAG → deployment interface in the physical planning design doc #971 documents the boundary).

4. "memo" / "cache"

• None found at this boundary.

5. Rules over memo groups

• A state group chosen once in the memo becomes one materialized view, read by every reader; check this on the installed DAG.

Correctness and efficiency (executor)

• Execution gaps that make a selected plan fail at bind or run time: [Bug] Nested histogram_quantile and histogram_quantiles still get no plan (executor compile) #911, PromQL aggregates grouped with without (...) get no plan: the executor rejects "grouping without requires resolved label columns" #861, A Sum over a grouped PromQL sum fails to execute: the update reads a sample-value column the grouped output doesn't have #720, Exact Sum over an Int64 column is priced by Stage 3 but does not bind in the executor #626, SQL top-k ordered by an alias (ORDER BY c DESC LIMIT k) does not bind #619, Top-k results drop the item columns (port/job); heavy-hitter candidates resolve columns by name #672, Execute scalar operator references in Project/Filter expressions of the physical DAG #668, Runtime: sum/avg over empty input, avg overflow, and unsupported SQL shapes #659, Typed workload binding returns no physical expression for frequency and heap top-k sketches #340, bind: a SQL aggregate root never binds — DataFusion's identity Project wrapper defeats implement_tree's Aggregate check #236, SQL: implement downstream execution semantics for window frames #281, Plain Count-Min still uses the portable CountMinSketch: drops nonpositive weights, reuses row seeds past 20 rows #725, KMV insert and merge are O(k) and O(k²) in asap_sketchlib; unusable at the k the planner sizes #709, DDSketch maintenance rejects 0 and negative inputs that asap_sketchlib supports #527.
• Operator and kernel performance: Physical operator library performance: columnar, batched, parallel and efficient basic operations #778, Flat-buffer state columns for fixed-size sketches need slice kernels in asap_sketchlib #764; sketch library: Move the asap_sketchlib pin to a released sketchlib main #785; serialization formats owned by the planner: Move summary kernel serialization formats out of Planner #471; runtime tests: Runtime: tests for the native lowering of PhysicalASAPDAG #662.

Input/output check with the DAG viewer

• Input: the selected plan. Output: the installed DAG (with expanded panes, materialized views and retention) and what each DataStore read returns.
• Check: every materialized view in the installed DAG corresponds to a node Stage 3 chose to materialize; every root answers its query text; installation does not change Stage 3's choices.

Related issues

Sub-issues: #968, #999, #747, #656, #701, #911, #861, #720, #626, #619, #672, #668, #659, #662, #340, #236, #281, #725, #709, #764, #778, #785, #471, #527.

Referenced: #971 (docs, under #968), #949 (compile check, under #1015), #525 (deployment inputs, under #1017), #948 (naming).

🤖 Generated with Claude Code