From d61a61ad11f367020fa0b283911055908cab48a0 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 13:43:38 +0000 Subject: [PATCH 01/29] docs: explain conditional DP and MILP decomposition --- .../proposals/joint-planning-search.md | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 1198ece2d..19f5dbbf1 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -494,6 +494,66 @@ Let `n` be the number of physical candidates, `m` the number of compatible produ Search reductions must preserve joint choices. A candidate that costs more alone may enable useful sharing, so local cost dominance is insufficient. Independent components can be solved separately only when dependencies, sharing, and global constraints also separate. Generation caching and sparse compatibility edges reduce overhead without requiring enumeration of complete workload plans. +### Conditional optimal substructure + +Memo groups alone do not make the workload separable. A group's cheapest candidate can depend on which other consumers share its producers. However, a region has optimal substructure when its interaction with the rest of the plan is completely described by fixed **boundary conditions**. These include exposed inputs and outputs, producer identity/configuration, accuracy and freshness contracts, execution demand, and any resource or latency information needed outside the region. Given the same boundary conditions, replacing the interior with a cheaper feasible implementation must preserve every external cost and constraint. + +For a region without shared internal work, additive costs permit the familiar dynamic program (DP): + +```text +Best(group, requirements) = min over compatible candidates c: + LocalCost(c, requirements) + + sum over required inputs i: Best(input_group(i), input_requirements(c, i)) +``` + +The requirements must include relevant physical properties and execution context. This recurrence applies only when input subproblems are independent: summing their costs is invalid if they share a paid producer. Potential candidate cycles also require care; apply this simple recurrence to acyclic subproblems rather than recursively following cyclic memo references. Regions that do not meet these conditions remain in joint selection. + +For example, P95 and P99 can share one KLL producer. Once its configuration, materialization, supplied guarantees, and consumer execution demand are fixed, their downstream extraction regions can be optimized independently if no other constraints couple them. The shared build cost belongs to the joint selection problem and is excluded from both local costs. With evaluations every 1m and 5m, the shared producer's demand is their compatible evaluation-time union. Frequencies alone do not determine that union: alignment and freshness also matter. A boundary must preserve the required schedule information or an exact equivalent representation. + +### Local DP with joint MILP selection + +Use DP to compress only regions with a proven sufficient boundary. A local result stores its boundary contract, incremental cost, resource/accuracy/latency profile, and enough information to reconstruct the selected candidates and connections. If global budgets can distinguish local plans, retain their Pareto frontier rather than only the cheapest one. Dominance comparisons require equivalent external sharing interfaces and no weaker guarantees; a cheaper plan with fewer reuse opportunities does not dominate another plan. + +The proposed workflow is: + +```text +Generate memo candidates and compatible connections + → Identify independent components and shared boundaries + → Optimize eligible local regions with DP + → Retain boundary-compatible local plans / Pareto frontiers + → MILP selects local plans, shared candidates, and connections jointly + → Expand selected local plans and validate the execution DAG +``` + +The MILP retains shared producer activation, configuration, consumer assignment, connection-dependent demand, and global resource constraints. Local plans can replace their internal decisions only if boundary compatibility, cost ownership, and all cross-region constraints are preserved, including acyclicity after expansion. Shared raw retention is a boundary decision even when the computation regions otherwise appear independent. Nonseparable pricing or scheduling interactions must likewise remain represented jointly. + +Pane width follows the same rule. A pane chain used exclusively inside a region can be optimized locally over its generated legal widths. A chain potentially shared across regions must expose its width, coverage, materialization, and guarantees as boundary configurations. Fixing each query's locally cheapest width before joint selection can discard the globally cheapest shared chain. + +### Independent components and global budgets + +Components may be solved and combined independently only when both the feasible set and objective separate. No possible selected connection, shared resource, pricing term, or global constraint may couple them. Inspect the candidate dependency and constraint structure, not just one provisional selected plan. + +When only global budgets couple otherwise independent components, local optimization can still reduce the master problem. For example: + +| Component | Local plan | Cost | Resident memory | +| --- | --- | ---: | ---: | +| A | A1 | 3 | 8 GB | +| A | A2 | 5 | 2 GB | +| B | B1 | 3 | 8 GB | +| B | B2 | 5 | 2 GB | + +Under a 10 GB budget, independently choosing A1 and B1 is infeasible. Retaining both local choices lets a small MILP select A1+B2 or A2+B1 at cost 8. This example assumes additive simultaneous residency and identical external contracts. More complex peak-memory or latency interactions require a sufficient profile rather than a single scalar. + +### Exactness and expected benefit + +Decomposition preserves the finite-model optimum only if every relevant boundary state and nondominated local choice is retained, local subproblems are solved exactly, and the joint selection is solved to optimality. Restricting boundary configurations, capping frontiers, discretizing resources approximately, or stopping early requires a corresponding qualified result; final validation establishes feasibility, not optimality. + +DP does not remove the worst-case combinatorial difficulty. Boundary states can grow exponentially with the number of shared producers, consumer subsets, and demand patterns. The useful case is a large amount of local computation connected through a small boundary. Dense sharing may offer little compression, and frontier construction itself may cost more than it saves. + +Implement strict independent-component splitting and acyclic, unshared local DP first. Extend to shared-boundary DP only where measurements justify the larger state space. This is a proposed acceleration of the same candidate-selection problem, not a requirement to use LBBD or a claim that DP compression is already implemented. + +### Reconstruction and validation + After selection, the planner reconstructs the DAG and evaluates accuracy, latency, and resource cost. A complete MILP does not require LBBD. Lazy refinement is useful when difficult checks are left outside the initial model, provided the resulting feasibility or cost cuts are sound. The report includes the validated cost, solver status and gap, and generation limits; validating one solution alone does not prove a relaxed model optimal. ## 8. Implementation and Evaluation @@ -504,6 +564,8 @@ Implementation proceeds from initial memo identities and rule expansion to physi Evaluation must distinguish generation coverage from selection correctness. Small workloads are exhaustively enumerated over candidate subsets and compatible connections under the same inventory, then compared with MILP results. Cases include multiple selected candidates in one group, shared build costs, separate execution scopes, fusion outputs, and composed accuracy/resource requirements. Scaling experiments measure candidate and connection counts, generation and solve time, planner memory, and validated execution savings. Together these measurements establish whether the representation avoids repeated search work and whether joint selection justifies its planning cost. +For decomposition, compare the original MILP, DP-compressed MILP, and exhaustive enumeration on small inventories. They must agree on feasible optima under the same models. Include shared build ownership, differing evaluation alignments, global memory budgets, competing pane widths, external reuse opportunities, and expansion-time acyclicity. Measure local DP time, boundary-state/frontier sizes, candidate and connection counts before/after compression, and total planning time; fewer MILP variables alone do not establish a speedup. + ### Expected planning performance and execution benefits Planning performance and the selected plan's execution performance are separate outcomes. The following expectations are hypotheses to test, not measured speedups or solver-time guarantees. From 7bc7c4aacd7fce2676d20374cd6e24b44b262700 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 13:45:29 +0000 Subject: [PATCH 02/29] docs: clarify how DP reduces MILP candidate choices --- docs/design_docs/proposals/joint-planning-search.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 19f5dbbf1..2ff66d28b 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -514,6 +514,16 @@ For example, P95 and P99 can share one KLL producer. Once its configuration, mat Use DP to compress only regions with a proven sufficient boundary. A local result stores its boundary contract, incremental cost, resource/accuracy/latency profile, and enough information to reconstruct the selected candidates and connections. If global budgets can distinguish local plans, retain their Pareto frontier rather than only the cheapest one. Dominance comparisons require equivalent external sharing interfaces and no weaker guarantees; a cheaper plan with fewer reuse opportunities does not dominate another plan. +DP can reduce the candidates and connections passed to MILP without reducing the candidates initially generated. These are distinct mechanisms: + +| Mechanism | What it reduces | +| --- | --- | +| Memo interning and rule-generation caching | Duplicate candidate derivation and repeated exploration. | +| DP caching under identical boundary conditions | Repeated solution of the same local optimization subproblem. | +| DP with safe dominance pruning and region compression | Local choices and internal connections that the MILP must explicitly select. | + +For example, a local region may have 20 implementations with equivalent external interfaces, sharing opportunities, and guarantees. If there are no remaining resource or other external tradeoffs, DP can retain its cheapest implementation as one representative local plan. The MILP then selects that plan through its boundary rather than selecting its internal implementation. If one plan costs 3 and uses 8 GB while another costs 5 and uses 2 GB, both may be needed under a global memory budget. Likewise, a locally more expensive candidate cannot be removed if it offers a distinct sharing opportunity. DP alone does not guarantee a smaller candidate inventory: compression comes from proving that discarded choices cannot improve any relevant external decision. + The proposed workflow is: ```text @@ -550,7 +560,7 @@ Decomposition preserves the finite-model optimum only if every relevant boundary DP does not remove the worst-case combinatorial difficulty. Boundary states can grow exponentially with the number of shared producers, consumer subsets, and demand patterns. The useful case is a large amount of local computation connected through a small boundary. Dense sharing may offer little compression, and frontier construction itself may cost more than it saves. -Implement strict independent-component splitting and acyclic, unshared local DP first. Extend to shared-boundary DP only where measurements justify the larger state space. This is a proposed acceleration of the same candidate-selection problem, not a requirement to use LBBD or a claim that DP compression is already implemented. +Complete and validate the current MILP implementation before implementing DP compression; use the uncompressed MILP as a correctness baseline. For subsequent DP work, implement strict independent-component splitting and acyclic, unshared local DP first. Extend to shared-boundary DP only where measurements justify the larger state space. This is a proposed acceleration of the same candidate-selection problem, not a requirement to use LBBD or a claim that DP compression is already implemented. ### Reconstruction and validation From 50f6228ac1a0363c13a56e63cf7fcfc4b09011e7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 02:31:49 +0000 Subject: [PATCH 03/29] docs: require feasibility-only accuracy and direct DAG costing --- docs/design_docs/proposals/joint-planning-search.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 2ff66d28b..96b63cae9 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -194,6 +194,14 @@ The following cases make the accounting concrete. Let `B` be the cost of one who Memory-time and capacity answer different questions. A state occupying `m` bytes for duration `t` contributes `m × t` to memory-time. A capacity limit instead considers simultaneous residency, including temporary allocations; it cannot be inferred from memory-time alone. +### Direct modeled cost in the MILP + +The selection objective must express the modeled cost of the selected complete DAG, rather than rank plans with an exactness preference and discover their prices only after selection. This does not require enumerating complete plans. Factor the objective into node activation and state, execution demand, consumer connections and reads, shared resources, and query overhead. Each term has one owner; execution-dependent costs use the selected consumers' compatible evaluation schedules. Raw retention is charged once per source and storage scope at the maximum required horizon. Fixed query overhead can be omitted during minimization but must be restored in reported totals. + +For a shared KMV rollup, charge one build at each required common evaluation, one fine-group extraction, the coarse regrouping merge, and its extraction. Two independent builds incur two build charges. Different consumer schedules change the union of execution demand, not merely the number of activated nodes. Supplier-dependent cardinality, state size, and lifetime require explicit finite contexts or connection-dependent variables; do not replace an unknown coefficient with zero and report the objective as complete. + +Within a supported cost context, the decoded MILP objective plus omitted constants must agree with the independent physical-plan evaluator within numerical tolerance. Runtime measurements calibrate this model; neither solver optimality nor evaluator agreement establishes actual runtime cost. Contexts not yet fully encoded must report evaluator-refinement costing and must not claim a direct complete-cost formulation. + The configured Stage 3 objective and evaluators remain authoritative. Current memo MILP supports usage-over-time pricing; broader cost-model support is an extension. Accuracy is an output requirement, since locally acceptable errors may compose into an unacceptable query result. Latency and resource limits likewise depend on the configured timing and capacity assumptions. ## 5. MILP Formulation @@ -416,6 +424,8 @@ Consumer-dependent freshness, invalidation, and lifetime checks may involve seve For each query `q`, the configured accuracy contract specifies an error metric and target, and, when probabilistic, a failure-probability target. A plan is feasible only if its composed output error and failure probability satisfy those targets. These are end-to-end checks, not a sum of arbitrary local sketch errors. +Accuracy is a feasibility condition, not an approximate-candidate surcharge. Exact and approximate candidates satisfying the requested contract compete on resource cost without an exactness penalty. Prune a candidate or connection combination only when its incompatibility is established; local uncertainty is not proof that every completion fails. If accuracy depends on suppliers or downstream composition, constrain that combination. In strict mode, an uncertified plan cannot be returned as satisfying the target, and the user's failure probability must not be silently ignored. + When proven composition rules exist, encode their bounds directly. Otherwise use the exact finite-assignment exclusion below for each plan whose validated accuracy fails. For example, if a selected P99 estimator and its small KLL producer violate the target regardless of all other choices, the following focused cut is sound: ```text From 0f88c4e0977d0e301d917f8321148957535a1517 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 02:34:13 +0000 Subject: [PATCH 04/29] fix: remove approximate-candidate accuracy surcharges --- .../src/search/refinement/planner.rs | 43 ++++--------------- 1 file changed, 9 insertions(+), 34 deletions(-) diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index 64b819483..15ded0e81 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -423,33 +423,8 @@ fn accuracy_baseline( let node = template .as_ref() .ok_or("baseline candidate has no operator template")?; - let penalty = if context.models.plan_selection.ignore_accuracy_guarantees { - // The configured policy accepts approximate outputs without a - // guarantee; preferring exact contracts would hide cheaper builds. - 0.0 - } else if let Some(family) = node.asap().and_then(ASAPOp::produced_state) { - if matches!(family, FieldDataType::ExactAggregate(..)) { - 0.0 - } else { - preference - } - } else if let Some(guarantee) = &node.guarantee { - if guarantee.is_exact() { - 0.0 - } else if guarantee.spec.accuracy_metric == AccuracyMetric::Rank - && !rank_requested[id] - { - preference * preference - } else { - preference - } - } else if matches!(node.asap(), Some(ASAPOp::SummaryEstimate { .. })) { - preference * preference - } else { - // Exact operators may have no template guarantee until their - // selected input contracts are recomposed; do not penalize that. - 0.0 - }; + // Accuracy determines admissibility through composition and validation. + // It must not add a price surcharge to an approximate implementation. let cost_priority = if reference_cost > 0.0 { let relative = proposal_costs[id] / reference_cost; // Preserve ordering beyond the normal linear range, without @@ -510,7 +485,7 @@ fn accuracy_baseline( if matches!(kind.params(), asap_types::ir::schema::SketchParams::DDSketch { alpha } if *alpha > dds_budgets[id]))); Ok(( - 1.0 + penalty + cost_priority + unknown_work_priority, + 1.0 + cost_priority + unknown_work_priority, contract_risk || contextual_materialization || projected_raw_source || insufficient_composed_budget, )) }) @@ -1388,10 +1363,10 @@ mod tests { ); } - // Ignoring accuracy guarantees removes exact/sketch preference penalties; + // Accuracy policy changes feasibility, not the price of a summary candidate; // costs and final evaluation still determine which implementation is kept. #[test] - fn ignored_accuracy_does_not_penalize_summary_candidates() { + fn accuracy_policy_does_not_penalize_summary_candidates() { let (configured, roots, entries, data) = fixture(); let settings = crate::PlanSelectionSettings { ignore_accuracy_guarantees: true, @@ -1433,9 +1408,9 @@ mod tests { }, ) .unwrap(); - assert!( - ignored[id] < enforced[id], - "ignored guarantees must not retain an exactness penalty" + assert_eq!( + ignored[id], enforced[id], + "enforcing accuracy must not add an exactness penalty" ); } @@ -1637,7 +1612,7 @@ mod tests { assert!(rank_nodes .iter() - .all(|&id| rank_weights[id] < value_weights[id])); + .all(|&id| rank_weights[id] == value_weights[id])); assert!(configured .inventory .graph From 0a177b86fdec4192c291f8d2ea0e172ddc8a3e5a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 02:36:16 +0000 Subject: [PATCH 05/29] test: validate production KMV rollup selection under budget --- .../tests/rollup_controlled.rs | 204 ++++++++++++++++++ 1 file changed, 204 insertions(+) create mode 100644 crates/integration-tests/tests/rollup_controlled.rs diff --git a/crates/integration-tests/tests/rollup_controlled.rs b/crates/integration-tests/tests/rollup_controlled.rs new file mode 100644 index 000000000..966611126 --- /dev/null +++ b/crates/integration-tests/tests/rollup_controlled.rs @@ -0,0 +1,204 @@ +//! Controlled plans for the viewer's actual distinct-count rollup workload. +use asap_logical_asap_optimization::{ + asap_aware_cse::{ + rollup_candidates, rollup_pass2_candidates, sharing_candidates, with_shared_candidates, + }, + per_computation_candidates::{ + compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, + }, +}; +use asap_plan_selection::{CostModel, PlanSelectionSettings, PlanningModels, ResourceUnitCost}; +use asap_types::{ + ir::{ + schema::{DataType, Field, Schema, SketchAlgorithm}, + ASAPOp, OperatorNode, QueryRoot, + }, + physical::{Materialization, Medium}, + workload::PlanningWorkload, +}; +use std::{collections::HashSet, rc::Rc}; + +fn fixture() -> (PlanningWorkload, Vec) { + let workload: PlanningWorkload = serde_json::from_str(include_str!( + "../../../tools/dag-viewer/memo-examples/sql/rollupheavy-t11-service-endpoint-5m.json" + )) + .unwrap(); + let mut schema = Schema::new(vec![ + Field::plain("ts", DataType::Timestamp, false), + Field::plain("region", DataType::Utf8, false), + Field::plain("service", DataType::Utf8, false), + Field::plain("endpoint", DataType::Utf8, false), + Field::plain("status", DataType::Utf8, false), + Field::plain("user_id", DataType::Float64, false), + Field::plain("latency", DataType::Float64, false), + ]); + schema.time_index = Some(0); + let catalog = asap_frontend_sql::SqlCatalog::new().with_table("http", schema); + let roots = asap_devtools::planning_stages::lower_sql_roots(&workload, &catalog).unwrap(); + (workload, roots) +} +fn nodes(roots: &[QueryRoot]) -> Vec> { + roots + .iter() + .flat_map(|r| match r { + QueryRoot::Operator(n) => OperatorNode::reachable(n), + _ => panic!("operator root"), + }) + .collect() +} + +#[test] +fn controlled_distinct_rollup_prices_the_original_workload() { + let (workload, roots) = fixture(); + let entries: Vec<_> = workload.query_workload.entries().collect(); + let data = workload.data_workload.as_ref().unwrap(); + let generated = + generate_per_computation_candidates(roots.into_iter().enumerate().collect()).unwrap(); + let mut settings = PlanSelectionSettings::default(); + settings.cost_model = CostModel::UsageOverTime(ResourceUnitCost::aws_fargate_us_east_1( + settings.resource_speed, + )); + let models = PlanningModels::builtin().with_plan_selection(&settings); + let mut accepted = Vec::new(); + for algorithm in [None, Some(SketchAlgorithm::Hll), Some(SketchAlgorithm::Kmv)] { + let choices: Vec<_> = generated + .targets + .iter() + .map(|target| match &algorithm { + None => 0, + Some(a) => target + .choices + .candidates + .iter() + .position(|c| matches!(c,ComputationCandidate::Summary(found) if found==a)) + .expect("family generated"), + }) + .collect(); + let roots: Vec<_> = compose_logical_candidate(&generated, &choices) + .unwrap() + .into_iter() + .map(|(_, r)| r) + .collect(); + let sharing = sharing_candidates(&roots); + let roots = with_shared_candidates(&roots, &sharing, &vec![true; sharing.len()]); + let mut variants = vec![("separate", roots.clone())]; + if algorithm.is_some() { + let rules = rollup_candidates(&roots); + assert_eq!(rules.len(), 1, "one service rollup for {algorithm:?}"); + for rolled in rollup_pass2_candidates(&roots, &rules).unwrap() { + variants.push(("rollup", rolled.roots.unwrap())); + } + } + for (kind, roots) in variants { + let family = algorithm + .as_ref() + .map_or("exact".to_string(), |a| format!("{a:?}")); + let id = format!("{family}-{kind}"); + let builds: HashSet<_> = nodes(&roots) + .iter() + .filter(|n| matches!(n.asap(), Some(ASAPOp::SummaryAgg { .. }))) + .map(Rc::as_ptr) + .collect(); + let mut plan = asap_physical_asap_optimization::optimize_physical_asap_chosen( + &id, + &roots, + &entries, + data, + models.deployment_capabilities, + &mut |node, _, options| { + let mat = Materialization::MatAtQuery(Medium::Memory); + if builds.contains(&node) && options.contains(&mat) { + mat + } else { + Materialization::NotMaterialized + } + }, + ) + .unwrap() + .expect("fixed query-time implementation"); + plan.id = id.clone(); + let result = asap_plan_selection::select_plan(&[plan.clone()], &entries, data, models); + match result { + Ok(selection) => { + println!( + "{id}: builds={} cost={:?} accepted={:?}", + builds.len(), + selection.costs[&id].cost_per_second, + selection.accepted + ); + accepted.push(plan); + } + Err(error) => println!("{id}: REJECTED {error}"), + } + for n in nodes(&roots) { + if let Some(ASAPOp::SummaryAgg { family, .. }) = n.asap() { + println!("{id} family={family:?}"); + } + } + } + } + let chosen = asap_plan_selection::select_plan(&accepted, &entries, data, models).unwrap(); + println!("CONTROLLED WINNER {}", chosen.selected); + assert_eq!( + chosen.selected, "Kmv-rollup", + "shared KMV must win the controlled cost comparison" + ); +} + +// The production entrypoint must find a sketch rollup under the viewer budget. +#[test] +fn production_distinct_rollup_finds_shared_kmv() { + let (workload, roots) = fixture(); + let entries: Vec<_> = workload.query_workload.entries().collect(); + let data = workload.data_workload.as_ref().unwrap(); + let mut settings = PlanSelectionSettings::default(); + settings.cost_model = CostModel::UsageOverTime(ResourceUnitCost::aws_fargate_us_east_1( + settings.resource_speed, + )); + let run = asap_plan_selection::run_stage1_to_stage3_with( + roots.into_iter().enumerate().collect::>(), + &entries, + data, + PlanningModels::builtin().with_plan_selection(&settings), + asap_plan_selection::Stage1ToStage3Search { + milp_time_limit_ms: Some(30_000), + ..Default::default() + }, + ) + .unwrap(); + let selection = run + .stage3_plan_selection + .as_ref() + .expect("validated plan within budget"); + println!( + "production cost={:?} report={:?}", + selection.costs[&selection.selected].cost_per_second, + selection.search.candidate_connections + ); + let accepted = &selection.accepted[&selection.selected]; + assert_eq!(accepted.len(), 2); + assert!(accepted + .iter() + .all(|accuracy| !accuracy.accepted_regardless_of_guarantee + && accuracy.sketches.iter().any(|sketch| sketch == "Kmv"))); + assert!( + selection.costs[&selection.selected] + .cost_per_second + .unwrap() + < 2.0e-6 + ); + let plan = run.selected().unwrap(); + let builds: HashSet<_> = nodes(&plan.roots) + .iter() + .filter(|node| matches!(node.asap(), Some(ASAPOp::SummaryAgg { .. }))) + .map(Rc::as_ptr) + .collect(); + assert_eq!(builds.len(), 1, "both queries reuse a single state build"); + let rollup = nodes(&plan.roots) + .iter() + .any(|n| matches!(n.asap(), Some(ASAPOp::SummaryMerge { .. }))); + assert!( + rollup, + "production search must select the cheaper legal rollup" + ); +} From 5009efc50f8265ccb558eb4de5f7f1467af4ae43 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 02:41:45 +0000 Subject: [PATCH 06/29] fix: keep unresolved estimator contracts out of feasibility seed --- .../src/search/refinement/planner.rs | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index 15ded0e81..c816b8354 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -442,8 +442,23 @@ fn accuracy_baseline( } else { 0.0 }; + // A strict feasibility seed must not spend its limited attempts on + // estimators whose error still needs unavailable workload facts. + // This is seed-only: another supplier may repair the contract, so + // the complete graph retains these candidates and validates them. + let unresolved_estimator_error = + matches!(node.asap(), Some(ASAPOp::SummaryEstimate { .. })) + && { + let root = QueryRoot::Operator(std::rc::Rc::clone(node)); + crate::bound_root_guarantee( + &root, + crate::series_read(&root, context.data), + context.data, + &context.models, + ).is_none_or(|guarantee| guarantee.spec.error.evaluate().is_none()) + }; let contract_risk = !context.models.plan_selection.ignore_accuracy_guarantees - && (node.guarantee.as_ref().is_some_and(|guarantee| { + && (unresolved_estimator_error || node.guarantee.as_ref().is_some_and(|guarantee| { !guarantee.is_exact() && guarantee.spec.accuracy_metric == AccuracyMetric::Rank && !rank_requested[id] From 8b7a68aaca345e7e1c6fbe70750a806a3e891ca5 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 03:13:16 +0000 Subject: [PATCH 07/29] docs: explain jointly computed execution DAG costs and coverage --- docs/design_docs/proposals/joint-planning-search.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 41db27871..af980a666 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -198,6 +198,10 @@ Memory-time and capacity answer different questions. A state occupying `m` bytes ### Direct modeled cost in the MILP +The direct-cost objective computes the total modeled cost of the selected execution DAG. It does not precompute one fixed total cost for each candidate or enumerate every complete DAG. Before solving, the planner computes each supported candidate's local cost parameters from its configuration and compatible suppliers. Inside MILP, the selected nodes and connections determine execution frequency and repeated work, consumer-specific operations, state holding costs, shared raw retention, and query overhead. A shared producer owns its charge once, including every required execution; its consumers' work is charged separately. **Local candidate cost parameters are precomputed; the complete selected DAG's cost is determined jointly by MILP.** + +The first implementation admits a restricted query-time exact/KMV subspace with fixed common evaluation schedules and certified supplier price interfaces. It includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; this is an implementation limitation to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. + The selection objective must express the modeled cost of the selected complete DAG, rather than rank plans with an exactness preference and discover their prices only after selection. This does not require enumerating complete plans. Factor the objective into node activation and state, execution demand, consumer connections and reads, shared resources, and query overhead. Each term has one owner; execution-dependent costs use the selected consumers' compatible evaluation schedules. Raw retention is charged once per source and storage scope at the maximum required horizon. Fixed query overhead can be omitted during minimization but must be restored in reported totals. For a shared KMV rollup, charge one build at each required common evaluation, one fine-group extraction, the coarse regrouping merge, and its extraction. Two independent builds incur two build charges. Different consumer schedules change the union of execution demand, not merely the number of activated nodes. Supplier-dependent cardinality, state size, and lifetime require explicit finite contexts or connection-dependent variables; do not replace an unknown coefficient with zero and report the objective as complete. From dd5e6501d27be9c55a54fd639d48e15381515ca4 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 03:15:29 +0000 Subject: [PATCH 08/29] feat: price query-time execution DAGs directly in joint MILP --- .../tests/rollup_controlled.rs | 11 + crates/plan-selection/src/lib.rs | 11 + .../src/search/candidates/graph.rs | 2 + .../src/search/costing/direct_profile.rs | 759 ++++++++++++++++++ .../plan-selection/src/search/costing/mod.rs | 2 + .../src/search/milp/direct_cost.rs | 626 +++++++++++++++ crates/plan-selection/src/search/milp/mod.rs | 2 + .../plan-selection/src/search/milp/model.rs | 132 ++- .../src/search/refinement/components.rs | 17 + .../src/search/refinement/planner.rs | 85 +- .../src/search/refinement/search.rs | 248 ++++-- crates/plan-selection/src/search/route.rs | 20 + 12 files changed, 1843 insertions(+), 72 deletions(-) create mode 100644 crates/plan-selection/src/search/costing/direct_profile.rs create mode 100644 crates/plan-selection/src/search/milp/direct_cost.rs diff --git a/crates/integration-tests/tests/rollup_controlled.rs b/crates/integration-tests/tests/rollup_controlled.rs index 966611126..406fea84f 100644 --- a/crates/integration-tests/tests/rollup_controlled.rs +++ b/crates/integration-tests/tests/rollup_controlled.rs @@ -175,6 +175,17 @@ fn production_distinct_rollup_finds_shared_kmv() { selection.costs[&selection.selected].cost_per_second, selection.search.candidate_connections ); + let report = selection.search.candidate_connections.as_ref().unwrap(); + assert!( + report.direct_cost_validated_plans > 0, + "production must validate a plan priced directly by MILP: {report:?}" + ); + assert!( + report + .direct_cost_best_validated_cost + .is_some_and(|cost| cost < 2.0e-6), + "direct MILP must price a plan cheaper than independent KMV" + ); let accepted = &selection.accepted[&selection.selected]; assert_eq!(accepted.len(), 2); assert!(accepted diff --git a/crates/plan-selection/src/lib.rs b/crates/plan-selection/src/lib.rs index 46c9e32e2..d1276fd16 100644 --- a/crates/plan-selection/src/lib.rs +++ b/crates/plan-selection/src/lib.rs @@ -1098,6 +1098,17 @@ pub struct CandidateConnectionReport { pub solver_ms: f64, pub validation_ms: f64, pub solve_calls: usize, + /// Restricted query-time proposals priced completely inside the MILP. + #[serde(default)] + pub direct_cost_solve_calls: usize, + /// Feasible direct proposals whose objective agrees with the evaluator. + #[serde(default)] + pub direct_cost_validated_plans: usize, + #[serde(default)] + pub direct_cost_best_validated_cost: Option, + /// Direct-cost contexts rejected by numerical or modeling support checks. + #[serde(default)] + pub direct_cost_fallback_reasons: Vec, pub peak_milp_variables: usize, pub peak_milp_constraints: usize, #[serde(default)] diff --git a/crates/plan-selection/src/search/candidates/graph.rs b/crates/plan-selection/src/search/candidates/graph.rs index f8b55e82a..998c158db 100644 --- a/crates/plan-selection/src/search/candidates/graph.rs +++ b/crates/plan-selection/src/search/candidates/graph.rs @@ -100,6 +100,8 @@ pub(crate) struct SelectedCandidateGraph { #[derive(Debug, thiserror::Error)] pub(crate) enum CandidateSelectionError { + #[error("direct cost model is not representable: {0}")] + UnsupportedDirectCost(String), #[error("invalid candidate dependency graph: {0}")] InvalidGraph(String), #[error("candidate selection solver: {0}")] diff --git a/crates/plan-selection/src/search/costing/direct_profile.rs b/crates/plan-selection/src/search/costing/direct_profile.rs new file mode 100644 index 000000000..cc938981c --- /dev/null +++ b/crates/plan-selection/src/search/costing/direct_profile.rs @@ -0,0 +1,759 @@ +//! Exact local profiles for a restricted query-time proposal domain. +//! +//! Each accepted input is priced from its actual recursively certified supplier. +//! Different priced interfaces are excluded from this proposal, not the full +//! catalog. No original-template child is used as workload statistics evidence. +use crate::search::candidates::configurations::CandidateConfigurationInventory; +use crate::search::candidates::graph::PhysicalCandidateId; +use crate::search::milp::direct_cost::{DirectCostProfile, NodeCostProfile}; +use crate::search::refinement::planner::CandidatePlanningContext; +use crate::{CostModel, PricedNode, ProducerOutput}; +use asap_types::ir::physical_export::PhysicalASAPDAGNode; +use asap_types::ir::properties::execution_data_state::{DataPrimitive, ExecutionDataState}; +use asap_types::ir::properties::{AccuracyScope, ExecutionTiming}; +use asap_types::ir::schema::{FieldDataType, SketchParams}; +use asap_types::ir::{ASAPOp, NonASAPOp, QueryRoot}; +use asap_types::physical::{Materialization, Medium}; +use asap_types::workload::{QueryRecurrence, RepeatedDemand}; +use std::collections::HashSet; + +#[derive(Debug, thiserror::Error)] +pub(in crate::search) enum DirectProfileError { + #[error(transparent)] + DeadlineExceeded(#[from] asap_logical_asap_optimization::memo::CandidateGenerationTimeout), + #[error("direct cost profile: {0}")] + Invalid(String), +} + +pub(in crate::search) struct DirectProfileExtraction { + pub profile: DirectCostProfile, + pub supported: Vec, + /// The sole input's certified suppliers; empty for scans/unsupported nodes. + pub suppliers: Vec>, +} + +fn supported_node(node: &asap_types::ir::OperatorNode) -> bool { + matches!( + node.non_asap(), + Some( + NonASAPOp::Scan { .. } + | NonASAPOp::TimeRange { .. } + | NonASAPOp::Filter { .. } + | NonASAPOp::Project { .. } + | NonASAPOp::Aggregate { .. } + ) + ) || matches!( + node.asap(), + Some( + ASAPOp::SummaryAgg { .. } + | ASAPOp::SummaryEstimate { .. } + | ASAPOp::FinalizeExactAccumulator { .. } + | ASAPOp::SummaryMerge { .. } + ) + ) +} + +/// Only fixed, identical request schedules, no range requests and one operator +/// root per query are supported. Every returned profile is priced at one +/// execution; the MILP supplies demand, held lifetime, retention and overhead. +pub(in crate::search) fn extract( + configured: &CandidateConfigurationInventory, + context: &CandidatePlanningContext<'_>, +) -> Result, DirectProfileError> { + let settings = context.models.plan_selection; + let CostModel::UsageOverTime(prices) = &settings.cost_model else { + return Ok(None); + }; + let Some(first) = context.entries.first() else { + return Ok(None); + }; + if !matches!( + first.recurrence, + QueryRecurrence::Repeated( + RepeatedDemand::FixedInterval(_) | RepeatedDemand::FixedIntervalAt { .. } + ) + ) || context + .entries + .iter() + .any(|entry| entry.recurrence != first.recurrence || entry.range_evaluation.is_some()) + || context + .original_roots + .iter() + .any(|root| !matches!(root, QueryRoot::Operator(_))) + || context.original_roots.len() != configured.inventory.graph.query_roots.len() + { + return Ok(None); + } + let rate = crate::cost::execution_demand::evaluation_rate( + [(&first.recurrence, None)], + settings.non_repeating_query_horizon_seconds, + ) + .map_err(|error| DirectProfileError::Invalid(error.to_string()))?; + if !rate.is_finite() || rate <= 0.0 { + return Ok(None); + } + if context.entries.iter().any(|entry| { + !entry.requirements.accuracy.is_exact() + && (entry.requirements.accuracy.accuracy_scope != AccuracyScope::PerValue + || entry + .requirements + .accuracy + .failure_probability + .is_none_or(|delta| !delta.is_finite() || delta <= 0.0)) + }) { + return Ok(None); + } + let delta = context + .entries + .iter() + .filter(|entry| !entry.requirements.accuracy.is_exact()) + .filter_map(|entry| entry.requirements.accuracy.failure_probability) + .reduce(f64::min); + let count = configured.configurations.len(); + let mut nodes = Vec::with_capacity(count); + for (id, configuration) in configured.configurations.iter().enumerate() { + check_deadline(context.deadline)?; + let node = configured.inventory.templates[id].as_ref(); + nodes.push( + node.filter(|node| { + supported_node(node) + && stable_size(node, delta) + && configuration.timing == ExecutionTiming::QueryTime + && matches!( + configuration.materialization, + Materialization::NotMaterialized + | Materialization::MatAtQuery(Medium::Memory) + ) + && configured.inventory.graph.candidates[id] + .producer_requirements + .len() + <= 1 + }) + .map(|node| { + let mut position = 0; + PhysicalASAPDAGNode { + id, + payload: node.operator.map_children(|_| { + let current = position; + position += 1; + current + }), + output_state: ExecutionDataState { + timing: ExecutionTiming::QueryTime, + primitive: if node.asap().and_then(ASAPOp::produced_state).is_some() { + DataPrimitive::SummaryState + } else { + DataPrimitive::Raw + }, + }, + materialization: configuration.materialization, + output_schema: node.schema.clone(), + guarantee: node.guarantee.clone(), + } + }), + ); + } + // The evaluator inspects immediate scan consumers (and TimeShift, excluded + // here). Different possible window/no-window contexts cannot share a price. + let mut lookbacks = vec![None; count]; + for id in 0..count { + if !nodes[id].as_ref().is_some_and(|node| { + matches!( + node.payload, + asap_types::ir::Operator::NonASAP(NonASAPOp::Scan { .. }) + ) + }) { + continue; + } + let mut windows = HashSet::new(); + for (consumer, candidate) in configured.inventory.graph.candidates.iter().enumerate() { + check_deadline(context.deadline)?; + let Some(node) = &nodes[consumer] else { + continue; + }; + if candidate.producer_requirements.iter().any(|input| { + input + .compatible_producers + .contains(&PhysicalCandidateId(id)) + }) { + windows.insert(match &node.payload { + asap_types::ir::Operator::NonASAP(NonASAPOp::TimeRange { range, .. }) => { + Some(range.as_millis() as u64) + } + _ => None, + }); + } + } + if configured.inventory.graph.query_roots.iter().any(|root| { + root.compatible_candidates + .contains(&PhysicalCandidateId(id)) + }) { + windows.insert(None); + } + if windows.len() == 1 { + lookbacks[id] = *windows.iter().next().unwrap(); + } else { + nodes[id] = None; + } + } + let mut cache = vec![None; count]; + let mut visited = vec![false; count]; + let mut visiting = vec![false; count]; + let mut suppliers = vec![Vec::new(); count]; + let mut sketch_counts = vec![0; count]; + for id in 0..count { + price( + id, + configured, + context, + &nodes, + &lookbacks, + &mut cache, + &mut visited, + &mut visiting, + &mut suppliers, + &mut sketch_counts, + )?; + } + let supported: Vec<_> = cache.iter().map(Option::is_some).collect(); + + let zero = || NodeCostProfile { + execution_cost: 0.0, + work_seconds: 0.0, + held_memory_cost_per_second: 0.0, + materialized: false, + raw_retention: None, + }; + let profiles = cache + .iter() + .enumerate() + .map(|(id, priced)| { + let Some(priced) = priced else { + return Ok(zero()); + }; + let raw_retention = priced + .raw_scan + .map(|(source, bytes)| { + serde_json::to_string(source) + .map(|identity| (identity, bytes as f64 * prices.cost_per_disk_byte_second)) + .map_err(|error| DirectProfileError::Invalid(error.to_string())) + }) + .transpose()?; + Ok(NodeCostProfile { + execution_cost: priced.cost.cost_per_second.unwrap(), + work_seconds: priced.work_ms / 1000.0, + held_memory_cost_per_second: priced.held.map_or(0.0, |(bytes, _, _)| { + bytes as f64 * prices.cost_per_memory_byte_second + }), + materialized: nodes[id].as_ref().unwrap().materialization + != Materialization::NotMaterialized, + raw_retention, + }) + }) + .collect::, DirectProfileError>>()?; + let request_ms = settings.measured_operation_costs.ns_per_evaluation * 1e-6; + Ok(Some(DirectProfileExtraction { + profile: DirectCostProfile { + nodes: profiles, + query_rate: rate, + request_latency_seconds: request_ms / 1000.0, + request_cost: request_ms / settings.resource_speed.ms_per_cpu_op + * prices.cost_per_cpu_op + * rate + * context.entries.len() as f64, + }, + supported, + suppliers, + })) +} + +#[allow(clippy::too_many_arguments)] +fn price<'a>( + id: usize, + configured: &CandidateConfigurationInventory, + context: &'a CandidatePlanningContext<'_>, + nodes: &'a [Option], + lookbacks: &[Option], + cache: &mut [Option>], + visited: &mut [bool], + visiting: &mut [bool], + suppliers: &mut [Vec], + sketch_counts: &mut [usize], +) -> Result<(), DirectProfileError> { + check_deadline(context.deadline)?; + if visited[id] || visiting[id] { + return Ok(()); + } + let Some(node) = &nodes[id] else { + visited[id] = true; + return Ok(()); + }; + visiting[id] = true; + let requirements = &configured.inventory.graph.candidates[id].producer_requirements; + let options: Vec<_> = requirements.first().map_or_else( + || vec![None], + |input| { + input + .compatible_producers + .iter() + .copied() + .map(Some) + .collect() + }, + ); + for supplier in options { + let mut views = Vec::new(); + if let Some(supplier) = supplier { + if visiting[supplier.0] { + continue; + } + price( + supplier.0, + configured, + context, + nodes, + lookbacks, + cache, + visited, + visiting, + suppliers, + sketch_counts, + )?; + let Some(priced) = &cache[supplier.0] else { + continue; + }; + let producer = nodes[supplier.0].as_ref().unwrap(); + views.push(ProducerOutput { + payload: &producer.payload, + output_schema: &producer.output_schema, + output: priced.output, + sources: priced.sources.clone(), + family: priced.family, + unchanged_columns: priced.unchanged_columns.clone(), + }); + } + let sketches = supplier.map_or(0, |supplier| sketch_counts[supplier.0]) + + usize::from(matches!( + &node.payload, + asap_types::ir::Operator::ASAP(ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(..), + .. + }) + )); + // Failure-budget sizing splits delta across approximate inputs. This + // slice admits at most one sketch build along each unary output path. + if sketches > 1 { + continue; + } + let Ok(priced) = crate::price_node( + node, + &views, + (1, 1.0), + lookbacks[id], + 1, + None, + context.data, + context.models.plan_selection, + context.calibration.factor(), + ) else { + continue; + }; + // Compare the actual evaluator outputs, including every piece of the + // downstream pricing interface. This is not a template-row equivalence. + if cache[id] + .as_ref() + .is_some_and(|prior| !same_price(prior, &priced)) + { + continue; + } + if priced.cost.cost_per_second.is_none() + || priced.average_memory_bytes != 0.0 + || priced.disk_bytes != 0 + || priced.maintained.is_some() + { + continue; + } + if let Some(supplier) = supplier { + suppliers[id].push(supplier); + } + sketch_counts[id] = sketch_counts[id].max(sketches); + if cache[id].is_none() { + cache[id] = Some(priced); + } + } + visiting[id] = false; + visited[id] = true; + Ok(()) +} + +fn stable_size(node: &asap_types::ir::OperatorNode, delta: Option) -> bool { + let Some(ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(kind, _), + .. + }) = node.asap() + else { + return true; + }; + if !matches!( + kind.params(), + SketchParams::Kmv { + relative_error: Some(_), + .. + } + ) { + return false; + } + let Some(delta) = delta else { return false }; + let sized = asap_logical_asap_optimization::accuracy::estimators::with_failure_probability( + kind.params(), + delta, + ); + match (rmp_serde::to_vec(kind.params()), rmp_serde::to_vec(&sized)) { + (Ok(before), Ok(after)) => before == after, + _ => false, + } +} + +fn check_deadline(deadline: Option) -> Result<(), DirectProfileError> { + if deadline.is_some_and(|deadline| std::time::Instant::now() >= deadline) { + Err(asap_logical_asap_optimization::memo::CandidateGenerationTimeout.into()) + } else { + Ok(()) + } +} + +fn same_price(a: &PricedNode<'_>, b: &PricedNode<'_>) -> bool { + let family = match (a.family, b.family) { + (crate::StateFamily::None, crate::StateFamily::None) => true, + (crate::StateFamily::Exact(a), crate::StateFamily::Exact(b)) => a == b, + (crate::StateFamily::Sketch(a), crate::StateFamily::Sketch(b)) => { + // SketchParams PartialEq intentionally omits some confidence fields. + match (rmp_serde::to_vec(a), rmp_serde::to_vec(b)) { + (Ok(a), Ok(b)) => a == b, + _ => false, + } + } + _ => false, + }; + let bits = |p: &PricedNode<'_>| { + [ + p.work_ms.to_bits(), + p.cpu_ops_per_second.to_bits(), + p.scanned_bytes_per_second.to_bits(), + p.average_memory_bytes.to_bits(), + ] + }; + let held = |p: &PricedNode<'_>| { + p.held + .map(|(bytes, rate, per_reader)| (bytes, rate.to_bits(), per_reader)) + }; + family + && a.output == b.output + && a.sources == b.sources + && a.unchanged_columns == b.unchanged_columns + && bits(a) == bits(b) + && a.peak_memory_bytes == b.peak_memory_bytes + && a.disk_bytes == b.disk_bytes + && held(a) == held(b) + && a.maintained == b.maintained + && a.raw_scan == b.raw_scan + && a.cost.cost_per_second.map(f64::to_bits) == b.cost.cost_per_second.map(f64::to_bits) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::search::candidates::configurations::CandidateConfiguration; + use crate::search::candidates::graph::{ + CandidateDependencyGraph, PhysicalCandidate, ProducerRequirement, QueryRootCandidates, + }; + use crate::search::candidates::inventory::CandidateInventory; + use asap_logical_asap_optimization::memo::MemoGroupId; + use asap_logical_asap_optimization::test_support::lower_promql; + use asap_types::ir::properties::AccuracyTarget; + use asap_types::ir::{Operator, OperatorNode, ProjectItem, ScalarExpr}; + use asap_types::workload::{ + DataWorkload, Predictability, Query, QueryRequirements, QueryWorkloadEntry, + RepetitionInterval, + }; + + // Direct pricing must not certify a state that Stage 3 would enlarge. + #[test] + fn kmv_profiles_require_already_sized_failure_probability() { + use asap_types::ir::schema::{SketchAlgorithm, SketchKind}; + let mut node = asap_logical_asap_optimization::test_support::cms_summary("count(m)"); + let small = SketchParams::Kmv { + k: 2, + relative_error: Some(0.05), + }; + let sized = asap_logical_asap_optimization::accuracy::estimators::with_failure_probability( + &small, 0.05, + ); + let set = |node: &mut OperatorNode, params| { + let Operator::ASAP(ASAPOp::SummaryAgg { family, .. }) = &mut node.operator else { + panic!("summary fixture") + }; + let FieldDataType::Sketch(_, grouping) = family else { + panic!("sketch fixture") + }; + *family = FieldDataType::Sketch( + SketchKind::new(SketchAlgorithm::Kmv, params), + grouping.clone(), + ); + }; + set(&mut node, small); + assert!(!stable_size(&node, Some(0.05))); + set(&mut node, sized); + assert!(stable_size(&node, Some(0.05))); + assert!(!stable_size(&node, Some(0.00001))); + assert!(!stable_size(&node, None)); + } + + // Equal-priced plain-output consumers retain both exact and sketch paths. + // The sketch-count bound controls later resizing, not the price interface. + #[test] + fn plain_reader_keeps_exact_and_kmv_supplier_interfaces() { + use asap_types::ir::schema::{SketchAlgorithm, SketchKind, SketchStatistic}; + use std::rc::Rc; + let target = AccuracyTarget::max_error(0.05) + .failure_probability(0.05) + .accuracy_scope(AccuracyScope::PerValue); + let exact = OperatorNode::reachable(&lower_promql("count(m)", target.clone())) + .into_iter() + .find(|node| matches!(node.non_asap(), Some(NonASAPOp::Aggregate { .. }))) + .unwrap(); + let mut summary = asap_logical_asap_optimization::test_support::cms_summary("count(m)"); + let params = asap_logical_asap_optimization::accuracy::estimators::with_failure_probability( + &SketchParams::Kmv { + k: 2, + relative_error: Some(0.05), + }, + 0.05, + ); + let Operator::ASAP(ASAPOp::SummaryAgg { family, .. }) = &mut summary.operator else { + panic!("summary") + }; + let FieldDataType::Sketch(_, grouping) = family else { + panic!("sketch") + }; + *family = FieldDataType::Sketch( + SketchKind::new(SketchAlgorithm::Kmv, params), + grouping.clone(), + ); + let summary = OperatorNode::new_shared(summary.operator).unwrap(); + let estimate = Rc::new(OperatorNode::with_schema( + Operator::ASAP(ASAPOp::SummaryEstimate { + summary_input: summary, + query: SketchStatistic::Cardinality, + }), + exact.schema.clone(), + )); + let project = OperatorNode::new_shared(Operator::NonASAP(NonASAPOp::Project { + cols: exact + .schema + .fields + .iter() + .enumerate() + .map(|(index, field)| ProjectItem { + alias: Some(field.name.clone()), + expr: ScalarExpr::Column(index), + }) + .collect(), + qualifier: None, + child: exact.clone(), + })) + .unwrap(); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph::default(), + templates: vec![], + group_candidates: vec![], + }, + configurations: vec![], + }; + fn append( + node: Rc, + configured: &mut CandidateConfigurationInventory, + ) -> PhysicalCandidateId { + let mut inputs = Vec::new(); + node.operator + .map_children(|child| inputs.push(append(child.clone(), configured))); + let id = PhysicalCandidateId(configured.configurations.len()); + configured.configurations.push(CandidateConfiguration { + source_candidate: id, + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::QueryTime, + }); + configured.inventory.templates.push(Some(node)); + configured + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: MemoGroupId(id.0 as u32), + local_cost: 0.0, + producer_requirements: inputs + .into_iter() + .map(|supplier| ProducerRequirement { + compatible_producers: vec![supplier].into(), + }) + .collect(), + }); + id + } + let exact_id = append(exact, &mut configured); + let estimate_id = append(estimate, &mut configured); + let project_id = append(project.clone(), &mut configured); + configured.inventory.graph.candidates[project_id.0].producer_requirements[0] + .compatible_producers = vec![exact_id, estimate_id].into(); + configured + .inventory + .graph + .query_roots + .push(QueryRootCandidates { + compatible_candidates: vec![project_id], + }); + let roots = vec![QueryRoot::Operator(project)]; + let entries = vec![QueryWorkloadEntry { + query: Query("count(m)".into()), + requirements: QueryRequirements { + accuracy: target, + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval( + RepetitionInterval(60_000), + )), + range_evaluation: None, + }]; + let data: DataWorkload = serde_json::from_value(serde_json::json!({ "arrival": {"ingesting": {"rows_per_second":100.0}}, "sources":{"m":{"rows_held":1000,"series_count":1}} })).unwrap(); + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &roots, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let extracted = extract(&configured, &context).unwrap().unwrap(); + assert!(extracted.supported[exact_id.0] && extracted.supported[estimate_id.0]); + assert_eq!( + extracted.suppliers[project_id.0], + vec![exact_id, estimate_id] + ); + } + + // A template child is not evidence that every supplier has the same rows. + // Two deliberately different source alternatives exercise cost-interface + // filtering independently of the logical compatibility check upstream. + #[test] + fn different_supplier_statistics_are_not_priced_as_the_template_child() { + let scan = |metric| { + OperatorNode::reachable(&lower_promql(metric, AccuracyTarget::exact())) + .into_iter() + .find(|node| matches!(node.non_asap(), Some(NonASAPOp::Scan { .. }))) + .unwrap() + }; + let m = scan("m"); + let n = scan("n"); + let project = OperatorNode::new_shared(Operator::NonASAP(NonASAPOp::Project { + cols: m + .schema + .fields + .iter() + .enumerate() + .map(|(index, field)| ProjectItem { + alias: Some(field.name.clone()), + expr: ScalarExpr::Column(index), + }) + .collect(), + qualifier: None, + child: m.clone(), + })) + .unwrap(); + let configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph { + candidates: (0..3) + .map(|id| PhysicalCandidate { + group: MemoGroupId(id as u32), + local_cost: 0.0, + producer_requirements: if id == 2 { + vec![ProducerRequirement { + compatible_producers: vec![ + PhysicalCandidateId(0), + PhysicalCandidateId(1), + ] + .into(), + }] + } else { + vec![] + }, + }) + .collect(), + query_roots: vec![QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(2)], + }], + ..Default::default() + }, + templates: vec![Some(m), Some(n), Some(project.clone())], + group_candidates: vec![], + }, + configurations: (0..3) + .map(|id| CandidateConfiguration { + source_candidate: PhysicalCandidateId(id), + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::QueryTime, + }) + .collect(), + }; + let roots = vec![QueryRoot::Operator(project)]; + let entries = vec![QueryWorkloadEntry { + query: Query("m".into()), + requirements: QueryRequirements { + accuracy: AccuracyTarget::exact(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval( + RepetitionInterval(60_000), + )), + range_evaluation: None, + }]; + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival": {"ingesting": {"rows_per_second": 100.0}}, + "sources": {"m": {"rows_held":10,"series_count":1},"n":{"rows_held":100,"series_count":1}} + })).unwrap(); + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &roots, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let extracted = extract(&configured, &context).unwrap().unwrap(); + assert_eq!(extracted.supported, vec![true, true, true]); + assert_eq!(extracted.suppliers[2], vec![PhysicalCandidateId(0)]); + assert!( + extracted.profile.nodes[1].execution_cost > extracted.profile.nodes[0].execution_cost + ); + assert_eq!( + configured.inventory.graph.candidates[2].producer_requirements[0] + .compatible_producers + .len(), + 2, + "full catalog remains unchanged" + ); + let timed = CandidatePlanningContext { + deadline: Some(std::time::Instant::now()), + ..context + }; + assert!(matches!( + extract(&configured, &timed), + Err(DirectProfileError::DeadlineExceeded(_)) + )); + } +} diff --git a/crates/plan-selection/src/search/costing/mod.rs b/crates/plan-selection/src/search/costing/mod.rs index 93d6a380f..377519094 100644 --- a/crates/plan-selection/src/search/costing/mod.rs +++ b/crates/plan-selection/src/search/costing/mod.rs @@ -6,3 +6,5 @@ pub(crate) mod calibration; pub(crate) mod demand; pub(crate) mod lower_bounds; pub(crate) mod proposals; + +pub(crate) mod direct_profile; diff --git a/crates/plan-selection/src/search/milp/direct_cost.rs b/crates/plan-selection/src/search/milp/direct_cost.rs new file mode 100644 index 000000000..db6aa8a84 --- /dev/null +++ b/crates/plan-selection/src/search/milp/direct_cost.rs @@ -0,0 +1,626 @@ +//! Exact objective encoding for certified equal-schedule query-time profiles. +//! Profiles are supplied by physical costing; this module never infers prices +//! from a logical template or treats accuracy as a monetary penalty. +use good_lp::{constraint, variable, Constraint, Expression, ProblemVariables, Variable}; +use std::collections::BTreeMap; + +use crate::search::candidates::graph::*; + +/// Smallest/largest monetary coefficient ratio after lifetime normalization. +/// The minimum is scaled to one; no positive cost is rounded away. +pub(super) const MINIMUM_DIRECT_RELATIVE_COEFFICIENT: f64 = 1e-12; + +#[derive(Debug, Clone, Default)] +pub(crate) struct NodeCostProfile { + pub execution_cost: f64, + pub work_seconds: f64, + pub held_memory_cost_per_second: f64, + pub materialized: bool, + pub raw_retention: Option<(String, f64)>, +} + +#[derive(Debug, Clone)] +pub(crate) struct DirectCostProfile { + pub nodes: Vec, + pub query_rate: f64, + pub request_latency_seconds: f64, + /// Total request fee per second across all queries. + pub request_cost: f64, +} + +impl DirectCostProfile { + pub(crate) fn validate( + &self, + graph: &CandidateDependencyGraph, + ) -> Result<(), CandidateSelectionError> { + let valid = |v: f64| v.is_finite() && v >= 0.0; + if self.nodes.len() != graph.candidates.len() + || !valid(self.query_rate) + || self.query_rate == 0.0 + || !valid(self.request_latency_seconds) + || !valid(self.request_cost) + || self.nodes.iter().any(|n| { + !valid(n.execution_cost) + || !valid(n.work_seconds) + || !valid(n.held_memory_cost_per_second) + || n.raw_retention + .as_ref() + .is_some_and(|(_, cost)| !valid(*cost)) + }) + { + return Err(CandidateSelectionError::InvalidGraph( + "invalid direct-cost profile".into(), + )); + } + let maximum_latency = self.maximum_latency(); + if self.coefficients().any(|cost| !cost.is_finite()) + || !maximum_latency.is_finite() + || !(maximum_latency * graph.query_roots.len() as f64).is_finite() + { + return Err(CandidateSelectionError::InvalidGraph( + "direct-cost latency bound overflow".into(), + )); + } + Ok(()) + } + + /// Recompute the complete selected objective independently of LP auxiliaries. + /// A time-limited incumbent may leave nonminimal epigraph variables. + pub(crate) fn selected_cost( + &self, + graph: &CandidateDependencyGraph, + selected: &SelectedCandidateGraph, + ) -> f64 { + let mut inputs = vec![Vec::new(); self.nodes.len()]; + for edge in &selected.connections { + inputs[edge.consumer.0].push(edge.producer.0); + } + let mut reached = vec![vec![false; self.nodes.len()]; graph.query_roots.len()]; + for root in &selected.query_roots { + let mut pending = vec![root.candidate.0]; + while let Some(id) = pending.pop() { + if !std::mem::replace(&mut reached[root.query_root][id], true) { + pending.extend(&inputs[id]); + } + } + } + let latency: Vec = reached + .iter() + .map(|row| { + self.request_latency_seconds + + row + .iter() + .enumerate() + .filter(|(_, yes)| **yes) + .map(|(id, _)| self.nodes[id].work_seconds) + .sum::() + }) + .collect(); + let mut counts = vec![0usize; self.nodes.len()]; + let boundaries = selected + .query_roots + .iter() + .map(|root| vec![root.candidate.0]) + .chain( + selected + .selected_candidates + .iter() + .filter(|id| self.nodes[id.0].materialized) + .map(|id| inputs[id.0].clone()), + ); + for mut pending in boundaries { + let mut seen = vec![false; self.nodes.len()]; + while let Some(id) = pending.pop() { + if self.nodes[id].materialized || std::mem::replace(&mut seen[id], true) { + continue; + } + counts[id] += 1; + pending.extend(&inputs[id]); + } + } + let mut total = self.request_cost; + let mut raw = BTreeMap::<&str, f64>::new(); + for id in &selected.selected_candidates { + let node = &self.nodes[id.0]; + let readers: Vec<_> = reached + .iter() + .enumerate() + .filter(|(_, row)| row[id.0]) + .map(|(q, _)| latency[q]) + .collect(); + let executions = if node.materialized { 1 } else { counts[id.0] } as f64; + let lifetime = if node.materialized { + readers.iter().copied().fold(0.0, f64::max) + } else { + readers.iter().sum::() / readers.len().max(1) as f64 + }; + total += self.query_rate + * executions + * (node.execution_cost + node.held_memory_cost_per_second * lifetime); + if let Some((source, cost)) = &node.raw_retention { + let value = raw.entry(source).or_default(); + *value = value.max(*cost); + } + } + total + raw.values().sum::() + } + + fn maximum_latency(&self) -> f64 { + self.request_latency_seconds + self.nodes.iter().map(|n| n.work_seconds).sum::() + } + + pub(crate) fn coefficients(&self) -> impl Iterator + '_ { + // Constant request fees cannot influence selection and need not limit + // the coefficient range. Lifetimes use a dimensionless [0,1] scale. + let latency_scale = self.maximum_latency().max(f64::MIN_POSITIVE); + self.nodes.iter().flat_map(move |node| { + [ + node.execution_cost * self.query_rate, + node.held_memory_cost_per_second * self.query_rate * latency_scale, + node.raw_retention.as_ref().map_or(0.0, |(_, cost)| *cost), + ] + }) + } +} + +fn gated( + variables: &mut ProblemVariables, + constraints: &mut Vec, + value: Expression, + enabled: Variable, + maximum: f64, +) -> Variable { + let product = variables.add(variable().min(0).max(maximum)); + constraints.push(constraint!(product <= value.clone())); + constraints.push(constraint!(product <= maximum * enabled)); + constraints.push(constraint!(product >= value - maximum * (1.0 - enabled))); + product +} + +/// Exact on integer activation/connection decisions. Reach variables are +/// implied integral by the selected acyclic graph, as in the main model. +#[allow(clippy::too_many_arguments)] +pub(super) fn encode( + graph: &CandidateDependencyGraph, + profile: &DirectCostProfile, + variables: &mut ProblemVariables, + constraints: &mut Vec, + active: &[Variable], + edges: &[(ProducerConsumerConnection, Variable)], + roots: &[(QueryRootAssignment, Variable)], + reach: &[Vec>], + divisor: f64, + check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, +) -> Result { + let count = graph.candidates.len(); + let queries = graph.query_roots.len(); + let latency_scale = profile.maximum_latency().max(f64::MIN_POSITIVE); + let maximum_latency = 1.0; + let latency: Vec = reach + .iter() + .map(|row| { + Expression::from(profile.request_latency_seconds / latency_scale) + + row + .iter() + .enumerate() + .filter_map(|(id, reached)| { + reached.map(|r| profile.nodes[id].work_seconds / latency_scale * r) + }) + .sum::() + }) + .collect(); + let mut boundary_count = vec![Expression::from(0.0); count]; + let mut boundary_limit = vec![0usize; count]; + // Each root query and each selected materialized computation is one + // independent execution boundary. Materialized nodes stop propagation. + for boundary in 0..queries + count { + check_deadline()?; + if boundary >= queries && !profile.nodes[boundary - queries].materialized { + continue; + } + let mut possible = vec![false; count]; + let mut pending = if boundary < queries { + graph.query_roots[boundary].compatible_candidates.clone() + } else { + graph.candidates[boundary - queries] + .producer_requirements + .iter() + .flat_map(|input| input.compatible_producers.iter().copied()) + .collect() + }; + while let Some(id) = pending.pop() { + check_deadline()?; + if profile.nodes[id.0].materialized || std::mem::replace(&mut possible[id.0], true) { + continue; + } + pending.extend( + graph.candidates[id.0] + .producer_requirements + .iter() + .flat_map(|input| input.compatible_producers.iter().copied()), + ); + } + let used: Vec<_> = possible + .iter() + .map(|possible| possible.then(|| variables.add(variable().min(0).max(1)))) + .collect(); + let mut support = vec![Expression::from(0.0); count]; + if boundary < queries { + for (root, selected) in roots.iter().filter(|(root, _)| root.query_root == boundary) { + if let Some(reached) = used[root.candidate.0] { + constraints.push(constraint!(*selected <= reached)); + support[root.candidate.0] += *selected; + } + } + } + for (edge, connected) in edges { + check_deadline()?; + let Some(producer) = used[edge.producer.0] else { + continue; + }; + if boundary >= queries && edge.consumer.0 == boundary - queries { + constraints.push(constraint!(*connected <= producer)); + support[edge.producer.0] += *connected; + } else if let Some(consumer) = used[edge.consumer.0] { + let traversed = variables.add(variable().min(0).max(1)); + constraints.push(constraint!(traversed <= consumer)); + constraints.push(constraint!(traversed <= *connected)); + constraints.push(constraint!(traversed >= consumer + *connected - 1)); + constraints.push(constraint!(traversed <= producer)); + support[edge.producer.0] += traversed; + } + } + for (id, used) in used.into_iter().enumerate() { + if let Some(used) = used { + boundary_limit[id] += 1; + constraints.push(constraint!(used <= support[id].clone())); + constraints.push(constraint!(used <= active[id])); + boundary_count[id] += used; + } + } + } + let mut objective = Expression::from(profile.request_cost / divisor); + let mut retained: BTreeMap<&str, Vec<(usize, f64)>> = BTreeMap::new(); + for (id, node) in profile.nodes.iter().enumerate() { + check_deadline()?; + let executions = if node.materialized { + Expression::from(active[id]) + } else { + boundary_count[id].clone() + }; + objective += node.execution_cost * profile.query_rate / divisor * executions.clone(); + if let Some((source, cost)) = &node.raw_retention { + retained.entry(source).or_default().push((id, *cost)); + } + if node.held_memory_cost_per_second == 0.0 { + continue; + } + let held_latency = if node.materialized { + let longest = variables.add(variable().min(0).max(maximum_latency)); + for (query, row) in reach.iter().enumerate() { + if let Some(reached) = row[id] { + constraints.push(constraint!( + longest >= latency[query].clone() - maximum_latency * (1.0 - reached) + )); + } + } + Expression::from(longest) + } else { + let sum: Expression = reach + .iter() + .enumerate() + .filter_map(|(query, row)| { + row[id].map(|reached| { + gated( + variables, + constraints, + latency[query].clone(), + reached, + maximum_latency, + ) + }) + }) + .sum(); + let reader_count: Expression = reach.iter().filter_map(|row| row[id]).sum(); + let sizes: Vec<_> = (0..=queries) + .map(|_| variables.add(variable().binary())) + .collect(); + constraints.push(constraint!(sizes.iter().copied().sum::() == 1)); + constraints.push(constraint!( + sizes + .iter() + .enumerate() + .map(|(k, v)| k as f64 * *v) + .sum::() + == reader_count + )); + let mut weighted = Expression::from(0.0); + // The boundary count is an integer in [0, roots + states]. Gate + // its possible values, then multiply only by a bounded continuous + // lifetime. This reproduces B/Q without dividing by a variable. + let boundaries: Vec<_> = (0..=boundary_limit[id]) + .map(|_| variables.add(variable().binary())) + .collect(); + constraints.push(constraint!( + boundaries.iter().copied().sum::() == 1 + )); + constraints.push(constraint!( + boundaries + .iter() + .enumerate() + .map(|(b, v)| b as f64 * *v) + .sum::() + == executions + )); + for (k, size) in sizes.iter().enumerate().skip(1) { + check_deadline()?; + let selected_sum = gated( + variables, + constraints, + sum.clone(), + *size, + maximum_latency * queries as f64, + ); + for (b, boundary) in boundaries.iter().enumerate().skip(1) { + check_deadline()?; + let value = gated( + variables, + constraints, + selected_sum.into(), + *boundary, + maximum_latency * queries as f64, + ); + weighted += b as f64 / k as f64 * value; + } + } + weighted + }; + objective += node.held_memory_cost_per_second * profile.query_rate * latency_scale + / divisor + * held_latency; + } + for entries in retained.into_values() { + let cost = variables.add(variable().min(0)); + for (id, bytes_cost) in entries { + constraints.push(constraint!(cost >= bytes_cost / divisor * active[id])); + } + objective += cost; + } + Ok(objective) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::search::milp::model::{ + solve_direct_cost_proposal_with_metrics, CandidateSolveMetrics, + }; + use asap_logical_asap_optimization::memo::MemoGroupId; + + fn node(input: Option) -> PhysicalCandidate { + PhysicalCandidate { + group: MemoGroupId(0), + local_cost: 0.0, + producer_requirements: input + .into_iter() + .map(|id| ProducerRequirement { + compatible_producers: vec![PhysicalCandidateId(id)].into(), + }) + .collect(), + } + } + fn root(id: usize) -> QueryRootCandidates { + QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(id)], + } + } + fn solve(graph: &CandidateDependencyGraph, profile: &DirectCostProfile) -> f64 { + solve_direct_cost_proposal_with_metrics( + graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + profile, + ) + .unwrap() + .unwrap() + .model_lower_bound + .unwrap() + } + + // Shared build demand is one; unmaterialized demand is two. Raw data is + // retained once at the maximum horizon, while fees remain per workload. + #[test] + fn complete_cost_counts_shared_demand_held_memory_retention_and_fees() { + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(Some(0)), node(Some(0))], + query_roots: vec![root(1), root(2)], + ..Default::default() + }; + let mut profile = DirectCostProfile { + query_rate: 2.0, + request_latency_seconds: 0.5, + request_cost: 7.0, + nodes: vec![ + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 2.0, + raw_retention: Some(("source".into(), 4.0)), + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 3.0, + raw_retention: Some(("source".into(), 6.0)), + ..Default::default() + }, + ], + }; + assert!((solve(&graph, &profile) - 48.0).abs() < 1e-7); + profile.nodes[0].materialized = false; + assert!((solve(&graph, &profile) - 73.0).abs() < 1e-7); + } + + // Two queries behind one materialized computation produce B=1,Q=2 for + // its unmaterialized input. Charging it once per query would double memory. + #[test] + fn upstream_lifetime_uses_boundary_count_divided_by_query_count() { + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(Some(0)), node(Some(1)), node(Some(1))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + let profile = DirectCostProfile { + query_rate: 2.0, + request_latency_seconds: 0.5, + request_cost: 7.0, + nodes: vec![ + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 2.0, + held_memory_cost_per_second: 4.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + work_seconds: 3.0, + ..Default::default() + }, + NodeCostProfile { + work_seconds: 4.0, + ..Default::default() + }, + ], + }; + assert!((solve(&graph, &profile) - 115.0).abs() < 1e-7); + } + // Independently enumerate both readers' choices. Sharing changes execution + // counts and held lifetime; the least-node graph alone cannot price these. + #[test] + fn direct_selection_matches_four_assignment_cost_oracle() { + let mut graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None), node(Some(0)), node(Some(0))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + for reader in [2, 3] { + graph.candidates[reader].producer_requirements[0].compatible_producers = + vec![PhysicalCandidateId(0), PhysicalCandidateId(1)].into(); + } + for materialized_build_cost in [1.0, 10.0, 100.0] { + let profile = DirectCostProfile { + query_rate: 2.0, + request_latency_seconds: 0.5, + request_cost: 7.0, + nodes: vec![ + NodeCostProfile { + execution_cost: materialized_build_cost, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 0.5, + held_memory_cost_per_second: 1.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 2.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 3.0, + ..Default::default() + }, + ], + }; + let mut minimum = f64::INFINITY; + for assignment in [[0, 0], [0, 1], [1, 0], [1, 1]] { + let durations = [ + 2.5 + [1.0, 0.5][assignment[0]], + 3.5 + [1.0, 0.5][assignment[1]], + ]; + let mut cost = 7.0 + 4.0; + for producer in [0, 1] { + let times: Vec = assignment + .iter() + .enumerate() + .filter(|(_, supplier)| **supplier == producer) + .map(|(q, _)| durations[q]) + .collect(); + if times.is_empty() { + continue; + } + cost += if producer == 0 { + 2.0 * (materialized_build_cost + + 3.0 * times.iter().copied().fold(0.0, f64::max)) + } else { + 2.0 * (2.0 * times.len() as f64 + times.iter().sum::()) + }; + } + minimum = minimum.min(cost); + } + assert!((solve(&graph, &profile) - minimum).abs() < 1e-7); + } + } + // Small reader fees remain meaningful alongside a 2.2-billion-times larger + // possible raw-buffer cost; currency normalization must not drop the fee. + #[test] + fn direct_currency_range_preserves_tiny_meaningful_costs() { + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None)], + query_roots: vec![QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(0), PhysicalCandidateId(1)], + }], + ..Default::default() + }; + let profile = DirectCostProfile { + nodes: vec![ + NodeCostProfile { + execution_cost: 7.1e-13, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 7.1e-13 * 2.2e9, + ..Default::default() + }, + ], + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 1.2e-15, + }; + let proposal = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert_eq!( + proposal.selected.selected_candidates, + vec![PhysicalCandidateId(0)] + ); + let expected = 7.1e-13 + 1.2e-15; + assert!((proposal.direct_modeled_cost.unwrap() / expected - 1.0).abs() < 1e-9); + assert!((proposal.model_lower_bound.unwrap() / expected - 1.0).abs() < 1e-9); + } +} diff --git a/crates/plan-selection/src/search/milp/mod.rs b/crates/plan-selection/src/search/milp/mod.rs index 48ad39e10..d459feae3 100644 --- a/crates/plan-selection/src/search/milp/mod.rs +++ b/crates/plan-selection/src/search/milp/mod.rs @@ -1,2 +1,4 @@ //! MILP encoding and complete feasible initialization. pub(crate) mod model; + +pub(crate) mod direct_cost; diff --git a/crates/plan-selection/src/search/milp/model.rs b/crates/plan-selection/src/search/milp/model.rs index 0ae521214..15d9f2b1f 100644 --- a/crates/plan-selection/src/search/milp/model.rs +++ b/crates/plan-selection/src/search/milp/model.rs @@ -34,6 +34,7 @@ use good_lp::{ /// at a constrained producer and follows possible consumers back toward roots. fn required_reader_reach( graph: &CandidateDependencyGraph, + all_outputs: bool, check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, ) -> Result>, CandidateSelectionError> { let count = graph.candidates.len(); @@ -61,6 +62,9 @@ fn required_reader_reach( } } } + if all_outputs { + return Ok(potential); + } let mut needed = vec![vec![false; count]; graph.query_roots.len()]; let mut pending = Vec::new(); for constraint in &graph.reader_constraints { @@ -146,6 +150,8 @@ pub(crate) struct ObservedAssignment { #[derive(Debug)] pub(crate) struct CandidateProposal { pub selected: SelectedCandidateGraph, + /// Complete modeled cost of this selected DAG, only for a certified direct profile. + pub direct_modeled_cost: Option, /// Certified optimal relaxation objective, distinct from the local sum. /// A feasible gap/time-limited incumbent objective is NOT a lower bound. pub model_lower_bound: Option, @@ -157,6 +163,10 @@ pub(crate) struct CandidateSolveMetrics { pub model_construction_seconds: f64, pub solver_seconds: f64, pub solve_calls: usize, + pub direct_cost_solve_calls: usize, + pub direct_cost_validated_plans: usize, + pub direct_cost_best_validated_cost: Option, + pub direct_cost_fallback_reasons: Vec, /// Largest model submitted across refinement rounds, before solver presolve. pub peak_variables: usize, pub peak_constraints: usize, @@ -184,9 +194,38 @@ pub(crate) fn solve_candidate_proposal_with_metrics( observations: &[ObservedAssignment], remaining_seconds: Option, metrics: &mut CandidateSolveMetrics, +) -> Result, CandidateSelectionError> { + solve_with_objective(graph, observations, remaining_seconds, metrics, None) +} + +pub(crate) fn solve_direct_cost_proposal_with_metrics( + graph: &CandidateDependencyGraph, + observations: &[ObservedAssignment], + remaining_seconds: Option, + metrics: &mut CandidateSolveMetrics, + profile: &super::direct_cost::DirectCostProfile, +) -> Result, CandidateSelectionError> { + solve_with_objective( + graph, + observations, + remaining_seconds, + metrics, + Some(profile), + ) +} + +fn solve_with_objective( + graph: &CandidateDependencyGraph, + observations: &[ObservedAssignment], + remaining_seconds: Option, + metrics: &mut CandidateSolveMetrics, + direct_profile: Option<&super::direct_cost::DirectCostProfile>, ) -> Result, CandidateSelectionError> { let started = std::time::Instant::now(); graph.validate()?; + if let Some(profile) = direct_profile { + profile.validate(graph)?; + } if remaining_seconds.is_some_and(|seconds| !seconds.is_finite() || seconds <= 0.0) { return Err(CandidateSelectionError::Solver( "no positive solver time remains".into(), @@ -231,6 +270,7 @@ pub(crate) fn solve_candidate_proposal_with_metrics( return Ok(None); } return Ok(Some(CandidateProposal { + direct_modeled_cost: direct_profile.map(|profile| profile.request_cost), selected: SelectedCandidateGraph { selected_candidates: Vec::new(), connections: Vec::new(), @@ -241,7 +281,10 @@ pub(crate) fn solve_candidate_proposal_with_metrics( observations .iter() .filter_map(|observation| observation.exact_cost) - .fold(0.0, f64::max), + .fold( + direct_profile.map_or(0.0, |profile| profile.request_cost), + f64::max, + ), ), })); } @@ -253,7 +296,11 @@ pub(crate) fn solve_candidate_proposal_with_metrics( return Ok(None); } - let warm_start = warm_start::cheapest_start(graph, observations); + // Direct objective auxiliaries have no complete feasible start yet. + let warm_start = direct_profile + .is_none() + .then(|| warm_start::cheapest_start(graph, observations)) + .flatten(); let count = graph.candidates.len(); let mut variables = ProblemVariables::new(); let candidate_selected: Vec = (0..count) @@ -347,7 +394,8 @@ pub(crate) fn solve_candidate_proposal_with_metrics( } constraints.push(constraint!(root_selected == 1)); } - if !graph.reader_constraints.is_empty() { + let mut objective_reach = Vec::new(); + if !graph.reader_constraints.is_empty() || direct_profile.is_some() { // Reachability is implied integral: roots and selected edges are binary, // and selected edges form a DAG. Induction from query roots makes each // AND/OR below exactly zero or one, so these auxiliaries need no integer @@ -356,9 +404,10 @@ pub(crate) fn solve_candidate_proposal_with_metrics( // edges determine actual reach; incompatible potential readers do not // rule out a candidate used by a compatible subset. let mut reach = Vec::with_capacity(graph.query_roots.len()); - for (output, potential) in required_reader_reach(graph, check_deadline)? - .into_iter() - .enumerate() + for (output, potential) in + required_reader_reach(graph, direct_profile.is_some(), check_deadline)? + .into_iter() + .enumerate() { reach.push( potential @@ -419,6 +468,9 @@ pub(crate) fn solve_candidate_proposal_with_metrics( } } } + if direct_profile.is_some() { + objective_reach = reach.clone(); + } for requirement in &graph.reader_constraints { match requirement { ReaderConstraint::ActivationRequires { @@ -523,6 +575,10 @@ pub(crate) fn solve_candidate_proposal_with_metrics( .iter() .map(|c| c.local_cost) .fold(0.0, f64::max); + let maximum_cost = direct_profile + .into_iter() + .flat_map(|profile| profile.coefficients()) + .fold(maximum_cost, f64::max); let maximum_cost = observations .iter() .filter_map(|observation| observation.exact_cost) @@ -540,9 +596,23 @@ pub(crate) fn solve_candidate_proposal_with_metrics( .iter() .filter_map(|observation| observation.exact_cost), ) + .chain( + direct_profile + .into_iter() + .flat_map(|profile| profile.coefficients()), + ) .filter(|cost| *cost > 0.0) .reduce(f64::min) .unwrap_or(1.0); + // Direct held-memory profiles can span tiny reader buffers and large raw + // windows. Their lifetimes are dimensionless and the smallest currency + // coefficient is scaled to one; permit up to 1e12 here without rounding + // away fees. The fixed-node graph keeps its original stricter range. + let relative_coefficient_limit = if direct_profile.is_some() { + super::direct_cost::MINIMUM_DIRECT_RELATIVE_COEFFICIENT + } else { + MINIMUM_RELATIVE_COEFFICIENT + }; if graph .candidates .iter() @@ -552,11 +622,30 @@ pub(crate) fn solve_candidate_proposal_with_metrics( .iter() .filter_map(|observation| observation.exact_cost), ) - .any(|cost| cost > 0.0 && cost / maximum_cost < MINIMUM_RELATIVE_COEFFICIENT) + .chain( + direct_profile + .into_iter() + .flat_map(|profile| profile.coefficients()), + ) + .any(|cost| cost > 0.0 && cost / maximum_cost < relative_coefficient_limit) { - return Err(CandidateSelectionError::InvalidGraph( - "observed costs and local bounds exceed supported numerical range".into(), - )); + let local_minimum = graph + .candidates + .iter() + .map(|candidate| candidate.local_cost) + .filter(|cost| *cost > 0.0) + .reduce(f64::min); + let direct_minimum = direct_profile + .into_iter() + .flat_map(|profile| profile.coefficients()) + .filter(|cost| *cost > 0.0) + .reduce(f64::min); + let reason = format!("observed costs and objective coefficients exceed supported numerical range: min={divisor:e}, max={maximum_cost:e}, local_min={local_minimum:?}, direct_min={direct_minimum:?}"); + return Err(if direct_profile.is_some() { + CandidateSelectionError::UnsupportedDirectCost(reason) + } else { + CandidateSelectionError::InvalidGraph(reason) + }); } let local_bound: Expression = graph .candidates @@ -569,6 +658,21 @@ pub(crate) fn solve_candidate_proposal_with_metrics( warm_start.as_ref().map(|start| start.cost / divisor), )); constraints.push(constraint!(total_cost >= local_bound)); + if let Some(profile) = direct_profile { + let objective = super::direct_cost::encode( + graph, + profile, + &mut variables, + &mut constraints, + &candidate_selected, + &connections, + &root_assignments, + &objective_reach, + divisor, + check_deadline, + )?; + constraints.push(constraint!(total_cost >= objective)); + } for observation in observations { check_deadline()?; for witness in &observation.rejected_queries { @@ -661,6 +765,7 @@ pub(crate) fn solve_candidate_proposal_with_metrics( } let solve_started = std::time::Instant::now(); metrics.solve_calls += 1; + metrics.direct_cost_solve_calls += usize::from(direct_profile.is_some()); let result = model.solve(); metrics.solver_seconds += solve_started.elapsed().as_secs_f64(); let solution = match result { @@ -701,13 +806,17 @@ pub(crate) fn solve_candidate_proposal_with_metrics( .collect(), modeled_cost, }; + let direct_modeled_cost = direct_profile.map(|profile| profile.selected_cost(graph, &selected)); decoded_proposal( graph, selected, solution.status(), solution.value(total_cost) * divisor, ) - .map(Some) + .map(|mut proposal| { + proposal.direct_modeled_cost = direct_modeled_cost; + Some(proposal) + }) } fn validate_binary_value(value: f64) -> Result<(), CandidateSelectionError> { @@ -732,6 +841,7 @@ fn decoded_proposal( )); } Ok(CandidateProposal { + direct_modeled_cost: None, selected, model_lower_bound: matches!(status, SolutionStatus::Optimal).then_some(objective), }) diff --git a/crates/plan-selection/src/search/refinement/components.rs b/crates/plan-selection/src/search/refinement/components.rs index 8141ab858..fcfca13b2 100644 --- a/crates/plan-selection/src/search/refinement/components.rs +++ b/crates/plan-selection/src/search/refinement/components.rs @@ -437,6 +437,23 @@ pub(in crate::search) fn optimize_independent_components( summary.solve_metrics.model_construction_seconds += metrics.model_construction_seconds; summary.solve_metrics.solver_seconds += metrics.solver_seconds; summary.solve_metrics.solve_calls += metrics.solve_calls; + summary.solve_metrics.direct_cost_solve_calls += metrics.direct_cost_solve_calls; + summary.solve_metrics.direct_cost_validated_plans += metrics.direct_cost_validated_plans; + // Component proposal costs are not complete workload costs; do not + // expose their minimum as the price of a validated whole-workload plan. + for reason in &metrics.direct_cost_fallback_reasons { + if summary.solve_metrics.direct_cost_fallback_reasons.len() < 8 + && !summary + .solve_metrics + .direct_cost_fallback_reasons + .contains(reason) + { + summary + .solve_metrics + .direct_cost_fallback_reasons + .push(reason.clone()); + } + } summary.solve_metrics.peak_variables = summary .solve_metrics .peak_variables diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index c816b8354..5cfa95d95 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -13,7 +13,7 @@ use crate::search::candidates::physical::{ materialize_selected_candidates, PhysicalCandidateError, }; use crate::search::refinement::search::{ - search_candidates_with_restricted_feedback, CandidateEvaluation, CandidateSearchLimits, + search_candidates_with_direct_feedback, CandidateEvaluation, CandidateSearchLimits, CandidateSearchResult, }; use crate::{CalibrationFit, CostModel, PlanSelector, PlanningModels, Selection, SelectionError}; @@ -122,7 +122,8 @@ pub(in crate::search) fn optimize_configured_candidates( } else { Vec::new() }; - let search = search_candidates_with_restricted_feedback( + let direct = direct_cost_seed(configured, context, &exclusions)?; + let search = search_candidates_with_direct_feedback( &configured.inventory.graph, CandidateSearchLimits { wall_time: limits.wall_time.saturating_sub(weighting_started.elapsed()), @@ -131,6 +132,7 @@ pub(in crate::search) fn optimize_configured_candidates( &weights, &exclusions, &changes, + direct.as_ref().map(|(graph, profile)| (graph, profile)), |selected| { evaluation_number += 1; let mut rejected_queries = Vec::new(); @@ -183,9 +185,82 @@ pub(in crate::search) fn optimize_configured_candidates( }) } -/// Prefer contracts that can survive downstream value computations. These are -/// proposal priorities, not feasibility rules: recomposition and evaluation -/// still decide the selected graph's actual accuracy. +/// Restrict only the direct-price proposal. Unsupported configurations and +/// supplier interfaces remain available in the full evaluator-refined search. +fn direct_cost_seed( + configured: &CandidateConfigurationInventory, + context: &CandidatePlanningContext<'_>, + seed_exclusions: &[crate::search::candidates::graph::PhysicalCandidateId], +) -> Result< + Option<( + crate::search::candidates::graph::CandidateDependencyGraph, + crate::search::milp::direct_cost::DirectCostProfile, + )>, + String, +> { + use crate::search::candidates::graph::{PhysicalCandidateId, ReaderConstraint}; + use crate::search::costing::direct_profile::{extract, DirectProfileError}; + let mut extracted = match extract(configured, context) { + Ok(Some(extracted)) => extracted, + Ok(None) | Err(DirectProfileError::DeadlineExceeded(_)) => return Ok(None), + Err(error) => return Err(error.to_string()), + }; + if context + .deadline + .is_some_and(|deadline| std::time::Instant::now() >= deadline) + { + return Ok(None); + } + let mut graph = configured.inventory.graph.clone(); + let mut supported = extracted.supported; + // One interchangeable execution-scope copy is sufficient for this seed. + // Keep all copies in the full graph, where independent input bindings may + // require them; avoiding their permutations bounds the direct-price solve. + for constraint in &graph.reader_constraints { + if let ReaderConstraint::ActivationRequires { candidate, .. } = constraint { + supported[candidate.0] = false; + } + } + for excluded in seed_exclusions { + supported[excluded.0] = false; + } + for (id, candidate) in graph.candidates.iter_mut().enumerate() { + if context + .deadline + .is_some_and(|deadline| std::time::Instant::now() >= deadline) + { + return Ok(None); + } + if !supported[id] { + extracted.profile.nodes[id] = Default::default(); + candidate.local_cost = 0.0; + candidate.producer_requirements.clear(); + graph + .reader_constraints + .push(ReaderConstraint::AllowedOutputs { + candidate: PhysicalCandidateId(id), + outputs: Vec::new(), + }); + } else { + for input in &mut candidate.producer_requirements { + input.compatible_producers.retain(|supplier| { + supported[supplier.0] && extracted.suppliers[id].contains(supplier) + }); + } + } + } + for root in &mut graph.query_roots { + root.compatible_candidates + .retain(|candidate| supported[candidate.0]); + if root.compatible_candidates.is_empty() { + return Ok(None); + } + } + Ok(Some((graph, extracted.profile))) +} + +/// Price work and state for proposal ordering without an exactness surcharge. +/// Recomposition and validation still decide the selected graph's accuracy. pub(in crate::search) fn accuracy_baseline_weights( configured: &CandidateConfigurationInventory, context: &CandidatePlanningContext<'_>, diff --git a/crates/plan-selection/src/search/refinement/search.rs b/crates/plan-selection/src/search/refinement/search.rs index 145a60c77..739d97683 100644 --- a/crates/plan-selection/src/search/refinement/search.rs +++ b/crates/plan-selection/src/search/refinement/search.rs @@ -2,6 +2,7 @@ //! complete plans up front. Certified query cuts reuse size-independent accuracy //! failures; full-assignment cuts remain a correctness fallback; //! they make no claim of scalable demand modeling or support for every cost model. +use crate::search::milp::direct_cost::DirectCostProfile; use std::time::{Duration, Instant}; use crate::search::candidates::graph::{ @@ -9,8 +10,8 @@ use crate::search::candidates::graph::{ SelectedCandidateGraph, }; use crate::search::milp::model::{ - solve_candidate_proposal_with_metrics, CandidateProposal, CandidateSolveMetrics, - ObservedAssignment, + solve_candidate_proposal_with_metrics, solve_direct_cost_proposal_with_metrics, + CandidateProposal, CandidateSolveMetrics, ObservedAssignment, }; #[derive(Debug, Default)] @@ -76,6 +77,7 @@ pub(in crate::search) struct CandidateSearchResult { struct SeedRestrictions<'a> { excluded: &'a [PhysicalCandidateId], materialization_changes: &'a [(PhysicalCandidateId, PhysicalCandidateId)], + direct: Option<(&'a CandidateDependencyGraph, &'a DirectCostProfile)>, } /// The callback returns an actual total cost for a validated feasible plan, or @@ -143,6 +145,7 @@ pub(in crate::search) fn search_candidates_with_feedback( ) } +#[cfg(test)] pub(in crate::search) fn search_candidates_with_restricted_feedback( graph: &CandidateDependencyGraph, limits: CandidateSearchLimits, @@ -150,6 +153,26 @@ pub(in crate::search) fn search_candidates_with_restricted_feedback( excluded_from_seed: &[PhysicalCandidateId], materialization_changes: &[(PhysicalCandidateId, PhysicalCandidateId)], evaluate: impl FnMut(&SelectedCandidateGraph) -> Result, +) -> Result { + search_candidates_with_direct_feedback( + graph, + limits, + weights, + excluded_from_seed, + materialization_changes, + None, + evaluate, + ) +} + +pub(in crate::search) fn search_candidates_with_direct_feedback( + graph: &CandidateDependencyGraph, + limits: CandidateSearchLimits, + weights: &[f64], + excluded_from_seed: &[PhysicalCandidateId], + materialization_changes: &[(PhysicalCandidateId, PhysicalCandidateId)], + direct: Option<(&CandidateDependencyGraph, &DirectCostProfile)>, + evaluate: impl FnMut(&SelectedCandidateGraph) -> Result, ) -> Result { search_with_structural_baseline( graph, @@ -159,6 +182,7 @@ pub(in crate::search) fn search_candidates_with_restricted_feedback( SeedRestrictions { excluded: excluded_from_seed, materialization_changes, + direct, }, evaluate, ) @@ -276,6 +300,10 @@ fn search_with_solver( // capped at two seconds; stop the baseline once it supplies an incumbent. let baseline_budget = (limits.wall_time / 5).min(Duration::from_secs(2)); let mut baseline_attempts = 0; + let mut baseline_started: Option = None; + let mut direct_attempts = 0; + let mut direct_finished = restrictions.direct.is_none(); + let direct_budget = (limits.wall_time / 3).min(Duration::from_secs(2)); // Each local proposal uses the same solver and evaluator. Limit the phase // to four validations and one fifth of this domain's existing wall budget. let neighborhood_budget = (limits.wall_time / 5).min(Duration::from_secs(1)); @@ -290,15 +318,26 @@ fn search_with_solver( if refinements >= limits.max_refinements { break CandidateSearchStatus::RefinementLimit; } - let use_baseline = heuristic_graph.is_some() - && incumbent.is_none() + let use_direct = + !direct_finished && direct_attempts < 3 && started.elapsed() < direct_budget; + if !use_direct { + baseline_started.get_or_insert_with(Instant::now); + } + let baseline_elapsed = baseline_started.map_or(Duration::ZERO, |start| start.elapsed()); + let use_baseline = !use_direct + && heuristic_graph.is_some() + && (incumbent.is_none() || solve_metrics.direct_cost_validated_plans > 0) && baseline_attempts < 3 - && started.elapsed() < baseline_budget; + && baseline_elapsed < baseline_budget; let neighborhood_started = Instant::now(); let neighborhood_deadline = neighborhood_started + remaining.min(neighborhood_budget.saturating_sub(neighborhood_spent)); let mut neighborhood = None; - if !use_baseline && neighborhood_attempts < 4 && neighborhood_spent < neighborhood_budget { + if !use_direct + && !use_baseline + && neighborhood_attempts < 4 + && neighborhood_spent < neighborhood_budget + { if let Some(best) = &incumbent { for &(from, to) in restrictions.materialization_changes { if Instant::now() >= neighborhood_deadline { @@ -331,58 +370,95 @@ fn search_with_solver( if remaining.is_zero() { break CandidateSearchStatus::TimeLimit; } - let (proposal_graph, proposal_observations, solve_budget) = - if let Some(restricted) = &neighborhood { - ( - restricted, - Vec::new(), - remaining.min(neighborhood_deadline.saturating_duration_since(Instant::now())), - ) - } else if use_baseline { - baseline_attempts += 1; - // A structural proposal is never repriced by monetary observations. - // Tried assignments are skipped here only; their validated costs - // remain in the certified model used after this heuristic phase. - let skipped = observations - .iter() - .map(|observation: &ObservedAssignment| { - let mut selected = observation.selected.clone(); - selected.modeled_cost = selected - .selected_candidates - .iter() - .map(|id| { - heuristic_graph - .as_ref() - .expect("baseline graph exists") - .candidates[id.0] - .local_cost - }) - .sum(); - ObservedAssignment { - selected, - exact_cost: None, - rejected_queries: observation.rejected_queries.clone(), - } - }) - .collect::>(); - ( - heuristic_graph.as_ref().expect("baseline graph exists"), - skipped, - remaining.min(baseline_budget.saturating_sub(started.elapsed())), - ) - } else { - (graph, observations.clone(), remaining) - }; + let (proposal_graph, proposal_observations, solve_budget) = if use_direct { + direct_attempts += 1; + ( + restrictions.direct.expect("direct profile").0, + observations.clone(), + remaining.min(direct_budget.saturating_sub(started.elapsed())), + ) + } else if let Some(restricted) = &neighborhood { + ( + restricted, + Vec::new(), + remaining.min(neighborhood_deadline.saturating_duration_since(Instant::now())), + ) + } else if use_baseline { + baseline_attempts += 1; + // A structural proposal is never repriced by monetary observations. + // Tried assignments are skipped here only; their validated costs + // remain in the certified model used after this heuristic phase. + let skipped = observations + .iter() + .map(|observation: &ObservedAssignment| { + let mut selected = observation.selected.clone(); + selected.modeled_cost = selected + .selected_candidates + .iter() + .map(|id| { + heuristic_graph + .as_ref() + .expect("baseline graph exists") + .candidates[id.0] + .local_cost + }) + .sum(); + ObservedAssignment { + selected, + exact_cost: None, + rejected_queries: observation.rejected_queries.clone(), + } + }) + .collect::>(); + ( + heuristic_graph.as_ref().expect("baseline graph exists"), + skipped, + remaining.min(baseline_budget.saturating_sub(baseline_elapsed)), + ) + } else { + (graph, observations.clone(), remaining) + }; if use_neighborhood && solve_budget.is_zero() { neighborhood_spent += neighborhood_started.elapsed(); continue; } - let mut proposal = match propose( - proposal_graph, - &proposal_observations, - Some(solve_budget.as_secs_f64()), - &mut solve_metrics, - ) { + let proposed = if use_direct { + solve_direct_cost_proposal_with_metrics( + proposal_graph, + &proposal_observations, + Some(solve_budget.as_secs_f64()), + &mut solve_metrics, + restrictions.direct.expect("direct profile").1, + ) + } else { + propose( + proposal_graph, + &proposal_observations, + Some(solve_budget.as_secs_f64()), + &mut solve_metrics, + ) + }; + let mut proposal = match proposed { + Ok(None) if use_direct => { + direct_finished = true; + continue; + } + Err(CandidateSelectionError::UnsupportedDirectCost(reason)) if use_direct => { + if solve_metrics.direct_cost_fallback_reasons.len() < 8 { + solve_metrics.direct_cost_fallback_reasons.push(reason); + } + direct_finished = true; + continue; + } + Err(CandidateSelectionError::Solver(reason)) if use_direct => { + if solve_metrics.direct_cost_fallback_reasons.len() < 8 { + solve_metrics + .direct_cost_fallback_reasons + .push(format!("direct solver stopped: {reason}")); + } + direct_finished = true; + continue; + } Ok(Some(proposal)) => proposal, Ok(None) if use_neighborhood => { neighborhood_spent += neighborhood_started.elapsed(); @@ -399,7 +475,8 @@ fn search_with_solver( break CandidateSearchStatus::Infeasible; } Err(CandidateSelectionError::Solver(_)) - if use_baseline && started.elapsed() >= baseline_budget => + if use_baseline + && baseline_started.is_some_and(|start| start.elapsed() >= baseline_budget) => { heuristic_graph = None; continue; @@ -418,7 +495,8 @@ fn search_with_solver( }; } }; - if use_baseline { + let direct_modeled_cost = proposal.direct_modeled_cost; + if use_baseline || use_direct { proposal.selected.modeled_cost = proposal .selected .selected_candidates @@ -443,6 +521,10 @@ fn search_with_solver( if observations.iter().any(|observation: &ObservedAssignment| { same_assignment(&observation.selected, &proposal.selected) }) { + if use_direct { + direct_finished = true; + continue; + } if use_neighborhood { neighborhood_spent += neighborhood_started.elapsed(); continue; @@ -490,6 +572,21 @@ fn search_with_solver( }); } } + if let (Some(modeled), Some(actual)) = (direct_modeled_cost, actual_cost) { + let tolerance = 1e-7 * modeled.abs().max(actual.abs()).max(1e-15); + if (modeled - actual).abs() > tolerance { + return Err(format!( + "direct-cost objective {modeled} disagrees with evaluator {actual}" + )); + } + direct_finished = true; + solve_metrics.direct_cost_validated_plans += 1; + solve_metrics.direct_cost_best_validated_cost = Some( + solve_metrics + .direct_cost_best_validated_cost + .map_or(actual, |previous| previous.min(actual)), + ); + } observations.push(ObservedAssignment { selected: proposal.selected, exact_cost: actual_cost, @@ -578,6 +675,44 @@ mod tests { } } + // A restricted direct-cost seed supplies an incumbent, never a full-domain bound. + #[test] + fn direct_cost_seed_validates_price_without_claiming_global_optimum() { + use crate::search::milp::direct_cost::NodeCostProfile; + let graph = choices(&[0.0, 0.0]); + let mut restricted = graph.clone(); + restricted.query_roots[0].compatible_candidates = vec![PhysicalCandidateId(1)]; + let profile = DirectCostProfile { + nodes: vec![ + NodeCostProfile { + execution_cost: 1.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 2.0, + ..Default::default() + }, + ], + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + let result = search_candidates_with_direct_feedback( + &graph, + limits(1), + &[1.0, 1.0], + &[], + &[], + Some((&restricted, &profile)), + |selected| Ok(Some(profile.selected_cost(&restricted, selected)).into()), + ) + .unwrap(); + assert_eq!(result.incumbent.unwrap().actual_cost, 2.0); + assert_eq!(result.lower_bound, 0.0); + assert_eq!(result.status, CandidateSearchStatus::RefinementLimit); + assert_eq!(result.solve_metrics.direct_cost_validated_plans, 1); + } + // A later unproven solve can improve the validated incumbent without replacing // the earlier certified bound or falsely closing the optimality gap. #[test] @@ -616,6 +751,7 @@ mod tests { Ok(Some(CandidateProposal { selected, model_lower_bound, + direct_modeled_cost: None, })) }, ) diff --git a/crates/plan-selection/src/search/route.rs b/crates/plan-selection/src/search/route.rs index f2e3b6e65..d5c278ac5 100644 --- a/crates/plan-selection/src/search/route.rs +++ b/crates/plan-selection/src/search/route.rs @@ -595,6 +595,22 @@ fn add_phase_metrics( } } report.solve_calls += phase.solve_calls; + report.direct_cost_solve_calls += phase.direct_cost_solve_calls; + report.direct_cost_validated_plans += phase.direct_cost_validated_plans; + if let Some(cost) = phase.direct_cost_best_validated_cost { + report.direct_cost_best_validated_cost = Some( + report + .direct_cost_best_validated_cost + .map_or(cost, |previous| previous.min(cost)), + ); + } + for reason in &phase.direct_cost_fallback_reasons { + if report.direct_cost_fallback_reasons.len() < 8 + && !report.direct_cost_fallback_reasons.contains(reason) + { + report.direct_cost_fallback_reasons.push(reason.clone()); + } + } report.refinements += phase.refinements; report.certified_query_cuts += phase.certified_query_cuts; report.peak_milp_variables = report.peak_milp_variables.max(phase.peak_milp_variables); @@ -1041,6 +1057,10 @@ fn search_domain( solver_ms: result.metrics.solver_seconds * 1000.0, validation_ms: result.validation_seconds * 1000.0, solve_calls: result.metrics.solve_calls, + direct_cost_solve_calls: result.metrics.direct_cost_solve_calls, + direct_cost_validated_plans: result.metrics.direct_cost_validated_plans, + direct_cost_best_validated_cost: result.metrics.direct_cost_best_validated_cost, + direct_cost_fallback_reasons: result.metrics.direct_cost_fallback_reasons.clone(), peak_milp_variables: result.metrics.peak_variables, peak_milp_constraints: result.metrics.peak_constraints, validation_rejection_counts: result.rejections.counts, From 7efd233e288b4b6cb71bfe2fd80a0321b684b6ec Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 03:36:14 +0000 Subject: [PATCH 09/29] docs: describe expanded direct pricing and remaining coverage --- docs/design_docs/proposals/joint-planning-search.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index af980a666..ecc0b5ad3 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -200,7 +200,9 @@ Memory-time and capacity answer different questions. A state occupying `m` bytes The direct-cost objective computes the total modeled cost of the selected execution DAG. It does not precompute one fixed total cost for each candidate or enumerate every complete DAG. Before solving, the planner computes each supported candidate's local cost parameters from its configuration and compatible suppliers. Inside MILP, the selected nodes and connections determine execution frequency and repeated work, consumer-specific operations, state holding costs, shared raw retention, and query overhead. A shared producer owns its charge once, including every required execution; its consumers' work is charged separately. **Local candidate cost parameters are precomputed; the complete selected DAG's cost is determined jointly by MILP.** -The first implementation admits a restricted query-time exact/KMV subspace with fixed common evaluation schedules and certified supplier price interfaces. It includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; this is an implementation limitation to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. +Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Matching schedules use a compact demand model; up to three differing query schedules use evaluator-derived consumer-subset rates, including phase offsets. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. + +Ingestion-maintained and retained-query-time states, range-evaluation contexts, uncertified supplier interfaces, and larger heterogeneous schedule tables are not yet covered by this direct objective. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. Numerically ill-conditioned rate tables also fall back explicitly rather than silently dropping rare-query charges. The selection objective must express the modeled cost of the selected complete DAG, rather than rank plans with an exactness preference and discover their prices only after selection. This does not require enumerating complete plans. Factor the objective into node activation and state, execution demand, consumer connections and reads, shared resources, and query overhead. Each term has one owner; execution-dependent costs use the selected consumers' compatible evaluation schedules. Raw retention is charged once per source and storage scope at the maximum required horizon. Fixed query overhead can be omitted during minimization but must be restored in reported totals. From 3b5b216f2bc9355e8d188538cc072834daab9eda Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 03:36:14 +0000 Subject: [PATCH 10/29] feat: extend direct pricing to fixed sketches and query schedules --- .../src/search/costing/direct_profile.rs | 609 +++++++++++++++++- .../src/search/milp/direct_cost.rs | 250 +++++-- .../src/search/milp/direct_schedule.rs | 474 ++++++++++++++ .../src/search/refinement/search.rs | 1 + 4 files changed, 1257 insertions(+), 77 deletions(-) create mode 100644 crates/plan-selection/src/search/milp/direct_schedule.rs diff --git a/crates/plan-selection/src/search/costing/direct_profile.rs b/crates/plan-selection/src/search/costing/direct_profile.rs index cc938981c..27f7fd16f 100644 --- a/crates/plan-selection/src/search/costing/direct_profile.rs +++ b/crates/plan-selection/src/search/costing/direct_profile.rs @@ -5,7 +5,7 @@ //! catalog. No original-template child is used as workload statistics evidence. use crate::search::candidates::configurations::CandidateConfigurationInventory; use crate::search::candidates::graph::PhysicalCandidateId; -use crate::search::milp::direct_cost::{DirectCostProfile, NodeCostProfile}; +use crate::search::milp::direct_cost::{DirectCostProfile, NodeCostProfile, QueryScheduleRates}; use crate::search::refinement::planner::CandidatePlanningContext; use crate::{CostModel, PricedNode, ProducerOutput}; use asap_types::ir::physical_export::PhysicalASAPDAGNode; @@ -53,7 +53,7 @@ fn supported_node(node: &asap_types::ir::OperatorNode) -> bool { ) } -/// Only fixed, identical request schedules, no range requests and one operator +/// Identical fixed schedules or up to three heterogeneous schedules, no range requests and one operator /// root per query are supported. Every returned profile is priced at one /// execution; the MILP supplies demand, held lifetime, retention and overhead. pub(in crate::search) fn extract( @@ -67,48 +67,74 @@ pub(in crate::search) fn extract( let Some(first) = context.entries.first() else { return Ok(None); }; - if !matches!( - first.recurrence, - QueryRecurrence::Repeated( - RepeatedDemand::FixedInterval(_) | RepeatedDemand::FixedIntervalAt { .. } - ) - ) || context + if context .entries .iter() - .any(|entry| entry.recurrence != first.recurrence || entry.range_evaluation.is_some()) + .any(|entry| entry.range_evaluation.is_some()) || context .original_roots .iter() .any(|root| !matches!(root, QueryRoot::Operator(_))) + || context.original_roots.len() != context.entries.len() || context.original_roots.len() != configured.inventory.graph.query_roots.len() { return Ok(None); } - let rate = crate::cost::execution_demand::evaluation_rate( - [(&first.recurrence, None)], - settings.non_repeating_query_horizon_seconds, - ) - .map_err(|error| DirectProfileError::Invalid(error.to_string()))?; - if !rate.is_finite() || rate <= 0.0 { - return Ok(None); - } - if context.entries.iter().any(|entry| { - !entry.requirements.accuracy.is_exact() - && (entry.requirements.accuracy.accuracy_scope != AccuracyScope::PerValue - || entry - .requirements - .accuracy - .failure_probability - .is_none_or(|delta| !delta.is_finite() || delta <= 0.0)) - }) { + let equal_fixed = matches!( + first.recurrence, + QueryRecurrence::Repeated( + RepeatedDemand::FixedInterval(_) | RepeatedDemand::FixedIntervalAt { .. } + ) + ) && context + .entries + .iter() + .all(|entry| entry.recurrence == first.recurrence); + if !equal_fixed && context.entries.len() > 3 { return Ok(None); } - let delta = context + let rates: Vec<_> = context .entries .iter() - .filter(|entry| !entry.requirements.accuracy.is_exact()) - .filter_map(|entry| entry.requirements.accuracy.failure_probability) - .reduce(f64::min); + .map(|entry| { + crate::cost::execution_demand::evaluation_rate( + [(&entry.recurrence, None)], + settings.non_repeating_query_horizon_seconds, + ) + .map_err(|error| DirectProfileError::Invalid(error.to_string())) + }) + .collect::>()?; + if rates.iter().any(|rate| !rate.is_finite() || *rate <= 0.0) { + return Ok(None); + } + let rate = rates[0]; + let request_rate: f64 = rates.iter().sum(); + let schedules = if equal_fixed { + None + } else { + let mut materialized_subset_rates = Vec::with_capacity(1 << context.entries.len()); + for mask in 0..(1usize << context.entries.len()) { + materialized_subset_rates.push( + crate::cost::execution_demand::evaluation_rate( + context + .entries + .iter() + .enumerate() + .filter(|(query, _)| mask & (1 << query) != 0) + .map(|(_, entry)| (&entry.recurrence, None)), + settings.non_repeating_query_horizon_seconds, + ) + .map_err(|error| DirectProfileError::Invalid(error.to_string()))?, + ); + } + Some(QueryScheduleRates { + per_query_rates: rates, + materialized_subset_rates, + }) + }; + let delta = match failure_budget(context) { + Some(delta) => delta, + None => return Ok(None), + }; let count = configured.configurations.len(); let mut nodes = Vec::with_capacity(count); for (id, configuration) in configured.configurations.iter().enumerate() { @@ -117,7 +143,7 @@ pub(in crate::search) fn extract( nodes.push( node.filter(|node| { supported_node(node) - && stable_size(node, delta) + && (requires_source_sizing(node) || stable_size(node, delta)) && configuration.timing == ExecutionTiming::QueryTime && matches!( configuration.materialization, @@ -201,6 +227,7 @@ pub(in crate::search) fn extract( let mut visiting = vec![false; count]; let mut suppliers = vec![Vec::new(); count]; let mut sketch_counts = vec![0; count]; + let mut raw_sources = vec![None; count]; for id in 0..count { price( id, @@ -213,6 +240,8 @@ pub(in crate::search) fn extract( &mut visiting, &mut suppliers, &mut sketch_counts, + &mut raw_sources, + delta, )?; } let supported: Vec<_> = cache.iter().map(Option::is_some).collect(); @@ -256,11 +285,11 @@ pub(in crate::search) fn extract( profile: DirectCostProfile { nodes: profiles, query_rate: rate, + schedules, request_latency_seconds: request_ms / 1000.0, request_cost: request_ms / settings.resource_speed.ms_per_cpu_op * prices.cost_per_cpu_op - * rate - * context.entries.len() as f64, + * request_rate, }, supported, suppliers, @@ -279,6 +308,8 @@ fn price<'a>( visiting: &mut [bool], suppliers: &mut [Vec], sketch_counts: &mut [usize], + raw_sources: &mut [Option], + delta: Option, ) -> Result<(), DirectProfileError> { check_deadline(context.deadline)?; if visited[id] || visiting[id] { @@ -318,6 +349,8 @@ fn price<'a>( visiting, suppliers, sketch_counts, + raw_sources, + delta, )?; let Some(priced) = &cache[supplier.0] else { continue; @@ -332,6 +365,45 @@ fn price<'a>( unchanged_columns: priced.unchanged_columns.clone(), }); } + if let asap_types::ir::Operator::ASAP(ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(kind, _), + .. + }) = &node.payload + { + if let SketchParams::Hll { + precision, + relative_error, + } = kind.params() + { + let Some(source) = supplier.and_then(|supplier| raw_sources[supplier.0].as_deref()) + else { + continue; + }; + let Some(bound) = context + .data + .sources + .get(source) + .and_then(|facts| facts.max_distinct_per_evaluation) + else { + continue; + }; + let Some(delta) = delta else { continue }; + if !hll_size_certified(*precision, *relative_error, delta, bound) { + continue; + } + } + } + let raw_source = match &node.payload { + asap_types::ir::Operator::NonASAP(NonASAPOp::Scan { source, .. }) => { + Some(crate::source_name(source).to_string()) + } + asap_types::ir::Operator::NonASAP( + NonASAPOp::Filter { .. } | NonASAPOp::TimeRange { .. }, + ) if views.len() == 1 && *views[0].output_schema == node.output_schema => { + supplier.and_then(|supplier| raw_sources[supplier.0].clone()) + } + _ => None, + }; let sketches = supplier.map_or(0, |supplier| sketch_counts[supplier.0]) + usize::from(matches!( &node.payload, @@ -379,6 +451,11 @@ fn price<'a>( sketch_counts[id] = sketch_counts[id].max(sketches); if cache[id].is_none() { cache[id] = Some(priced); + raw_sources[id] = raw_source; + } else if raw_sources[id] != raw_source { + // The same priced rows need not carry the same raw-population proof. + // Keep no proof unless every admitted binding preserves it. + raw_sources[id] = None; } } visiting[id] = false; @@ -386,6 +463,77 @@ fn price<'a>( Ok(()) } +/// A conservative delta valid for every selected unary output path, including +/// rollups whose finer grouping the current evaluator uses for its row bound. +/// Unknown population bounds reject this direct proposal, not the full search. +fn failure_budget(context: &CandidatePlanningContext<'_>) -> Option> { + let population = || { + let mut sources = HashSet::new(); + for root in context.original_roots { + let QueryRoot::Operator(root) = root else { + return None; + }; + for node in asap_types::ir::OperatorNode::reachable(root) { + if let Some(NonASAPOp::Scan { source, .. }) = node.non_asap() { + sources.insert(crate::source_name(source).to_string()); + } + } + } + let counts: Option> = sources + .iter() + .map(|source| { + context + .data + .sources + .get(source)? + .series_count + .filter(|count| *count > 0) + }) + .collect(); + counts?.into_iter().max() + }; + let mut delta: Option = None; + for entry in context.entries { + let target = &entry.requirements.accuracy; + if target.is_exact() { + continue; + } + let requested = target + .failure_probability + .filter(|delta| delta.is_finite() && *delta > 0.0)?; + let values = match target.accuracy_scope { + AccuracyScope::WholeResult => population()?, + scope => scope.max_values_covered(None)?, + } + .max(1); + let required = requested / values as f64; + if !required.is_finite() || required <= 0.0 { + return None; + } + delta = Some(delta.map_or(required, |prior| prior.min(required))); + } + Some(delta) +} + +fn requires_source_sizing(node: &asap_types::ir::OperatorNode) -> bool { + matches!(node.asap(), Some(ASAPOp::SummaryAgg { family: FieldDataType::Sketch(kind, _), .. }) + if matches!(kind.params(), SketchParams::Hll { .. })) +} + +fn hll_size_certified( + precision: u8, + relative_error: Option, + delta: f64, + distinct: u64, +) -> bool { + use asap_logical_asap_optimization::accuracy::estimators::hll; + let error = hll::classic_relative_error(precision, relative_error); + // Unsupported populations can make min_precision return its input unchanged. + // The actual confidence certificate is essential, not merely size equality. + hll::min_precision(error, delta, distinct, precision) == precision + && hll::classic_failure_probability(precision, error, distinct) <= delta +} + fn stable_size(node: &asap_types::ir::OperatorNode, delta: Option) -> bool { let Some(ASAPOp::SummaryAgg { family: FieldDataType::Sketch(kind, _), @@ -399,7 +547,11 @@ fn stable_size(node: &asap_types::ir::OperatorNode, delta: Option) -> bool SketchParams::Kmv { relative_error: Some(_), .. - } + } | SketchParams::Kll { .. } + | SketchParams::DDSketch { .. } + | SketchParams::Cms { .. } + | SketchParams::CountSketch { .. } + | SketchParams::Theta { .. } ) { return false; } @@ -462,6 +614,15 @@ fn same_price(a: &PricedNode<'_>, b: &PricedNode<'_>) -> bool { #[cfg(test)] mod tests { + #[test] + fn hll_admission_requires_stable_precision_and_a_supported_population() { + assert!(super::hll_size_certified(15, Some(0.05), 0.05, 1000)); + assert!(!super::hll_size_certified(12, Some(0.05), 0.05, 1000)); + // The sizing API returns the original precision when no supported + // precision proves the target. That must not admit an unproved sketch. + assert!(!super::hll_size_certified(15, Some(0.05), 0.05, 1_000_000)); + } + use super::*; use crate::search::candidates::configurations::CandidateConfiguration; use crate::search::candidates::graph::{ @@ -644,6 +805,386 @@ mod tests { ); } + // Certified KLL/DDS configurations keep their sizes after reconstruction; + // direct costs include working memory and agree with the complete evaluator. + #[test] + fn fixed_sketch_profiles_match_the_real_evaluator() { + use asap_logical_asap_optimization::per_computation_candidates::{ + compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, + }; + use asap_types::ir::schema::SketchAlgorithm; + use std::rc::Rc; + fn append( + node: Rc, + configured: &mut CandidateConfigurationInventory, + ) -> PhysicalCandidateId { + let mut inputs = Vec::new(); + node.operator + .map_children(|child| inputs.push(append(child.clone(), configured))); + let id = PhysicalCandidateId(configured.configurations.len()); + configured.configurations.push(CandidateConfiguration { + source_candidate: id, + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::QueryTime, + }); + configured.inventory.templates.push(Some(node)); + configured + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: MemoGroupId(id.0 as u32), + local_cost: 0.0, + producer_requirements: inputs + .into_iter() + .map(|supplier| ProducerRequirement { + compatible_producers: vec![supplier].into(), + }) + .collect(), + }); + id + } + for algorithm in [ + SketchAlgorithm::Kll, + SketchAlgorithm::DDSketch, + SketchAlgorithm::Hll, + ] { + for scope in [ + AccuracyScope::PerValue, + AccuracyScope::WholeResult, + AccuracyScope::GroupsWithRecordShareAtLeast(0.5), + ] { + let target = AccuracyTarget::max_error(0.05) + .failure_probability(0.01) + .accuracy_scope(scope); + let query = if algorithm == SketchAlgorithm::Hll { + "distinct_over_time(latency[5m])" + } else { + "quantile_over_time(0.95, latency[5m])" + }; + let original = lower_promql(query, target.clone()); + let generated = generate_per_computation_candidates(vec![( + 0, + QueryRoot::Operator(original.clone()), + )]) + .unwrap(); + let choice = generated.targets[0].choices.candidates.iter().position(|candidate| + matches!(candidate, ComputationCandidate::Summary(found) if *found == algorithm)).unwrap(); + let roots: Vec<_> = compose_logical_candidate(&generated, &[choice]) + .unwrap() + .into_iter() + .map(|(_, root)| root) + .collect(); + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival":{"ingesting":{"rows_per_second":100.0}}, + "sources":{"latency":{"series_count":2,"rows_held":30000,"values_non_negative":true,"max_distinct_per_evaluation":1000}} + })).unwrap(); + // A legal finite configuration is sized before direct pricing. + let sized = crate::resize_for_failure_probability( + &roots, + std::slice::from_ref(&target), + &data, + ) + .unwrap(); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph::default(), + templates: vec![], + group_candidates: vec![], + }, + configurations: vec![], + }; + let QueryRoot::Operator(root) = &sized[0] else { + panic!("operator") + }; + let id = append(root.clone(), &mut configured); + configured + .inventory + .graph + .query_roots + .push(QueryRootCandidates { + compatible_candidates: vec![id], + }); + let originals = vec![QueryRoot::Operator(original)]; + let entries = vec![QueryWorkloadEntry { + query: Query(query.into()), + requirements: QueryRequirements { + accuracy: target, + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval( + RepetitionInterval(60_000), + )), + range_evaluation: None, + }]; + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &originals, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let extracted = extract(&configured, &context).unwrap().unwrap(); + assert!( + extracted.supported.iter().all(|supported| *supported), + "{algorithm:?} {scope:?}" + ); + let proposal = crate::search::milp::model::solve_direct_cost_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &extracted.profile, + ) + .unwrap() + .unwrap(); + let mut rejections = + crate::search::refinement::planner::CandidateRejectionSummary::default(); + let evaluated = crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("{algorithm:?} {scope:?}: {rejections:?}")); + let cost = evaluated.0.costs[&evaluated.0.selected] + .cost_per_second + .unwrap(); + let direct = proposal.direct_modeled_cost.unwrap(); + assert!( + (cost - direct).abs() <= 1e-9 * cost.max(1e-15), + "{algorithm:?} {scope:?}: {direct} vs {cost}" + ); + if scope == AccuracyScope::WholeResult { + let mut unknown = data.clone(); + unknown.sources.get_mut("latency").unwrap().series_count = None; + let unproved = CandidatePlanningContext { + data: &unknown, + ..context + }; + assert!( + extract(&configured, &unproved).unwrap().is_none(), + "unknown population cannot certify a whole-result sizing budget" + ); + } + let too_small_delta = match algorithm { + SketchAlgorithm::Kll => Some(1e-12), + _ => None, + }; + if let Some(delta) = too_small_delta { + assert!(configured + .inventory + .templates + .iter() + .flatten() + .any( + |node| matches!(node.asap(), Some(ASAPOp::SummaryAgg { .. })) + && !stable_size(node, Some(delta)) + )); + } + } + } + } + + // Real shared KLL/DDS plans verify schedule unions and held-memory lifetime, + // including two equal frequencies whose different phases cannot coalesce. + #[test] + fn heterogeneous_shared_sketch_profiles_match_the_real_evaluator() { + use asap_logical_asap_optimization::{ + asap_aware_cse::{sharing_candidates, with_shared_candidates}, + per_computation_candidates::{ + compose_logical_candidate, generate_per_computation_candidates, + ComputationCandidate, + }, + }; + use asap_types::ir::schema::SketchAlgorithm; + use asap_types::workload::TimestampMs; + use std::rc::Rc; + fn append( + node: Rc, + inventory: &mut CandidateConfigurationInventory, + ) -> PhysicalCandidateId { + if let Some(id) = inventory + .inventory + .templates + .iter() + .position(|old| old.as_ref().is_some_and(|old| Rc::ptr_eq(old, &node))) + { + return PhysicalCandidateId(id); + } + let mut children = Vec::new(); + node.operator + .map_children(|child| children.push(append(child.clone(), inventory))); + let id = PhysicalCandidateId(inventory.configurations.len()); + inventory.configurations.push(CandidateConfiguration { + source_candidate: id, + timing: ExecutionTiming::QueryTime, + materialization: if matches!(node.asap(), Some(ASAPOp::SummaryAgg { .. })) { + Materialization::MatAtQuery(Medium::Memory) + } else { + Materialization::NotMaterialized + }, + }); + inventory.inventory.templates.push(Some(node)); + inventory + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: MemoGroupId(id.0 as u32), + local_cost: 0.0, + producer_requirements: children + .into_iter() + .map(|id| ProducerRequirement { + compatible_producers: vec![id].into(), + }) + .collect(), + }); + id + } + for algorithm in [SketchAlgorithm::Kll, SketchAlgorithm::DDSketch] { + for phases in [false, true] { + let target = AccuracyTarget::max_error(0.05) + .failure_probability(0.01) + .accuracy_scope(AccuracyScope::PerValue); + let originals: Vec<_> = [0.95, 0.99] + .iter() + .map(|q| { + QueryRoot::Operator(lower_promql( + &format!("quantile_over_time({q}, latency[5m])"), + target.clone(), + )) + }) + .collect(); + let generated = generate_per_computation_candidates( + originals.iter().cloned().enumerate().collect(), + ) + .unwrap(); + let choices: Vec<_> = generated.targets.iter().map(|target| target.choices.candidates.iter().position(|candidate| + matches!(candidate,ComputationCandidate::Summary(found) if *found==algorithm)).unwrap()).collect(); + let roots: Vec<_> = compose_logical_candidate(&generated, &choices) + .unwrap() + .into_iter() + .map(|(_, root)| root) + .collect(); + let candidates = sharing_candidates(&roots); + let shared = + with_shared_candidates(&roots, &candidates, &vec![true; candidates.len()]); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph::default(), + templates: vec![], + group_candidates: vec![], + }, + configurations: vec![], + }; + for root in shared { + let QueryRoot::Operator(root) = root else { + panic!("operator") + }; + let id = append(root, &mut configured); + configured + .inventory + .graph + .query_roots + .push(QueryRootCandidates { + compatible_candidates: vec![id], + }); + } + assert_eq!( + configured + .inventory + .templates + .iter() + .flatten() + .filter(|node| matches!(node.asap(), Some(ASAPOp::SummaryAgg { .. }))) + .count(), + 1 + ); + let entries: Vec<_> = (0..2) + .map(|q| QueryWorkloadEntry { + query: Query(format!("quantile {q}")), + requirements: QueryRequirements { + accuracy: target.clone(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(if phases { + RepeatedDemand::FixedIntervalAt { + interval: RepetitionInterval(1000), + evaluation_phase: TimestampMs(q * 500), + } + } else { + RepeatedDemand::FixedInterval(RepetitionInterval(if q == 0 { + 500 + } else { + 1000 + })) + }), + range_evaluation: None, + }) + .collect(); + let data:DataWorkload=serde_json::from_value(serde_json::json!({"arrival":{"ingesting":{"rows_per_second":1.0}},"sources":{"latency":{"series_count":2,"rows_held":300,"values_non_negative":true}}})).unwrap(); + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &originals, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let extracted = extract(&configured, &context).unwrap().unwrap(); + assert!(extracted.supported.iter().all(|supported| *supported)); + assert_eq!( + extracted + .profile + .schedules + .as_ref() + .unwrap() + .materialized_subset_rates[3], + if phases { 2.0 } else { 3.0 } + ); + let proposal = crate::search::milp::model::solve_direct_cost_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &extracted.profile, + ) + .unwrap() + .unwrap(); + let mut rejections = + crate::search::refinement::planner::CandidateRejectionSummary::default(); + let (selection, _) = + crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("{algorithm:?} phases={phases}: {rejections:?}")); + let actual = selection.costs[&selection.selected] + .cost_per_second + .unwrap(); + let direct = proposal.direct_modeled_cost.unwrap(); + assert!( + (actual - direct).abs() <= 1e-9 * actual.max(1e-15), + "{algorithm:?} phases={phases}: {direct} vs {actual}" + ); + } + } + } + // A template child is not evidence that every supplier has the same rows. // Two deliberately different source alternatives exercise cost-interface // filtering independently of the logical compatibility check upstream. diff --git a/crates/plan-selection/src/search/milp/direct_cost.rs b/crates/plan-selection/src/search/milp/direct_cost.rs index db6aa8a84..60d6cbc02 100644 --- a/crates/plan-selection/src/search/milp/direct_cost.rs +++ b/crates/plan-selection/src/search/milp/direct_cost.rs @@ -4,12 +4,20 @@ use good_lp::{constraint, variable, Constraint, Expression, ProblemVariables, Variable}; use std::collections::BTreeMap; +#[path = "direct_schedule.rs"] +mod schedules; + use crate::search::candidates::graph::*; /// Smallest/largest monetary coefficient ratio after lifetime normalization. /// The minimum is scaled to one; no positive cost is rounded away. pub(super) const MINIMUM_DIRECT_RELATIVE_COEFFICIENT: f64 = 1e-12; +// Weighted-lifetime equalities use dimensionless rates. Their small entries +// need a stronger guard than monetary objective coefficients, since absolute +// LP feasibility tolerances can erase a rare reader's otherwise large cost. +const MINIMUM_QUERY_RATE_WEIGHT: f64 = 1e-6; + #[derive(Debug, Clone, Default)] pub(crate) struct NodeCostProfile { pub execution_cost: f64, @@ -19,9 +27,18 @@ pub(crate) struct NodeCostProfile { pub raw_retention: Option<(String, f64)>, } +/// Exact evaluator demand for each nonempty selected query subset. Only small +/// heterogeneous domains are admitted; this is not a workload-plan product. +#[derive(Debug, Clone)] +pub(crate) struct QueryScheduleRates { + pub per_query_rates: Vec, + pub materialized_subset_rates: Vec, +} + #[derive(Debug, Clone)] pub(crate) struct DirectCostProfile { pub nodes: Vec, + pub schedules: Option, pub query_rate: f64, pub request_latency_seconds: f64, /// Total request fee per second across all queries. @@ -52,6 +69,51 @@ impl DirectCostProfile { "invalid direct-cost profile".into(), )); } + if let Some(schedule) = &self.schedules { + let queries = graph.query_roots.len(); + if queries > 3 { + return Err(CandidateSelectionError::UnsupportedDirectCost( + "heterogeneous query schedule table supports at most three output queries" + .into(), + )); + } + if !schedule.per_query_rates.iter().sum::().is_finite() + || schedule.per_query_rates.len() != queries + || schedule.materialized_subset_rates.len() != (1usize << queries) + || schedule.materialized_subset_rates[0] != 0.0 + || schedule + .per_query_rates + .iter() + .any(|rate| !rate.is_finite() || *rate <= 0.0) + || schedule + .materialized_subset_rates + .iter() + .skip(1) + .any(|rate| !rate.is_finite() || *rate <= 0.0) + || schedule + .per_query_rates + .iter() + .enumerate() + .any(|(q, rate)| *rate != schedule.materialized_subset_rates[1 << q]) + { + return Err(CandidateSelectionError::InvalidGraph( + "invalid direct query-schedule rates".into(), + )); + } + } + if let Some(schedule) = &self.schedules { + let total_rate: f64 = schedule.per_query_rates.iter().sum(); + if schedule + .per_query_rates + .iter() + .any(|rate| rate / total_rate < MINIMUM_QUERY_RATE_WEIGHT) + { + return Err(CandidateSelectionError::UnsupportedDirectCost( + "query-rate weights are below the supported weighted-lifetime numerical range" + .into(), + )); + } + } let maximum_latency = self.maximum_latency(); if self.coefficients().any(|cost| !cost.is_finite()) || !maximum_latency.is_finite() @@ -96,25 +158,40 @@ impl DirectCostProfile { .sum::() }) .collect(); - let mut counts = vec![0usize; self.nodes.len()]; + let rates = self.schedules.as_ref().map_or_else( + || vec![self.query_rate; graph.query_roots.len()], + |schedule| schedule.per_query_rates.clone(), + ); + let union_rate = |id: usize| { + let Some(schedule) = &self.schedules else { + return self.query_rate; + }; + let mask = reached + .iter() + .enumerate() + .filter(|(_, row)| row[id]) + .fold(0usize, |mask, (q, _)| mask | (1 << q)); + schedule.materialized_subset_rates[mask] + }; + let mut execution_rates = vec![0.0; self.nodes.len()]; let boundaries = selected .query_roots .iter() - .map(|root| vec![root.candidate.0]) + .map(|root| (vec![root.candidate.0], rates[root.query_root])) .chain( selected .selected_candidates .iter() .filter(|id| self.nodes[id.0].materialized) - .map(|id| inputs[id.0].clone()), + .map(|id| (inputs[id.0].clone(), union_rate(id.0))), ); - for mut pending in boundaries { + for (mut pending, rate) in boundaries { let mut seen = vec![false; self.nodes.len()]; while let Some(id) = pending.pop() { if self.nodes[id].materialized || std::mem::replace(&mut seen[id], true) { continue; } - counts[id] += 1; + execution_rates[id] += rate; pending.extend(&inputs[id]); } } @@ -128,14 +205,26 @@ impl DirectCostProfile { .filter(|(_, row)| row[id.0]) .map(|(q, _)| latency[q]) .collect(); - let executions = if node.materialized { 1 } else { counts[id.0] } as f64; + let execution_rate = if node.materialized { + union_rate(id.0) + } else { + execution_rates[id.0] + }; let lifetime = if node.materialized { readers.iter().copied().fold(0.0, f64::max) } else { - readers.iter().sum::() / readers.len().max(1) as f64 + let reader_rates = reached.iter().enumerate().filter(|(_, row)| row[id.0]); + let denominator: f64 = reader_rates.clone().map(|(q, _)| rates[q]).sum(); + if denominator > 0.0 { + reader_rates + .map(|(q, _)| rates[q] * latency[q]) + .sum::() + / denominator + } else { + 0.0 + } }; - total += self.query_rate - * executions + total += execution_rate * (node.execution_cost + node.held_memory_cost_per_second * lifetime); if let Some((source, cost)) = &node.raw_retention { let value = raw.entry(source).or_default(); @@ -153,12 +242,27 @@ impl DirectCostProfile { // Constant request fees cannot influence selection and need not limit // the coefficient range. Lifetimes use a dimensionless [0,1] scale. let latency_scale = self.maximum_latency().max(f64::MIN_POSITIVE); + let rates = self.schedules.as_ref().map_or_else( + || vec![self.query_rate], + |schedule| { + schedule + .materialized_subset_rates + .iter() + .copied() + .filter(|rate| *rate > 0.0) + .collect() + }, + ); self.nodes.iter().flat_map(move |node| { - [ - node.execution_cost * self.query_rate, - node.held_memory_cost_per_second * self.query_rate * latency_scale, - node.raw_retention.as_ref().map_or(0.0, |(_, cost)| *cost), - ] + let mut coefficients = Vec::with_capacity(rates.len() * 2 + 1); + for rate in &rates { + coefficients.extend([ + node.execution_cost * rate, + node.held_memory_cost_per_second * rate * latency_scale, + ]); + } + coefficients.push(node.raw_retention.as_ref().map_or(0.0, |(_, cost)| *cost)); + coefficients }) } } @@ -177,10 +281,13 @@ fn gated( product } -/// Exact on integer activation/connection decisions. Reach variables are -/// implied integral by the selected acyclic graph, as in the main model. +struct BoundaryReach { + boundary: usize, + reached: Vec>, +} + #[allow(clippy::too_many_arguments)] -pub(super) fn encode( +fn boundary_flows( graph: &CandidateDependencyGraph, profile: &DirectCostProfile, variables: &mut ProblemVariables, @@ -188,29 +295,11 @@ pub(super) fn encode( active: &[Variable], edges: &[(ProducerConsumerConnection, Variable)], roots: &[(QueryRootAssignment, Variable)], - reach: &[Vec>], - divisor: f64, - check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, -) -> Result { + check_deadline: &impl Fn() -> Result<(), CandidateSelectionError>, +) -> Result, CandidateSelectionError> { let count = graph.candidates.len(); let queries = graph.query_roots.len(); - let latency_scale = profile.maximum_latency().max(f64::MIN_POSITIVE); - let maximum_latency = 1.0; - let latency: Vec = reach - .iter() - .map(|row| { - Expression::from(profile.request_latency_seconds / latency_scale) - + row - .iter() - .enumerate() - .filter_map(|(id, reached)| { - reached.map(|r| profile.nodes[id].work_seconds / latency_scale * r) - }) - .sum::() - }) - .collect(); - let mut boundary_count = vec![Expression::from(0.0); count]; - let mut boundary_limit = vec![0usize; count]; + let mut flows = Vec::new(); // Each root query and each selected materialized computation is one // independent execution boundary. Materialized nodes stop propagation. for boundary in 0..queries + count { @@ -270,12 +359,83 @@ pub(super) fn encode( support[edge.producer.0] += traversed; } } - for (id, used) in used.into_iter().enumerate() { - if let Some(used) = used { + for (id, reached) in used.iter().enumerate() { + if let Some(reached) = reached { + constraints.push(constraint!(*reached <= support[id].clone())); + constraints.push(constraint!(*reached <= active[id])); + } + } + flows.push(BoundaryReach { + boundary, + reached: used, + }); + } + Ok(flows) +} + +/// Exact on integer activation/connection decisions. Reach variables are +/// implied integral by the selected acyclic graph, as in the main model. +#[allow(clippy::too_many_arguments)] +pub(super) fn encode( + graph: &CandidateDependencyGraph, + profile: &DirectCostProfile, + variables: &mut ProblemVariables, + constraints: &mut Vec, + active: &[Variable], + edges: &[(ProducerConsumerConnection, Variable)], + roots: &[(QueryRootAssignment, Variable)], + reach: &[Vec>], + divisor: f64, + check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, +) -> Result { + if let Some(schedule) = &profile.schedules { + return schedules::encode( + graph, + profile, + schedule, + variables, + constraints, + active, + edges, + roots, + reach, + divisor, + check_deadline, + ); + } + let count = graph.candidates.len(); + let queries = graph.query_roots.len(); + let latency_scale = profile.maximum_latency().max(f64::MIN_POSITIVE); + let maximum_latency = 1.0; + let latency: Vec = reach + .iter() + .map(|row| { + Expression::from(profile.request_latency_seconds / latency_scale) + + row + .iter() + .enumerate() + .filter_map(|(id, reached)| { + reached.map(|r| profile.nodes[id].work_seconds / latency_scale * r) + }) + .sum::() + }) + .collect(); + let mut boundary_count = vec![Expression::from(0.0); count]; + let mut boundary_limit = vec![0usize; count]; + for flow in boundary_flows( + graph, + profile, + variables, + constraints, + active, + edges, + roots, + &check_deadline, + )? { + for (id, reached) in flow.reached.into_iter().enumerate() { + if let Some(reached) = reached { + boundary_count[id] += reached; boundary_limit[id] += 1; - constraints.push(constraint!(used <= support[id].clone())); - constraints.push(constraint!(used <= active[id])); - boundary_count[id] += used; } } } @@ -438,6 +598,7 @@ mod tests { ..Default::default() }; let mut profile = DirectCostProfile { + schedules: None, query_rate: 2.0, request_latency_seconds: 0.5, request_cost: 7.0, @@ -478,6 +639,7 @@ mod tests { ..Default::default() }; let profile = DirectCostProfile { + schedules: None, query_rate: 2.0, request_latency_seconds: 0.5, request_cost: 7.0, @@ -522,6 +684,7 @@ mod tests { } for materialized_build_cost in [1.0, 10.0, 100.0] { let profile = DirectCostProfile { + schedules: None, query_rate: 2.0, request_latency_seconds: 0.5, request_cost: 7.0, @@ -592,6 +755,7 @@ mod tests { ..Default::default() }; let profile = DirectCostProfile { + schedules: None, nodes: vec![ NodeCostProfile { execution_cost: 7.1e-13, diff --git a/crates/plan-selection/src/search/milp/direct_schedule.rs b/crates/plan-selection/src/search/milp/direct_schedule.rs new file mode 100644 index 000000000..ffabd6133 --- /dev/null +++ b/crates/plan-selection/src/search/milp/direct_schedule.rs @@ -0,0 +1,474 @@ +//! Heterogeneous execution demand for a small query-subset domain. +//! Subset rates come from the evaluator, not a second scheduling policy. +use super::*; + +#[allow(clippy::too_many_arguments)] +pub(super) fn encode( + graph: &CandidateDependencyGraph, + profile: &DirectCostProfile, + schedule: &QueryScheduleRates, + variables: &mut ProblemVariables, + constraints: &mut Vec, + active: &[Variable], + edges: &[(ProducerConsumerConnection, Variable)], + roots: &[(QueryRootAssignment, Variable)], + reach: &[Vec>], + divisor: f64, + check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, +) -> Result { + let count = graph.candidates.len(); + let queries = graph.query_roots.len(); + let subsets = 1usize << queries; + let latency_scale = profile.maximum_latency().max(f64::MIN_POSITIVE); + let latency: Vec = reach + .iter() + .map(|row| { + Expression::from(profile.request_latency_seconds / latency_scale) + + row + .iter() + .enumerate() + .filter_map(|(id, reached)| { + reached.map(|r| profile.nodes[id].work_seconds / latency_scale * r) + }) + .sum::() + }) + .collect(); + let mut masks = Vec::with_capacity(count); + for id in 0..count { + check_deadline()?; + let allowed = reach + .iter() + .enumerate() + .filter(|(_, row)| row[id].is_some()) + .fold(0usize, |mask, (q, _)| mask | (1 << q)); + let row: Vec<_> = (0..subsets) + .map(|mask| (mask & !allowed == 0).then(|| variables.add(variable().min(0).max(1)))) + .collect(); + // Integral query reach fixes exactly one subset, so these selector + // auxiliaries need no additional integer branching. + constraints.push(constraint!( + row.iter().flatten().copied().sum::() == 1 + )); + constraints.push(constraint!( + row.iter().skip(1).flatten().copied().sum::() == active[id] + )); + for (query, reachable) in reach.iter().enumerate() { + let included: Expression = row + .iter() + .enumerate() + .filter(|(mask, _)| mask & (1 << query) != 0) + .filter_map(|(_, variable)| *variable) + .sum(); + constraints.push(constraint!( + included == reachable[id].map_or_else(|| Expression::from(0.0), Expression::from) + )); + } + masks.push(row); + } + // Each pair is (binary demand indicator, executions per second). A query + // boundary has its own rate; a materialized boundary uses its selected + // downstream query subset's union rate and stops upstream propagation. + let mut demands = vec![Vec::<(Variable, f64)>::new(); count]; + for flow in boundary_flows( + graph, + profile, + variables, + constraints, + active, + edges, + roots, + &check_deadline, + )? { + let boundary = flow.boundary; + for (id, reaches) in flow.reached.into_iter().enumerate() { + check_deadline()?; + let Some(reaches) = reaches else { continue }; + if boundary < queries { + demands[id].push((reaches, schedule.per_query_rates[boundary])); + } else { + for (mask, selector) in masks[boundary - queries].iter().enumerate().skip(1) { + let Some(selector) = selector else { continue }; + let used = variables.add(variable().min(0).max(1)); + constraints.push(constraint!(used <= reaches)); + constraints.push(constraint!(used <= *selector)); + constraints.push(constraint!(used >= reaches + *selector - 1)); + demands[id].push((used, schedule.materialized_subset_rates[mask])); + } + } + } + } + let mut objective = Expression::from(profile.request_cost / divisor); + let mut retained = BTreeMap::<&str, Vec<(usize, f64)>>::new(); + for (id, node) in profile.nodes.iter().enumerate() { + check_deadline()?; + let executions: Expression = if node.materialized { + masks[id] + .iter() + .enumerate() + .skip(1) + .filter_map(|(mask, selector)| { + selector.map(|s| schedule.materialized_subset_rates[mask] * s) + }) + .sum() + } else { + demands[id].iter().map(|(used, rate)| *rate * *used).sum() + }; + objective += node.execution_cost / divisor * executions; + if let Some((source, cost)) = &node.raw_retention { + retained.entry(source).or_default().push((id, *cost)); + } + if node.held_memory_cost_per_second == 0.0 { + continue; + } + let mut lifetime_demand = Expression::from(0.0); + if node.materialized { + let longest = variables.add(variable().min(0).max(1)); + for (query, row) in reach.iter().enumerate() { + if let Some(reached) = row[id] { + constraints.push(constraint!( + longest >= latency[query].clone() - (1.0 - reached) + )); + } + } + for (mask, selector) in masks[id].iter().enumerate().skip(1) { + if let Some(selector) = selector { + lifetime_demand += schedule.materialized_subset_rates[mask] + * gated(variables, constraints, longest.into(), *selector, 1.0); + } + } + } else { + for (mask, selector) in masks[id].iter().enumerate().skip(1) { + check_deadline()?; + let Some(selector) = selector else { continue }; + let total_rate: f64 = schedule + .per_query_rates + .iter() + .enumerate() + .filter(|(q, _)| mask & (1 << q) != 0) + .map(|(_, rate)| *rate) + .sum(); + let mean: Expression = latency + .iter() + .enumerate() + .filter(|(q, _)| mask & (1 << q) != 0) + .map(|(q, latency)| schedule.per_query_rates[q] / total_rate * latency.clone()) + .sum(); + let selected_mean = gated(variables, constraints, mean, *selector, 1.0); + for (demand, rate) in &demands[id] { + check_deadline()?; + lifetime_demand += + *rate * gated(variables, constraints, selected_mean.into(), *demand, 1.0); + } + } + } + objective += node.held_memory_cost_per_second * latency_scale / divisor * lifetime_demand; + } + for entries in retained.into_values() { + let retained_cost = variables.add(variable().min(0)); + for (id, cost) in entries { + constraints.push(constraint!(retained_cost >= cost / divisor * active[id])); + } + objective += retained_cost; + } + Ok(objective) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::search::milp::model::{ + solve_direct_cost_proposal_with_metrics, CandidateSolveMetrics, + }; + use asap_logical_asap_optimization::memo::MemoGroupId; + use asap_types::workload::{QueryRecurrence, RepeatedDemand, RepetitionInterval}; + + fn schedules(queries: &[QueryRecurrence]) -> QueryScheduleRates { + let union = |mask: usize| { + crate::cost::execution_demand::evaluation_rate( + queries + .iter() + .enumerate() + .filter(|(q, _)| mask & (1 << q) != 0) + .map(|(_, query)| (query, None)), + 3600.0, + ) + .unwrap() + }; + QueryScheduleRates { + per_query_rates: (0..queries.len()).map(|q| union(1 << q)).collect(), + materialized_subset_rates: (0..1 << queries.len()).map(union).collect(), + } + } + fn node(input: Option) -> PhysicalCandidate { + PhysicalCandidate { + group: MemoGroupId(0), + local_cost: 0.0, + producer_requirements: input + .into_iter() + .map(|id| ProducerRequirement { + compatible_producers: vec![PhysicalCandidateId(id)].into(), + }) + .collect(), + } + } + fn root(id: usize) -> QueryRootCandidates { + QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(id)], + } + } + + // Boundary demand is the evaluator's union rate; upstream held memory uses + // the rate-weighted mean, not the arithmetic mean or the sum of root rates. + #[test] + fn heterogeneous_intervals_price_boundary_rates_and_weighted_lifetimes() { + let queries = [500, 1000].map(|ms| { + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms))) + }); + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(Some(0)), node(Some(1)), node(Some(1))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + let profile = DirectCostProfile { + schedules: Some(schedules(&queries)), + query_rate: 1.0, + request_latency_seconds: 0.5, + request_cost: 7.0, + nodes: vec![ + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 2.0, + held_memory_cost_per_second: 4.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + work_seconds: 3.0, + ..Default::default() + }, + NodeCostProfile { + work_seconds: 4.0, + ..Default::default() + }, + ], + }; + assert_eq!( + profile + .schedules + .as_ref() + .unwrap() + .materialized_subset_rates[3], + 3.0 + ); + let result = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((result.direct_modeled_cost.unwrap() - 167.5).abs() < 1e-8); + assert!((result.model_lower_bound.unwrap() - 167.5).abs() < 1e-8); + } + + // Same-interval readers can share one evaluation only at compatible phases. + // Use the actual evaluator's table rather than assuming all equal rates merge. + #[test] + fn aligned_and_offset_phases_use_evaluator_union_policy() { + use asap_types::workload::TimestampMs; + let graph = CandidateDependencyGraph { + candidates: vec![node(None)], + query_roots: vec![root(0), root(0)], + ..Default::default() + }; + for (second_phase, expected) in [(0, 7.0), (500, 14.0)] { + let queries = [0, second_phase].map(|phase| { + QueryRecurrence::Repeated(RepeatedDemand::FixedIntervalAt { + interval: RepetitionInterval(1000), + evaluation_phase: TimestampMs(phase), + }) + }); + let profile = DirectCostProfile { + schedules: Some(schedules(&queries)), + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + nodes: vec![NodeCostProfile { + execution_cost: 3.0, + work_seconds: 2.0, + held_memory_cost_per_second: 2.0, + materialized: true, + raw_retention: None, + }], + }; + let result = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((result.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + // HiGHS epigraph feasibility is tolerance-based; the decoded selected + // cost above is recomputed independently and remains exact here. + assert!( + (result.model_lower_bound.unwrap() - expected).abs() <= 1e-7 * expected, + "bound={:?}, expected={expected}", + result.model_lower_bound + ); + } + } + + // Every reader can independently choose either materialized producer. + // Compare the joint optimum with four independently priced assignments. + #[test] + fn heterogeneous_selection_matches_exhaustive_reader_assignments() { + let queries = [500, 1000].map(|ms| { + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms))) + }); + let schedule = schedules(&queries); + let mut graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None), node(Some(0)), node(Some(0))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + for id in [2, 3] { + graph.candidates[id].producer_requirements[0].compatible_producers = + vec![PhysicalCandidateId(0), PhysicalCandidateId(1)].into(); + } + let profile = DirectCostProfile { + schedules: Some(schedule.clone()), + query_rate: 1.0, + request_latency_seconds: 0.5, + request_cost: 7.0, + nodes: vec![ + NodeCostProfile { + execution_cost: 10.0, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 0.5, + held_memory_cost_per_second: 1.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 2.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 3.0, + ..Default::default() + }, + ], + }; + let expected = [[0, 0], [0, 1], [1, 0], [1, 1]] + .into_iter() + .map(|assignment| { + let latency = [ + 2.5 + [1.0, 0.5][assignment[0]], + 3.5 + [1.0, 0.5][assignment[1]], + ]; + let mut cost = 7.0 + 3.0; + for supplier in [0, 1] { + let mask = assignment + .iter() + .enumerate() + .filter(|(_, chosen)| **chosen == supplier) + .fold(0, |mask, (q, _)| mask | (1 << q)); + let longest = latency + .iter() + .enumerate() + .filter(|(q, _)| mask & (1 << q) != 0) + .map(|(_, time)| *time) + .fold(0.0, f64::max); + cost += schedule.materialized_subset_rates[mask] + * ([10.0, 2.0][supplier] + [3.0, 1.0][supplier] * longest); + } + cost + }) + .fold(f64::INFINITY, f64::min); + let result = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((result.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + // HiGHS epigraph feasibility is tolerance-based; the decoded selected + // cost above is recomputed independently and remains exact here. + assert!( + (result.model_lower_bound.unwrap() - expected).abs() <= 1e-7 * expected, + "bound={:?}, expected={expected}", + result.model_lower_bound + ); + } + // A rare slow reader's weighted held lifetime can dominate cost despite a + // tiny rate. Reject that numerically unsupported mean before optimization, + // rather than erase its contribution in a relaxed equality. + #[test] + fn extreme_reader_rates_are_explicitly_unsupported() { + let queries = [1, u32::MAX].map(|ms| { + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms))) + }); + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None), node(Some(0)), node(Some(0))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + let profile = DirectCostProfile { + schedules: Some(schedules(&queries)), + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + nodes: vec![ + NodeCostProfile { + execution_cost: 1.0, + held_memory_cost_per_second: 1e6, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 2.0, + held_memory_cost_per_second: 1e6, + ..Default::default() + }, + NodeCostProfile { + work_seconds: 1.0, + ..Default::default() + }, + NodeCostProfile { + work_seconds: 1e9, + ..Default::default() + }, + ], + }; + let rates = &profile.schedules.as_ref().unwrap().per_query_rates; + assert!( + rates[1] * 1e9 * 1e6 > 1e8, + "the rare reader has meaningful memory cost" + ); + assert!(matches!( + profile.validate(&graph), + Err(CandidateSelectionError::UnsupportedDirectCost(_)) + )); + } +} diff --git a/crates/plan-selection/src/search/refinement/search.rs b/crates/plan-selection/src/search/refinement/search.rs index 739d97683..b67b2258a 100644 --- a/crates/plan-selection/src/search/refinement/search.rs +++ b/crates/plan-selection/src/search/refinement/search.rs @@ -683,6 +683,7 @@ mod tests { let mut restricted = graph.clone(); restricted.query_roots[0].compatible_candidates = vec![PhysicalCandidateId(1)]; let profile = DirectCostProfile { + schedules: None, nodes: vec![ NodeCostProfile { execution_cost: 1.0, From 71160892a3a69104e3e6440a7cdbdc6ace8deb74 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 03:57:50 +0000 Subject: [PATCH 11/29] docs: record direct ingestion pricing and its certificates --- docs/design_docs/proposals/joint-planning-search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index ecc0b5ad3..5762d8ef8 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -202,7 +202,7 @@ The direct-cost objective computes the total modeled cost of the selected execut Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Matching schedules use a compact demand model; up to three differing query schedules use evaluator-derived consumer-subset rates, including phase offsets. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. -Ingestion-maintained and retained-query-time states, range-evaluation contexts, uncertified supplier interfaces, and larger heterogeneous schedule tables are not yet covered by this direct objective. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. Numerically ill-conditioned rate tables also fall back explicitly rather than silently dropping rare-query charges. +Fixed-retention ingestion panes are also supported when their actual unary supplier closures certify a single ingest worker. Updates run at ingestion demand, resident state is charged once, and ingestion update work is excluded from query latency. More workers require explicit compaction-latency modeling before admission. Retained-query-time states, range-evaluation contexts, uncertified supplier interfaces, and larger heterogeneous schedule tables are not yet covered by this direct objective. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. Numerically ill-conditioned cost profiles or rate tables also fall back explicitly rather than silently dropping rare-query charges. The selection objective must express the modeled cost of the selected complete DAG, rather than rank plans with an exactness preference and discover their prices only after selection. This does not require enumerating complete plans. Factor the objective into node activation and state, execution demand, consumer connections and reads, shared resources, and query overhead. Each term has one owner; execution-dependent costs use the selected consumers' compatible evaluation schedules. Raw retention is charged once per source and storage scope at the maximum required horizon. Fixed query overhead can be omitted during minimization but must be restored in reported totals. From 411159757bf5e9102bf8613f5d381a1d13135058 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 03:57:48 +0000 Subject: [PATCH 12/29] feat: directly price fixed-retention ingestion panes --- .../src/search/costing/direct_profile.rs | 428 +++++++++++++++++- .../src/search/milp/direct_cost.rs | 290 +++++++++++- .../src/search/milp/direct_schedule.rs | 35 +- .../src/search/refinement/search.rs | 4 + 4 files changed, 726 insertions(+), 31 deletions(-) diff --git a/crates/plan-selection/src/search/costing/direct_profile.rs b/crates/plan-selection/src/search/costing/direct_profile.rs index 27f7fd16f..d8e97d910 100644 --- a/crates/plan-selection/src/search/costing/direct_profile.rs +++ b/crates/plan-selection/src/search/costing/direct_profile.rs @@ -1,8 +1,10 @@ -//! Exact local profiles for a restricted query-time proposal domain. +//! Exact local profiles for query-time plans and single-worker maintained panes. //! //! Each accepted input is priced from its actual recursively certified supplier. //! Different priced interfaces are excluded from this proposal, not the full //! catalog. No original-template child is used as workload statistics evidence. +//! Ingestion support is limited to fixed-retention pane chains with certified +//! single-worker unary input closures; multiworker compaction remains excluded. use crate::search::candidates::configurations::CandidateConfigurationInventory; use crate::search::candidates::graph::PhysicalCandidateId; use crate::search::milp::direct_cost::{DirectCostProfile, NodeCostProfile, QueryScheduleRates}; @@ -46,6 +48,7 @@ fn supported_node(node: &asap_types::ir::OperatorNode) -> bool { node.asap(), Some( ASAPOp::SummaryAgg { .. } + | ASAPOp::PaneChain { .. } | ASAPOp::SummaryEstimate { .. } | ASAPOp::FinalizeExactAccumulator { .. } | ASAPOp::SummaryMerge { .. } @@ -143,13 +146,41 @@ pub(in crate::search) fn extract( nodes.push( node.filter(|node| { supported_node(node) + && !(configuration.timing == ExecutionTiming::IngestionTime + && configuration.materialization == Materialization::NotMaterialized + && configured.inventory.graph.query_roots.iter().any(|root| { + root.compatible_candidates + .contains(&PhysicalCandidateId(id)) + })) && (requires_source_sizing(node) || stable_size(node, delta)) - && configuration.timing == ExecutionTiming::QueryTime - && matches!( - configuration.materialization, - Materialization::NotMaterialized - | Materialization::MatAtQuery(Medium::Memory) - ) + && match configuration.timing { + ExecutionTiming::QueryTime => { + !matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) + && matches!( + configuration.materialization, + Materialization::NotMaterialized + | Materialization::MatAtQuery(Medium::Memory) + ) + } + ExecutionTiming::IngestionTime => match node.asap() { + Some(ASAPOp::PaneChain { .. }) => { + configuration.materialization + == Materialization::MatAtIngestion(Medium::Memory) + } + _ => { + matches!( + node.non_asap(), + Some( + NonASAPOp::Scan { .. } + | NonASAPOp::Filter { .. } + | NonASAPOp::Project { .. } + | NonASAPOp::TimeRange { .. } + ) + ) && configuration.materialization + == Materialization::NotMaterialized + } + }, + } && configured.inventory.graph.candidates[id] .producer_requirements .len() @@ -165,7 +196,7 @@ pub(in crate::search) fn extract( current }), output_state: ExecutionDataState { - timing: ExecutionTiming::QueryTime, + timing: configuration.timing, primitive: if node.asap().and_then(ASAPOp::produced_state).is_some() { DataPrimitive::SummaryState } else { @@ -228,6 +259,7 @@ pub(in crate::search) fn extract( let mut suppliers = vec![Vec::new(); count]; let mut sketch_counts = vec![0; count]; let mut raw_sources = vec![None; count]; + let mut closure_work_ms = vec![0.0; count]; for id in 0..count { price( id, @@ -241,6 +273,7 @@ pub(in crate::search) fn extract( &mut suppliers, &mut sketch_counts, &mut raw_sources, + &mut closure_work_ms, delta, )?; } @@ -251,6 +284,8 @@ pub(in crate::search) fn extract( work_seconds: 0.0, held_memory_cost_per_second: 0.0, materialized: false, + ingestion: false, + standing_cost_per_second: 0.0, raw_retention: None, }; let profiles = cache @@ -269,7 +304,13 @@ pub(in crate::search) fn extract( }) .transpose()?; Ok(NodeCostProfile { - execution_cost: priced.cost.cost_per_second.unwrap(), + execution_cost: priced.cpu_ops_per_second * prices.cost_per_cpu_op + + priced.scanned_bytes_per_second * prices.cost_per_scanned_byte, + ingestion: nodes[id].as_ref().unwrap().output_state.timing + == ExecutionTiming::IngestionTime, + standing_cost_per_second: priced.average_memory_bytes + * prices.cost_per_memory_byte_second + + priced.disk_bytes as f64 * prices.cost_per_disk_byte_second, work_seconds: priced.work_ms / 1000.0, held_memory_cost_per_second: priced.held.map_or(0.0, |(bytes, _, _)| { bytes as f64 * prices.cost_per_memory_byte_second @@ -309,6 +350,7 @@ fn price<'a>( suppliers: &mut [Vec], sketch_counts: &mut [usize], raw_sources: &mut [Option], + closure_work_ms: &mut [f64], delta: Option, ) -> Result<(), DirectProfileError> { check_deadline(context.deadline)?; @@ -350,12 +392,22 @@ fn price<'a>( suppliers, sketch_counts, raw_sources, + closure_work_ms, delta, )?; let Some(priced) = &cache[supplier.0] else { continue; }; let producer = nodes[supplier.0].as_ref().unwrap(); + let consumer_ingests = node.output_state.timing == ExecutionTiming::IngestionTime; + let producer_ingests = producer.output_state.timing == ExecutionTiming::IngestionTime; + if (consumer_ingests && !producer_ingests) + || (!consumer_ingests + && producer_ingests + && producer.materialization != Materialization::MatAtIngestion(Medium::Memory)) + { + continue; + } views.push(ProducerOutput { payload: &producer.payload, output_schema: &producer.output_schema, @@ -407,29 +459,55 @@ fn price<'a>( let sketches = supplier.map_or(0, |supplier| sketch_counts[supplier.0]) + usize::from(matches!( &node.payload, - asap_types::ir::Operator::ASAP(ASAPOp::SummaryAgg { - family: FieldDataType::Sketch(..), - .. - }) + asap_types::ir::Operator::ASAP( + ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(..), + .. + } | ASAPOp::PaneChain { + family: FieldDataType::Sketch(..), + .. + } + ) )); // Failure-budget sizing splits delta across approximate inputs. This // slice admits at most one sketch build along each unary output path. if sketches > 1 { continue; } + let (retained_panes, ingestion_window) = match &node.payload { + asap_types::ir::Operator::ASAP(ASAPOp::PaneChain { + pane_count, + pane_width_ms, + .. + }) => ( + u64::from(*pane_count), + pane_width_ms.checked_mul(u64::from(*pane_count)), + ), + _ => (1, None), + }; let Ok(priced) = crate::price_node( node, &views, (1, 1.0), lookbacks[id], - 1, - None, + retained_panes, + ingestion_window, context.data, context.models.plan_selection, context.calibration.factor(), ) else { continue; }; + let closure_work = (supplier.map_or(0.0, |supplier| closure_work_ms[supplier.0]) + + priced.work_ms) + .next_up(); + // With a unary closure this is the evaluator's distinct upstream sum. + // A larger worker count introduces compaction work and state copies. + if node.output_state.timing == ExecutionTiming::IngestionTime + && !single_worker(closure_work, nodes.len()) + { + continue; + } // Compare the actual evaluator outputs, including every piece of the // downstream pricing interface. This is not a template-row equivalence. if cache[id] @@ -439,9 +517,10 @@ fn price<'a>( continue; } if priced.cost.cost_per_second.is_none() - || priced.average_memory_bytes != 0.0 - || priced.disk_bytes != 0 - || priced.maintained.is_some() + || (node.output_state.timing == ExecutionTiming::QueryTime + && (priced.average_memory_bytes != 0.0 + || priced.disk_bytes != 0 + || priced.maintained.is_some())) { continue; } @@ -449,6 +528,7 @@ fn price<'a>( suppliers[id].push(supplier); } sketch_counts[id] = sketch_counts[id].max(sketches); + closure_work_ms[id] = closure_work_ms[id].max(closure_work); if cache[id].is_none() { cache[id] = Some(priced); raw_sources[id] = raw_source; @@ -535,10 +615,16 @@ fn hll_size_certified( } fn stable_size(node: &asap_types::ir::OperatorNode, delta: Option) -> bool { - let Some(ASAPOp::SummaryAgg { - family: FieldDataType::Sketch(kind, _), - .. - }) = node.asap() + let Some( + ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(kind, _), + .. + } + | ASAPOp::PaneChain { + family: FieldDataType::Sketch(kind, _), + .. + }, + ) = node.asap() else { return true; }; @@ -566,6 +652,14 @@ fn stable_size(node: &asap_types::ir::OperatorNode, delta: Option) -> bool } } +// Include a bound on roundoff when the evaluator sums the same positive work +// in reverse traversal order. Outward rounding of our path sum bounds the real +// sum; gamma_n bounds the evaluator's finite-precision accumulation. +fn single_worker(upper_work_ms: f64, node_count: usize) -> bool { + let error = node_count as f64 * f64::EPSILON; + upper_work_ms.is_finite() && error < 0.5 && (upper_work_ms / (1.0 - error)).next_up() < 1000.0 +} + fn check_deadline(deadline: Option) -> Result<(), DirectProfileError> { if deadline.is_some_and(|deadline| std::time::Instant::now() >= deadline) { Err(asap_logical_asap_optimization::memo::CandidateGenerationTimeout.into()) @@ -614,6 +708,14 @@ fn same_price(a: &PricedNode<'_>, b: &PricedNode<'_>) -> bool { #[cfg(test)] mod tests { + #[test] + fn ingestion_worker_bound_accounts_for_summation_roundoff() { + assert!(super::single_worker(999.0, 100)); + assert!(!super::single_worker(1000.0, 100)); + assert!(!super::single_worker(1000.0_f64.next_down(), 100)); + assert!(!super::single_worker(1001.0, 1)); + } + #[test] fn hll_admission_requires_stable_precision_and_a_supported_population() { assert!(super::hll_size_certified(15, Some(0.05), 0.05, 1000)); @@ -992,6 +1094,288 @@ mod tests { } } + #[test] + fn ingestion_pane_profiles_match_the_real_evaluator_and_reject_extra_workers() { + use asap_logical_asap_optimization::per_computation_candidates::{ + compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, + }; + use asap_types::ir::schema::SketchAlgorithm; + use std::rc::Rc; + fn append( + node: Rc, + configured: &mut CandidateConfigurationInventory, + ingestion: bool, + ) -> PhysicalCandidateId { + let ingestion = ingestion || matches!(node.asap(), Some(ASAPOp::PaneChain { .. })); + let pane = matches!(node.asap(), Some(ASAPOp::PaneChain { .. })); + let mut inputs = Vec::new(); + node.operator + .map_children(|child| inputs.push(append(child.clone(), configured, ingestion))); + let id = PhysicalCandidateId(configured.configurations.len()); + configured.configurations.push(CandidateConfiguration { + source_candidate: id, + materialization: if pane { + Materialization::MatAtIngestion(Medium::Memory) + } else { + Materialization::NotMaterialized + }, + timing: if ingestion { + ExecutionTiming::IngestionTime + } else { + ExecutionTiming::QueryTime + }, + }); + configured.inventory.templates.push(Some(node)); + configured + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: MemoGroupId(id.0 as u32), + local_cost: 0.0, + producer_requirements: inputs + .into_iter() + .map(|supplier| ProducerRequirement { + compatible_producers: vec![supplier].into(), + }) + .collect(), + }); + id + } + for (series_count, rows_per_second, speed_multiplier) in + [(2_u64, 100_u64, 1.0), (100, 1000, 1.0), (2, 100, 1000.0)] + { + for width in [60_000, 30_000] { + for algorithm in [SketchAlgorithm::Kll, SketchAlgorithm::DDSketch] { + let scope = AccuracyScope::PerValue; + let target = AccuracyTarget::max_error(0.05) + .failure_probability(0.01) + .accuracy_scope(scope); + let query = if algorithm == SketchAlgorithm::Hll { + "distinct_over_time(latency[5m])" + } else { + "quantile_over_time(0.95, latency[5m])" + }; + let original = lower_promql(query, target.clone()); + let generated = generate_per_computation_candidates(vec![( + 0, + QueryRoot::Operator(original.clone()), + )]) + .unwrap(); + let choice = generated.targets[0].choices.candidates.iter().position(|candidate| + matches!(candidate, ComputationCandidate::Summary(found) if *found == algorithm)).unwrap(); + let roots: Vec<_> = compose_logical_candidate(&generated, &[choice]) + .unwrap() + .into_iter() + .map(|(_, root)| root) + .collect(); + let pane_rules = + asap_logical_asap_optimization::asap_aware_cse::pane_candidates( + &roots, + &[Some(width)], + ); + let roots = asap_logical_asap_optimization::asap_aware_cse::window_composition_pass2_candidates(&roots, &pane_rules).unwrap().pop().unwrap().roots.unwrap(); + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival":{"ingesting":{"rows_per_second":rows_per_second}}, + "sources":{"latency":{"series_count":series_count,"rows_held":rows_per_second * 300,"values_non_negative":true,"max_distinct_per_evaluation":1000}} + })).unwrap(); + // A legal finite configuration is sized before direct pricing. + let sized = crate::resize_for_failure_probability( + &roots, + std::slice::from_ref(&target), + &data, + ) + .unwrap(); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph::default(), + templates: vec![], + group_candidates: vec![], + }, + configurations: vec![], + }; + let QueryRoot::Operator(root) = &sized[0] else { + panic!("operator") + }; + let id = append(root.clone(), &mut configured, false); + configured + .inventory + .graph + .query_roots + .push(QueryRootCandidates { + compatible_candidates: vec![id], + }); + let originals = vec![QueryRoot::Operator(original)]; + let entries = vec![QueryWorkloadEntry { + query: Query(query.into()), + requirements: QueryRequirements { + accuracy: target, + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval( + RepetitionInterval(60_000), + )), + range_evaluation: None, + }]; + let mut settings = crate::PlanSelectionSettings::default(); + settings.resource_speed.ms_per_cpu_op *= speed_multiplier; + settings.resource_speed.ms_per_scanned_byte *= speed_multiplier; + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &originals, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin().with_plan_selection(&settings), + calibration: crate::CalibrationFit::default(), + }; + let extracted = extract(&configured, &context).unwrap().unwrap(); + assert!( + extracted.supported.iter().all(|supported| *supported), + "{algorithm:?} {scope:?}" + ); + let direct_proposal = + crate::search::milp::model::solve_direct_cost_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &extracted.profile, + ); + let proposal = if speed_multiplier == 1.0 { + // Default nanosecond operation latency gives sub-threshold + // held-memory terms. Keep the explicit unsupported outcome; + // fixed-binding profile/evaluator parity remains meaningful. + assert!(matches!(direct_proposal, Err(crate::search::candidates::graph::CandidateSelectionError::UnsupportedDirectCost(_)))); + crate::search::milp::model::solve_candidate_proposal( + &configured.inventory.graph, + &[], + None, + ) + .unwrap() + .unwrap() + } else { + // An explicit microsecond operation calibration fits + // the same guard, with unchanged prices. + direct_proposal.unwrap().unwrap() + }; + let mut rejections = + crate::search::refinement::planner::CandidateRejectionSummary::default(); + let evaluated = + crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("{algorithm:?} {scope:?}: {rejections:?}")); + let cost = evaluated.0.costs[&evaluated.0.selected] + .cost_per_second + .unwrap(); + let direct = extracted + .profile + .selected_cost(&configured.inventory.graph, &proposal.selected); + assert!( + (cost - direct).abs() <= 1e-9 * cost.max(1e-15), + "{algorithm:?} {scope:?}: {direct} vs {cost}" + ); + let chain = configured + .inventory + .templates + .iter() + .position(|node| { + node.as_ref().is_some_and(|node| { + matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) + }) + }) + .unwrap(); + let Some(ASAPOp::PaneChain { + family, pane_count, .. + }) = configured.inventory.templates[chain] + .as_ref() + .unwrap() + .asap() + else { + unreachable!() + }; + let (_, state_bytes) = crate::summary_shape( + family, + crate::StateInputRows { + rows_per_state: Some( + (width * rows_per_second) + .div_ceil(1000) + .div_ceil(series_count), + ), + non_negative: true, + }, + ); + let CostModel::UsageOverTime(prices) = + &context.models.plan_selection.cost_model + else { + unreachable!() + }; + let expected_residency = + (u64::from(*pane_count) + 1) * series_count * state_bytes; + let expected_standing = + expected_residency as f64 * prices.cost_per_memory_byte_second; + assert!( + (extracted.profile.nodes[chain].standing_cost_per_second + - expected_standing) + .abs() + <= 1e-12 * expected_standing.max(1e-15) + ); + assert_eq!( + extracted.profile.nodes[chain].held_memory_cost_per_second, + 0.0 + ); + let mut overloaded = data.clone(); + overloaded.arrival = serde_json::from_value( + serde_json::json!({"ingesting":{"rows_per_second":1e12}}), + ) + .unwrap(); + let overloaded_context = CandidatePlanningContext { + data: &overloaded, + ..context + }; + let rejected = extract(&configured, &overloaded_context).unwrap().unwrap(); + assert!( + !rejected.supported[chain], + "multiworker compaction is outside this certified slice" + ); + if scope == AccuracyScope::WholeResult { + let mut unknown = data.clone(); + unknown.sources.get_mut("latency").unwrap().series_count = None; + let unproved = CandidatePlanningContext { + data: &unknown, + ..context + }; + assert!( + extract(&configured, &unproved).unwrap().is_none(), + "unknown population cannot certify a whole-result sizing budget" + ); + } + let too_small_delta = match algorithm { + SketchAlgorithm::Kll => Some(1e-12), + _ => None, + }; + if let Some(delta) = too_small_delta { + assert!(configured + .inventory + .templates + .iter() + .flatten() + .any(|node| matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) + && !stable_size(node, Some(delta)))); + } + } + } + } + } + // Real shared KLL/DDS plans verify schedule unions and held-memory lifetime, // including two equal frequencies whose different phases cannot coalesce. #[test] diff --git a/crates/plan-selection/src/search/milp/direct_cost.rs b/crates/plan-selection/src/search/milp/direct_cost.rs index 60d6cbc02..f6acbbccb 100644 --- a/crates/plan-selection/src/search/milp/direct_cost.rs +++ b/crates/plan-selection/src/search/milp/direct_cost.rs @@ -21,12 +21,24 @@ const MINIMUM_QUERY_RATE_WEIGHT: f64 = 1e-6; #[derive(Debug, Clone, Default)] pub(crate) struct NodeCostProfile { pub execution_cost: f64, + pub ingestion: bool, + pub standing_cost_per_second: f64, pub work_seconds: f64, pub held_memory_cost_per_second: f64, pub materialized: bool, pub raw_retention: Option<(String, f64)>, } +impl NodeCostProfile { + fn query_work_seconds(&self) -> f64 { + if self.ingestion { + 0.0 + } else { + self.work_seconds + } + } +} + /// Exact evaluator demand for each nonempty selected query subset. Only small /// heterogeneous domains are admitted; this is not a workload-plan product. #[derive(Debug, Clone)] @@ -58,6 +70,8 @@ impl DirectCostProfile { || !valid(self.request_cost) || self.nodes.iter().any(|n| { !valid(n.execution_cost) + || !valid(n.standing_cost_per_second) + || (n.ingestion && n.held_memory_cost_per_second != 0.0) || !valid(n.work_seconds) || !valid(n.held_memory_cost_per_second) || n.raw_retention @@ -69,6 +83,32 @@ impl DirectCostProfile { "invalid direct-cost profile".into(), )); } + for (consumer, candidate) in graph.candidates.iter().enumerate() { + for producer in candidate + .producer_requirements + .iter() + .flat_map(|input| input.compatible_producers.iter()) + { + let source = &self.nodes[producer.0]; + if (self.nodes[consumer].ingestion && !source.ingestion) + || (!self.nodes[consumer].ingestion && source.ingestion && !source.materialized) + { + return Err(CandidateSelectionError::InvalidGraph( + "direct-cost edge crosses an unsupported ingestion boundary".into(), + )); + } + } + } + if graph + .query_roots + .iter() + .flat_map(|root| &root.compatible_candidates) + .any(|id| self.nodes[id.0].ingestion && !self.nodes[id.0].materialized) + { + return Err(CandidateSelectionError::InvalidGraph( + "query output cannot directly read unmaterialized ingestion work".into(), + )); + } if let Some(schedule) = &self.schedules { let queries = graph.query_roots.len(); if queries > 3 { @@ -154,7 +194,7 @@ impl DirectCostProfile { .iter() .enumerate() .filter(|(_, yes)| **yes) - .map(|(id, _)| self.nodes[id].work_seconds) + .map(|(id, _)| self.nodes[id].query_work_seconds()) .sum::() }) .collect(); @@ -163,6 +203,9 @@ impl DirectCostProfile { |schedule| schedule.per_query_rates.clone(), ); let union_rate = |id: usize| { + if self.nodes[id].ingestion { + return 1.0; + } let Some(schedule) = &self.schedules else { return self.query_rate; }; @@ -224,8 +267,9 @@ impl DirectCostProfile { 0.0 } }; - total += execution_rate - * (node.execution_cost + node.held_memory_cost_per_second * lifetime); + total += node.standing_cost_per_second + + execution_rate + * (node.execution_cost + node.held_memory_cost_per_second * lifetime); if let Some((source, cost)) = &node.raw_retention { let value = raw.entry(source).or_default(); *value = value.max(*cost); @@ -235,7 +279,12 @@ impl DirectCostProfile { } fn maximum_latency(&self) -> f64 { - self.request_latency_seconds + self.nodes.iter().map(|n| n.work_seconds).sum::() + self.request_latency_seconds + + self + .nodes + .iter() + .map(NodeCostProfile::query_work_seconds) + .sum::() } pub(crate) fn coefficients(&self) -> impl Iterator + '_ { @@ -255,12 +304,18 @@ impl DirectCostProfile { ); self.nodes.iter().flat_map(move |node| { let mut coefficients = Vec::with_capacity(rates.len() * 2 + 1); - for rate in &rates { + let node_rates = if node.ingestion { + &[1.0][..] + } else { + &rates[..] + }; + for rate in node_rates { coefficients.extend([ node.execution_cost * rate, node.held_memory_cost_per_second * rate * latency_scale, ]); } + coefficients.push(node.standing_cost_per_second); coefficients.push(node.raw_retention.as_ref().map_or(0.0, |(_, cost)| *cost)); coefficients }) @@ -415,7 +470,7 @@ pub(super) fn encode( .iter() .enumerate() .filter_map(|(id, reached)| { - reached.map(|r| profile.nodes[id].work_seconds / latency_scale * r) + reached.map(|r| profile.nodes[id].query_work_seconds() / latency_scale * r) }) .sum::() }) @@ -448,7 +503,13 @@ pub(super) fn encode( } else { boundary_count[id].clone() }; - objective += node.execution_cost * profile.query_rate / divisor * executions.clone(); + let execution_rate = if node.ingestion { + 1.0 + } else { + profile.query_rate + }; + objective += node.standing_cost_per_second / divisor * active[id]; + objective += node.execution_cost * execution_rate / divisor * executions.clone(); if let Some((source, cost)) = &node.raw_retention { retained.entry(source).or_default().push((id, *cost)); } @@ -604,6 +665,8 @@ mod tests { request_cost: 7.0, nodes: vec![ NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 1.0, held_memory_cost_per_second: 3.0, @@ -611,12 +674,16 @@ mod tests { raw_retention: None, }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, raw_retention: Some(("source".into(), 4.0)), ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 3.0, raw_retention: Some(("source".into(), 6.0)), @@ -645,12 +712,16 @@ mod tests { request_cost: 7.0, nodes: vec![ NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 1.0, held_memory_cost_per_second: 3.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, held_memory_cost_per_second: 4.0, @@ -658,10 +729,14 @@ mod tests { raw_retention: None, }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, work_seconds: 3.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, work_seconds: 4.0, ..Default::default() }, @@ -690,6 +765,8 @@ mod tests { request_cost: 7.0, nodes: vec![ NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: materialized_build_cost, work_seconds: 1.0, held_memory_cost_per_second: 3.0, @@ -697,17 +774,23 @@ mod tests { raw_retention: None, }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 0.5, held_memory_cost_per_second: 1.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 3.0, ..Default::default() @@ -758,10 +841,14 @@ mod tests { schedules: None, nodes: vec![ NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 7.1e-13, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 7.1e-13 * 2.2e9, ..Default::default() }, @@ -787,4 +874,193 @@ mod tests { assert!((proposal.direct_modeled_cost.unwrap() / expected - 1.0).abs() < 1e-9); assert!((proposal.model_lower_bound.unwrap() / expected - 1.0).abs() < 1e-9); } + fn ingestion_profile( + two_states: bool, + query_rate: f64, + ) -> (CandidateDependencyGraph, DirectCostProfile) { + let mut candidates = vec![node(None), node(Some(0))]; + let mut nodes = vec![ + NodeCostProfile { + ingestion: true, + execution_cost: 2.0, + work_seconds: 0.2, + ..Default::default() + }, + NodeCostProfile { + ingestion: true, + materialized: true, + execution_cost: 3.0, + standing_cost_per_second: 7.0, + work_seconds: 0.4, + ..Default::default() + }, + ]; + if two_states { + candidates.push(node(Some(0))); + nodes.push(nodes[1].clone()); + } + let first_reader = candidates.len(); + candidates.extend([node(Some(1)), node(Some(if two_states { 2 } else { 1 }))]); + nodes.extend((0..2).map(|_| NodeCostProfile { + execution_cost: 1.0, + work_seconds: 0.5, + held_memory_cost_per_second: 2.0, + ..Default::default() + })); + ( + CandidateDependencyGraph { + candidates, + query_roots: vec![root(first_reader), root(first_reader + 1)], + ..Default::default() + }, + DirectCostProfile { + nodes, + schedules: None, + query_rate, + request_latency_seconds: 0.1, + request_cost: 1.0, + }, + ) + } + + // Update work and standing state are charged once regardless of query + // frequency; ingestion work contributes no query-held evaluation latency. + #[test] + fn maintained_state_cost_is_independent_of_reader_frequency() { + for rate in [1.0, 5.0] { + let (graph, profile) = ingestion_profile(false, rate); + let expected = 13.0 + 4.4 * rate; + let proposal = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((proposal.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-6); + } + } + + // Two maintained states independently update a shared unmaterialized scan. + // Both equal and heterogeneous query schedules leave that ingestion rate2. + #[test] + fn two_ingestion_boundaries_recompute_their_shared_input() { + let (graph, mut profile) = ingestion_profile(true, 1.0); + assert!((solve(&graph, &profile) - 29.4).abs() < 1e-6); + profile.schedules = Some(QueryScheduleRates { + per_query_rates: vec![1.0, 2.0], + materialized_subset_rates: vec![0.0, 1.0, 2.0, 3.0], + }); + let proposal = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((proposal.direct_modeled_cost.unwrap() - 31.6).abs() < 1e-8); + assert!((proposal.model_lower_bound.unwrap() - 31.6).abs() < 1e-6); + } + + // Enumerate shared or separate state choices without charging a shared + // producer twice; standing memory can change which choice wins. + #[test] + fn ingestion_selection_matches_shared_state_cost_oracle() { + let graph = CandidateDependencyGraph { + candidates: vec![ + node(None), + node(Some(0)), + node(Some(0)), + node(Some(0)), + PhysicalCandidate { + group: MemoGroupId(1), + local_cost: 0.0, + producer_requirements: vec![ProducerRequirement { + compatible_producers: vec![PhysicalCandidateId(1), PhysicalCandidateId(2)] + .into(), + }], + }, + PhysicalCandidate { + group: MemoGroupId(1), + local_cost: 0.0, + producer_requirements: vec![ProducerRequirement { + compatible_producers: vec![PhysicalCandidateId(1), PhysicalCandidateId(3)] + .into(), + }], + }, + ], + query_roots: vec![root(4), root(5)], + ..Default::default() + }; + for shared_standing in [1.0, 20.0] { + let profile = DirectCostProfile { + query_rate: 5.0, + schedules: None, + request_latency_seconds: 0.0, + request_cost: 0.0, + nodes: vec![ + NodeCostProfile { + ingestion: true, + execution_cost: 2.0, + ..Default::default() + }, + NodeCostProfile { + ingestion: true, + materialized: true, + execution_cost: 3.0, + standing_cost_per_second: shared_standing, + ..Default::default() + }, + NodeCostProfile { + ingestion: true, + materialized: true, + execution_cost: 3.0, + standing_cost_per_second: 7.0, + ..Default::default() + }, + NodeCostProfile { + ingestion: true, + materialized: true, + execution_cost: 3.0, + standing_cost_per_second: 7.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + ..Default::default() + }, + ], + }; + let expected = [[1, 1], [1, 3], [2, 1], [2, 3]] + .into_iter() + .map(|assignment| { + let states: std::collections::HashSet<_> = assignment.into_iter().collect(); + 10.0 + states + .iter() + .map(|state| 2.0 + 3.0 + if *state == 1 { shared_standing } else { 7.0 }) + .sum::() + }) + .fold(f64::INFINITY, f64::min); + let proposal = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((proposal.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-6); + } + } } diff --git a/crates/plan-selection/src/search/milp/direct_schedule.rs b/crates/plan-selection/src/search/milp/direct_schedule.rs index ffabd6133..727278842 100644 --- a/crates/plan-selection/src/search/milp/direct_schedule.rs +++ b/crates/plan-selection/src/search/milp/direct_schedule.rs @@ -28,7 +28,7 @@ pub(super) fn encode( .iter() .enumerate() .filter_map(|(id, reached)| { - reached.map(|r| profile.nodes[id].work_seconds / latency_scale * r) + reached.map(|r| profile.nodes[id].query_work_seconds() / latency_scale * r) }) .sum::() }) @@ -85,6 +85,8 @@ pub(super) fn encode( let Some(reaches) = reaches else { continue }; if boundary < queries { demands[id].push((reaches, schedule.per_query_rates[boundary])); + } else if profile.nodes[boundary - queries].ingestion { + demands[id].push((reaches, 1.0)); } else { for (mask, selector) in masks[boundary - queries].iter().enumerate().skip(1) { let Some(selector) = selector else { continue }; @@ -101,7 +103,9 @@ pub(super) fn encode( let mut retained = BTreeMap::<&str, Vec<(usize, f64)>>::new(); for (id, node) in profile.nodes.iter().enumerate() { check_deadline()?; - let executions: Expression = if node.materialized { + let executions: Expression = if node.materialized && node.ingestion { + active[id].into() + } else if node.materialized { masks[id] .iter() .enumerate() @@ -113,6 +117,7 @@ pub(super) fn encode( } else { demands[id].iter().map(|(used, rate)| *rate * *used).sum() }; + objective += node.standing_cost_per_second / divisor * active[id]; objective += node.execution_cost / divisor * executions; if let Some((source, cost)) = &node.raw_retention { retained.entry(source).or_default().push((id, *cost)); @@ -236,12 +241,16 @@ mod tests { request_cost: 7.0, nodes: vec![ NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 1.0, held_memory_cost_per_second: 3.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, held_memory_cost_per_second: 4.0, @@ -249,10 +258,14 @@ mod tests { raw_retention: None, }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, work_seconds: 3.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, work_seconds: 4.0, ..Default::default() }, @@ -302,6 +315,8 @@ mod tests { request_latency_seconds: 0.0, request_cost: 0.0, nodes: vec![NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 3.0, work_seconds: 2.0, held_memory_cost_per_second: 2.0, @@ -353,6 +368,8 @@ mod tests { request_cost: 7.0, nodes: vec![ NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 10.0, work_seconds: 1.0, held_memory_cost_per_second: 3.0, @@ -360,6 +377,8 @@ mod tests { raw_retention: None, }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 0.5, held_memory_cost_per_second: 1.0, @@ -367,11 +386,15 @@ mod tests { raw_retention: None, }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 3.0, ..Default::default() @@ -442,20 +465,28 @@ mod tests { request_cost: 0.0, nodes: vec![ NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, held_memory_cost_per_second: 1e6, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 2.0, held_memory_cost_per_second: 1e6, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, work_seconds: 1.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, work_seconds: 1e9, ..Default::default() }, diff --git a/crates/plan-selection/src/search/refinement/search.rs b/crates/plan-selection/src/search/refinement/search.rs index b67b2258a..f3365b66f 100644 --- a/crates/plan-selection/src/search/refinement/search.rs +++ b/crates/plan-selection/src/search/refinement/search.rs @@ -686,10 +686,14 @@ mod tests { schedules: None, nodes: vec![ NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 1.0, ..Default::default() }, NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, execution_cost: 2.0, ..Default::default() }, From 369d96a7f4e230349e75fffba69b3e8e6ae02e89 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 04:13:03 +0000 Subject: [PATCH 13/29] feat: directly price rebuilt and retained query-time panes --- .../src/search/costing/direct_profile.rs | 359 ++++++++++++++---- .../src/search/milp/direct_cost.rs | 76 +++- .../plan-selection/src/search/milp/model.rs | 18 +- .../src/search/refinement/planner.rs | 33 +- .../src/search/refinement/search.rs | 90 ++++- 5 files changed, 482 insertions(+), 94 deletions(-) diff --git a/crates/plan-selection/src/search/costing/direct_profile.rs b/crates/plan-selection/src/search/costing/direct_profile.rs index d8e97d910..bef12924a 100644 --- a/crates/plan-selection/src/search/costing/direct_profile.rs +++ b/crates/plan-selection/src/search/costing/direct_profile.rs @@ -62,6 +62,23 @@ fn supported_node(node: &asap_types::ir::OperatorNode) -> bool { pub(in crate::search) fn extract( configured: &CandidateConfigurationInventory, context: &CandidatePlanningContext<'_>, +) -> Result, DirectProfileError> { + extract_scan_class(configured, context, false) +} + +/// A second restricted proposal keeps the newest-pane scan class separate from +/// ordinary full-window scans. The full candidate graph is never changed. +pub(in crate::search) fn extract_latest_panes( + configured: &CandidateConfigurationInventory, + context: &CandidatePlanningContext<'_>, +) -> Result, DirectProfileError> { + extract_scan_class(configured, context, true) +} + +fn extract_scan_class( + configured: &CandidateConfigurationInventory, + context: &CandidatePlanningContext<'_>, + prefer_latest_pane: bool, ) -> Result, DirectProfileError> { let settings = context.models.plan_selection; let CostModel::UsageOverTime(prices) = &settings.cost_model else { @@ -155,12 +172,11 @@ pub(in crate::search) fn extract( && (requires_source_sizing(node) || stable_size(node, delta)) && match configuration.timing { ExecutionTiming::QueryTime => { - !matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) - && matches!( - configuration.materialization, - Materialization::NotMaterialized - | Materialization::MatAtQuery(Medium::Memory) - ) + matches!(configuration.materialization, Materialization::NotMaterialized | Materialization::MatAtQuery(Medium::Memory)) + || (configuration.materialization == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) + && matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) + && context.models.deployment_capabilities.query_time_results_retained + && (0..context.entries.len()).any(|reader| asap_physical_asap_optimization::summary_materialization_eligibility(node, &[reader], context.entries, context.data, context.models.deployment_capabilities).is_ok_and(|eligibility| eligibility.may_keep_across_evaluations))) } ExecutionTiming::IngestionTime => match node.asap() { Some(ASAPOp::PaneChain { .. }) => { @@ -210,47 +226,64 @@ pub(in crate::search) fn extract( }), ); } - // The evaluator inspects immediate scan consumers (and TimeShift, excluded - // here). Different possible window/no-window contexts cannot share a price. + // Each scan keeps one explicit lookback class, not a maximum requiring an + // optionally selected witness. Every admitted consumer edge must use it. let mut lookbacks = vec![None; count]; for id in 0..count { - if !nodes[id].as_ref().is_some_and(|node| { - matches!( - node.payload, + let Some(scan) = &nodes[id] else { continue }; + if scan.output_state.timing != ExecutionTiming::QueryTime + || !matches!( + scan.payload, asap_types::ir::Operator::NonASAP(NonASAPOp::Scan { .. }) ) - }) { + { continue; } - let mut windows = HashSet::new(); + let mut ordinary = HashSet::new(); + let mut kept = HashSet::new(); for (consumer, candidate) in configured.inventory.graph.candidates.iter().enumerate() { check_deadline(context.deadline)?; let Some(node) = &nodes[consumer] else { continue; }; + if node.output_state.timing != ExecutionTiming::QueryTime { + continue; + } if candidate.producer_requirements.iter().any(|input| { input .compatible_producers .contains(&PhysicalCandidateId(id)) }) { - windows.insert(match &node.payload { - asap_types::ir::Operator::NonASAP(NonASAPOp::TimeRange { range, .. }) => { - Some(range.as_millis() as u64) - } - _ => None, - }); + if node.materialization + == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) + { + kept.insert(scan_consumer_lookback(node)); + } else { + ordinary.insert(scan_consumer_lookback(node)); + } } } - if configured.inventory.graph.query_roots.iter().any(|root| { + let raw_output = configured.inventory.graph.query_roots.iter().any(|root| { root.compatible_candidates .contains(&PhysicalCandidateId(id)) - }) { - windows.insert(None); - } - if windows.len() == 1 { - lookbacks[id] = *windows.iter().next().unwrap(); + }); + if raw_output { + lookbacks[id] = None; } else { - nodes[id] = None; + let choices = if prefer_latest_pane && !kept.is_empty() { + &kept + } else if !ordinary.is_empty() { + &ordinary + } else { + &kept + }; + if choices.is_empty() { + nodes[id] = None; + } else if choices.contains(&None) { + lookbacks[id] = None; + } else { + lookbacks[id] = choices.iter().copied().max().flatten(); + } } } let mut cache = vec![None; count]; @@ -399,6 +432,16 @@ fn price<'a>( continue; }; let producer = nodes[supplier.0].as_ref().unwrap(); + if producer.output_state.timing == ExecutionTiming::QueryTime + && matches!( + producer.payload, + asap_types::ir::Operator::NonASAP(NonASAPOp::Scan { .. }) + ) + && lookbacks[supplier.0] != scan_consumer_lookback(node) + { + continue; + } + let consumer_ingests = node.output_state.timing == ExecutionTiming::IngestionTime; let producer_ingests = producer.output_state.timing == ExecutionTiming::IngestionTime; if (consumer_ingests && !producer_ingests) @@ -518,6 +561,8 @@ fn price<'a>( } if priced.cost.cost_per_second.is_none() || (node.output_state.timing == ExecutionTiming::QueryTime + && node.materialization + != Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) && (priced.average_memory_bytes != 0.0 || priced.disk_bytes != 0 || priced.maintained.is_some())) @@ -652,6 +697,28 @@ fn stable_size(node: &asap_types::ir::OperatorNode, delta: Option) -> bool } } +fn scan_consumer_lookback(node: &PhysicalASAPDAGNode) -> Option { + match &node.payload { + asap_types::ir::Operator::NonASAP(NonASAPOp::TimeRange { range, .. }) => { + Some(range.as_millis() as u64) + } + asap_types::ir::Operator::ASAP(ASAPOp::PaneChain { + pane_width_ms, + pane_count, + .. + }) => Some( + if node.materialization + == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) + { + *pane_width_ms + } else { + pane_width_ms.saturating_mul(u64::from(*pane_count)) + }, + ), + _ => None, + } +} + // Include a bound on roundoff when the evaluator sums the same positive work // in reverse traversal order. Outward rounding of our path sum bounds the real // sum; gamma_n bounds the evaluator's finite-precision accumulation. @@ -1095,7 +1162,24 @@ mod tests { } #[test] - fn ingestion_pane_profiles_match_the_real_evaluator_and_reject_extra_workers() { + fn ingestion_pane_profiles_match_the_real_evaluator() { + check_pane_profile(Materialization::MatAtIngestion(Medium::Memory)); + } + + #[test] + fn rebuilt_pane_profiles_match_the_real_evaluator() { + check_pane_profile(Materialization::NotMaterialized); + check_pane_profile(Materialization::MatAtQuery(Medium::Memory)); + } + + #[test] + fn kept_pane_profiles_and_scan_classes_match_the_real_evaluator() { + check_pane_profile(Materialization::MatAtQueryKeptAcrossEvaluations( + Medium::Memory, + )); + } + + fn check_pane_profile(mode: Materialization) { use asap_logical_asap_optimization::per_computation_candidates::{ compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, }; @@ -1105,17 +1189,21 @@ mod tests { node: Rc, configured: &mut CandidateConfigurationInventory, ingestion: bool, + mode: Materialization, ) -> PhysicalCandidateId { - let ingestion = ingestion || matches!(node.asap(), Some(ASAPOp::PaneChain { .. })); + let ingestion = ingestion + || (matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) + && mode == Materialization::MatAtIngestion(Medium::Memory)); let pane = matches!(node.asap(), Some(ASAPOp::PaneChain { .. })); let mut inputs = Vec::new(); - node.operator - .map_children(|child| inputs.push(append(child.clone(), configured, ingestion))); + node.operator.map_children(|child| { + inputs.push(append(child.clone(), configured, ingestion, mode)) + }); let id = PhysicalCandidateId(configured.configurations.len()); configured.configurations.push(CandidateConfiguration { source_candidate: id, materialization: if pane { - Materialization::MatAtIngestion(Medium::Memory) + mode } else { Materialization::NotMaterialized }, @@ -1197,7 +1285,7 @@ mod tests { let QueryRoot::Operator(root) = &sized[0] else { panic!("operator") }; - let id = append(root.clone(), &mut configured, false); + let id = append(root.clone(), &mut configured, false, mode); configured .inventory .graph @@ -1205,8 +1293,16 @@ mod tests { .push(QueryRootCandidates { compatible_candidates: vec![id], }); - let originals = vec![QueryRoot::Operator(original)]; - let entries = vec![QueryWorkloadEntry { + configured + .inventory + .graph + .query_roots + .push(configured.inventory.graph.query_roots[0].clone()); + let originals = vec![ + QueryRoot::Operator(original.clone()), + QueryRoot::Operator(original), + ]; + let mut entries = vec![QueryWorkloadEntry { query: Query(query.into()), requirements: QueryRequirements { accuracy: target, @@ -1214,20 +1310,28 @@ mod tests { }, predictability: Predictability::Predictable { known_at: None }, recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval( - RepetitionInterval(60_000), + RepetitionInterval(width as u32), )), range_evaluation: None, }]; + entries.push(entries[0].clone()); let mut settings = crate::PlanSelectionSettings::default(); settings.resource_speed.ms_per_cpu_op *= speed_multiplier; settings.resource_speed.ms_per_scanned_byte *= speed_multiplier; + let deployment = asap_logical_asap_optimization::DeploymentCapabilities { + query_time_results_retained: true, + ..Default::default() + }; let context = CandidatePlanningContext { deadline: None, refine_materialization: false, original_roots: &originals, entries: &entries, data: &data, - models: crate::PlanningModels::builtin().with_plan_selection(&settings), + models: crate::PlanningModels { + deployment_capabilities: &deployment, + ..crate::PlanningModels::builtin().with_plan_selection(&settings) + }, calibration: crate::CalibrationFit::default(), }; let extracted = extract(&configured, &context).unwrap().unwrap(); @@ -1243,22 +1347,14 @@ mod tests { &mut Default::default(), &extracted.profile, ); - let proposal = if speed_multiplier == 1.0 { - // Default nanosecond operation latency gives sub-threshold - // held-memory terms. Keep the explicit unsupported outcome; - // fixed-binding profile/evaluator parity remains meaningful. - assert!(matches!(direct_proposal, Err(crate::search::candidates::graph::CandidateSelectionError::UnsupportedDirectCost(_)))); - crate::search::milp::model::solve_candidate_proposal( - &configured.inventory.graph, - &[], - None, - ) - .unwrap() - .unwrap() - } else { - // An explicit microsecond operation calibration fits - // the same guard, with unchanged prices. - direct_proposal.unwrap().unwrap() + let proposal = match direct_proposal { + Ok(Some(proposal)) => proposal, + Err(crate::search::candidates::graph::CandidateSelectionError::UnsupportedDirectCost(_)) if speed_multiplier == 1.0 => { + // Default nanosecond work can give sub-threshold + // held-memory coefficients. Preserve that fallback. + crate::search::milp::model::solve_candidate_proposal(&configured.inventory.graph, &[], None).unwrap().unwrap() + } + result => panic!("{mode:?} {algorithm:?} width={width} series={series_count} speed={speed_multiplier}: {result:?}"), }; let mut rejections = crate::search::refinement::planner::CandidateRejectionSummary::default(); @@ -1318,8 +1414,14 @@ mod tests { else { unreachable!() }; - let expected_residency = - (u64::from(*pane_count) + 1) * series_count * state_bytes; + let expected_residency = match mode { + Materialization::MatAtIngestion(_) => u64::from(*pane_count) + 1, + Materialization::MatAtQueryKeptAcrossEvaluations(_) => { + u64::from(*pane_count) - 1 + } + _ => 0, + } * series_count + * state_bytes; let expected_standing = expected_residency as f64 * prices.cost_per_memory_byte_second; assert!( @@ -1328,24 +1430,137 @@ mod tests { .abs() <= 1e-12 * expected_standing.max(1e-15) ); - assert_eq!( - extracted.profile.nodes[chain].held_memory_cost_per_second, - 0.0 - ); - let mut overloaded = data.clone(); - overloaded.arrival = serde_json::from_value( - serde_json::json!({"ingesting":{"rows_per_second":1e12}}), - ) - .unwrap(); - let overloaded_context = CandidatePlanningContext { - data: &overloaded, - ..context - }; - let rejected = extract(&configured, &overloaded_context).unwrap().unwrap(); - assert!( - !rejected.supported[chain], - "multiworker compaction is outside this certified slice" - ); + if matches!( + mode, + Materialization::MatAtIngestion(_) + | Materialization::MatAtQueryKeptAcrossEvaluations(_) + ) { + assert_eq!( + extracted.profile.nodes[chain].held_memory_cost_per_second, + 0.0 + ); + } else { + assert!(extracted.profile.nodes[chain].held_memory_cost_per_second > 0.0); + } + if mode == Materialization::MatAtIngestion(Medium::Memory) { + let mut overloaded = data.clone(); + overloaded.arrival = serde_json::from_value( + serde_json::json!({"ingesting":{"rows_per_second":1e12}}), + ) + .unwrap(); + let overloaded_context = CandidatePlanningContext { + data: &overloaded, + ..context + }; + let rejected = extract(&configured, &overloaded_context).unwrap().unwrap(); + assert!( + !rejected.supported[chain], + "multiworker compaction is outside this certified slice" + ); + } + if mode == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) { + // The same physical scan may feed either a full-window + // rebuild or a newest-pane refresh. Keep these priced + // domains distinct, and validate each selected binding. + let normal_chain = PhysicalCandidateId(configured.configurations.len()); + configured.configurations.push(CandidateConfiguration { + source_candidate: normal_chain, + timing: ExecutionTiming::QueryTime, + materialization: Materialization::MatAtQuery(Medium::Memory), + }); + configured + .inventory + .templates + .push(configured.inventory.templates[chain].clone()); + configured + .inventory + .graph + .candidates + .push(configured.inventory.graph.candidates[chain].clone()); + for candidate in &mut configured.inventory.graph.candidates { + for input in &mut candidate.producer_requirements { + if input + .compatible_producers + .contains(&PhysicalCandidateId(chain)) + { + input.compatible_producers.push(normal_chain); + } + } + } + let normal = extract(&configured, &context).unwrap().unwrap(); + let latest = extract_latest_panes(&configured, &context) + .unwrap() + .unwrap(); + assert!(!normal.supported[chain] && normal.supported[normal_chain.0]); + assert!(latest.supported[chain] && !latest.supported[normal_chain.0]); + let scan = configured + .inventory + .templates + .iter() + .position(|node| { + node.as_ref().is_some_and(|node| { + matches!(node.non_asap(), Some(NonASAPOp::Scan { .. })) + }) + }) + .unwrap(); + let ratio = 300_000.0 / width as f64; + assert!( + (normal.profile.nodes[scan].execution_cost + - ratio * latest.profile.nodes[scan].execution_cost) + .abs() + <= 1e-12 * normal.profile.nodes[scan].execution_cost + ); + let mut full_selection = proposal.selected.clone(); + for id in &mut full_selection.selected_candidates { + if id.0 == chain { + *id = normal_chain; + } + } + for edge in &mut full_selection.connections { + if edge.producer.0 == chain { + edge.producer = normal_chain; + } + if edge.consumer.0 == chain { + edge.consumer = normal_chain; + } + } + let full_evaluated = + crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &full_selection, + 1, + &mut Default::default(), + &mut Vec::new(), + ) + .unwrap() + .unwrap(); + let actual = full_evaluated.0.costs[&full_evaluated.0.selected] + .cost_per_second + .unwrap(); + let modeled = normal + .profile + .selected_cost(&configured.inventory.graph, &full_selection); + assert!( + (actual - modeled).abs() <= 1e-9 * actual.max(1e-15), + "full-window class: {actual} vs {modeled}" + ); + let forbidden = + asap_logical_asap_optimization::DeploymentCapabilities::default(); + let no_retention = CandidatePlanningContext { + models: crate::PlanningModels { + deployment_capabilities: &forbidden, + ..context.models + }, + ..context + }; + assert!( + !extract(&configured, &no_retention) + .unwrap() + .unwrap() + .supported[chain] + ); + } if scope == AccuracyScope::WholeResult { let mut unknown = data.clone(); unknown.sources.get_mut("latency").unwrap().series_count = None; diff --git a/crates/plan-selection/src/search/milp/direct_cost.rs b/crates/plan-selection/src/search/milp/direct_cost.rs index f6acbbccb..7ebe39ed1 100644 --- a/crates/plan-selection/src/search/milp/direct_cost.rs +++ b/crates/plan-selection/src/search/milp/direct_cost.rs @@ -1,4 +1,6 @@ -//! Exact objective encoding for certified equal-schedule query-time profiles. +//! Exact objective encoding for certified query-time and ingestion profiles. +//! Fixed evaluation schedules may be equal or use a bounded union-rate table; +//! maintained and retained states carry separate standing resource charges. //! Profiles are supplied by physical costing; this module never infers prices //! from a logical template or treats accuracy as a monetary penalty. use good_lp::{constraint, variable, Constraint, Expression, ProblemVariables, Variable}; @@ -1063,4 +1065,76 @@ mod tests { assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-6); } } + // Query-kept state refreshes once at its readers' union rate. Residency is + // charged once, while its unmaterialized input is held for the weighted + // reader lifetime. This differs from charging the kept state per reader. + #[test] + fn retained_query_state_prices_residency_refresh_and_input_lifetime() { + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(Some(0)), node(Some(1)), node(Some(1))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + let mut profile = DirectCostProfile { + query_rate: 2.0, + schedules: None, + request_latency_seconds: 0.1, + request_cost: 0.0, + nodes: vec![ + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 2.0, + materialized: true, + standing_cost_per_second: 7.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 0.5, + held_memory_cost_per_second: 2.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 1.5, + held_memory_cost_per_second: 2.0, + ..Default::default() + }, + ], + }; + // Latencies are 3.6 and 4.6 seconds. Equal-rate readers share one + // refresh; distinct fixed intervals add under the evaluator's policy. + for (rates, union, expected) in [(vec![2.0, 2.0], 2.0, 74.4), (vec![1.0, 2.0], 3.0, 83.0)] { + if rates[0] != rates[1] { + profile.schedules = Some(QueryScheduleRates { + per_query_rates: rates.clone(), + materialized_subset_rates: vec![0.0, rates[0], rates[1], union], + }); + } + let weighted_latency = rates[0] * 3.6 + rates[1] * 4.6; + let rate_sum: f64 = rates.iter().sum(); + let independent = 7.0 + + union * (3.0 + 3.0 * weighted_latency / rate_sum) + + rate_sum + + 2.0 * weighted_latency; + assert!((independent - expected).abs() < 1e-10); + let proposal = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((proposal.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-7 * expected); + } + } } diff --git a/crates/plan-selection/src/search/milp/model.rs b/crates/plan-selection/src/search/milp/model.rs index 15d9f2b1f..c13e04bfe 100644 --- a/crates/plan-selection/src/search/milp/model.rs +++ b/crates/plan-selection/src/search/milp/model.rs @@ -658,8 +658,8 @@ fn solve_with_objective( warm_start.as_ref().map(|start| start.cost / divisor), )); constraints.push(constraint!(total_cost >= local_bound)); - if let Some(profile) = direct_profile { - let objective = super::direct_cost::encode( + let objective = if let Some(profile) = direct_profile { + super::direct_cost::encode( graph, profile, &mut variables, @@ -670,9 +670,12 @@ fn solve_with_objective( &objective_reach, divisor, check_deadline, - )?; - constraints.push(constraint!(total_cost >= objective)); - } + )? + } else { + Expression::from(total_cost) + }; + // A certified direct expression needs no epigraph row. Keeping its large + // currency sum in the objective avoids postsolve cancellation in that row. for observation in observations { check_deadline()?; for witness in &observation.rejected_queries { @@ -716,6 +719,7 @@ fn solve_with_objective( } } match observation.exact_cost { + Some(_) if direct_profile.is_some() => {} // Already priced exactly. Some(cost) => constraints.push(constraint!( total_cost >= (cost / divisor) * (1.0 - mismatch) )), @@ -748,7 +752,7 @@ fn solve_with_objective( metrics.peak_variables = metrics.peak_variables.max(variables.len()); metrics.peak_constraints = metrics.peak_constraints.max(constraints.len()); let mut model = variables - .minimise(total_cost) + .minimise(objective.clone()) .using(highs) .set_option("mip_rel_gap", 0.0) .set_option("mip_abs_gap", 0.0) @@ -811,7 +815,7 @@ fn solve_with_objective( graph, selected, solution.status(), - solution.value(total_cost) * divisor, + solution.eval(&objective) * divisor, ) .map(|mut proposal| { proposal.direct_modeled_cost = direct_modeled_cost; diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index 5cfa95d95..fe0320b7e 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -122,7 +122,24 @@ pub(in crate::search) fn optimize_configured_candidates( } else { Vec::new() }; - let direct = direct_cost_seed(configured, context, &exclusions)?; + let mut direct = Vec::new(); + if let Some(seed) = direct_cost_seed(configured, context, &exclusions, false)? { + direct.push(seed); + } + if configured.configurations.iter().any(|configuration| { + matches!( + configuration.materialization, + asap_types::physical::Materialization::MatAtQueryKeptAcrossEvaluations(_) + ) + }) { + if let Some(seed) = direct_cost_seed(configured, context, &exclusions, true)? { + direct.push(seed); + } + } + let direct_refs: Vec<_> = direct + .iter() + .map(|(graph, profile)| (graph, profile)) + .collect(); let search = search_candidates_with_direct_feedback( &configured.inventory.graph, CandidateSearchLimits { @@ -132,7 +149,7 @@ pub(in crate::search) fn optimize_configured_candidates( &weights, &exclusions, &changes, - direct.as_ref().map(|(graph, profile)| (graph, profile)), + &direct_refs, |selected| { evaluation_number += 1; let mut rejected_queries = Vec::new(); @@ -191,6 +208,7 @@ fn direct_cost_seed( configured: &CandidateConfigurationInventory, context: &CandidatePlanningContext<'_>, seed_exclusions: &[crate::search::candidates::graph::PhysicalCandidateId], + latest_panes: bool, ) -> Result< Option<( crate::search::candidates::graph::CandidateDependencyGraph, @@ -199,8 +217,15 @@ fn direct_cost_seed( String, > { use crate::search::candidates::graph::{PhysicalCandidateId, ReaderConstraint}; - use crate::search::costing::direct_profile::{extract, DirectProfileError}; - let mut extracted = match extract(configured, context) { + use crate::search::costing::direct_profile::{ + extract, extract_latest_panes, DirectProfileError, + }; + let extracted = if latest_panes { + extract_latest_panes(configured, context) + } else { + extract(configured, context) + }; + let mut extracted = match extracted { Ok(Some(extracted)) => extracted, Ok(None) | Err(DirectProfileError::DeadlineExceeded(_)) => return Ok(None), Err(error) => return Err(error.to_string()), diff --git a/crates/plan-selection/src/search/refinement/search.rs b/crates/plan-selection/src/search/refinement/search.rs index f3365b66f..ce24c36c6 100644 --- a/crates/plan-selection/src/search/refinement/search.rs +++ b/crates/plan-selection/src/search/refinement/search.rs @@ -77,7 +77,7 @@ pub(in crate::search) struct CandidateSearchResult { struct SeedRestrictions<'a> { excluded: &'a [PhysicalCandidateId], materialization_changes: &'a [(PhysicalCandidateId, PhysicalCandidateId)], - direct: Option<(&'a CandidateDependencyGraph, &'a DirectCostProfile)>, + direct: &'a [(&'a CandidateDependencyGraph, &'a DirectCostProfile)], } /// The callback returns an actual total cost for a validated feasible plan, or @@ -160,7 +160,7 @@ pub(in crate::search) fn search_candidates_with_restricted_feedback( weights, excluded_from_seed, materialization_changes, - None, + &[], evaluate, ) } @@ -171,7 +171,7 @@ pub(in crate::search) fn search_candidates_with_direct_feedback( weights: &[f64], excluded_from_seed: &[PhysicalCandidateId], materialization_changes: &[(PhysicalCandidateId, PhysicalCandidateId)], - direct: Option<(&CandidateDependencyGraph, &DirectCostProfile)>, + direct: &[(&CandidateDependencyGraph, &DirectCostProfile)], evaluate: impl FnMut(&SelectedCandidateGraph) -> Result, ) -> Result { search_with_structural_baseline( @@ -302,8 +302,11 @@ fn search_with_solver( let mut baseline_attempts = 0; let mut baseline_started: Option = None; let mut direct_attempts = 0; - let mut direct_finished = restrictions.direct.is_none(); + let mut direct_finished = false; + let mut direct_index = 0; let direct_budget = (limits.wall_time / 3).min(Duration::from_secs(2)); + let per_direct_budget = direct_budget / restrictions.direct.len().max(1) as u32; + let mut direct_started = Instant::now(); // Each local proposal uses the same solver and evaluator. Limit the phase // to four validations and one fifth of this domain's existing wall budget. let neighborhood_budget = (limits.wall_time / 5).min(Duration::from_secs(1)); @@ -318,8 +321,18 @@ fn search_with_solver( if refinements >= limits.max_refinements { break CandidateSearchStatus::RefinementLimit; } + if direct_index < restrictions.direct.len() + && (direct_finished + || direct_attempts >= 3 + || direct_started.elapsed() >= per_direct_budget) + { + direct_index += 1; + direct_attempts = 0; + direct_finished = false; + direct_started = Instant::now(); + } let use_direct = - !direct_finished && direct_attempts < 3 && started.elapsed() < direct_budget; + direct_index < restrictions.direct.len() && started.elapsed() < direct_budget; if !use_direct { baseline_started.get_or_insert_with(Instant::now); } @@ -372,10 +385,26 @@ fn search_with_solver( } let (proposal_graph, proposal_observations, solve_budget) = if use_direct { direct_attempts += 1; + let restricted = restrictions.direct[direct_index].0; + // A different scan-price class may forbid earlier bindings. Keep + // their observations for the full graph, not this restricted model. + let admitted = observations + .iter() + .filter(|observation: &&ObservedAssignment| { + crate::search::candidates::graph::validate_selected_graph( + restricted, + &observation.selected, + ) + .is_ok() + }) + .cloned() + .collect(); ( - restrictions.direct.expect("direct profile").0, - observations.clone(), - remaining.min(direct_budget.saturating_sub(started.elapsed())), + restricted, + admitted, + remaining + .min(direct_budget.saturating_sub(started.elapsed())) + .min(per_direct_budget.saturating_sub(direct_started.elapsed())), ) } else if let Some(restricted) = &neighborhood { ( @@ -428,7 +457,7 @@ fn search_with_solver( &proposal_observations, Some(solve_budget.as_secs_f64()), &mut solve_metrics, - restrictions.direct.expect("direct profile").1, + restrictions.direct[direct_index].1, ) } else { propose( @@ -708,7 +737,7 @@ mod tests { &[1.0, 1.0], &[], &[], - Some((&restricted, &profile)), + &[(&restricted, &profile)], |selected| Ok(Some(profile.selected_cost(&restricted, selected)).into()), ) .unwrap(); @@ -718,6 +747,47 @@ mod tests { assert_eq!(result.solve_metrics.direct_cost_validated_plans, 1); } + // Separate scan-price classes share one budget and retain the best validated DAG. + #[test] + fn direct_price_classes_preserve_the_best_incumbent() { + use crate::search::milp::direct_cost::NodeCostProfile; + let graph = choices(&[0.0, 0.0]); + let mut first = graph.clone(); + first.query_roots[0].compatible_candidates = vec![PhysicalCandidateId(0)]; + let mut second = graph.clone(); + second.query_roots[0].compatible_candidates = vec![PhysicalCandidateId(1)]; + let profile = DirectCostProfile { + schedules: None, + nodes: vec![ + NodeCostProfile { + execution_cost: 1.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 2.0, + ..Default::default() + }, + ], + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + let result = search_candidates_with_direct_feedback( + &graph, + limits(2), + &[1.0, 1.0], + &[], + &[], + &[(&first, &profile), (&second, &profile)], + |selected| Ok(Some(profile.selected_cost(&graph, selected)).into()), + ) + .unwrap(); + assert_eq!(result.incumbent.unwrap().actual_cost, 1.0); + assert_eq!(result.solve_metrics.direct_cost_validated_plans, 2); + assert_eq!(result.lower_bound, 0.0); + assert_eq!(result.status, CandidateSearchStatus::RefinementLimit); + } + // A later unproven solve can improve the validated incumbent without replacing // the earlier certified bound or falsely closing the optimality gap. #[test] From 1f250c723aadc666536da41570d94e6d930a3a0d Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 04:13:24 +0000 Subject: [PATCH 14/29] docs: describe direct pane cost coverage and remaining contexts --- docs/design_docs/proposals/joint-planning-search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 5762d8ef8..94760b127 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -202,7 +202,7 @@ The direct-cost objective computes the total modeled cost of the selected execut Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Matching schedules use a compact demand model; up to three differing query schedules use evaluator-derived consumer-subset rates, including phase offsets. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. -Fixed-retention ingestion panes are also supported when their actual unary supplier closures certify a single ingest worker. Updates run at ingestion demand, resident state is charged once, and ingestion update work is excluded from query latency. More workers require explicit compaction-latency modeling before admission. Retained-query-time states, range-evaluation contexts, uncertified supplier interfaces, and larger heterogeneous schedule tables are not yet covered by this direct objective. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. Numerically ill-conditioned cost profiles or rate tables also fall back explicitly rather than silently dropping rare-query charges. +Direct pane pricing covers query-time rebuilding, eligible state retained across evaluations, and fixed-retention ingestion maintenance when the actual unary supplier closure certifies a single ingest worker. Rebuilt state pays for its execution demand and temporary lifetime; retained query-time state pays for resident panes and required refreshes. Ingestion updates run at ingestion demand, resident state is charged once, and ingestion update work is excluded from query latency. Ordinary full-window scans and latest-pane refresh scans use separate restricted pricing classes: a consumer can connect only to suppliers with the matching scan context. These classes share the planning budget and preserve the best validated plan; they do not enumerate every mixed-context DAG. More ingest workers require explicit compaction-latency modeling before admission. Range-evaluation contexts, uncertified supplier interfaces, and larger heterogeneous schedule tables are not yet covered by this direct objective. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. Numerically ill-conditioned cost profiles or rate tables also fall back explicitly rather than silently dropping rare-query charges. The selection objective must express the modeled cost of the selected complete DAG, rather than rank plans with an exactness preference and discover their prices only after selection. This does not require enumerating complete plans. Factor the objective into node activation and state, execution demand, consumer connections and reads, shared resources, and query overhead. Each term has one owner; execution-dependent costs use the selected consumers' compatible evaluation schedules. Raw retention is charged once per source and storage scope at the maximum required horizon. Fixed query overhead can be omitted during minimization but must be restored in reported totals. From df5fe9bc9684719a93d3a6f3d24067eca65ff35e Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 04:31:44 +0000 Subject: [PATCH 15/29] feat: price HLL panes and fixed-worker compaction directly --- .../src/search/costing/direct_profile.rs | 259 +++++++-- .../src/search/milp/direct_cost.rs | 494 +++++++++++++++++- .../src/search/milp/direct_schedule.rs | 34 +- .../plan-selection/src/search/milp/model.rs | 32 +- 4 files changed, 734 insertions(+), 85 deletions(-) diff --git a/crates/plan-selection/src/search/costing/direct_profile.rs b/crates/plan-selection/src/search/costing/direct_profile.rs index bef12924a..5b1991290 100644 --- a/crates/plan-selection/src/search/costing/direct_profile.rs +++ b/crates/plan-selection/src/search/costing/direct_profile.rs @@ -1,10 +1,10 @@ -//! Exact local profiles for query-time plans and single-worker maintained panes. +//! Exact local profiles for query-time plans and fixed-worker maintained panes. //! //! Each accepted input is priced from its actual recursively certified supplier. //! Different priced interfaces are excluded from this proposal, not the full //! catalog. No original-template child is used as workload statistics evidence. //! Ingestion support is limited to fixed-retention pane chains with certified -//! single-worker unary input closures; multiworker compaction remains excluded. +//! unary input closures whose complete range certifies one worker-count bucket. use crate::search::candidates::configurations::CandidateConfigurationInventory; use crate::search::candidates::graph::PhysicalCandidateId; use crate::search::milp::direct_cost::{DirectCostProfile, NodeCostProfile, QueryScheduleRates}; @@ -292,7 +292,9 @@ fn extract_scan_class( let mut suppliers = vec![Vec::new(); count]; let mut sketch_counts = vec![0; count]; let mut raw_sources = vec![None; count]; - let mut closure_work_ms = vec![0.0; count]; + let mut closure_work_ms = vec![WorkBounds::default(); count]; + let mut worker_counts = vec![None; count]; + let mut compaction_seconds = vec![0.0; count]; for id in 0..count { price( id, @@ -307,6 +309,8 @@ fn extract_scan_class( &mut sketch_counts, &mut raw_sources, &mut closure_work_ms, + &mut worker_counts, + &mut compaction_seconds, delta, )?; } @@ -315,6 +319,7 @@ fn extract_scan_class( let zero = || NodeCostProfile { execution_cost: 0.0, work_seconds: 0.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 0.0, materialized: false, ingestion: false, @@ -345,6 +350,7 @@ fn extract_scan_class( * prices.cost_per_memory_byte_second + priced.disk_bytes as f64 * prices.cost_per_disk_byte_second, work_seconds: priced.work_ms / 1000.0, + compaction_seconds: compaction_seconds[id], held_memory_cost_per_second: priced.held.map_or(0.0, |(bytes, _, _)| { bytes as f64 * prices.cost_per_memory_byte_second }), @@ -383,7 +389,9 @@ fn price<'a>( suppliers: &mut [Vec], sketch_counts: &mut [usize], raw_sources: &mut [Option], - closure_work_ms: &mut [f64], + closure_work_ms: &mut [WorkBounds], + worker_counts: &mut [Option], + compaction_seconds: &mut [f64], delta: Option, ) -> Result<(), DirectProfileError> { check_deadline(context.deadline)?; @@ -426,6 +434,8 @@ fn price<'a>( sketch_counts, raw_sources, closure_work_ms, + worker_counts, + compaction_seconds, delta, )?; let Some(priced) = &cache[supplier.0] else { @@ -444,7 +454,8 @@ fn price<'a>( let consumer_ingests = node.output_state.timing == ExecutionTiming::IngestionTime; let producer_ingests = producer.output_state.timing == ExecutionTiming::IngestionTime; - if (consumer_ingests && !producer_ingests) + if (consumer_ingests && producer.materialization != Materialization::NotMaterialized) + || (consumer_ingests && !producer_ingests) || (!consumer_ingests && producer_ingests && producer.materialization != Materialization::MatAtIngestion(Medium::Memory)) @@ -460,10 +471,16 @@ fn price<'a>( unchanged_columns: priced.unchanged_columns.clone(), }); } - if let asap_types::ir::Operator::ASAP(ASAPOp::SummaryAgg { - family: FieldDataType::Sketch(kind, _), - .. - }) = &node.payload + if let asap_types::ir::Operator::ASAP( + ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(kind, _), + .. + } + | ASAPOp::PaneChain { + family: FieldDataType::Sketch(kind, _), + .. + }, + ) = &node.payload { if let SketchParams::Hll { precision, @@ -541,14 +558,35 @@ fn price<'a>( ) else { continue; }; - let closure_work = (supplier.map_or(0.0, |supplier| closure_work_ms[supplier.0]) - + priced.work_ms) - .next_up(); - // With a unary closure this is the evaluator's distinct upstream sum. - // A larger worker count introduces compaction work and state copies. - if node.output_state.timing == ExecutionTiming::IngestionTime - && !single_worker(closure_work, nodes.len()) + let prior = supplier.map_or(WorkBounds::default(), |supplier| { + closure_work_ms[supplier.0] + }); + let closure_work = WorkBounds { + minimum: (prior.minimum + priced.work_ms).next_down().max(0.0), + maximum: (prior.maximum + priced.work_ms).next_up(), + }; + if !closure_work.maximum.is_finite() { + continue; + } + let workers = if node.output_state.timing == ExecutionTiming::IngestionTime + && node.materialization == Materialization::MatAtIngestion(Medium::Memory) { + let Some(workers) = certified_workers(closure_work, nodes.len()) else { + continue; + }; + // Same local price does not imply the same upstream worker count. + // Restrict this producer to one proven bucket of supplier bindings. + if worker_counts[id].is_some_and(|prior| prior != workers) { + continue; + } + Some(workers) + } else { + None + }; + let compaction = workers.map_or(0.0, |workers| { + state_compaction_seconds(node, &priced, workers, context) + }); + if !compaction.is_finite() { continue; } // Compare the actual evaluator outputs, including every piece of the @@ -573,7 +611,16 @@ fn price<'a>( suppliers[id].push(supplier); } sketch_counts[id] = sketch_counts[id].max(sketches); - closure_work_ms[id] = closure_work_ms[id].max(closure_work); + closure_work_ms[id] = if cache[id].is_none() { + closure_work + } else { + WorkBounds { + minimum: closure_work_ms[id].minimum.min(closure_work.minimum), + maximum: closure_work_ms[id].maximum.max(closure_work.maximum), + } + }; + worker_counts[id] = workers; + compaction_seconds[id] = compaction; if cache[id].is_none() { cache[id] = Some(priced); raw_sources[id] = raw_source; @@ -641,7 +688,7 @@ fn failure_budget(context: &CandidatePlanningContext<'_>) -> Option> } fn requires_source_sizing(node: &asap_types::ir::OperatorNode) -> bool { - matches!(node.asap(), Some(ASAPOp::SummaryAgg { family: FieldDataType::Sketch(kind, _), .. }) + matches!(node.asap(), Some(ASAPOp::SummaryAgg { family: FieldDataType::Sketch(kind, _), .. } | ASAPOp::PaneChain { family: FieldDataType::Sketch(kind, _), .. }) if matches!(kind.params(), SketchParams::Hll { .. })) } @@ -719,12 +766,46 @@ fn scan_consumer_lookback(node: &PhysicalASAPDAGNode) -> Option { } } -// Include a bound on roundoff when the evaluator sums the same positive work -// in reverse traversal order. Outward rounding of our path sum bounds the real -// sum; gamma_n bounds the evaluator's finite-precision accumulation. -fn single_worker(upper_work_ms: f64, node_count: usize) -> bool { +#[derive(Clone, Copy, Default)] +struct WorkBounds { + minimum: f64, + maximum: f64, +} + +// Outward path sums bound every admitted binding. Account also for the +// evaluator summing those positive values in a different traversal order. +fn certified_workers(work: WorkBounds, node_count: usize) -> Option { let error = node_count as f64 * f64::EPSILON; - upper_work_ms.is_finite() && error < 0.5 && (upper_work_ms / (1.0 - error)).next_up() < 1000.0 + if error >= 0.5 || !work.maximum.is_finite() { + return None; + } + let lower = (work.minimum * (1.0 - error)).next_down().max(0.0); + let upper = (work.maximum / (1.0 - error)).next_up(); + let first = (lower / 1000.0).ceil().max(1.0); + let last = (upper / 1000.0).ceil().max(1.0); + (first == last && last < (1_u64 << 53) as f64).then_some(last as u64) +} + +fn state_compaction_seconds( + node: &PhysicalASAPDAGNode, + priced: &PricedNode<'_>, + workers: u64, + context: &CandidatePlanningContext<'_>, +) -> f64 { + let settings = context.models.plan_selection; + let measured = crate::lookup( + &settings.measured_operation_costs, + crate::OperatorKind::SummaryMerge, + ExecutionTiming::QueryTime, + priced.family, + |params| crate::sketch_shape(params, crate::StateInputRows::UNKNOWN).1 as f64, + ); + let merge_ms = measured.map_or( + settings.resource_speed.ms_per_cpu_op * context.calibration.factor(), + |unit| (unit.ns_per_unit + unit.ns_per_invocation) * 1e-6, + ); + let states = priced.output.rows / crate::panes_of(node); + (workers - 1) as f64 * states as f64 * merge_ms / 1000.0 } fn check_deadline(deadline: Option) -> Result<(), DirectProfileError> { @@ -777,10 +858,14 @@ fn same_price(a: &PricedNode<'_>, b: &PricedNode<'_>) -> bool { mod tests { #[test] fn ingestion_worker_bound_accounts_for_summation_roundoff() { - assert!(super::single_worker(999.0, 100)); - assert!(!super::single_worker(1000.0, 100)); - assert!(!super::single_worker(1000.0_f64.next_down(), 100)); - assert!(!super::single_worker(1001.0, 1)); + let workers = |minimum, maximum| { + super::certified_workers(super::WorkBounds { minimum, maximum }, 100) + }; + assert_eq!(workers(999.0, 999.0), Some(1)); + assert_eq!(workers(1000.0, 1000.0), None); + assert_eq!(workers(1000.0_f64.next_down(), 1000.0), None); + assert_eq!(workers(1001.0, 1999.0), Some(2)); + assert_eq!(workers(999.0, 1001.0), None); } #[test] @@ -1232,9 +1317,21 @@ mod tests { } for (series_count, rows_per_second, speed_multiplier) in [(2_u64, 100_u64, 1.0), (100, 1000, 1.0), (2, 100, 1000.0)] + .into_iter() + .chain( + (mode == Materialization::MatAtIngestion(Medium::Memory)).then_some(( + 2, + 100, + 12_345_670.0, + )), + ) { for width in [60_000, 30_000] { - for algorithm in [SketchAlgorithm::Kll, SketchAlgorithm::DDSketch] { + for algorithm in [ + SketchAlgorithm::Kll, + SketchAlgorithm::DDSketch, + SketchAlgorithm::Hll, + ] { let scope = AccuracyScope::PerValue; let target = AccuracyTarget::max_error(0.05) .failure_probability(0.01) @@ -1443,20 +1540,87 @@ mod tests { assert!(extracted.profile.nodes[chain].held_memory_cost_per_second > 0.0); } if mode == Materialization::MatAtIngestion(Medium::Memory) { - let mut overloaded = data.clone(); - overloaded.arrival = serde_json::from_value( - serde_json::json!({"ingesting":{"rows_per_second":1e12}}), - ) - .unwrap(); - let overloaded_context = CandidatePlanningContext { - data: &overloaded, + if speed_multiplier > 1e6 { + assert!(extracted.profile.nodes[chain].compaction_seconds > 0.0); + } + let work_ms: f64 = extracted + .profile + .nodes + .iter() + .filter(|node| node.ingestion) + .map(|node| node.work_seconds * 1000.0) + .sum(); + let mut boundary_settings = settings.clone(); + boundary_settings.resource_speed.ms_per_cpu_op *= 1000.0 / work_ms; + boundary_settings.resource_speed.ms_per_scanned_byte *= 1000.0 / work_ms; + let boundary_context = CandidatePlanningContext { + models: crate::PlanningModels { + plan_selection: &boundary_settings, + ..context.models + }, ..context }; - let rejected = extract(&configured, &overloaded_context).unwrap().unwrap(); assert!( - !rejected.supported[chain], - "multiworker compaction is outside this certified slice" + !extract(&configured, &boundary_context) + .unwrap() + .unwrap() + .supported[chain], + "a worker threshold cannot certify one fixed bucket" ); + if speed_multiplier > 1e6 { + let FieldDataType::Sketch(kind, _) = family else { + unreachable!() + }; + let mut measured = settings.clone(); + measured.measured_operation_costs.operators.push( + crate::cost::measured::OperatorCost { + operator: crate::OperatorKind::SummaryMerge, + phase: Some(ExecutionTiming::QueryTime), + sketch: Some(kind.params().clone()), + exact: None, + measure: None, + ns_per_unit: 2_000_000.0, + ns_per_invocation: 500_000.0, + memory_bytes_per_state: None, + }, + ); + let measured_context = CandidatePlanningContext { + models: crate::PlanningModels { + plan_selection: &measured, + ..context.models + }, + ..context + }; + let profile = extract(&configured, &measured_context) + .unwrap() + .unwrap() + .profile; + assert!(profile.nodes[chain].compaction_seconds > 0.0); + assert_ne!( + profile.nodes[chain].compaction_seconds, + extracted.profile.nodes[chain].compaction_seconds + ); + let measured_plan = + crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &measured_context, + &proposal.selected, + 2, + &mut Default::default(), + &mut Vec::new(), + ) + .unwrap() + .unwrap(); + let actual = measured_plan.0.costs[&measured_plan.0.selected] + .cost_per_second + .unwrap(); + let modeled = profile + .selected_cost(&configured.inventory.graph, &proposal.selected); + assert!( + (actual - modeled).abs() <= 1e-9 * actual.max(1e-15), + "measured compaction: {actual} vs {modeled}" + ); + } } if mode == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) { // The same physical scan may feed either a full-window @@ -1561,6 +1725,25 @@ mod tests { .supported[chain] ); } + if algorithm == SketchAlgorithm::Hll { + let mut unsupported = data.clone(); + unsupported + .sources + .get_mut("latency") + .unwrap() + .max_distinct_per_evaluation = Some(1_000_000); + let outside_certificate = CandidatePlanningContext { + data: &unsupported, + ..context + }; + assert!( + !extract(&configured, &outside_certificate) + .unwrap() + .unwrap() + .supported[chain], + "unchanged HLL precision cannot certify an unsupported population" + ); + } if scope == AccuracyScope::WholeResult { let mut unknown = data.clone(); unknown.sources.get_mut("latency").unwrap().series_count = None; diff --git a/crates/plan-selection/src/search/milp/direct_cost.rs b/crates/plan-selection/src/search/milp/direct_cost.rs index 7ebe39ed1..3b6b21f8c 100644 --- a/crates/plan-selection/src/search/milp/direct_cost.rs +++ b/crates/plan-selection/src/search/milp/direct_cost.rs @@ -19,6 +19,9 @@ pub(super) const MINIMUM_DIRECT_RELATIVE_COEFFICIENT: f64 = 1e-12; // need a stronger guard than monetary objective coefficients, since absolute // LP feasibility tolerances can erase a rare reader's otherwise large cost. const MINIMUM_QUERY_RATE_WEIGHT: f64 = 1e-6; +// Lifetime gates use unit-scale bounds. Smaller positive terms can disappear +// within LP feasibility tolerances and then be amplified by a memory price. +const MINIMUM_NORMALIZED_LATENCY_TERM: f64 = 1e-6; #[derive(Debug, Clone, Default)] pub(crate) struct NodeCostProfile { @@ -26,6 +29,8 @@ pub(crate) struct NodeCostProfile { pub ingestion: bool, pub standing_cost_per_second: f64, pub work_seconds: f64, + /// Certified fixed-worker merge delay; the query pays the maximum reached delay. + pub compaction_seconds: f64, pub held_memory_cost_per_second: f64, pub materialized: bool, pub raw_retention: Option<(String, f64)>, @@ -74,6 +79,8 @@ impl DirectCostProfile { !valid(n.execution_cost) || !valid(n.standing_cost_per_second) || (n.ingestion && n.held_memory_cost_per_second != 0.0) + || !valid(n.compaction_seconds) + || (n.compaction_seconds > 0.0 && !(n.ingestion && n.materialized)) || !valid(n.work_seconds) || !valid(n.held_memory_cost_per_second) || n.raw_retention @@ -92,7 +99,10 @@ impl DirectCostProfile { .flat_map(|input| input.compatible_producers.iter()) { let source = &self.nodes[producer.0]; - if (self.nodes[consumer].ingestion && !source.ingestion) + // Query latency stops at the first maintained ingestion state. + // This certified slice admits only raw upstream closures, not + // nested maintained states whose deeper compaction is hidden. + if (self.nodes[consumer].ingestion && (!source.ingestion || source.materialized)) || (!self.nodes[consumer].ingestion && source.ingestion && !source.materialized) { return Err(CandidateSelectionError::InvalidGraph( @@ -156,8 +166,8 @@ impl DirectCostProfile { )); } } - let maximum_latency = self.maximum_latency(); - if self.coefficients().any(|cost| !cost.is_finite()) + let maximum_latency = self.maximum_latency(graph); + if self.coefficients(graph).any(|cost| !cost.is_finite()) || !maximum_latency.is_finite() || !(maximum_latency * graph.query_roots.len() as f64).is_finite() { @@ -165,6 +175,48 @@ impl DirectCostProfile { "direct-cost latency bound overflow".into(), )); } + if self + .nodes + .iter() + .any(|node| node.held_memory_cost_per_second > 0.0) + { + let mut compaction_levels: Vec<_> = self + .nodes + .iter() + .map(|node| node.compaction_seconds) + .filter(|delay| *delay > 0.0) + .collect(); + compaction_levels.sort_by(f64::total_cmp); + compaction_levels.dedup(); + let mut previous = 0.0; + let increments = compaction_levels.into_iter().map(|delay| { + let increment = delay - previous; + previous = delay; + increment + }); + let minimum = std::iter::once(self.request_latency_seconds) + .chain(self.nodes.iter().map(NodeCostProfile::query_work_seconds)) + .chain(increments) + .filter(|term| *term > 0.0) + .reduce(f64::min); + let minimum_rate_weight = self.schedules.as_ref().map_or(1.0, |schedule| { + let total: f64 = schedule.per_query_rates.iter().sum(); + schedule + .per_query_rates + .iter() + .map(|rate| rate / total) + .fold(1.0, f64::min) + }); + // Heterogeneous mean-lifetime rows multiply these coefficients; + // checking each factor separately would miss their small product. + if minimum.is_some_and(|term| { + term / maximum_latency * minimum_rate_weight < MINIMUM_NORMALIZED_LATENCY_TERM + }) { + return Err(CandidateSelectionError::UnsupportedDirectCost( + format!("normalized latency terms are below the supported held-memory numerical range: minimum={minimum:?}, maximum={maximum_latency:e}, rate_weight={minimum_rate_weight:e}, request={:e}", self.request_latency_seconds), + )); + } + } Ok(()) } @@ -198,6 +250,12 @@ impl DirectCostProfile { .filter(|(_, yes)| **yes) .map(|(id, _)| self.nodes[id].query_work_seconds()) .sum::() + + row + .iter() + .enumerate() + .filter(|(_, yes)| **yes) + .map(|(id, _)| self.nodes[id].compaction_seconds) + .fold(0.0, f64::max) }) .collect(); let rates = self.schedules.as_ref().map_or_else( @@ -280,19 +338,82 @@ impl DirectCostProfile { total + raw.values().sum::() } - fn maximum_latency(&self) -> f64 { + /// Sum all required input roles, but take the maximum compatible supplier + /// for each role. This bounds every selected DAG (shared work may be counted + /// twice) without summing mutually exclusive catalog choices. Potential + /// cycles conservatively use the old all-node sum, never a cyclic DP value. + fn maximum_latency(&self, graph: &CandidateDependencyGraph) -> f64 { + let count = self.nodes.len(); + let mut pending = vec![0usize; count]; + let mut parents = vec![Vec::new(); count]; + for (id, candidate) in graph.candidates.iter().enumerate() { + if self.nodes[id].ingestion { + continue; + } + for supplier in candidate + .producer_requirements + .iter() + .flat_map(|input| input.compatible_producers.iter()) + { + pending[id] += 1; + parents[supplier.0].push(id); + } + } + let mut ready: Vec<_> = (0..count).filter(|id| pending[*id] == 0).collect(); + let mut bounds = vec![0.0_f64; count]; + let mut processed = 0; + while let Some(id) = ready.pop() { + processed += 1; + if !self.nodes[id].ingestion { + bounds[id] = self.nodes[id].query_work_seconds() + + graph.candidates[id] + .producer_requirements + .iter() + .map(|input| { + input + .compatible_producers + .iter() + .map(|supplier| bounds[supplier.0]) + .fold(0.0, f64::max) + }) + .sum::(); + } + for parent in &parents[id] { + pending[*parent] -= 1; + if pending[*parent] == 0 { + ready.push(*parent); + } + } + } + let work = if processed == count { + graph + .query_roots + .iter() + .flat_map(|root| &root.compatible_candidates) + .map(|id| bounds[id.0]) + .fold(0.0, f64::max) + } else { + self.nodes + .iter() + .map(NodeCostProfile::query_work_seconds) + .sum() + }; self.request_latency_seconds + + work + self .nodes .iter() - .map(NodeCostProfile::query_work_seconds) - .sum::() + .map(|node| node.compaction_seconds) + .fold(0.0, f64::max) } - pub(crate) fn coefficients(&self) -> impl Iterator + '_ { + pub(crate) fn coefficients<'a>( + &'a self, + graph: &CandidateDependencyGraph, + ) -> impl Iterator + 'a { // Constant request fees cannot influence selection and need not limit // the coefficient range. Lifetimes use a dimensionless [0,1] scale. - let latency_scale = self.maximum_latency().max(f64::MIN_POSITIVE); + let latency_scale = self.maximum_latency(graph).max(f64::MIN_POSITIVE); let rates = self.schedules.as_ref().map_or_else( || vec![self.query_rate], |schedule| { @@ -430,6 +551,60 @@ fn boundary_flows( Ok(flows) } +/// Encode max compaction exactly, including when no memory cost prices latency. +/// Descending threshold ORs are implied integral by binary query reach. Their +/// telescoping weighted sum equals the largest reached delay, with O(N) rows. +fn query_latency_expressions( + graph: &CandidateDependencyGraph, + profile: &DirectCostProfile, + variables: &mut ProblemVariables, + constraints: &mut Vec, + reach: &[Vec>], + check_deadline: &impl Fn() -> Result<(), CandidateSelectionError>, +) -> Result, CandidateSelectionError> { + let scale = profile.maximum_latency(graph).max(f64::MIN_POSITIVE); + let mut result = Vec::with_capacity(reach.len()); + for row in reach { + check_deadline()?; + let mut latency = Expression::from(profile.request_latency_seconds / scale); + let mut delays = Vec::new(); + for (id, reached) in row.iter().enumerate() { + if let Some(reached) = reached { + latency += profile.nodes[id].query_work_seconds() / scale * *reached; + let delay = profile.nodes[id].compaction_seconds; + if delay > 0.0 { + delays.push((delay, *reached)); + } + } + } + delays.sort_by(|left, right| right.0.total_cmp(&left.0)); + let mut previous = None; + let mut index = 0; + while index < delays.len() { + check_deadline()?; + let delay = delays[index].0; + let threshold_reached = variables.add(variable().min(0.0).max(1.0)); + let mut support = Expression::from(0.0); + if let Some(previous) = previous { + constraints.push(constraint!(threshold_reached >= previous)); + support += previous; + } + while index < delays.len() && delays[index].0 == delay { + let reached = delays[index].1; + constraints.push(constraint!(threshold_reached >= reached)); + support += reached; + index += 1; + } + constraints.push(constraint!(threshold_reached <= support)); + let next_delay = delays.get(index).map_or(0.0, |entry| entry.0); + latency += (delay - next_delay) / scale * threshold_reached; + previous = Some(threshold_reached); + } + result.push(latency); + } + Ok(result) +} + /// Exact on integer activation/connection decisions. Reach variables are /// implied integral by the selected acyclic graph, as in the main model. #[allow(clippy::too_many_arguments)] @@ -462,21 +637,16 @@ pub(super) fn encode( } let count = graph.candidates.len(); let queries = graph.query_roots.len(); - let latency_scale = profile.maximum_latency().max(f64::MIN_POSITIVE); + let latency_scale = profile.maximum_latency(graph).max(f64::MIN_POSITIVE); let maximum_latency = 1.0; - let latency: Vec = reach - .iter() - .map(|row| { - Expression::from(profile.request_latency_seconds / latency_scale) - + row - .iter() - .enumerate() - .filter_map(|(id, reached)| { - reached.map(|r| profile.nodes[id].query_work_seconds() / latency_scale * r) - }) - .sum::() - }) - .collect(); + let latency = query_latency_expressions( + graph, + profile, + variables, + constraints, + reach, + &check_deadline, + )?; let mut boundary_count = vec![Expression::from(0.0); count]; let mut boundary_limit = vec![0usize; count]; for flow in boundary_flows( @@ -671,6 +841,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 1.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 3.0, materialized: true, raw_retention: None, @@ -680,6 +851,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, + compaction_seconds: 0.0, raw_retention: Some(("source".into(), 4.0)), ..Default::default() }, @@ -688,6 +860,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 3.0, + compaction_seconds: 0.0, raw_retention: Some(("source".into(), 6.0)), ..Default::default() }, @@ -718,6 +891,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 1.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 3.0, ..Default::default() }, @@ -726,6 +900,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 4.0, materialized: true, raw_retention: None, @@ -734,12 +909,14 @@ mod tests { ingestion: false, standing_cost_per_second: 0.0, work_seconds: 3.0, + compaction_seconds: 0.0, ..Default::default() }, NodeCostProfile { ingestion: false, standing_cost_per_second: 0.0, work_seconds: 4.0, + compaction_seconds: 0.0, ..Default::default() }, ], @@ -771,6 +948,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: materialized_build_cost, work_seconds: 1.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 3.0, materialized: true, raw_retention: None, @@ -780,6 +958,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 0.5, + compaction_seconds: 0.0, held_memory_cost_per_second: 1.0, ..Default::default() }, @@ -788,6 +967,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, + compaction_seconds: 0.0, ..Default::default() }, NodeCostProfile { @@ -795,6 +975,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 3.0, + compaction_seconds: 0.0, ..Default::default() }, ], @@ -886,6 +1067,7 @@ mod tests { ingestion: true, execution_cost: 2.0, work_seconds: 0.2, + compaction_seconds: 0.0, ..Default::default() }, NodeCostProfile { @@ -894,6 +1076,7 @@ mod tests { execution_cost: 3.0, standing_cost_per_second: 7.0, work_seconds: 0.4, + compaction_seconds: 0.0, ..Default::default() }, ]; @@ -906,6 +1089,7 @@ mod tests { nodes.extend((0..2).map(|_| NodeCostProfile { execution_cost: 1.0, work_seconds: 0.5, + compaction_seconds: 0.0, held_memory_cost_per_second: 2.0, ..Default::default() })); @@ -1084,12 +1268,14 @@ mod tests { NodeCostProfile { execution_cost: 2.0, work_seconds: 1.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 3.0, ..Default::default() }, NodeCostProfile { execution_cost: 1.0, work_seconds: 2.0, + compaction_seconds: 0.0, materialized: true, standing_cost_per_second: 7.0, ..Default::default() @@ -1097,12 +1283,14 @@ mod tests { NodeCostProfile { execution_cost: 1.0, work_seconds: 0.5, + compaction_seconds: 0.0, held_memory_cost_per_second: 2.0, ..Default::default() }, NodeCostProfile { execution_cost: 1.0, work_seconds: 1.5, + compaction_seconds: 0.0, held_memory_cost_per_second: 2.0, ..Default::default() }, @@ -1137,4 +1325,266 @@ mod tests { assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-7 * expected); } } + // Max is fixed by reach even without held-memory pricing: minimizing or + // maximizing the latency expression gives the same value for every mask. + #[test] + fn compaction_max_is_exact_without_a_held_memory_objective() { + let graph = CandidateDependencyGraph { + candidates: vec![node(None); 3], + query_roots: vec![root(0), root(1), root(2)], + ..Default::default() + }; + use good_lp::{highs, Solution, SolverModel}; + let profile = DirectCostProfile { + nodes: [2.0, 5.0, 5.0] + .into_iter() + .map(|delay| NodeCostProfile { + ingestion: true, + materialized: true, + compaction_seconds: delay, + ..Default::default() + }) + .collect(), + query_rate: 1.0, + schedules: None, + request_latency_seconds: 1.0, + request_cost: 0.0, + }; + for mask in 0..8 { + let expected = 1.0 + + [2.0_f64, 5.0, 5.0] + .iter() + .enumerate() + .filter(|(id, _)| mask & (1 << id) != 0) + .map(|(_, delay)| *delay) + .fold(0.0, f64::max); + for sign in [-1.0, 1.0] { + let mut variables = ProblemVariables::new(); + let mut constraints = Vec::new(); + let row = (0..3) + .map(|id| { + let reached = variables.add(variable().binary()); + constraints.push(constraint!(reached == f64::from(mask & (1 << id) != 0))); + Some(reached) + }) + .collect(); + let latency = query_latency_expressions( + &graph, + &profile, + &mut variables, + &mut constraints, + &[row], + &|| Ok(()), + ) + .unwrap() + .remove(0); + let solution = variables + .minimise(sign * latency.clone()) + .using(highs) + .with_all(constraints) + .solve() + .unwrap(); + assert!( + (solution.eval(&latency) * profile.maximum_latency(&graph) - expected).abs() + < 1e-9 + ); + } + } + } + + // Exhaustive reader bindings distinguish max compaction from summing + // delays, and preserve once-only standing/update costs for shared states. + #[test] + fn compaction_selection_matches_exhaustive_reader_bindings() { + let mut reader = node(Some(0)); + reader.producer_requirements[0].compatible_producers = + vec![PhysicalCandidateId(0), PhysicalCandidateId(1)].into(); + reader + .producer_requirements + .push(node(Some(2)).producer_requirements.remove(0)); + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None), node(None), reader.clone(), reader], + query_roots: vec![root(3), root(4)], + ..Default::default() + }; + let mut profile = DirectCostProfile { + nodes: [(2.0, 4.0), (5.0, 1.0), (3.0, 2.0)] + .into_iter() + .map(|(delay, cost)| NodeCostProfile { + ingestion: true, + materialized: true, + execution_cost: cost, + compaction_seconds: delay, + ..Default::default() + }) + .chain((0..2).map(|_| NodeCostProfile { + work_seconds: 1.0, + held_memory_cost_per_second: 2.0, + ..Default::default() + })) + .collect(), + query_rate: 1.0, + schedules: None, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + for rates in [[1.0, 1.0], [1.0, 2.0]] { + if rates[0] != rates[1] { + profile.schedules = Some(QueryScheduleRates { + per_query_rates: rates.to_vec(), + materialized_subset_rates: vec![0.0, rates[0], rates[1], rates[0] + rates[1]], + }); + } + let expected = (0..4) + .map(|mask| { + let chosen = [mask & 1, (mask >> 1) & 1]; + let update = 2.0 + + if chosen.contains(&0) { 4.0 } else { 0.0 } + + if chosen.contains(&1) { 1.0 } else { 0.0 }; + update + + (0..2) + .map(|q| { + rates[q] * 2.0 * (1.0 + if chosen[q] == 0 { 3.0 } else { 5.0 }) + }) + .sum::() + }) + .fold(f64::INFINITY, f64::min); + let proposal = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((proposal.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-7); + } + } + + // Hidden compaction below an already maintained state is not query-visible; + // reject that unsupported profile instead of charging all ancestor maxima. + #[test] + fn nested_maintained_ingestion_states_are_not_admitted() { + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(Some(0))], + query_roots: vec![root(1)], + ..Default::default() + }; + let profile = DirectCostProfile { + nodes: (0..2) + .map(|_| NodeCostProfile { + ingestion: true, + materialized: true, + compaction_seconds: 1.0, + ..Default::default() + }) + .collect(), + query_rate: 1.0, + schedules: None, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + assert!(matches!( + profile.validate(&graph), + Err(CandidateSelectionError::InvalidGraph(_)) + )); + } + // An inactive huge-delay candidate must not erase a selected tiny delay: + // its held-memory charge remains significant in actual currency units. + #[test] + fn extreme_unselected_latency_does_not_erase_selected_memory_cost() { + let mut graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None), node(Some(0))], + query_roots: vec![root(2)], + ..Default::default() + }; + graph.candidates[2].producer_requirements[0].compatible_producers = + vec![PhysicalCandidateId(0), PhysicalCandidateId(1)].into(); + for tiny_compaction in [false, true] { + let profile = DirectCostProfile { + nodes: vec![ + NodeCostProfile { + ingestion: true, + materialized: true, + compaction_seconds: if tiny_compaction { 1e-6 } else { 0.0 }, + ..Default::default() + }, + NodeCostProfile { + ingestion: true, + materialized: true, + compaction_seconds: 1e9, + ..Default::default() + }, + NodeCostProfile { + work_seconds: if tiny_compaction { 0.0 } else { 1e-6 }, + held_memory_cost_per_second: 1e6, + ..Default::default() + }, + ], + query_rate: 1.0, + schedules: None, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + assert!(matches!( + solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile + ), + Err(CandidateSelectionError::UnsupportedDirectCost(_)) + )); + // Ordinary microsecond-scale candidates remain admitted and priced; + // only the extreme dynamic range requires evaluator-based fallback. + let mut supported = profile.clone(); + supported.nodes[1].compaction_seconds = 2e-6; + let proposal = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &supported, + ) + .unwrap() + .unwrap(); + assert!((proposal.direct_modeled_cost.unwrap() - 1.0).abs() < 1e-8); + assert!((proposal.model_lower_bound.unwrap() - 1.0).abs() < 1e-8); + } + } + // Supplier choices use max, required positions use sum, and potential + // cycles conservatively fall back instead of inventing a smaller bound. + #[test] + fn latency_bound_respects_choices_required_inputs_and_potential_cycles() { + let mut graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None), node(Some(0))], + query_roots: vec![root(2)], + ..Default::default() + }; + graph.candidates[2].producer_requirements[0].compatible_producers = + vec![PhysicalCandidateId(0), PhysicalCandidateId(1)].into(); + let profile = DirectCostProfile { + nodes: [1.0, 3.0, 2.0] + .into_iter() + .map(|work_seconds| NodeCostProfile { + work_seconds, + ..Default::default() + }) + .collect(), + query_rate: 1.0, + schedules: None, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + assert_eq!(profile.maximum_latency(&graph), 5.0); + graph.candidates[2] + .producer_requirements + .push(node(Some(0)).producer_requirements.remove(0)); + assert_eq!(profile.maximum_latency(&graph), 6.0); + graph.candidates[0].producer_requirements = node(Some(2)).producer_requirements; + assert_eq!(profile.maximum_latency(&graph), 6.0); + } } diff --git a/crates/plan-selection/src/search/milp/direct_schedule.rs b/crates/plan-selection/src/search/milp/direct_schedule.rs index 727278842..720695ee1 100644 --- a/crates/plan-selection/src/search/milp/direct_schedule.rs +++ b/crates/plan-selection/src/search/milp/direct_schedule.rs @@ -19,20 +19,15 @@ pub(super) fn encode( let count = graph.candidates.len(); let queries = graph.query_roots.len(); let subsets = 1usize << queries; - let latency_scale = profile.maximum_latency().max(f64::MIN_POSITIVE); - let latency: Vec = reach - .iter() - .map(|row| { - Expression::from(profile.request_latency_seconds / latency_scale) - + row - .iter() - .enumerate() - .filter_map(|(id, reached)| { - reached.map(|r| profile.nodes[id].query_work_seconds() / latency_scale * r) - }) - .sum::() - }) - .collect(); + let latency_scale = profile.maximum_latency(graph).max(f64::MIN_POSITIVE); + let latency = query_latency_expressions( + graph, + profile, + variables, + constraints, + reach, + &check_deadline, + )?; let mut masks = Vec::with_capacity(count); for id in 0..count { check_deadline()?; @@ -245,6 +240,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 1.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 3.0, ..Default::default() }, @@ -253,6 +249,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 4.0, materialized: true, raw_retention: None, @@ -261,12 +258,14 @@ mod tests { ingestion: false, standing_cost_per_second: 0.0, work_seconds: 3.0, + compaction_seconds: 0.0, ..Default::default() }, NodeCostProfile { ingestion: false, standing_cost_per_second: 0.0, work_seconds: 4.0, + compaction_seconds: 0.0, ..Default::default() }, ], @@ -319,6 +318,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 3.0, work_seconds: 2.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 2.0, materialized: true, raw_retention: None, @@ -372,6 +372,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 10.0, work_seconds: 1.0, + compaction_seconds: 0.0, held_memory_cost_per_second: 3.0, materialized: true, raw_retention: None, @@ -381,6 +382,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 2.0, work_seconds: 0.5, + compaction_seconds: 0.0, held_memory_cost_per_second: 1.0, materialized: true, raw_retention: None, @@ -390,6 +392,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 2.0, + compaction_seconds: 0.0, ..Default::default() }, NodeCostProfile { @@ -397,6 +400,7 @@ mod tests { standing_cost_per_second: 0.0, execution_cost: 1.0, work_seconds: 3.0, + compaction_seconds: 0.0, ..Default::default() }, ], @@ -482,12 +486,14 @@ mod tests { ingestion: false, standing_cost_per_second: 0.0, work_seconds: 1.0, + compaction_seconds: 0.0, ..Default::default() }, NodeCostProfile { ingestion: false, standing_cost_per_second: 0.0, work_seconds: 1e9, + compaction_seconds: 0.0, ..Default::default() }, ], diff --git a/crates/plan-selection/src/search/milp/model.rs b/crates/plan-selection/src/search/milp/model.rs index c13e04bfe..e2f0b629d 100644 --- a/crates/plan-selection/src/search/milp/model.rs +++ b/crates/plan-selection/src/search/milp/model.rs @@ -577,7 +577,7 @@ fn solve_with_objective( .fold(0.0, f64::max); let maximum_cost = direct_profile .into_iter() - .flat_map(|profile| profile.coefficients()) + .flat_map(|profile| profile.coefficients(graph)) .fold(maximum_cost, f64::max); let maximum_cost = observations .iter() @@ -599,7 +599,7 @@ fn solve_with_objective( .chain( direct_profile .into_iter() - .flat_map(|profile| profile.coefficients()), + .flat_map(|profile| profile.coefficients(graph)), ) .filter(|cost| *cost > 0.0) .reduce(f64::min) @@ -625,7 +625,7 @@ fn solve_with_objective( .chain( direct_profile .into_iter() - .flat_map(|profile| profile.coefficients()), + .flat_map(|profile| profile.coefficients(graph)), ) .any(|cost| cost > 0.0 && cost / maximum_cost < relative_coefficient_limit) { @@ -637,7 +637,7 @@ fn solve_with_objective( .reduce(f64::min); let direct_minimum = direct_profile .into_iter() - .flat_map(|profile| profile.coefficients()) + .flat_map(|profile| profile.coefficients(graph)) .filter(|cost| *cost > 0.0) .reduce(f64::min); let reason = format!("observed costs and objective coefficients exceed supported numerical range: min={divisor:e}, max={maximum_cost:e}, local_min={local_minimum:?}, direct_min={direct_minimum:?}"); @@ -811,13 +811,23 @@ fn solve_with_objective( modeled_cost, }; let direct_modeled_cost = direct_profile.map(|profile| profile.selected_cost(graph, &selected)); - decoded_proposal( - graph, - selected, - solution.status(), - solution.eval(&objective) * divisor, - ) - .map(|mut proposal| { + let solved_objective = solution.eval(&objective) * divisor; + if matches!(solution.status(), SolutionStatus::Optimal) { + if let Some(decoded) = direct_modeled_cost { + // Decoding checks the selected graph independently of LP lifetime + // auxiliaries. A lost small term must not become a false certificate. + let tolerance = 1e-7 * decoded.abs().max(solved_objective.abs()); + if !decoded.is_finite() + || !solved_objective.is_finite() + || (decoded - solved_objective).abs() > tolerance + { + return Err(CandidateSelectionError::UnsupportedDirectCost(format!( + "direct solver objective disagrees with decoded cost: solver={solved_objective:e}, decoded={decoded:e}" + ))); + } + } + } + decoded_proposal(graph, selected, solution.status(), solved_objective).map(|mut proposal| { proposal.direct_modeled_cost = direct_modeled_cost; Some(proposal) }) From ab3e66d50f68cfce474ac7e734d410ec16d03a99 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 04:31:48 +0000 Subject: [PATCH 16/29] docs: clarify direct DAG pricing coverage and numerical validation --- docs/design_docs/proposals/joint-planning-search.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 94760b127..fd2f892d2 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -202,9 +202,13 @@ The direct-cost objective computes the total modeled cost of the selected execut Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Matching schedules use a compact demand model; up to three differing query schedules use evaluator-derived consumer-subset rates, including phase offsets. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. -Direct pane pricing covers query-time rebuilding, eligible state retained across evaluations, and fixed-retention ingestion maintenance when the actual unary supplier closure certifies a single ingest worker. Rebuilt state pays for its execution demand and temporary lifetime; retained query-time state pays for resident panes and required refreshes. Ingestion updates run at ingestion demand, resident state is charged once, and ingestion update work is excluded from query latency. Ordinary full-window scans and latest-pane refresh scans use separate restricted pricing classes: a consumer can connect only to suppliers with the matching scan context. These classes share the planning budget and preserve the best validated plan; they do not enumerate every mixed-context DAG. More ingest workers require explicit compaction-latency modeling before admission. Range-evaluation contexts, uncertified supplier interfaces, and larger heterogeneous schedule tables are not yet covered by this direct objective. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. Numerically ill-conditioned cost profiles or rate tables also fall back explicitly rather than silently dropping rare-query charges. +Direct pane pricing covers query-time rebuilding, eligible state retained across evaluations, and fixed-retention ingestion maintenance when every admitted unary supplier binding certifies the same ingest-worker count. Source-certified HLL panes use the same population and failure-probability checks as HLL whole-window builds. Rebuilt state pays for its execution demand and temporary lifetime; retained query-time state pays for resident panes and required refreshes. Ingestion updates run at ingestion demand, resident state is charged once, and ingestion update work is excluded from query latency. Ordinary full-window scans and latest-pane refresh scans use separate restricted pricing classes: a consumer can connect only to suppliers with the matching scan context. These classes share the planning budget and preserve the best validated plan; they do not enumerate every mixed-context DAG. For multiple ingest workers, the model includes the maximum compaction delay among the ingestion states reached by each query; this affects query-time state holding cost. Worker counts crossing a configuration boundary remain unsupported in that pricing class. -The selection objective must express the modeled cost of the selected complete DAG, rather than rank plans with an exactness preference and discover their prices only after selection. This does not require enumerating complete plans. Factor the objective into node activation and state, execution demand, consumer connections and reads, shared resources, and query overhead. Each term has one owner; execution-dependent costs use the selected consumers' compatible evaluation schedules. Raw retention is charged once per source and storage scope at the maximum required horizon. Fixed query overhead can be omitted during minimization but must be restored in reported totals. +Range-evaluation contexts, uncertified supplier interfaces, and larger heterogeneous schedule tables are not yet covered by this direct objective. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. + +Numerically ill-conditioned cost profiles, rate tables, or normalized latency terms also fall back explicitly rather than silently dropping charges. An optimal direct solve must additionally agree with an independently decoded selected-DAG cost before its restricted-domain certificate is accepted. + +Fixed query overhead can be omitted during minimization but must be restored in reported totals. For a shared KMV rollup, charge one build at each required common evaluation, one fine-group extraction, the coarse regrouping merge, and its extraction. Two independent builds incur two build charges. Different consumer schedules change the union of execution demand, not merely the number of activated nodes. Supplier-dependent cardinality, state size, and lifetime require explicit finite contexts or connection-dependent variables; do not replace an unknown coefficient with zero and report the objective as complete. From 6edb9c78634015076c7bcac8afa55020b4a66e87 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 05:17:49 +0000 Subject: [PATCH 17/29] feat: extend direct pricing and validate production rollup workloads --- .../tests/production_rollup_workloads.rs | 290 +++++++++ .../candidates/accuracy_configurations.rs | 85 +++ .../src/search/candidates/configurations.rs | 15 + .../src/search/candidates/graph.rs | 32 + .../src/search/candidates/mod.rs | 1 + .../src/search/candidates/scopes.rs | 7 + .../src/search/costing/direct_profile.rs | 150 ++++- .../src/search/milp/direct_cost.rs | 85 ++- .../src/search/milp/direct_fixed_schedule.rs | 567 ++++++++++++++++++ .../src/search/milp/direct_schedule.rs | 17 + .../plan-selection/src/search/milp/model.rs | 130 ++++ .../src/search/refinement/components.rs | 15 + .../src/search/refinement/planner.rs | 8 + .../src/search/refinement/pruning.rs | 1 + crates/plan-selection/src/search/route.rs | 20 +- 15 files changed, 1365 insertions(+), 58 deletions(-) create mode 100644 crates/integration-tests/tests/production_rollup_workloads.rs create mode 100644 crates/plan-selection/src/search/candidates/accuracy_configurations.rs create mode 100644 crates/plan-selection/src/search/milp/direct_fixed_schedule.rs diff --git a/crates/integration-tests/tests/production_rollup_workloads.rs b/crates/integration-tests/tests/production_rollup_workloads.rs new file mode 100644 index 000000000..2c7ed45ea --- /dev/null +++ b/crates/integration-tests/tests/production_rollup_workloads.rs @@ -0,0 +1,290 @@ +//! Sketch rollup in planning: Pass 2 offers answering a coarse sketch +//! grouping from the finest build of the same sketch over the same rows, +//! and Stage 3 selects it when one build and a merge cost less than two +//! builds over the raw rows. +use std::rc::Rc; + +use asap_executor::physical_planner::promql_rows; +use asap_frontend_sql::{lower_sql, SqlCatalog}; +use asap_integration_tests::fixtures::lower_promql; +use asap_plan_selection::{ + CostModel, PlanSelectionSettings, PlanningModels, ResourceUnitCost, Stage1ToStage3Run, +}; +use asap_types::ir::properties::{AccuracyMetric, AccuracyScope, AccuracyTarget}; +use asap_types::ir::schema::{DataType, Field, Schema}; +use asap_types::ir::{ASAPOp, OperatorNode, QueryRoot}; +use asap_types::workload::{ + DataWorkload, Predictability, QueryRecurrence, QueryRequirements, QueryWorkloadEntry, + RepeatedDemand, RepetitionInterval, +}; +use std::collections::HashSet; + +fn catalog() -> SqlCatalog { + SqlCatalog::new().with_table( + "http", + Schema::new(vec![ + Field::plain("ts", DataType::Timestamp, false), + Field::plain("region", DataType::Utf8, false), + Field::plain("service", DataType::Utf8, false), + Field::plain("user_id", DataType::Float64, false), + Field::plain("latency", DataType::Float64, false), + ]), + ) +} + +/// A predictable query repeating every `interval_ms` at `accuracy`. +fn entry(accuracy: &AccuracyTarget, interval_ms: u32) -> QueryWorkloadEntry { + QueryWorkloadEntry { + query: Default::default(), + requirements: QueryRequirements { + accuracy: accuracy.clone(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval( + interval_ms, + ))), + range_evaluation: None, + } +} + +/// `http` arrives at 2e6 rows/s (W-RollupHeavy's table), 10,000 series, +/// a million distinct users at most per evaluation, and holds a minute of +/// rows (the catalog has no time index, so a query's time predicate is a +/// filter over every row held). +fn data() -> DataWorkload { + serde_json::from_value(serde_json::json!({ + "arrival": {"ingesting": {"rows_per_second": 2_000_000.0}}, + "sources": {"http": { + "series_count": 10_000, + "sample_interval": 5, + "values_non_negative": true, + "max_distinct_per_evaluation": 1_000_000, + "rows_held": 120_000_000, + }}, + })) + .unwrap() +} + +fn production( + roots: Vec<(usize, QueryRoot)>, + entries: &[QueryWorkloadEntry], + data: &DataWorkload, +) -> Stage1ToStage3Run { + let mut settings = PlanSelectionSettings::default(); + settings.cost_model = CostModel::UsageOverTime(ResourceUnitCost::aws_fargate_us_east_1( + settings.resource_speed, + )); + asap_plan_selection::run_stage1_to_stage3_with( + roots, + entries, + data, + PlanningModels::builtin().with_plan_selection(&settings), + asap_plan_selection::Stage1ToStage3Search { + milp_time_limit_ms: Some(30_000), + ..Default::default() + }, + ) + .unwrap() +} + +fn check( + run: &Stage1ToStage3Run, + queries: usize, + expected_rollups: usize, + family: &str, + separate_cost: Option, +) { + let selection = run + .stage3_plan_selection + .as_ref() + .expect("validated production plan within 30s"); + let report = selection.search.candidate_connections.as_ref().unwrap(); + println!( + "selected cost={:?}; report={report:?}", + selection.costs[&selection.selected].cost_per_second + ); + assert!( + report.direct_cost_validated_plans > 0, + "no directly priced validated plan: {report:?}" + ); + assert!(report + .direct_cost_best_validated_cost + .is_some_and(|cost| cost.is_finite() && cost >= 0.0)); + let selected_cost = selection.costs[&selection.selected] + .cost_per_second + .unwrap(); + if let Some(separate) = separate_cost { + // Whole-result direct configurations are conservatively sized; full + // refinement may find a slightly smaller valid state. Both searches + // must improve on independently built KLLs at the same accuracy target. + assert!(report.direct_cost_best_validated_cost.unwrap() < separate); + assert!(selected_cost < separate); + } else { + assert!(report.direct_cost_best_validated_cost.unwrap() <= selected_cost * (1.0 + 1e-9)); + } + let accepted = &selection.accepted[&selection.selected]; + assert_eq!(accepted.len(), queries); + assert!(accepted.iter().all(|a| !a.accepted_regardless_of_guarantee)); + if separate_cost.is_some() && accepted.iter().all(|a| a.sketches.is_empty()) { + assert!(selected_cost <= report.direct_cost_best_validated_cost.unwrap() * (1.0 + 1e-9)); + println!("exact plan costs less than both the direct incumbent and independent KLLs"); + return; + } + assert!( + accepted + .iter() + .all(|a| !a.accepted_regardless_of_guarantee && a.sketches.iter().any(|s| s == family)), + "{accepted:?}" + ); + let selected = run.selected().unwrap(); + let mut seen = HashSet::new(); + let mut builds = 0; + let mut rollups = 0; + for root in &selected.roots { + let QueryRoot::Operator(root) = root else { + panic!("operator output") + }; + for node in OperatorNode::reachable(root) { + if !seen.insert(Rc::as_ptr(&node)) { + continue; + } + match node.asap() { + Some(ASAPOp::SummaryAgg { .. } | ASAPOp::PaneChain { .. }) => builds += 1, + Some(ASAPOp::SummaryMerge { + group_by: Some(_), .. + }) => rollups += 1, + _ => {} + } + } + } + assert_eq!( + (builds, rollups), + (1, expected_rollups), + "selected plan must share the finest grouping" + ); +} + +async fn sql(queries: &[String], accuracy: AccuracyTarget) -> Stage1ToStage3Run { + let mut roots = Vec::new(); + for (id, query) in queries.iter().enumerate() { + roots.push(( + id, + QueryRoot::Operator( + lower_sql(query, &catalog(), accuracy.clone()) + .await + .unwrap(), + ), + )); + } + production( + roots, + &vec![entry(&accuracy, 60_000); queries.len()], + &data(), + ) +} + +#[tokio::test] +async fn production_three_sql_groupings_share_kll() { + let accuracy = AccuracyTarget::max_error(0.05) + .failure_probability(0.05) + .accuracy_metric(AccuracyMetric::Rank) + .accuracy_scope(AccuracyScope::PerValue); + let queries: Vec<_> = ["region, service", "region", "service"].iter().map(|keys| format!("SELECT {keys}, approx_percentile_cont(latency, 0.99) AS value FROM http WHERE ts >= now() - INTERVAL '5 minutes' GROUP BY {keys}")).collect(); + check(&sql(&queries, accuracy).await, 3, 2, "Kll", None); +} + +#[tokio::test] +async fn production_sql_distinct_region_service_rollup() { + let accuracy = AccuracyTarget::max_error(0.05) + .failure_probability(0.05) + .accuracy_metric(AccuracyMetric::Relative) + .accuracy_scope(AccuracyScope::PerValue); + let queries: Vec<_> = ["region, service", "service"].iter().map(|keys| format!("SELECT {keys}, COUNT(DISTINCT user_id) AS value FROM http WHERE ts >= now() - INTERVAL '5 minutes' GROUP BY {keys}")).collect(); + check(&sql(&queries, accuracy).await, 2, 1, "Kmv", None); +} + +#[test] +// The earlier controlled fixture omitted raw retention. This production +// workload explicitly holds one current sample for each of its 10,000 series. +fn production_promql_compares_rollup_with_exact() { + let accuracy = AccuracyTarget::max_error(0.01) + .failure_probability(0.001) + .accuracy_metric(AccuracyMetric::Rank); + let roots: Vec<_> = [ + "quantile by (host, job) (0.5, data)", + "quantile by (job) (0.5, data)", + ] + .iter() + .enumerate() + .map(|(id, query)| { + let root = lower_promql(query, accuracy.clone()).unwrap(); + ( + id, + QueryRoot::Operator(promql_rows::with_series_identity(&root).unwrap()), + ) + }) + .collect(); + let data: DataWorkload = serde_json::from_value(serde_json::json!({"arrival":{"ingesting":{"rows_per_second":666.67}},"sources":{"data":{"series_count":10000,"metric_type":"gauge","sample_interval":15000,"rows_held":10000}}})).unwrap(); + let entries = [entry(&accuracy, 10000), entry(&accuracy, 10000)]; + let separate = independent_kll_cost(roots.clone(), &entries, &data); + println!("independent KLL cost={separate}"); + check( + &production(roots, &entries, &data), + 2, + 1, + "Kll", + Some(separate), + ); +} + +// Price the same workload with independently built, unmaterialized KLL states. +fn independent_kll_cost( + roots: Vec<(usize, QueryRoot)>, + entries: &[QueryWorkloadEntry], + data: &DataWorkload, +) -> f64 { + use asap_logical_asap_optimization::per_computation_candidates::{ + compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, + }; + let inventory = generate_per_computation_candidates(roots).unwrap(); + let choices: Vec<_> = inventory + .targets + .iter() + .map(|target| { + target + .choices + .candidates + .iter() + .position(|c| { + matches!( + c, + ComputationCandidate::Summary(asap_types::ir::schema::SketchAlgorithm::Kll) + ) + }) + .unwrap() + }) + .collect(); + let roots: Vec<_> = compose_logical_candidate(&inventory, &choices) + .unwrap() + .into_iter() + .map(|(_, root)| root) + .collect(); + let mut settings = PlanSelectionSettings::default(); + settings.cost_model = CostModel::UsageOverTime(ResourceUnitCost::aws_fargate_us_east_1( + settings.resource_speed, + )); + let models = PlanningModels::builtin().with_plan_selection(&settings); + let plan = asap_physical_asap_optimization::optimize_physical_asap_chosen( + "independent", + &roots, + entries, + data, + models.deployment_capabilities, + &mut |_, _, _| asap_types::physical::Materialization::NotMaterialized, + ) + .unwrap() + .unwrap(); + let result = asap_plan_selection::select_plan(&[plan], entries, data, models).unwrap(); + result.costs[&result.selected].cost_per_second.unwrap() +} diff --git a/crates/plan-selection/src/search/candidates/accuracy_configurations.rs b/crates/plan-selection/src/search/candidates/accuracy_configurations.rs new file mode 100644 index 000000000..cb1967bb8 --- /dev/null +++ b/crates/plan-selection/src/search/candidates/accuracy_configurations.rs @@ -0,0 +1,85 @@ +//! Additional finite capacities for whole-result failure budgets. +//! Original capacities remain available; final composed accuracy is still checked. +use asap_logical_asap_optimization::memo::candidates::CandidateMemo; +use asap_types::ir::{properties::AccuracyScope, QueryRoot}; +use asap_types::workload::{DataWorkload, QueryWorkloadEntry}; +use std::{rc::Rc, time::Instant}; + +pub(crate) fn add_scoped_candidates( + memo: &mut CandidateMemo, + entries: &[QueryWorkloadEntry], + data: &DataWorkload, + deadline: Option, +) -> Result<(), String> { + if !entries.iter().any(|entry| { + entry.requirements.accuracy.accuracy_scope == AccuracyScope::WholeResult + && !entry.requirements.accuracy.is_exact() + }) { + return Ok(()); + } + let mut sources = std::collections::HashSet::new(); + for group in &memo.groups { + if let Some(asap_types::ir::NonASAPOp::Scan { source, .. }) = group.node.non_asap() { + sources.insert(crate::source_name(source)); + } + } + let population = sources + .into_iter() + .map(|source| { + data.sources + .get(source) + .and_then(|facts| facts.series_count) + .filter(|count| *count > 0) + }) + .collect::>>() + .and_then(|counts| counts.into_iter().max()); + let mut budget: Option = None; + for entry in entries { + let target = &entry.requirements.accuracy; + if target.is_exact() { + continue; + } + let Some(delta) = target + .failure_probability + .filter(|d| d.is_finite() && *d > 0.0) + else { + return Ok(()); + }; + let count = match target.accuracy_scope { + AccuracyScope::WholeResult => population, + scope => scope.max_values_covered(None), + }; + let Some(count) = count.filter(|count| *count > 0) else { + return Ok(()); + }; + let delta = delta / count as f64; + if delta <= 0.0 { + return Ok(()); + } + budget = Some(budget.map_or(delta, |prior| prior.min(delta))); + } + let Some(budget) = budget else { + return Ok(()); + }; + for candidates in &mut memo.candidates { + let mut additional = Vec::new(); + for candidate in candidates.iter() { + super::check_deadline(deadline).map_err(|e| e.to_string())?; + let roots = crate::with_resized_summaries( + &[QueryRoot::Operator(candidate.node.clone())], + &|_, params| Some(asap_logical_asap_optimization::accuracy::estimators::with_failure_probability( + params, budget / candidate.budget_share.max(1) as f64)), + ).map_err(|e| e.to_string())?; + let QueryRoot::Operator(node) = &roots[0] else { + unreachable!() + }; + if !Rc::ptr_eq(node, &candidate.node) { + let mut sized = candidate.clone(); + sized.node = node.clone(); + additional.push(sized); + } + } + candidates.extend(additional); + } + Ok(()) +} diff --git a/crates/plan-selection/src/search/candidates/configurations.rs b/crates/plan-selection/src/search/candidates/configurations.rs index beed24f37..da83ed313 100644 --- a/crates/plan-selection/src/search/candidates/configurations.rs +++ b/crates/plan-selection/src/search/candidates/configurations.rs @@ -234,6 +234,14 @@ pub(in crate::search) fn enumerate_candidate_configurations_before( producer_requirements: Vec::new(), }); if materialization == Materialization::MatAtQuery(Medium::Memory) { + if !summary { + result.inventory.graph.reader_constraints.push( + ReaderConstraint::HighestSharedQueryMaterialization { + candidate: id, + query_outputs: query_outputs.clone(), + }, + ); + } result.inventory.graph.reader_constraints.push( ReaderConstraint::MinimumQueryReaders { candidate: id, @@ -383,6 +391,7 @@ pub(in crate::search) fn enumerate_candidate_configurations_before( } ReaderConstraint::AllowedOutputs { candidate, .. } | ReaderConstraint::IncompatibleOutputs { candidate, .. } + | ReaderConstraint::HighestSharedQueryMaterialization { candidate, .. } | ReaderConstraint::MinimumQueryReaders { candidate, .. } => candidate, }; for &candidate in &source_configurations[source_candidate.0] { @@ -409,6 +418,12 @@ pub(in crate::search) fn enumerate_candidate_configurations_before( right: *right, } } + ReaderConstraint::HighestSharedQueryMaterialization { + query_outputs, .. + } => ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs: query_outputs.clone(), + }, ReaderConstraint::MinimumQueryReaders { query_outputs, minimum, diff --git a/crates/plan-selection/src/search/candidates/graph.rs b/crates/plan-selection/src/search/candidates/graph.rs index 998c158db..8ddc30553 100644 --- a/crates/plan-selection/src/search/candidates/graph.rs +++ b/crates/plan-selection/src/search/candidates/graph.rs @@ -56,6 +56,12 @@ pub(crate) enum ReaderConstraint { query_outputs: Vec>, minimum: usize, }, + /// A non-summary query materialization must be a root or have an + /// immediate selected consumer reached by at most one query (Stage 2). + HighestSharedQueryMaterialization { + candidate: PhysicalCandidateId, + query_outputs: Vec>, + }, AllowedOutputs { candidate: PhysicalCandidateId, outputs: Vec, @@ -188,6 +194,10 @@ impl CandidateDependencyGraph { candidate, query_outputs, .. + } + | ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs, } => { let outputs: Vec<_> = query_outputs.iter().flatten().copied().collect(); let unique: HashSet<_> = outputs.iter().copied().collect(); @@ -473,6 +483,28 @@ pub(crate) fn validate_selected_graph( .count() < *minimum } + ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs, + } => { + active.contains(candidate) + && !selected + .query_roots + .iter() + .any(|root| root.candidate == *candidate) + && !selected.connections.iter().any(|edge| { + edge.producer == *candidate + && query_outputs + .iter() + .filter(|outputs| { + outputs + .iter() + .any(|output| output_reaches(*output, edge.consumer)) + }) + .count() + <= 1 + }) + } ReaderConstraint::AllowedOutputs { candidate, outputs } => (0..graph.query_roots.len()) .any(|output| !outputs.contains(&output) && output_reaches(output, *candidate)), ReaderConstraint::IncompatibleOutputs { diff --git a/crates/plan-selection/src/search/candidates/mod.rs b/crates/plan-selection/src/search/candidates/mod.rs index 21cc2ea45..0547d8d2b 100644 --- a/crates/plan-selection/src/search/candidates/mod.rs +++ b/crates/plan-selection/src/search/candidates/mod.rs @@ -1,4 +1,5 @@ //! Candidate graph representation and physical implementation construction. +pub(crate) mod accuracy_configurations; pub(crate) mod configurations; mod factoring; pub(crate) mod graph; diff --git a/crates/plan-selection/src/search/candidates/scopes.rs b/crates/plan-selection/src/search/candidates/scopes.rs index 83353c869..5356c2f66 100644 --- a/crates/plan-selection/src/search/candidates/scopes.rs +++ b/crates/plan-selection/src/search/candidates/scopes.rs @@ -224,6 +224,7 @@ pub(in crate::search) fn expand_execution_scopes_before( } ReaderConstraint::AllowedOutputs { candidate, .. } | ReaderConstraint::IncompatibleOutputs { candidate, .. } + | ReaderConstraint::HighestSharedQueryMaterialization { candidate, .. } | ReaderConstraint::MinimumQueryReaders { candidate, .. } => candidate, }; for &candidate in &copies[owner.0] { @@ -246,6 +247,12 @@ pub(in crate::search) fn expand_execution_scopes_before( right: *right, } } + ReaderConstraint::HighestSharedQueryMaterialization { + query_outputs, .. + } => ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs: query_outputs.clone(), + }, ReaderConstraint::MinimumQueryReaders { query_outputs, minimum, diff --git a/crates/plan-selection/src/search/costing/direct_profile.rs b/crates/plan-selection/src/search/costing/direct_profile.rs index 5b1991290..5369bbfb7 100644 --- a/crates/plan-selection/src/search/costing/direct_profile.rs +++ b/crates/plan-selection/src/search/costing/direct_profile.rs @@ -88,13 +88,9 @@ fn extract_scan_class( return Ok(None); }; if context - .entries + .original_roots .iter() - .any(|entry| entry.range_evaluation.is_some()) - || context - .original_roots - .iter() - .any(|root| !matches!(root, QueryRoot::Operator(_))) + .any(|root| !matches!(root, QueryRoot::Operator(_))) || context.original_roots.len() != context.entries.len() || context.original_roots.len() != configured.inventory.graph.query_roots.len() { @@ -109,7 +105,17 @@ fn extract_scan_class( .entries .iter() .all(|entry| entry.recurrence == first.recurrence); - if !equal_fixed && context.entries.len() > 3 { + let has_range = context + .entries + .iter() + .any(|entry| entry.range_evaluation.is_some()); + if has_range + && (!equal_fixed + || context + .entries + .iter() + .any(|entry| entry.range_evaluation != first.range_evaluation)) + { return Ok(None); } let rates: Vec<_> = context @@ -117,7 +123,7 @@ fn extract_scan_class( .iter() .map(|entry| { crate::cost::execution_demand::evaluation_rate( - [(&entry.recurrence, None)], + [(&entry.recurrence, entry.range_evaluation)], settings.non_repeating_query_horizon_seconds, ) .map_err(|error| DirectProfileError::Invalid(error.to_string())) @@ -127,9 +133,51 @@ fn extract_scan_class( return Ok(None); } let rate = rates[0]; - let request_rate: f64 = rates.iter().sum(); + let request_rate: f64 = context + .entries + .iter() + .map(|entry| { + crate::cost::execution_demand::evaluation_rate( + [(&entry.recurrence, None)], + settings.non_repeating_query_horizon_seconds, + ) + .map_err(|error| DirectProfileError::Invalid(error.to_string())) + }) + .collect::, _>>()? + .iter() + .sum(); let schedules = if equal_fixed { None + } else if context.entries.len() > 3 { + use crate::search::milp::direct_cost::FixedQuerySchedule; + let fixed: Option> = context + .entries + .iter() + .map(|entry| match &entry.recurrence { + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(interval)) => { + Some(FixedQuerySchedule { + interval_ms: interval.0, + phase_ms: None, + }) + } + QueryRecurrence::Repeated(RepeatedDemand::FixedIntervalAt { + interval, + evaluation_phase, + }) => Some(FixedQuerySchedule { + interval_ms: interval.0, + phase_ms: Some(evaluation_phase.0), + }), + _ => None, + }) + .collect(); + let Some(fixed) = fixed else { return Ok(None) }; + match QueryScheduleRates::from_fixed_schedules(fixed) { + Ok(schedule) => Some(schedule), + Err( + crate::search::candidates::graph::CandidateSelectionError::UnsupportedDirectCost(_), + ) => return Ok(None), + Err(error) => return Err(DirectProfileError::Invalid(error.to_string())), + } } else { let mut materialized_subset_rates = Vec::with_capacity(1 << context.entries.len()); for mask in 0..(1usize << context.entries.len()) { @@ -147,6 +195,7 @@ fn extract_scan_class( ); } Some(QueryScheduleRates { + compact_fixed: None, per_query_rates: rates, materialized_subset_rates, }) @@ -173,7 +222,7 @@ fn extract_scan_class( && match configuration.timing { ExecutionTiming::QueryTime => { matches!(configuration.materialization, Materialization::NotMaterialized | Materialization::MatAtQuery(Medium::Memory)) - || (configuration.materialization == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) + || (!has_range && configuration.materialization == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) && matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) && context.models.deployment_capabilities.query_time_results_retained && (0..context.entries.len()).any(|reader| asap_physical_asap_optimization::summary_materialization_eligibility(node, &[reader], context.entries, context.data, context.models.deployment_capabilities).is_ok_and(|eligibility| eligibility.may_keep_across_evaluations))) @@ -279,7 +328,9 @@ fn extract_scan_class( }; if choices.is_empty() { nodes[id] = None; - } else if choices.contains(&None) { + } else if choices.contains(&None) + && !(prefer_latest_pane && choices.iter().any(Option::is_some)) + { lookbacks[id] = None; } else { lookbacks[id] = choices.iter().copied().max().flatten(); @@ -539,10 +590,19 @@ fn price<'a>( pane_count, pane_width_ms, .. - }) => ( - u64::from(*pane_count), - pane_width_ms.checked_mul(u64::from(*pane_count)), - ), + }) => { + let window = pane_width_ms.checked_mul(u64::from(*pane_count)); + let span = context + .entries + .first() + .and_then(|entry| entry.range_evaluation) + .map_or(0, |range| range.span_ms); + let Some(retention) = window.and_then(|window| window.checked_add(span)) else { + continue; + }; + let retained = (retention / (*pane_width_ms).max(1)).max(u64::from(*pane_count)); + (retained, window) + } _ => (1, None), }; let Ok(priced) = crate::price_node( @@ -1265,6 +1325,28 @@ mod tests { } fn check_pane_profile(mode: Materialization) { + check_pane_profile_with_range(mode, None); + } + + #[test] + fn identical_range_pane_profiles_match_the_real_evaluator() { + let range = Some(asap_types::workload::RangeEvaluation { + span_ms: 3_600_000, + step_ms: 60_000, + }); + for mode in [ + Materialization::NotMaterialized, + Materialization::MatAtQuery(Medium::Memory), + Materialization::MatAtIngestion(Medium::Memory), + ] { + check_pane_profile_with_range(mode, range); + } + } + + fn check_pane_profile_with_range( + mode: Materialization, + range_evaluation: Option, + ) { use asap_logical_asap_optimization::per_computation_candidates::{ compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, }; @@ -1409,10 +1491,13 @@ mod tests { recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval( RepetitionInterval(width as u32), )), - range_evaluation: None, + range_evaluation, }]; entries.push(entries[0].clone()); let mut settings = crate::PlanSelectionSettings::default(); + if range_evaluation.is_some() { + settings.measured_operation_costs.ns_per_evaluation = 1_000_000.0; + } settings.resource_speed.ms_per_cpu_op *= speed_multiplier; settings.resource_speed.ms_per_scanned_byte *= speed_multiplier; let deployment = asap_logical_asap_optimization::DeploymentCapabilities { @@ -1512,7 +1597,11 @@ mod tests { unreachable!() }; let expected_residency = match mode { - Materialization::MatAtIngestion(_) => u64::from(*pane_count) + 1, + Materialization::MatAtIngestion(_) => { + u64::from(*pane_count) + + range_evaluation.map_or(0, |range| range.span_ms / width) + + 1 + } Materialization::MatAtQueryKeptAcrossEvaluations(_) => { u64::from(*pane_count) - 1 } @@ -1831,12 +1920,13 @@ mod tests { id } for algorithm in [SketchAlgorithm::Kll, SketchAlgorithm::DDSketch] { - for phases in [false, true] { + for (phases, query_count) in [(false, 2), (true, 2), (false, 5), (true, 5)] { let target = AccuracyTarget::max_error(0.05) .failure_probability(0.01) .accuracy_scope(AccuracyScope::PerValue); - let originals: Vec<_> = [0.95, 0.99] + let originals: Vec<_> = [0.95, 0.99, 0.5, 0.7, 0.9] .iter() + .take(query_count as usize) .map(|q| { QueryRoot::Operator(lower_promql( &format!("quantile_over_time({q}, latency[5m])"), @@ -1889,7 +1979,7 @@ mod tests { .count(), 1 ); - let entries: Vec<_> = (0..2) + let entries: Vec<_> = (0..query_count) .map(|q| QueryWorkloadEntry { query: Query(format!("quantile {q}")), requirements: QueryRequirements { @@ -1924,15 +2014,25 @@ mod tests { }; let extracted = extract(&configured, &context).unwrap().unwrap(); assert!(extracted.supported.iter().all(|supported| *supported)); - assert_eq!( - extracted + if query_count == 2 { + assert_eq!( + extracted + .profile + .schedules + .as_ref() + .unwrap() + .materialized_subset_rates[3], + if phases { 2.0 } else { 3.0 } + ); + } else { + assert!(extracted .profile .schedules .as_ref() .unwrap() - .materialized_subset_rates[3], - if phases { 2.0 } else { 3.0 } - ); + .compact_fixed + .is_some()); + } let proposal = crate::search::milp::model::solve_direct_cost_proposal_with_metrics( &configured.inventory.graph, &[], diff --git a/crates/plan-selection/src/search/milp/direct_cost.rs b/crates/plan-selection/src/search/milp/direct_cost.rs index 3b6b21f8c..5e4970fc6 100644 --- a/crates/plan-selection/src/search/milp/direct_cost.rs +++ b/crates/plan-selection/src/search/milp/direct_cost.rs @@ -6,8 +6,11 @@ use good_lp::{constraint, variable, Constraint, Expression, ProblemVariables, Variable}; use std::collections::BTreeMap; +#[path = "direct_fixed_schedule.rs"] +mod fixed_schedules; #[path = "direct_schedule.rs"] mod schedules; +pub(crate) use fixed_schedules::{FixedQuerySchedule, FixedScheduleLattice}; use crate::search::candidates::graph::*; @@ -52,6 +55,7 @@ impl NodeCostProfile { pub(crate) struct QueryScheduleRates { pub per_query_rates: Vec, pub materialized_subset_rates: Vec, + pub compact_fixed: Option, } #[derive(Debug, Clone)] @@ -123,34 +127,38 @@ impl DirectCostProfile { } if let Some(schedule) = &self.schedules { let queries = graph.query_roots.len(); - if queries > 3 { - return Err(CandidateSelectionError::UnsupportedDirectCost( - "heterogeneous query schedule table supports at most three output queries" - .into(), - )); - } - if !schedule.per_query_rates.iter().sum::().is_finite() - || schedule.per_query_rates.len() != queries - || schedule.materialized_subset_rates.len() != (1usize << queries) - || schedule.materialized_subset_rates[0] != 0.0 - || schedule - .per_query_rates - .iter() - .any(|rate| !rate.is_finite() || *rate <= 0.0) - || schedule - .materialized_subset_rates - .iter() - .skip(1) - .any(|rate| !rate.is_finite() || *rate <= 0.0) - || schedule - .per_query_rates - .iter() - .enumerate() - .any(|(q, rate)| *rate != schedule.materialized_subset_rates[1 << q]) - { - return Err(CandidateSelectionError::InvalidGraph( - "invalid direct query-schedule rates".into(), - )); + if let Some(lattice) = &schedule.compact_fixed { + lattice.validate(schedule, queries)?; + } else { + if queries > 3 { + return Err(CandidateSelectionError::UnsupportedDirectCost( + "heterogeneous query schedule table supports at most three output queries" + .into(), + )); + } + if !schedule.per_query_rates.iter().sum::().is_finite() + || schedule.per_query_rates.len() != queries + || schedule.materialized_subset_rates.len() != (1usize << queries) + || schedule.materialized_subset_rates[0] != 0.0 + || schedule + .per_query_rates + .iter() + .any(|rate| !rate.is_finite() || *rate <= 0.0) + || schedule + .materialized_subset_rates + .iter() + .skip(1) + .any(|rate| !rate.is_finite() || *rate <= 0.0) + || schedule + .per_query_rates + .iter() + .enumerate() + .any(|(q, rate)| *rate != schedule.materialized_subset_rates[1 << q]) + { + return Err(CandidateSelectionError::InvalidGraph( + "invalid direct query-schedule rates".into(), + )); + } } } if let Some(schedule) = &self.schedules { @@ -269,6 +277,9 @@ impl DirectCostProfile { let Some(schedule) = &self.schedules else { return self.query_rate; }; + if let Some(lattice) = &schedule.compact_fixed { + return lattice.union_rate(reached.iter().map(|row| row[id])); + } let mask = reached .iter() .enumerate() @@ -417,6 +428,9 @@ impl DirectCostProfile { let rates = self.schedules.as_ref().map_or_else( || vec![self.query_rate], |schedule| { + if schedule.compact_fixed.is_some() { + return schedule.per_query_rates.clone(); + } schedule .materialized_subset_rates .iter() @@ -425,6 +439,11 @@ impl DirectCostProfile { .collect() }, ); + let denominator_scale = self + .schedules + .as_ref() + .and_then(|s| s.compact_fixed.as_ref()) + .map_or(1.0, FixedScheduleLattice::maximum_denominator_scale); self.nodes.iter().flat_map(move |node| { let mut coefficients = Vec::with_capacity(rates.len() * 2 + 1); let node_rates = if node.ingestion { @@ -437,6 +456,13 @@ impl DirectCostProfile { node.execution_cost * rate, node.held_memory_cost_per_second * rate * latency_scale, ]); + // Compact mean-lifetime selectors multiply by total weight / + // selected weight; include that largest possible coefficient. + if !node.materialized && denominator_scale > 1.0 { + coefficients.push( + node.held_memory_cost_per_second * rate * latency_scale * denominator_scale, + ); + } } coefficients.push(node.standing_cost_per_second); coefficients.push(node.raw_retention.as_ref().map_or(0.0, |(_, cost)| *cost)); @@ -1137,6 +1163,7 @@ mod tests { let (graph, mut profile) = ingestion_profile(true, 1.0); assert!((solve(&graph, &profile) - 29.4).abs() < 1e-6); profile.schedules = Some(QueryScheduleRates { + compact_fixed: None, per_query_rates: vec![1.0, 2.0], materialized_subset_rates: vec![0.0, 1.0, 2.0, 3.0], }); @@ -1301,6 +1328,7 @@ mod tests { for (rates, union, expected) in [(vec![2.0, 2.0], 2.0, 74.4), (vec![1.0, 2.0], 3.0, 83.0)] { if rates[0] != rates[1] { profile.schedules = Some(QueryScheduleRates { + compact_fixed: None, per_query_rates: rates.clone(), materialized_subset_rates: vec![0.0, rates[0], rates[1], union], }); @@ -1431,6 +1459,7 @@ mod tests { for rates in [[1.0, 1.0], [1.0, 2.0]] { if rates[0] != rates[1] { profile.schedules = Some(QueryScheduleRates { + compact_fixed: None, per_query_rates: rates.to_vec(), materialized_subset_rates: vec![0.0, rates[0], rates[1], rates[0] + rates[1]], }); diff --git a/crates/plan-selection/src/search/milp/direct_fixed_schedule.rs b/crates/plan-selection/src/search/milp/direct_fixed_schedule.rs new file mode 100644 index 000000000..4174e57d0 --- /dev/null +++ b/crates/plan-selection/src/search/milp/direct_fixed_schedule.rs @@ -0,0 +1,567 @@ +//! Compact fixed-interval accounting follows the evaluator's policy: distinct +//! intervals add, distinct known phases add, unknown phase joins a known phase. +//! Weighted mean lifetimes use reachable integer rate totals, not reader subsets. +use super::*; + +const MAXIMUM_RATE_TICKS: u64 = 4096; + +#[derive(Debug, Clone, Copy)] +pub(crate) struct FixedQuerySchedule { + pub interval_ms: u32, + pub phase_ms: Option, +} + +#[derive(Debug, Clone)] +pub(crate) struct FixedScheduleLattice { + queries: Vec, + weights: Vec, +} + +fn gcd(mut left: u64, mut right: u64) -> u64 { + while right != 0 { + (left, right) = (right, left % right); + } + left +} + +impl QueryScheduleRates { + pub(crate) fn from_fixed_schedules( + queries: Vec, + ) -> Result { + if queries.is_empty() || queries.iter().any(|q| q.interval_ms == 0) { + return Err(CandidateSelectionError::InvalidGraph( + "empty or zero-interval fixed schedule".into(), + )); + } + let mut common = 1u64; + for query in &queries { + let interval = u64::from(query.interval_ms); + common = (common / gcd(common, interval)) + .checked_mul(interval) + .ok_or_else(|| { + CandidateSelectionError::UnsupportedDirectCost( + "fixed schedule lattice arithmetic overflow".into(), + ) + })?; + } + let mut weights: Vec<_> = queries + .iter() + .map(|q| common / u64::from(q.interval_ms)) + .collect(); + let divisor = weights.iter().copied().reduce(gcd).unwrap(); + for weight in &mut weights { + *weight /= divisor; + } + let total = weights + .iter() + .try_fold(0u64, |sum, weight| sum.checked_add(*weight)); + if total.is_none_or(|total| total > MAXIMUM_RATE_TICKS) { + return Err(CandidateSelectionError::UnsupportedDirectCost(format!( + "fixed schedule rate lattice exceeds {MAXIMUM_RATE_TICKS} total ticks" + ))); + } + Ok(Self { + per_query_rates: queries + .iter() + .map(|q| 1000.0 / f64::from(q.interval_ms)) + .collect(), + materialized_subset_rates: Vec::new(), + compact_fixed: Some(FixedScheduleLattice { + queries, + weights: weights.into_iter().map(|w| w as usize).collect(), + }), + }) + } +} + +impl FixedScheduleLattice { + pub(super) fn maximum_denominator_scale(&self) -> f64 { + self.weights.iter().sum::() as f64 + } + + pub(super) fn validate( + &self, + schedule: &QueryScheduleRates, + queries: usize, + ) -> Result<(), CandidateSelectionError> { + let expected = QueryScheduleRates::from_fixed_schedules(self.queries.clone())?; + if queries != self.queries.len() + || expected.per_query_rates != schedule.per_query_rates + || !schedule.materialized_subset_rates.is_empty() + || expected.compact_fixed.as_ref().unwrap().weights != self.weights + { + return Err(CandidateSelectionError::InvalidGraph( + "inconsistent compact fixed schedule".into(), + )); + } + Ok(()) + } + + pub(super) fn union_rate(&self, selected: impl Iterator) -> f64 { + let mut intervals = BTreeMap::)>::new(); + for (query, selected) in self.queries.iter().zip(selected) { + if !selected { + continue; + } + let (unknown, phases) = intervals.entry(query.interval_ms).or_default(); + if let Some(phase) = query.phase_ms { + phases.insert(phase % u64::from(query.interval_ms)); + } else { + *unknown = true; + } + } + intervals + .into_iter() + .map(|(interval, (unknown, phases))| { + phases.len().max(usize::from(unknown)) as f64 * 1000.0 / f64::from(interval) + }) + .sum() + } +} + +fn logical_or( + variables: &mut ProblemVariables, + constraints: &mut Vec, + inputs: &[Variable], +) -> Option { + if inputs.is_empty() { + return None; + } + if inputs.len() == 1 { + return Some(inputs[0]); + } + let value = variables.add(variable().min(0).max(1)); + for input in inputs { + constraints.push(constraint!(value >= *input)); + } + constraints.push(constraint!( + value <= inputs.iter().copied().sum::() + )); + Some(value) +} + +fn logical_and( + variables: &mut ProblemVariables, + constraints: &mut Vec, + left: Variable, + right: Variable, +) -> Variable { + let value = variables.add(variable().min(0).max(1)); + constraints.push(constraint!(value <= left)); + constraints.push(constraint!(value <= right)); + constraints.push(constraint!(value >= left + right - 1)); + value +} + +fn union_terms( + lattice: &FixedScheduleLattice, + node: usize, + reach: &[Vec>], + variables: &mut ProblemVariables, + constraints: &mut Vec, +) -> Vec<(Variable, f64)> { + let mut intervals = BTreeMap::, Vec>>::new(); + for (q, schedule) in lattice.queries.iter().enumerate() { + if let Some(reached) = reach[q][node] { + intervals + .entry(schedule.interval_ms) + .or_default() + .entry( + schedule + .phase_ms + .map(|p| p % u64::from(schedule.interval_ms)), + ) + .or_default() + .push(reached); + } + } + let mut terms = Vec::new(); + for (interval, mut phases) in intervals { + let rate = 1000.0 / f64::from(interval); + let unknown = logical_or( + variables, + constraints, + &phases.remove(&None).unwrap_or_default(), + ); + let known: Vec<_> = phases + .values() + .filter_map(|readers| logical_or(variables, constraints, readers)) + .collect(); + terms.extend(known.iter().map(|phase| (*phase, rate))); + if let Some(unknown) = unknown { + if let Some(any_known) = logical_or(variables, constraints, &known) { + let extra = variables.add(variable().min(0).max(1)); + constraints.push(constraint!(extra <= unknown)); + constraints.push(constraint!(extra <= 1.0 - any_known)); + constraints.push(constraint!(extra >= unknown - any_known)); + terms.push((extra, rate)); + } else { + terms.push((unknown, rate)); + } + } + } + terms +} + +#[allow(clippy::too_many_arguments)] +pub(super) fn encode( + graph: &CandidateDependencyGraph, + profile: &DirectCostProfile, + schedule: &QueryScheduleRates, + lattice: &FixedScheduleLattice, + variables: &mut ProblemVariables, + constraints: &mut Vec, + active: &[Variable], + edges: &[(ProducerConsumerConnection, Variable)], + roots: &[(QueryRootAssignment, Variable)], + reach: &[Vec>], + divisor: f64, + check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, +) -> Result { + let count = graph.candidates.len(); + let queries = graph.query_roots.len(); + let scale = profile.maximum_latency(graph).max(f64::MIN_POSITIVE); + let latency = query_latency_expressions( + graph, + profile, + variables, + constraints, + reach, + &check_deadline, + )?; + let mut unions = Vec::with_capacity(count); + for id in 0..count { + check_deadline()?; + unions.push(union_terms(lattice, id, reach, variables, constraints)); + } + let mut demands = vec![Vec::<(Variable, f64)>::new(); count]; + for flow in boundary_flows( + graph, + profile, + variables, + constraints, + active, + edges, + roots, + &check_deadline, + )? { + for (id, used) in flow.reached.into_iter().enumerate() { + check_deadline()?; + let Some(used) = used else { continue }; + if flow.boundary < queries { + demands[id].push((used, schedule.per_query_rates[flow.boundary])); + } else if profile.nodes[flow.boundary - queries].ingestion { + demands[id].push((used, 1.0)); + } else { + for (phase, rate) in &unions[flow.boundary - queries] { + demands[id].push((logical_and(variables, constraints, used, *phase), *rate)); + } + } + } + } + let mut objective = Expression::from(profile.request_cost / divisor); + let mut raw = BTreeMap::<&str, Vec<(usize, f64)>>::new(); + for (id, node) in profile.nodes.iter().enumerate() { + check_deadline()?; + let terms = if node.materialized { + if node.ingestion { + vec![(active[id], 1.0)] + } else { + unions[id].clone() + } + } else { + demands[id].clone() + }; + objective += node.standing_cost_per_second / divisor * active[id]; + objective += + node.execution_cost / divisor * terms.iter().map(|(v, r)| *r * *v).sum::(); + if let Some((source, cost)) = &node.raw_retention { + raw.entry(source).or_default().push((id, *cost)); + } + if node.held_memory_cost_per_second == 0.0 { + continue; + } + let mut lifetime = Expression::from(0.0); + if node.materialized { + let longest = variables.add(variable().min(0).max(1)); + for (q, row) in reach.iter().enumerate() { + if let Some(reached) = row[id] { + constraints.push(constraint!(longest >= latency[q].clone() - (1.0 - reached))); + } + } + for (used, rate) in terms { + lifetime += rate * gated(variables, constraints, longest.into(), used, 1.0); + } + } else { + let possible: Vec<_> = reach + .iter() + .enumerate() + .filter_map(|(q, row)| row[id].map(|r| (q, r))) + .collect(); + let maximum: usize = possible.iter().map(|(q, _)| lattice.weights[*q]).sum(); + if maximum == 0 { + continue; + } + let mut totals = vec![false; maximum + 1]; + totals[0] = true; + let mut accumulated = 0; + for (q, _) in &possible { + let weight = lattice.weights[*q]; + for sum in (0..=accumulated).rev() { + if totals[sum] { + totals[sum + weight] = true; + } + } + accumulated += weight; + } + // Denominator totals need binary selectors: unlike full reader + // subsets, their weighted-sum equality alone permits mixtures. + let selectors: Vec<_> = totals + .iter() + .enumerate() + .filter(|(_, yes)| **yes) + .map(|(total, _)| (total, variables.add(variable().binary()))) + .collect(); + constraints.push(constraint!( + selectors.iter().map(|(_, v)| *v).sum::() == 1 + )); + constraints.push(constraint!( + selectors + .iter() + .map(|(d, v)| *d as f64 * *v) + .sum::() + == possible + .iter() + .map(|(q, r)| lattice.weights[*q] as f64 * *r) + .sum::() + )); + let numerator: Expression = possible + .iter() + .map(|(q, r)| { + lattice.weights[*q] as f64 / maximum as f64 + * gated(variables, constraints, latency[*q].clone(), *r, 1.0) + }) + .sum(); + for (denominator, selector) in selectors.into_iter().filter(|(d, _)| *d > 0) { + check_deadline()?; + let chosen = gated(variables, constraints, numerator.clone(), selector, 1.0); + for (used, rate) in &terms { + lifetime += *rate * maximum as f64 / denominator as f64 + * gated(variables, constraints, chosen.into(), *used, 1.0); + } + } + } + objective += node.held_memory_cost_per_second * scale / divisor * lifetime; + } + for entries in raw.into_values() { + let held = variables.add(variable().min(0)); + for (id, cost) in entries { + constraints.push(constraint!(held >= cost / divisor * active[id])); + } + objective += held; + } + Ok(objective) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::search::milp::model::{ + solve_direct_cost_proposal_with_metrics, CandidateSolveMetrics, + }; + use asap_logical_asap_optimization::memo::MemoGroupId; + use asap_types::workload::{QueryRecurrence, RepeatedDemand, RepetitionInterval}; + + fn node(input: Option) -> PhysicalCandidate { + PhysicalCandidate { + group: MemoGroupId(0), + local_cost: 0.0, + producer_requirements: input + .into_iter() + .map(|id| ProducerRequirement { + compatible_producers: vec![PhysicalCandidateId(id)].into(), + }) + .collect(), + } + } + + // All reader subsets agree with the existing interval accounting policy; + // the compact representation never constructs a 2^Q rate table. + #[test] + fn compact_fixed_rates_match_evaluator_subsets() { + let intervals = [1000, 2000, 1000, 4000, 2000, 1000]; + let fixed: Vec<_> = intervals + .iter() + .map(|ms| FixedQuerySchedule { + interval_ms: *ms, + phase_ms: None, + }) + .collect(); + let schedule = QueryScheduleRates::from_fixed_schedules(fixed).unwrap(); + let recurrences: Vec<_> = intervals + .iter() + .map(|ms| { + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(*ms))) + }) + .collect(); + for mask in 0..1usize << intervals.len() { + let expected = crate::cost::execution_demand::evaluation_rate( + recurrences + .iter() + .enumerate() + .filter(|(q, _)| mask & (1 << q) != 0) + .map(|(_, r)| (r, None)), + 3600.0, + ) + .unwrap(); + let actual = schedule + .compact_fixed + .as_ref() + .unwrap() + .union_rate((0..intervals.len()).map(|q| mask & (1 << q) != 0)); + assert_eq!(actual, expected); + } + assert!(schedule.materialized_subset_rates.is_empty()); + let large = QueryScheduleRates::from_fixed_schedules(vec![ + FixedQuerySchedule { + interval_ms: 1000, + phase_ms: None + }; + 128 + ]) + .unwrap(); + assert_eq!(large.compact_fixed.unwrap().weights.len(), 128); + assert!(matches!( + QueryScheduleRates::from_fixed_schedules(vec![ + FixedQuerySchedule { + interval_ms: 1, + phase_ms: None + }, + FixedQuerySchedule { + interval_ms: 100000, + phase_ms: None + }, + ]), + Err(CandidateSelectionError::UnsupportedDirectCost(_)) + )); + } + + // Known phases add; unknown phase adds only when no known phase is active. + #[test] + fn compact_phase_activation_is_exact_for_every_subset() { + let schedule = QueryScheduleRates::from_fixed_schedules(vec![ + FixedQuerySchedule { + interval_ms: 1000, + phase_ms: None, + }, + FixedQuerySchedule { + interval_ms: 1000, + phase_ms: Some(0), + }, + FixedQuerySchedule { + interval_ms: 1000, + phase_ms: Some(500), + }, + FixedQuerySchedule { + interval_ms: 1000, + phase_ms: Some(1500), + }, + ]) + .unwrap(); + for mask in 0..16 { + let mut variables = ProblemVariables::new(); + let mut constraints = Vec::new(); + let reach: Vec<_> = (0..4) + .map(|q| { + let value = variables.add(variable().binary()); + constraints.push(constraint!(value == f64::from(mask & (1 << q) != 0))); + vec![Some(value)] + }) + .collect(); + let lattice = schedule.compact_fixed.as_ref().unwrap(); + let terms = union_terms(lattice, 0, &reach, &mut variables, &mut constraints); + let expression: Expression = terms.iter().map(|(v, r)| *r * *v).sum(); + use good_lp::{highs, Solution, SolverModel}; + let solution = variables + .minimise(-expression.clone()) + .using(highs) + .with_all(constraints) + .solve() + .unwrap(); + let expected = lattice.union_rate((0..4).map(|q| mask & (1 << q) != 0)); + assert_eq!(solution.eval(&expression), expected); + } + } + + // Five readers share a query-materialized build. Upstream lifetime uses + // their weighted mean; the build uses their maximum and union refresh rate. + #[test] + fn compact_schedule_prices_shared_refresh_and_weighted_memory() { + let intervals = [1000, 2000, 1000, 4000, 2000]; + let schedule = QueryScheduleRates::from_fixed_schedules( + intervals + .iter() + .map(|ms| FixedQuerySchedule { + interval_ms: *ms, + phase_ms: None, + }) + .collect(), + ) + .unwrap(); + let mut candidates = vec![node(None), node(Some(0))]; + candidates.extend((0..5).map(|_| node(Some(1)))); + let graph = CandidateDependencyGraph { + candidates, + query_roots: (2..7) + .map(|id| QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(id)], + }) + .collect(), + ..Default::default() + }; + let mut nodes = vec![ + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 4.0, + work_seconds: 2.0, + held_memory_cost_per_second: 2.0, + materialized: true, + ..Default::default() + }, + ]; + nodes.extend((0..5).map(|q| NodeCostProfile { + execution_cost: 1.0, + work_seconds: q as f64 + 1.0, + ..Default::default() + })); + let rates = schedule.per_query_rates.clone(); + let union = 1.75; + let weighted: f64 = rates + .iter() + .enumerate() + .map(|(q, r)| r * (q as f64 + 4.0)) + .sum(); + let sum: f64 = rates.iter().sum(); + let expected = union * (6.0 + 3.0 * weighted / sum + 2.0 * 8.0) + sum; + let profile = DirectCostProfile { + nodes, + schedules: Some(schedule), + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + let mut metrics = CandidateSolveMetrics::default(); + let proposal = + solve_direct_cost_proposal_with_metrics(&graph, &[], None, &mut metrics, &profile) + .unwrap() + .unwrap(); + assert!((proposal.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-6); + assert!(metrics.peak_variables < 5000); + } +} diff --git a/crates/plan-selection/src/search/milp/direct_schedule.rs b/crates/plan-selection/src/search/milp/direct_schedule.rs index 720695ee1..2c55dc2d1 100644 --- a/crates/plan-selection/src/search/milp/direct_schedule.rs +++ b/crates/plan-selection/src/search/milp/direct_schedule.rs @@ -16,6 +16,22 @@ pub(super) fn encode( divisor: f64, check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, ) -> Result { + if let Some(lattice) = &schedule.compact_fixed { + return fixed_schedules::encode( + graph, + profile, + schedule, + lattice, + variables, + constraints, + active, + edges, + roots, + reach, + divisor, + check_deadline, + ); + } let count = graph.candidates.len(); let queries = graph.query_roots.len(); let subsets = 1usize << queries; @@ -195,6 +211,7 @@ mod tests { .unwrap() }; QueryScheduleRates { + compact_fixed: None, per_query_rates: (0..queries.len()).map(|q| union(1 << q)).collect(), materialized_subset_rates: (0..1 << queries.len()).map(union).collect(), } diff --git a/crates/plan-selection/src/search/milp/model.rs b/crates/plan-selection/src/search/milp/model.rs index e2f0b629d..b4fe8f20d 100644 --- a/crates/plan-selection/src/search/milp/model.rs +++ b/crates/plan-selection/src/search/milp/model.rs @@ -71,6 +71,13 @@ fn required_reader_reach( check_deadline()?; match constraint { ReaderConstraint::ActivationRequires { .. } => {} + ReaderConstraint::HighestSharedQueryMaterialization { candidate, .. } => { + for (output, reachable) in potential.iter().enumerate() { + if reachable[candidate.0] { + pending.push((output, candidate.0)); + } + } + } ReaderConstraint::AllowedOutputs { candidate, outputs } => { for (output, reachable) in potential.iter().enumerate() { if reachable[candidate.0] && !outputs.contains(&output) { @@ -524,6 +531,69 @@ fn solve_with_objective( readers >= *minimum as f64 * candidate_selected[candidate.0] )); } + ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs, + } => { + let mut witnesses: Expression = root_assignments + .iter() + .filter(|(root, _)| root.candidate == *candidate) + .map(|(_, value)| *value) + .sum(); + for (edge, connected) in &connections { + if edge.producer != *candidate { + continue; + } + let witness = variables.add(initialized( + variable().binary(), + warm_start.as_ref().map(|start| { + f64::from( + start.selected.connections.contains(edge) + && query_outputs + .iter() + .filter(|outputs| { + outputs.iter().any(|output| { + start.reached[*output].contains(&edge.consumer) + }) + }) + .count() + <= 1, + ) + }), + )); + constraints.push(constraint!(witness <= *connected)); + let mut readers = Expression::from(0.0); + for outputs in query_outputs { + let reached: Vec<_> = outputs + .iter() + .filter_map(|output| reach[*output][edge.consumer.0]) + .collect(); + if reached.is_empty() { + continue; + } + let query = variables.add(initialized( + variable().min(0).max(1), + warm_start.as_ref().map(|start| { + f64::from(outputs.iter().any(|output| { + start.reached[*output].contains(&edge.consumer) + })) + }), + )); + for value in &reached { + constraints.push(constraint!(query >= *value)); + } + constraints.push(constraint!( + query <= reached.into_iter().sum::() + )); + readers += query; + } + constraints.push(constraint!( + readers <= 1.0 + query_outputs.len() as f64 * (1.0 - witness) + )); + witnesses += witness; + } + constraints.push(constraint!(candidate_selected[candidate.0] <= witnesses)); + } ReaderConstraint::AllowedOutputs { candidate, outputs } => { for (output, reachable) in reach.iter().enumerate() { if !outputs.contains(&output) { @@ -1347,6 +1417,41 @@ mod tests { .is_none()); } + // Stage 2 permits non-summary materialization only at the highest shared + // node: a shared parent blocks it, while distinct private parents allow it. + #[test] + fn highest_shared_materialization_matches_selected_connections() { + let mut graph = dependency_graph( + vec![ + candidate(0, 1.0, &[]), + candidate(0, 5.0, &[]), + candidate(1, 1.0, &[&[0, 1]]), + candidate(2, 1.0, &[&[0, 1]]), + ], + &[&[2], &[2]], + ); + graph + .reader_constraints + .push(ReaderConstraint::HighestSharedQueryMaterialization { + candidate: PhysicalCandidateId(0), + query_outputs: vec![vec![0], vec![1]], + }); + assert_eq!(compare_with_oracle(&graph).unwrap().modeled_cost, 6.0); + graph.query_roots[1].compatible_candidates = vec![PhysicalCandidateId(3)]; + assert_eq!(compare_with_oracle(&graph).unwrap().modeled_cost, 3.0); + graph.query_roots[0].compatible_candidates = vec![PhysicalCandidateId(0)]; + let selected = compare_with_oracle(&graph).unwrap(); + assert_eq!(selected.modeled_cost, 2.0); + let observation = ObservedAssignment { + selected, + exact_cost: Some(2.0), + rejected_queries: vec![], + }; + assert!(solve_candidate_proposal(&graph, &[observation], None) + .unwrap() + .is_some()); + } + fn candidate(group: u32, cost: f64, inputs: &[&[usize]]) -> PhysicalCandidate { PhysicalCandidate { group: MemoGroupId(group), @@ -1503,6 +1608,31 @@ mod tests { candidate, required_candidate, } => marks[candidate.0] == 2 && marks[required_candidate.0] != 2, + ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs, + } => { + marks[candidate.0] == 2 + && !selected_roots.iter().any(|(_, root)| *root == candidate.0) + && !dependencies + .iter() + .enumerate() + .any(|(consumer, producers)| { + marks[consumer] == 2 + && producers + .iter() + .any(|(_, producer)| *producer == candidate.0) + && query_outputs + .iter() + .filter(|outputs| { + outputs.iter().any(|q| { + reaches(*q, PhysicalCandidateId(consumer)) + }) + }) + .count() + <= 1 + }) + } ReaderConstraint::MinimumQueryReaders { candidate, query_outputs, diff --git a/crates/plan-selection/src/search/refinement/components.rs b/crates/plan-selection/src/search/refinement/components.rs index fcfca13b2..d95fe8172 100644 --- a/crates/plan-selection/src/search/refinement/components.rs +++ b/crates/plan-selection/src/search/refinement/components.rs @@ -242,6 +242,7 @@ fn project( ReaderConstraint::ActivationRequires { candidate, .. } | ReaderConstraint::AllowedOutputs { candidate, .. } | ReaderConstraint::IncompatibleOutputs { candidate, .. } + | ReaderConstraint::HighestSharedQueryMaterialization { candidate, .. } | ReaderConstraint::MinimumQueryReaders { candidate, .. } => candidate.0, }; let Some(&candidate) = candidates.get(&id) else { @@ -274,6 +275,20 @@ fn project( right, } } + ReaderConstraint::HighestSharedQueryMaterialization { query_outputs, .. } => { + ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs: query_outputs + .iter() + .map(|bucket| { + bucket + .iter() + .filter_map(|id| outputs.get(id).copied()) + .collect() + }) + .collect(), + } + } ReaderConstraint::MinimumQueryReaders { query_outputs, minimum, diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index fe0320b7e..6e02d228b 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -131,6 +131,14 @@ pub(in crate::search) fn optimize_configured_candidates( configuration.materialization, asap_types::physical::Materialization::MatAtQueryKeptAcrossEvaluations(_) ) + }) || configured.inventory.templates.iter().flatten().any(|node| { + matches!( + node.non_asap(), + Some(asap_types::ir::NonASAPOp::TimeRange { + kind: asap_types::ir::TimeRangeKind::Instant, + .. + }) + ) }) { if let Some(seed) = direct_cost_seed(configured, context, &exclusions, true)? { direct.push(seed); diff --git a/crates/plan-selection/src/search/refinement/pruning.rs b/crates/plan-selection/src/search/refinement/pruning.rs index 6f7aa771b..8ea2df048 100644 --- a/crates/plan-selection/src/search/refinement/pruning.rs +++ b/crates/plan-selection/src/search/refinement/pruning.rs @@ -84,6 +84,7 @@ pub(in crate::search) fn prune( .filter_map(|mut constraint| { let id = match &mut constraint { ReaderConstraint::AllowedOutputs { candidate, .. } + | ReaderConstraint::HighestSharedQueryMaterialization { candidate, .. } | ReaderConstraint::MinimumQueryReaders { candidate, .. } | ReaderConstraint::IncompatibleOutputs { candidate, .. } | ReaderConstraint::ActivationRequires { candidate, .. } => candidate, diff --git a/crates/plan-selection/src/search/route.rs b/crates/plan-selection/src/search/route.rs index d5c278ac5..06b2ca0ab 100644 --- a/crates/plan-selection/src/search/route.rs +++ b/crates/plan-selection/src/search/route.rs @@ -816,11 +816,21 @@ fn search_domain( let started = Instant::now(); let mut memo = match progress.local_memo.take() { Some(memo) => memo, - None => try_populate_pass1( - initial, - &pruned.as_ref().expect("first round pruned").inventory, - ) - .map_err(|error| unsupported(&error.to_string()))?, + None => { + let mut memo = try_populate_pass1( + initial, + &pruned.as_ref().expect("first round pruned").inventory, + ) + .map_err(|error| unsupported(&error.to_string()))?; + crate::search::candidates::accuracy_configurations::add_scoped_candidates( + &mut memo, + entries, + data, + Instant::now().checked_add(remaining()), + ) + .map_err(|error| unsupported(&error))?; + memo + } }; let deadline = Instant::now().checked_add(remaining()); asap_logical_asap_optimization::memo::window_candidates::extend_local_window_composition( From d712fa4d5d12a0b91d7af46b27fc72009c68f408 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 05:17:55 +0000 Subject: [PATCH 18/29] docs: update direct pricing coverage and materialization constraints --- docs/design_docs/proposals/joint-planning-search.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index fd2f892d2..850e13aed 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -125,6 +125,8 @@ G_KLL += MergeLatest15(G_panes) ### Stage 2: expand physical candidates +Non-summary query-time materialization must be a query output or have a selected immediate consumer used by only one query; the materialized node itself must have multiple query readers. These connection-dependent constraints match Stage 2's highest-shared-node rule and are enforced in MILP. Summary-state eligibility remains separate. + Physical rules provide executable operators, configurations, and materialization assignments for the generated computations. For example, a supported pane implementation can maintain state at ingestion and merge it at evaluation. This introduces two timing decisions within the same query, rather than one materialization choice for the whole query. Eligibility that depends on consumers is checked against the selected connections; considering every potential consumer at once could incorrectly reject a useful candidate. For the running example, assume evaluations occur at aligned minute boundaries and the deployment supports ingestion-time pane maintenance. Stage 2 can expand the state groups as follows; `s` is a generated KLL capacity: @@ -200,11 +202,11 @@ Memory-time and capacity answer different questions. A state occupying `m` bytes The direct-cost objective computes the total modeled cost of the selected execution DAG. It does not precompute one fixed total cost for each candidate or enumerate every complete DAG. Before solving, the planner computes each supported candidate's local cost parameters from its configuration and compatible suppliers. Inside MILP, the selected nodes and connections determine execution frequency and repeated work, consumer-specific operations, state holding costs, shared raw retention, and query overhead. A shared producer owns its charge once, including every required execution; its consumers' work is charged separately. **Local candidate cost parameters are precomputed; the complete selected DAG's cost is determined jointly by MILP.** -Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Matching schedules use a compact demand model; up to three differing query schedules use evaluator-derived consumer-subset rates, including phase offsets. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. +Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Whole-result targets therefore add conservatively sized capacity candidates before physical costing, preserving the original candidates. Matching schedules use a compact demand model. Larger sets of fixed-interval schedules use phase activation and bounded integer rate totals, avoiding enumeration of every reader subset; rates follow the evaluator's accounting policy, including unknown phases. The normalized rate total is bounded at 4,096; small heterogeneous workloads can also use evaluator-derived subset tables. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. Direct pane pricing covers query-time rebuilding, eligible state retained across evaluations, and fixed-retention ingestion maintenance when every admitted unary supplier binding certifies the same ingest-worker count. Source-certified HLL panes use the same population and failure-probability checks as HLL whole-window builds. Rebuilt state pays for its execution demand and temporary lifetime; retained query-time state pays for resident panes and required refreshes. Ingestion updates run at ingestion demand, resident state is charged once, and ingestion update work is excluded from query latency. Ordinary full-window scans and latest-pane refresh scans use separate restricted pricing classes: a consumer can connect only to suppliers with the matching scan context. These classes share the planning budget and preserve the best validated plan; they do not enumerate every mixed-context DAG. For multiple ingest workers, the model includes the maximum compaction delay among the ingestion states reached by each query; this affects query-time state holding cost. Worker counts crossing a configuration boundary remain unsupported in that pricing class. -Range-evaluation contexts, uncertified supplier interfaces, and larger heterogeneous schedule tables are not yet covered by this direct objective. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. +Matching range requests on the same fixed schedule are directly priced for rebuilt and ingestion-maintained state: evaluation frequency includes every range step, request overhead remains per request, and maintained panes retain the required historical span. Query-kept range state, differing range contexts, uncertified supplier interfaces, and schedules outside the bounded encodings still require evaluator refinement. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. Numerically ill-conditioned cost profiles, rate tables, or normalized latency terms also fall back explicitly rather than silently dropping charges. An optimal direct solve must additionally agree with an independently decoded selected-DAG cost before its restricted-domain certificate is accepted. From b70ebe2b623f2bc4eab7c8811ea9c96c5cc3cae5 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 05:45:30 +0000 Subject: [PATCH 19/29] test: isolate planner presentation checks from exhaustive panes --- crates/devtools/tests/stage_pipeline.rs | 31 +++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/crates/devtools/tests/stage_pipeline.rs b/crates/devtools/tests/stage_pipeline.rs index ae1f0cd1f..6e1e43e0d 100644 --- a/crates/devtools/tests/stage_pipeline.rs +++ b/crates/devtools/tests/stage_pipeline.rs @@ -162,7 +162,19 @@ fn example1_document_preserves_workload_and_validates_archived_plans() { /// the selection made over all candidates. #[test] fn display_cap_is_recorded_and_keeps_the_selection() { - let document = generate(&["--max-displayed", "1"]); + // Instant quantiles provide several implementations without expanding a + // pane-width domain. Exhaustive mode ensures the display cap is exercised. + let document = document_of(&[ + "--promql", + "quantile(0.5, data)", + "--promql", + "quantile(0.9, data)", + "--sample-interval-ms", + "15000", + "--exhaustive", + "--max-displayed", + "1", + ]); let logical = &document["stage1_logical_asap_optimization"]; assert_eq!(logical["capped"], true); // Logical parents of displayed physical plans are always retained; the @@ -171,7 +183,12 @@ fn display_cap_is_recorded_and_keeps_the_selection() { assert!(logical["combinations"].as_u64().unwrap() > 1); let selected = document["stage3_plan_selection"]["selected"] .as_str() - .expect("selection survives display cap"); + .unwrap_or_else(|| { + panic!( + "selection survives display cap: {}", + document["stage3_plan_selection"] + ) + }); let physical = document["stage2_physical_asap_optimization"]["candidates"] .as_array() .unwrap(); @@ -1889,11 +1906,17 @@ fn the_memo_search_document_names_the_selected_logical_candidate() { /// without `plan_installable`. Report-only: prices are unchanged. #[test] fn per_node_costs_show_operator_timing_and_predicted_cpu() { - let document = generate(&[]); + // The selected plan supplies the serialization evidence; exhaustive + // candidate coverage is tested separately by the archive tests. + let document = generate_bounded_example("planner-layering-1", &[]); let costs = document["stage3_plan_selection"]["costs"] .as_object() .unwrap(); - assert!(!costs.is_empty()); + assert!( + !costs.is_empty(), + "missing evaluated costs: {}", + document["stage3_plan_selection"] + ); let mut timings = HashSet::new(); for (id, cost) in costs { let per_node = cost["per_node"].as_object().unwrap(); From 92b492170c76da1cb9dc67c0896d8d6774c3cc9b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 05:59:18 +0000 Subject: [PATCH 20/29] feat: jointly price supplier and worker configurations --- .../tests/production_rollup_workloads.rs | 4 + crates/plan-selection/src/lib.rs | 3 + .../costing/direct_price_options_tests.rs | 700 ++++++++++++++++++ .../src/search/costing/direct_profile.rs | 167 +++-- .../search/costing/supplier_price_options.rs | 264 +++++++ .../src/search/milp/direct_price_options.rs | 539 ++++++++++++++ crates/plan-selection/src/search/milp/mod.rs | 1 + .../plan-selection/src/search/milp/model.rs | 71 +- .../src/search/refinement/components.rs | 1 + .../src/search/refinement/planner.rs | 108 ++- .../src/search/refinement/search.rs | 43 +- crates/plan-selection/src/search/route.rs | 2 + .../proposals/joint-planning-search.md | 8 +- 13 files changed, 1811 insertions(+), 100 deletions(-) create mode 100644 crates/plan-selection/src/search/costing/direct_price_options_tests.rs create mode 100644 crates/plan-selection/src/search/costing/supplier_price_options.rs create mode 100644 crates/plan-selection/src/search/milp/direct_price_options.rs diff --git a/crates/integration-tests/tests/production_rollup_workloads.rs b/crates/integration-tests/tests/production_rollup_workloads.rs index 2c7ed45ea..e00b2ba7c 100644 --- a/crates/integration-tests/tests/production_rollup_workloads.rs +++ b/crates/integration-tests/tests/production_rollup_workloads.rs @@ -104,6 +104,10 @@ fn check( "selected cost={:?}; report={report:?}", selection.costs[&selection.selected].cost_per_second ); + assert!( + report.joint_price_solve_calls > 0, + "joint supplier prices were not solved: {report:?}" + ); assert!( report.direct_cost_validated_plans > 0, "no directly priced validated plan: {report:?}" diff --git a/crates/plan-selection/src/lib.rs b/crates/plan-selection/src/lib.rs index d1276fd16..4899a1ee0 100644 --- a/crates/plan-selection/src/lib.rs +++ b/crates/plan-selection/src/lib.rs @@ -1101,6 +1101,9 @@ pub struct CandidateConnectionReport { /// Restricted query-time proposals priced completely inside the MILP. #[serde(default)] pub direct_cost_solve_calls: usize, + /// Direct solves jointly selecting supplier-dependent pricing options. + #[serde(default)] + pub joint_price_solve_calls: usize, /// Feasible direct proposals whose objective agrees with the evaluator. #[serde(default)] pub direct_cost_validated_plans: usize, diff --git a/crates/plan-selection/src/search/costing/direct_price_options_tests.rs b/crates/plan-selection/src/search/costing/direct_price_options_tests.rs new file mode 100644 index 000000000..70bca171b --- /dev/null +++ b/crates/plan-selection/src/search/costing/direct_price_options_tests.rs @@ -0,0 +1,700 @@ +//! Price-interface regressions: compatible inputs must not disappear merely +//! because their statistics differ from the template's input. +use super::*; +use crate::search::candidates::configurations::CandidateConfiguration; +use crate::search::candidates::graph::{ + CandidateDependencyGraph, PhysicalCandidate, ProducerRequirement, QueryRootCandidates, +}; +use crate::search::candidates::inventory::CandidateInventory; +use asap_logical_asap_optimization::memo::MemoGroupId; +use asap_logical_asap_optimization::test_support::lower_promql; +use asap_types::ir::properties::AccuracyTarget; +use asap_types::ir::{Operator, OperatorNode, ProjectItem, ScalarExpr}; +use asap_types::workload::{ + DataWorkload, Predictability, Query, QueryRequirements, QueryWorkloadEntry, RepetitionInterval, +}; + +#[test] +fn differing_supplier_statistics_keep_both_price_interfaces() { + let scan = |metric| { + OperatorNode::reachable(&lower_promql(metric, AccuracyTarget::exact())) + .into_iter() + .find(|node| matches!(node.non_asap(), Some(NonASAPOp::Scan { .. }))) + .unwrap() + }; + let m = scan("m"); + let n = scan("n"); + let project = OperatorNode::new_shared(Operator::NonASAP(NonASAPOp::Project { + cols: m + .schema + .fields + .iter() + .enumerate() + .map(|(index, field)| ProjectItem { + alias: Some(field.name.clone()), + expr: ScalarExpr::Column(index), + }) + .collect(), + qualifier: None, + child: m.clone(), + })) + .unwrap(); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph { + candidates: (0..3) + .map(|id| PhysicalCandidate { + group: MemoGroupId(id as u32), + local_cost: 0.0, + producer_requirements: if id == 2 { + vec![ProducerRequirement { + compatible_producers: vec![ + PhysicalCandidateId(0), + PhysicalCandidateId(1), + ] + .into(), + }] + } else { + vec![] + }, + }) + .collect(), + query_roots: vec![QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(2)], + }], + ..Default::default() + }, + templates: vec![Some(m), Some(n), Some(project.clone())], + group_candidates: vec![], + }, + configurations: (0..3) + .map(|id| CandidateConfiguration { + source_candidate: PhysicalCandidateId(id), + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::QueryTime, + }) + .collect(), + }; + let roots = vec![QueryRoot::Operator(project)]; + let entries = vec![QueryWorkloadEntry { + query: Query("m".into()), + requirements: QueryRequirements { + accuracy: AccuracyTarget::exact(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval( + 60_000, + ))), + range_evaluation: None, + }]; + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival": {"ingesting": {"rows_per_second": 100.0}}, + "sources": {"m": {"rows_held":10,"series_count":1},"n":{"rows_held":100,"series_count":1}} + })) + .unwrap(); + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &roots, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let extracted = extract_price_options(&configured, &context) + .unwrap() + .unwrap(); + assert_eq!(extracted.candidates[0].len(), 1); + assert_eq!(extracted.candidates[1].len(), 1); + let reader_options = &extracted.candidates[2]; + assert_eq!( + reader_options.len(), + 2, + "both legal supplier statistics must reach the joint model" + ); + for producer in [PhysicalCandidateId(0), PhysicalCandidateId(1)] { + let option = reader_options + .iter() + .find(|option| { + option + .compatible_suppliers + .iter() + .any(|binding| binding.producer == producer) + }) + .expect("missing compatible supplier price option"); + assert_eq!(option.compatible_suppliers.len(), 1); + assert_eq!(option.compatible_suppliers[0].price_option, 0); + } + let low = reader_options + .iter() + .find(|option| option.compatible_suppliers[0].producer == PhysicalCandidateId(0)) + .unwrap(); + let high = reader_options + .iter() + .find(|option| option.compatible_suppliers[0].producer == PhysicalCandidateId(1)) + .unwrap(); + assert!( + high.profile.work_seconds > low.profile.work_seconds, + "projection work must follow selected input rows, not template rows" + ); + assert_eq!( + configured.inventory.graph.candidates[2].producer_requirements[0] + .compatible_producers + .len(), + 2 + ); + // Force each structurally legal supplier in turn. The model must price + // the selected binding rather than only returning the cheapest interface. + for producer in [PhysicalCandidateId(0), PhysicalCandidateId(1)] { + configured.inventory.graph.reader_constraints = vec![ + crate::search::candidates::graph::ReaderConstraint::ActivationRequires { + candidate: PhysicalCandidateId(2), + required_candidate: producer, + }, + ]; + let proposal = + crate::search::milp::direct_price_options::solve_joint_price_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &extracted, + ) + .unwrap() + .expect("forced supplier must remain feasible"); + assert!(proposal + .selected + .connections + .iter() + .any(|edge| edge.producer == producer)); + let mut rejections = + crate::search::refinement::planner::CandidateRejectionSummary::default(); + let evaluated = crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("forced supplier rejected: {rejections:?}")); + let actual = evaluated.0.costs[&evaluated.0.selected] + .cost_per_second + .unwrap(); + let modeled = proposal + .direct_modeled_cost + .expect("joint direct objective"); + assert!( + (actual - modeled).abs() <= 1e-9 * actual.max(1e-15), + "supplier {producer:?}: modeled {modeled}, actual {actual}" + ); + } + let timed = CandidatePlanningContext { + deadline: Some(std::time::Instant::now()), + ..context + }; + assert!(matches!( + extract_price_options(&configured, &timed), + Err(DirectProfileError::DeadlineExceeded(_)) + )); +} + +#[test] +fn downstream_prices_follow_the_selected_supplier_interface() { + let scan = |metric| { + OperatorNode::reachable(&lower_promql(metric, AccuracyTarget::exact())) + .into_iter() + .find(|node| matches!(node.non_asap(), Some(NonASAPOp::Scan { .. }))) + .unwrap() + }; + let m = scan("m"); + let n = scan("n"); + let project = OperatorNode::new_shared(Operator::NonASAP(NonASAPOp::Project { + cols: m + .schema + .fields + .iter() + .enumerate() + .map(|(index, field)| ProjectItem { + alias: Some(field.name.clone()), + expr: ScalarExpr::Column(index), + }) + .collect(), + qualifier: None, + child: m.clone(), + })) + .unwrap(); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph { + candidates: (0..3) + .map(|id| PhysicalCandidate { + group: MemoGroupId(id as u32), + local_cost: 0.0, + producer_requirements: if id == 2 { + vec![ProducerRequirement { + compatible_producers: vec![ + PhysicalCandidateId(0), + PhysicalCandidateId(1), + ] + .into(), + }] + } else { + vec![] + }, + }) + .collect(), + query_roots: vec![QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(2)], + }], + ..Default::default() + }, + templates: vec![Some(m), Some(n), Some(project.clone())], + group_candidates: vec![], + }, + configurations: (0..3) + .map(|id| CandidateConfiguration { + source_candidate: PhysicalCandidateId(id), + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::QueryTime, + }) + .collect(), + }; + let downstream = OperatorNode::new_shared(Operator::NonASAP(NonASAPOp::Project { + cols: project + .schema + .fields + .iter() + .enumerate() + .map(|(index, field)| ProjectItem { + alias: Some(field.name.clone()), + expr: ScalarExpr::Column(index), + }) + .collect(), + qualifier: None, + child: project, + })) + .unwrap(); + configured + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: MemoGroupId(3), + local_cost: 0.0, + producer_requirements: vec![ProducerRequirement { + compatible_producers: vec![PhysicalCandidateId(2)].into(), + }], + }); + configured + .inventory + .templates + .push(Some(downstream.clone())); + configured.configurations.push(CandidateConfiguration { + source_candidate: PhysicalCandidateId(3), + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::QueryTime, + }); + configured.inventory.graph.query_roots[0].compatible_candidates = vec![PhysicalCandidateId(3)]; + let roots = vec![QueryRoot::Operator(downstream)]; + let entries = vec![QueryWorkloadEntry { + query: Query("m".into()), + requirements: QueryRequirements { + accuracy: AccuracyTarget::exact(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval( + 60_000, + ))), + range_evaluation: None, + }]; + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival": {"ingesting": {"rows_per_second": 100.0}}, + "sources": {"m": {"rows_held":10,"series_count":1},"n":{"rows_held":100,"series_count":1}} + })) + .unwrap(); + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &roots, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let extracted = extract_price_options(&configured, &context) + .unwrap() + .unwrap(); + assert_eq!(extracted.candidates[2].len(), 2); + assert_eq!( + extracted.candidates[3].len(), + 2, + "a single supplier candidate may expose multiple downstream price interfaces" + ); + let mut referenced = Vec::new(); + for option in &extracted.candidates[3] { + assert_eq!(option.compatible_suppliers.len(), 1); + let binding = &option.compatible_suppliers[0]; + assert_eq!(binding.producer, PhysicalCandidateId(2)); + referenced.push(binding.price_option); + let supplier = &extracted.candidates[2][binding.price_option]; + assert!( + (option.profile.work_seconds - supplier.profile.work_seconds).abs() < 1e-15, + "identical projections over the same rows must receive the same work price" + ); + } + referenced.sort_unstable(); + assert_eq!(referenced, vec![0, 1]); + // Optional identity cycles must not disable otherwise legal pricing paths + // or create a zero-input phantom plan. All selected DAGs still need a scan. + let solve = + |configured: &CandidateConfigurationInventory, + profile: &crate::search::milp::direct_price_options::JointPriceProfile| { + crate::search::milp::direct_price_options::solve_joint_price_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + profile, + ) + .unwrap() + .unwrap() + }; + let baseline = solve(&configured, &extracted); + configured.inventory.graph.candidates[2].producer_requirements[0].compatible_producers = vec![ + PhysicalCandidateId(0), + PhysicalCandidateId(1), + PhysicalCandidateId(2), + PhysicalCandidateId(3), + ] + .into(); + let cyclic = extract_price_options(&configured, &context) + .unwrap() + .unwrap(); + assert_eq!(cyclic.candidates[2].len(), 2); + assert_eq!(cyclic.candidates[3].len(), 2); + assert!(cyclic.candidates[2] + .iter() + .flat_map(|option| &option.compatible_suppliers) + .all(|binding| binding.producer != PhysicalCandidateId(2))); + let chosen = solve(&configured, &cyclic); + assert!( + (chosen.direct_modeled_cost.unwrap() - baseline.direct_modeled_cost.unwrap()).abs() < 1e-12 + ); + assert!(chosen + .selected + .connections + .iter() + .any(|edge| edge.producer.0 < 2)); +} + +#[test] +fn supplier_worker_buckets_remain_joint_choices_and_match_evaluator() { + use asap_logical_asap_optimization::per_computation_candidates::{ + compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, + }; + use asap_types::ir::schema::SketchAlgorithm; + use std::rc::Rc; + fn append( + node: Rc, + configured: &mut CandidateConfigurationInventory, + ingestion: bool, + mode: Materialization, + ) -> PhysicalCandidateId { + let ingestion = ingestion + || (matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) + && mode == Materialization::MatAtIngestion(Medium::Memory)); + let pane = matches!(node.asap(), Some(ASAPOp::PaneChain { .. })); + let mut inputs = Vec::new(); + node.operator + .map_children(|child| inputs.push(append(child.clone(), configured, ingestion, mode))); + let id = PhysicalCandidateId(configured.configurations.len()); + configured.configurations.push(CandidateConfiguration { + source_candidate: id, + materialization: if pane { + mode + } else { + Materialization::NotMaterialized + }, + timing: if ingestion { + ExecutionTiming::IngestionTime + } else { + ExecutionTiming::QueryTime + }, + }); + configured.inventory.templates.push(Some(node)); + configured + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: MemoGroupId(id.0 as u32), + local_cost: 0.0, + producer_requirements: inputs + .into_iter() + .map(|supplier| ProducerRequirement { + compatible_producers: vec![supplier].into(), + }) + .collect(), + }); + id + } + + let target = AccuracyTarget::max_error(0.05) + .failure_probability(0.01) + .accuracy_scope(AccuracyScope::PerValue); + let original = lower_promql("quantile_over_time(0.95, latency[5m])", target.clone()); + let generated = + generate_per_computation_candidates(vec![(0, QueryRoot::Operator(original.clone()))]) + .unwrap(); + let choice = generated.targets[0] + .choices + .candidates + .iter() + .position(|candidate| { + matches!( + candidate, + ComputationCandidate::Summary(SketchAlgorithm::Kll) + ) + }) + .unwrap(); + let roots: Vec<_> = compose_logical_candidate(&generated, &[choice]) + .unwrap() + .into_iter() + .map(|(_, root)| root) + .collect(); + let pane_rules = + asap_logical_asap_optimization::asap_aware_cse::pane_candidates(&roots, &[Some(60_000)]); + let roots = + asap_logical_asap_optimization::asap_aware_cse::window_composition_pass2_candidates( + &roots, + &pane_rules, + ) + .unwrap() + .pop() + .unwrap() + .roots + .unwrap(); + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival":{"ingesting":{"rows_per_second":100.0}}, + "sources":{ + "latency":{"series_count":1,"sample_interval":100,"rows_held":3000,"values_non_negative":true} + } + })).unwrap(); + let sized = crate::resize_for_failure_probability(&roots, std::slice::from_ref(&target), &data) + .unwrap(); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph::default(), + templates: vec![], + group_candidates: vec![], + }, + configurations: vec![], + }; + let QueryRoot::Operator(root) = &sized[0] else { + panic!("operator") + }; + let root_id = append( + root.clone(), + &mut configured, + false, + Materialization::MatAtIngestion(Medium::Memory), + ); + configured + .inventory + .graph + .query_roots + .push(QueryRootCandidates { + compatible_candidates: vec![root_id], + }); + let chain = configured + .inventory + .templates + .iter() + .position(|node| { + node.as_ref() + .is_some_and(|node| matches!(node.asap(), Some(ASAPOp::PaneChain { .. }))) + }) + .unwrap(); + let raw_supplier = configured.inventory.graph.candidates[chain].producer_requirements[0] + .compatible_producers[0]; + assert!(matches!( + configured.inventory.templates[raw_supplier.0] + .as_ref() + .unwrap() + .non_asap(), + Some(NonASAPOp::Scan { .. }) + )); + // This tests the pricing kernel, not executor installation of a projection + // below a pane. Both suppliers return exactly the same stream. The identity projection + // adds ingestion CPU without changing rows, values, or source identity. + let source = configured.inventory.templates[raw_supplier.0] + .as_ref() + .unwrap() + .clone(); + let projected = OperatorNode::new_shared(Operator::NonASAP(NonASAPOp::Project { + cols: source + .schema + .fields + .iter() + .enumerate() + .map(|(index, field)| ProjectItem { + alias: Some(field.name.clone()), + expr: ScalarExpr::Column(index), + }) + .collect(), + qualifier: None, + child: source, + })) + .unwrap(); + let projected_supplier = PhysicalCandidateId(configured.configurations.len()); + configured.inventory.templates.push(Some(projected)); + configured.configurations.push(CandidateConfiguration { + source_candidate: projected_supplier, + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::IngestionTime, + }); + configured + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: configured.inventory.graph.candidates[raw_supplier.0].group, + local_cost: 0.0, + producer_requirements: vec![ProducerRequirement { + compatible_producers: vec![raw_supplier].into(), + }], + }); + let originals = vec![QueryRoot::Operator(original)]; + let entries = vec![QueryWorkloadEntry { + query: Query("quantile_over_time(0.95, latency[5m])".into()), + requirements: QueryRequirements { + accuracy: target, + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval( + 60_000, + ))), + range_evaluation: None, + }]; + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &originals, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let initial = extract(&configured, &context).unwrap().unwrap(); + let raw_work: f64 = [raw_supplier.0, chain] + .into_iter() + .map(|id| initial.profile.nodes[id].work_seconds * 1000.0) + .sum(); + assert!(raw_work > 0.0); + let mut settings = crate::PlanSelectionSettings::default(); + settings.resource_speed.ms_per_cpu_op *= 850.0 / raw_work; + settings.resource_speed.ms_per_scanned_byte *= 850.0 / raw_work; + settings + .measured_operation_costs + .operators + .push(crate::cost::measured::OperatorCost { + operator: crate::OperatorKind::Project, + phase: Some(ExecutionTiming::IngestionTime), + sketch: None, + exact: None, + measure: None, + ns_per_unit: 0.0, + ns_per_invocation: 1_000_000_000.0, + memory_bytes_per_state: None, + }); + let context = CandidatePlanningContext { + models: crate::PlanningModels { + plan_selection: &settings, + ..context.models + }, + ..context + }; + configured.inventory.graph.candidates[chain].producer_requirements[0].compatible_producers = + vec![raw_supplier, projected_supplier].into(); + let old = extract(&configured, &context).unwrap().unwrap(); + assert_eq!( + old.suppliers[chain], + vec![raw_supplier], + "old fixed-bucket extraction loses the second worker choice" + ); + let joint = extract_price_options(&configured, &context) + .unwrap() + .unwrap(); + assert_eq!(joint.candidates[chain].len(), 2); + let raw_option = joint.candidates[chain] + .iter() + .find(|option| { + option + .compatible_suppliers + .iter() + .any(|binding| binding.producer == raw_supplier) + }) + .unwrap(); + let projected_option = joint.candidates[chain] + .iter() + .find(|option| { + option + .compatible_suppliers + .iter() + .any(|binding| binding.producer == projected_supplier) + }) + .unwrap(); + assert_eq!(raw_option.profile.compaction_seconds, 0.0); + assert!(projected_option.profile.compaction_seconds > 0.0); + for supplier in [raw_supplier, projected_supplier] { + configured.inventory.graph.reader_constraints = vec![ + crate::search::candidates::graph::ReaderConstraint::ActivationRequires { + candidate: root_id, + required_candidate: supplier, + }, + ]; + if supplier == raw_supplier { + configured.inventory.graph.reader_constraints.push( + crate::search::candidates::graph::ReaderConstraint::AllowedOutputs { + candidate: projected_supplier, + outputs: vec![], + }, + ); + } + let proposal = + crate::search::milp::direct_price_options::solve_joint_price_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &joint, + ) + .unwrap() + .unwrap(); + let mut rejections = + crate::search::refinement::planner::CandidateRejectionSummary::default(); + let evaluated = crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("{rejections:?}")); + let actual = evaluated.0.costs[&evaluated.0.selected] + .cost_per_second + .unwrap(); + let modeled = proposal.direct_modeled_cost.unwrap(); + assert!( + (actual - modeled).abs() <= 1e-9 * actual.max(1e-15), + "{supplier:?}: direct {modeled}, evaluator {actual}" + ); + } +} diff --git a/crates/plan-selection/src/search/costing/direct_profile.rs b/crates/plan-selection/src/search/costing/direct_profile.rs index 5369bbfb7..f431af283 100644 --- a/crates/plan-selection/src/search/costing/direct_profile.rs +++ b/crates/plan-selection/src/search/costing/direct_profile.rs @@ -1,10 +1,9 @@ -//! Exact local profiles for query-time plans and fixed-worker maintained panes. +//! Local cost certification shared by fixed and supplier-dependent profiles. //! -//! Each accepted input is priced from its actual recursively certified supplier. -//! Different priced interfaces are excluded from this proposal, not the full -//! catalog. No original-template child is used as workload statistics evidence. -//! Ingestion support is limited to fixed-retention pane chains with certified -//! unary input closures whose complete range certifies one worker-count bucket. +//! Price each operation from its actual supplier output and state configuration. +//! The fixed-profile adapter admits one interface; supplier_price_options keeps +//! compatible interfaces as joint MILP choices. Ingestion worker counts require +//! certified unary upstream work, never statistics from a template child. use crate::search::candidates::configurations::CandidateConfigurationInventory; use crate::search::candidates::graph::PhysicalCandidateId; use crate::search::milp::direct_cost::{DirectCostProfile, NodeCostProfile, QueryScheduleRates}; @@ -25,6 +24,8 @@ pub(in crate::search) enum DirectProfileError { DeadlineExceeded(#[from] asap_logical_asap_optimization::memo::CandidateGenerationTimeout), #[error("direct cost profile: {0}")] Invalid(String), + #[error("direct cost profile unsupported: {0}")] + Unsupported(String), } pub(in crate::search) struct DirectProfileExtraction { @@ -75,13 +76,23 @@ pub(in crate::search) fn extract_latest_panes( extract_scan_class(configured, context, true) } -fn extract_scan_class( +struct PreparedProfile { + nodes: Vec>, + lookbacks: Vec>, + all_lookbacks: Vec>>, + delta: Option, + schedules: Option, + rate: f64, + request_rate: f64, +} + +fn prepare_profile( configured: &CandidateConfigurationInventory, context: &CandidatePlanningContext<'_>, prefer_latest_pane: bool, -) -> Result, DirectProfileError> { +) -> Result, DirectProfileError> { let settings = context.models.plan_selection; - let CostModel::UsageOverTime(prices) = &settings.cost_model else { + let CostModel::UsageOverTime(_prices) = &settings.cost_model else { return Ok(None); }; let Some(first) = context.entries.first() else { @@ -278,6 +289,7 @@ fn extract_scan_class( // Each scan keeps one explicit lookback class, not a maximum requiring an // optionally selected witness. Every admitted consumer edge must use it. let mut lookbacks = vec![None; count]; + let mut all_lookbacks = vec![vec![None]; count]; for id in 0..count { let Some(scan) = &nodes[id] else { continue }; if scan.output_state.timing != ExecutionTiming::QueryTime @@ -316,6 +328,10 @@ fn extract_scan_class( root.compatible_candidates .contains(&PhysicalCandidateId(id)) }); + if !raw_output { + all_lookbacks[id] = ordinary.union(&kept).copied().collect(); + all_lookbacks[id].sort(); + } if raw_output { lookbacks[id] = None; } else { @@ -337,6 +353,39 @@ fn extract_scan_class( } } } + Ok(Some(PreparedProfile { + nodes, + lookbacks, + all_lookbacks, + delta, + schedules, + rate, + request_rate, + })) +} + +fn extract_scan_class( + configured: &CandidateConfigurationInventory, + context: &CandidatePlanningContext<'_>, + prefer_latest_pane: bool, +) -> Result, DirectProfileError> { + let Some(prepared) = prepare_profile(configured, context, prefer_latest_pane)? else { + return Ok(None); + }; + let PreparedProfile { + nodes, + lookbacks, + delta, + schedules, + rate, + request_rate, + .. + } = prepared; + let count = nodes.len(); + let settings = context.models.plan_selection; + let CostModel::UsageOverTime(prices) = &settings.cost_model else { + return Ok(None); + }; let mut cache = vec![None; count]; let mut visited = vec![false; count]; let mut visiting = vec![false; count]; @@ -363,6 +412,7 @@ fn extract_scan_class( &mut worker_counts, &mut compaction_seconds, delta, + None, )?; } let supported: Vec<_> = cache.iter().map(Option::is_some).collect(); @@ -384,31 +434,12 @@ fn extract_scan_class( let Some(priced) = priced else { return Ok(zero()); }; - let raw_retention = priced - .raw_scan - .map(|(source, bytes)| { - serde_json::to_string(source) - .map(|identity| (identity, bytes as f64 * prices.cost_per_disk_byte_second)) - .map_err(|error| DirectProfileError::Invalid(error.to_string())) - }) - .transpose()?; - Ok(NodeCostProfile { - execution_cost: priced.cpu_ops_per_second * prices.cost_per_cpu_op - + priced.scanned_bytes_per_second * prices.cost_per_scanned_byte, - ingestion: nodes[id].as_ref().unwrap().output_state.timing - == ExecutionTiming::IngestionTime, - standing_cost_per_second: priced.average_memory_bytes - * prices.cost_per_memory_byte_second - + priced.disk_bytes as f64 * prices.cost_per_disk_byte_second, - work_seconds: priced.work_ms / 1000.0, - compaction_seconds: compaction_seconds[id], - held_memory_cost_per_second: priced.held.map_or(0.0, |(bytes, _, _)| { - bytes as f64 * prices.cost_per_memory_byte_second - }), - materialized: nodes[id].as_ref().unwrap().materialization - != Materialization::NotMaterialized, - raw_retention, - }) + profile_for_priced( + priced, + nodes[id].as_ref().unwrap(), + compaction_seconds[id], + context, + ) }) .collect::, DirectProfileError>>()?; let request_ms = settings.measured_operation_costs.ns_per_evaluation * 1e-6; @@ -427,6 +458,41 @@ fn extract_scan_class( })) } +fn profile_for_priced( + priced: &PricedNode<'_>, + node: &PhysicalASAPDAGNode, + compaction_seconds: f64, + context: &CandidatePlanningContext<'_>, +) -> Result { + let CostModel::UsageOverTime(prices) = &context.models.plan_selection.cost_model else { + return Err(DirectProfileError::Unsupported( + "non-usage objective".into(), + )); + }; + let raw_retention = priced + .raw_scan + .map(|(source, bytes)| { + serde_json::to_string(source) + .map(|identity| (identity, bytes as f64 * prices.cost_per_disk_byte_second)) + .map_err(|error| DirectProfileError::Invalid(error.to_string())) + }) + .transpose()?; + Ok(NodeCostProfile { + execution_cost: priced.cpu_ops_per_second * prices.cost_per_cpu_op + + priced.scanned_bytes_per_second * prices.cost_per_scanned_byte, + ingestion: node.output_state.timing == ExecutionTiming::IngestionTime, + standing_cost_per_second: priced.average_memory_bytes * prices.cost_per_memory_byte_second + + priced.disk_bytes as f64 * prices.cost_per_disk_byte_second, + work_seconds: priced.work_ms / 1000.0, + compaction_seconds, + held_memory_cost_per_second: priced.held.map_or(0.0, |(bytes, _, _)| { + bytes as f64 * prices.cost_per_memory_byte_second + }), + materialized: node.materialization != Materialization::NotMaterialized, + raw_retention, + }) +} + #[allow(clippy::too_many_arguments)] fn price<'a>( id: usize, @@ -444,6 +510,7 @@ fn price<'a>( worker_counts: &mut [Option], compaction_seconds: &mut [f64], delta: Option, + supplier_override: Option, ) -> Result<(), DirectProfileError> { check_deadline(context.deadline)?; if visited[id] || visiting[id] { @@ -455,16 +522,21 @@ fn price<'a>( }; visiting[id] = true; let requirements = &configured.inventory.graph.candidates[id].producer_requirements; - let options: Vec<_> = requirements.first().map_or_else( - || vec![None], - |input| { - input - .compatible_producers - .iter() - .copied() - .map(Some) - .collect() + let options: Vec<_> = supplier_override.map_or_else( + || { + requirements.first().map_or_else( + || vec![None], + |input| { + input + .compatible_producers + .iter() + .copied() + .map(Some) + .collect() + }, + ) }, + |supplier| vec![Some(supplier)], ); for supplier in options { let mut views = Vec::new(); @@ -488,6 +560,7 @@ fn price<'a>( worker_counts, compaction_seconds, delta, + None, )?; let Some(priced) = &cache[supplier.0] else { continue; @@ -2180,3 +2253,11 @@ mod tests { )); } } + +#[path = "supplier_price_options.rs"] +mod supplier_price_options; +pub(in crate::search) use supplier_price_options::extract_price_options; + +#[cfg(test)] +#[path = "direct_price_options_tests.rs"] +mod price_options_tests; diff --git a/crates/plan-selection/src/search/costing/supplier_price_options.rs b/crates/plan-selection/src/search/costing/supplier_price_options.rs new file mode 100644 index 000000000..322e9f4c6 --- /dev/null +++ b/crates/plan-selection/src/search/costing/supplier_price_options.rs @@ -0,0 +1,264 @@ +//! Local pricing interfaces are generated from certified supplier interfaces. +//! No workload plans are enumerated. A candidate's distinct local options are +//! connected jointly by the MILP; identical interfaces retain every binding. +use super::*; +use crate::search::milp::direct_price_options::{ + CandidatePriceOption, JointPriceProfile, SupplierPriceBinding, +}; + +const MAXIMUM_PRICE_OPTIONS: usize = 50_000; + +#[derive(Clone)] +struct PriceClass<'a> { + priced: PricedNode<'a>, + lookback: Option, + sketches: usize, + raw_source: Option, + work: WorkBounds, + workers: Option, + compaction: f64, + bindings: Vec, +} + +impl PriceClass<'_> { + fn same_interface(&self, other: &Self) -> bool { + same_price(&self.priced, &other.priced) + && self.lookback == other.lookback + && self.sketches == other.sketches + && self.raw_source == other.raw_source + && self.workers == other.workers + && self.compaction.to_bits() == other.compaction.to_bits() + && self.work.minimum.to_bits() == other.work.minimum.to_bits() + && self.work.maximum.to_bits() == other.work.maximum.to_bits() + } +} + +/// Exact scan-lookback equality remains required: max-width sharing needs an +/// explicit selected-consumer witness. Cyclic catalogs use bounded local +/// interface propagation; selected physical graphs must still be acyclic. +pub(in crate::search) fn extract_price_options( + configured: &CandidateConfigurationInventory, + context: &CandidatePlanningContext<'_>, +) -> Result, DirectProfileError> { + let Some(prepared) = prepare_profile(configured, context, false)? else { + return Ok(None); + }; + let PreparedProfile { + nodes, + mut lookbacks, + all_lookbacks, + delta, + schedules, + rate, + request_rate, + } = prepared; + let count = nodes.len(); + let mut pending = vec![0usize; count]; + let mut parents = vec![Vec::new(); count]; + for (id, node) in nodes.iter().enumerate() { + if node.is_none() { + continue; + } + for supplier in configured.inventory.graph.candidates[id] + .producer_requirements + .iter() + .flat_map(|input| input.compatible_producers.iter()) + { + if supplier.0 != id && nodes[supplier.0].is_some() { + pending[id] += 1; + parents[supplier.0].push(id); + } + } + } + let mut ready: Vec<_> = (0..count).filter(|id| pending[*id] == 0).collect(); + let mut order = Vec::new(); + while let Some(id) = ready.pop() { + check_deadline(context.deadline)?; + order.push(id); + for parent in &parents[id] { + pending[*parent] -= 1; + if pending[*parent] == 0 { + ready.push(*parent); + } + } + } + let acyclic = order.len() == count; + if !acyclic { + let mut ordered = vec![false; count]; + for id in &order { + ordered[*id] = true; + } + order.extend((0..count).filter(|id| !ordered[*id])); + } + // Reuse the existing one-node certification/pricing path. Only one already + // priced supplier interface is present in this reusable scratch cache, so + // price() performs one local operation without recursive plan generation. + let mut cache = vec![None; count]; + let mut visited = vec![true; count]; + let mut visiting = vec![false; count]; + let mut suppliers = vec![Vec::new(); count]; + let mut sketch_counts = vec![0; count]; + let mut raw_sources = vec![None; count]; + let mut closure_work = vec![WorkBounds::default(); count]; + let mut worker_counts = vec![None; count]; + let mut compactions = vec![0.0; count]; + let mut classes: Vec>> = vec![Vec::new(); count]; + let mut total = 0; + // Every legal unary DAG path has at most N candidates. N monotone passes + // therefore cover every supported simple path even in a cyclic catalog. + // Walk-derived bindings are harmless: the solver enforces original-node + // uniqueness and acyclicity. Class IDs remain stable as options append. + for _round in 0..if acyclic { 1 } else { count } { + let before = total; + for &id in &order { + check_deadline(context.deadline)?; + if nodes[id].is_none() { + continue; + } + let mut supplier_counts = Vec::new(); + if let Some(input) = configured.inventory.graph.candidates[id] + .producer_requirements + .first() + { + for producer in &input.compatible_producers { + check_deadline(context.deadline)?; + if producer.0 != id { + supplier_counts.push((Some(*producer), classes[producer.0].len())); + } + } + } else { + supplier_counts.push((None, 1)); + } + // Stream local bindings instead of allocating a potentially enormous + // supplier-by-option product before the deadline can be checked. + let inputs = supplier_counts.into_iter().flat_map(|(producer, count)| { + (0..count).map(move |price_option| { + producer.map(|producer| SupplierPriceBinding { + producer, + price_option, + }) + }) + }); + for binding in inputs { + for &lookback in &all_lookbacks[id] { + check_deadline(context.deadline)?; + if let Some(binding) = binding { + let supplier = binding.producer.0; + let source = &classes[supplier][binding.price_option]; + cache[supplier] = Some(source.priced.clone()); + lookbacks[supplier] = source.lookback; + sketch_counts[supplier] = source.sketches; + raw_sources[supplier] = source.raw_source.clone(); + closure_work[supplier] = source.work; + worker_counts[supplier] = source.workers; + compactions[supplier] = source.compaction; + } + lookbacks[id] = lookback; + cache[id] = None; + visited[id] = false; + suppliers[id].clear(); + sketch_counts[id] = 0; + raw_sources[id] = None; + closure_work[id] = WorkBounds::default(); + worker_counts[id] = None; + compactions[id] = 0.0; + price( + id, + configured, + context, + &nodes, + &lookbacks, + &mut cache, + &mut visited, + &mut visiting, + &mut suppliers, + &mut sketch_counts, + &mut raw_sources, + &mut closure_work, + &mut worker_counts, + &mut compactions, + delta, + binding.map(|binding| binding.producer), + )?; + let priced = cache[id].take(); + if let Some(binding) = binding { + cache[binding.producer.0] = None; + } + let Some(priced) = priced else { continue }; + let candidate = PriceClass { + priced, + lookback, + sketches: sketch_counts[id], + raw_source: raw_sources[id].clone(), + // Query-time closure work never determines an ingestion + // worker count. Keeping it would distinguish every traversal + // of an identity cycle despite identical local interfaces. + work: if nodes[id].as_ref().unwrap().output_state.timing + == ExecutionTiming::IngestionTime + { + closure_work[id] + } else { + WorkBounds::default() + }, + workers: worker_counts[id], + compaction: compactions[id], + bindings: binding.into_iter().collect(), + }; + if let Some(existing) = classes[id] + .iter_mut() + .find(|known| known.same_interface(&candidate)) + { + for binding in candidate.bindings { + if !existing.bindings.contains(&binding) { + existing.bindings.push(binding); + } + } + } else { + total += 1; + if total > MAXIMUM_PRICE_OPTIONS { + return Err(DirectProfileError::Unsupported(format!("local pricing domain exceeds {MAXIMUM_PRICE_OPTIONS} interface options"))); + } + classes[id].push(candidate); + } + } + } + } + if total == before { + break; + } + } + let candidates = classes + .iter() + .enumerate() + .map(|(id, classes)| { + classes + .iter() + .map(|class| { + Ok(CandidatePriceOption { + profile: profile_for_priced( + &class.priced, + nodes[id].as_ref().unwrap(), + class.compaction, + context, + )?, + compatible_suppliers: class.bindings.clone(), + }) + }) + .collect::, DirectProfileError>>() + }) + .collect::, _>>()?; + let settings = context.models.plan_selection; + let CostModel::UsageOverTime(prices) = &settings.cost_model else { + return Ok(None); + }; + let request_ms = settings.measured_operation_costs.ns_per_evaluation * 1e-6; + Ok(Some(JointPriceProfile { + schedules, + query_rate: rate, + request_latency_seconds: request_ms / 1000.0, + request_cost: request_ms / settings.resource_speed.ms_per_cpu_op + * prices.cost_per_cpu_op + * request_rate, + candidates, + })) +} diff --git a/crates/plan-selection/src/search/milp/direct_price_options.rs b/crates/plan-selection/src/search/milp/direct_price_options.rs new file mode 100644 index 000000000..e734d49d2 --- /dev/null +++ b/crates/plan-selection/src/search/milp/direct_price_options.rs @@ -0,0 +1,539 @@ +//! Joint selection of supplier-dependent local pricing interfaces. +//! Pricing options describe one physical candidate, not independent executions. +use super::direct_cost::{NodeCostProfile, QueryScheduleRates}; +use crate::search::candidates::graph::PhysicalCandidateId; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct SupplierPriceBinding { + pub producer: PhysicalCandidateId, + pub price_option: usize, +} + +#[derive(Debug, Clone)] +pub(crate) struct CandidatePriceOption { + pub profile: NodeCostProfile, + pub compatible_suppliers: Vec, +} + +#[derive(Debug, Clone)] +pub(crate) struct JointPriceProfile { + pub schedules: Option, + pub query_rate: f64, + pub request_latency_seconds: f64, + pub request_cost: f64, + pub candidates: Vec>, +} + +use super::direct_cost::DirectCostProfile; +use super::model::{CandidateProposal, CandidateSolveMetrics, ObservedAssignment}; +use crate::search::candidates::graph::*; +use good_lp::{constraint, Constraint, Expression, Variable}; + +/// Internal option nodes share an original physical identity. At most one may +/// activate, so two readers cannot obtain incompatible views of one producer. +pub(super) struct PriceOptionProjection<'a> { + graph: &'a CandidateDependencyGraph, + origins: Vec, + options: Vec>, + observations: &'a [ObservedAssignment], +} + +impl PriceOptionProjection<'_> { + fn connection(&self, edge: ProducerConsumerConnection) -> ProducerConsumerConnection { + ProducerConsumerConnection { + producer: self.origins[edge.producer.0], + consumer: self.origins[edge.consumer.0], + requirement: edge.requirement, + } + } + + fn root(&self, root: QueryRootAssignment) -> QueryRootAssignment { + QueryRootAssignment { + candidate: self.origins[root.candidate.0], + ..root + } + } + + pub(super) fn constrain_activation( + &self, + active: &[Variable], + constraints: &mut Vec, + ) { + let sum = |id: PhysicalCandidateId| -> Expression { + self.options[id.0] + .iter() + .map(|option| active[option.0]) + .sum() + }; + for (id, options) in self.options.iter().enumerate() { + if options.len() > 1 { + constraints.push(constraint!(sum(PhysicalCandidateId(id)) <= 1.0)); + } + } + for rule in &self.graph.reader_constraints { + if let ReaderConstraint::ActivationRequires { + candidate, + required_candidate, + } = rule + { + constraints.push(constraint!(sum(*candidate) <= sum(*required_candidate))); + } + } + } + + /// A rejected physical DAG remains rejected across all of its price-option + /// encodings. Cuts refer to sums over the original activation/edge identity. + pub(super) fn constrain_observations( + &self, + active: &[Variable], + edges: &[(ProducerConsumerConnection, Variable)], + roots: &[(QueryRootAssignment, Variable)], + constraints: &mut Vec, + ) { + for observation in self.observations { + for witness in &observation.rejected_queries { + let mut changed = Expression::from( + (witness.connections.len() + witness.query_roots.len()) as f64, + ); + for (edge, variable) in edges { + if witness.connections.contains(&self.connection(*edge)) { + changed -= *variable; + } + } + for (root, variable) in roots { + if witness.query_roots.contains(&self.root(*root)) { + changed -= *variable; + } + } + constraints.push(constraint!(changed >= 1.0)); + } + if observation.exact_cost.is_some() { + continue; + } + let selected = &observation.selected; + let mut mismatch = Expression::from( + (selected.selected_candidates.len() + + selected.connections.len() + + selected.query_roots.len()) as f64, + ); + for (id, variable) in active.iter().enumerate() { + mismatch += if selected.selected_candidates.contains(&self.origins[id]) { + -1.0 + } else { + 1.0 + } * *variable; + } + for (edge, variable) in edges { + mismatch += if selected.connections.contains(&self.connection(*edge)) { + -1.0 + } else { + 1.0 + } * *variable; + } + for (root, variable) in roots { + mismatch += if selected.query_roots.contains(&self.root(*root)) { + -1.0 + } else { + 1.0 + } * *variable; + } + constraints.push(constraint!(mismatch >= 1.0)); + } + } +} + +pub(crate) fn solve_joint_price_proposal_with_metrics( + graph: &CandidateDependencyGraph, + observations: &[ObservedAssignment], + remaining_seconds: Option, + metrics: &mut CandidateSolveMetrics, + profile: &JointPriceProfile, +) -> Result, CandidateSelectionError> { + let started = std::time::Instant::now(); + graph.validate()?; + super::model::validate_observations(graph, observations)?; + let invalid = |message: &str| CandidateSelectionError::InvalidGraph(message.into()); + if profile.candidates.len() != graph.candidates.len() { + return Err(invalid( + "joint price options do not match candidate inventory", + )); + } + let mut projection = PriceOptionProjection { + graph, + origins: Vec::new(), + options: Vec::new(), + observations, + }; + let mut expanded = CandidateDependencyGraph::default(); + let mut nodes = Vec::new(); + for (id, options) in profile.candidates.iter().enumerate() { + let mut ids = Vec::new(); + for option in options { + ids.push(PhysicalCandidateId(nodes.len())); + projection.origins.push(PhysicalCandidateId(id)); + nodes.push(option.profile.clone()); + expanded.candidates.push(PhysicalCandidate { + producer_requirements: Vec::new(), + ..graph.candidates[id].clone() + }); + } + projection.options.push(ids); + } + for (id, options) in profile.candidates.iter().enumerate() { + if remaining_seconds.is_some_and(|s| started.elapsed().as_secs_f64() >= s) { + return Err(CandidateSelectionError::Solver( + "time budget exhausted expanding price options".into(), + )); + } + let required = &graph.candidates[id].producer_requirements; + if !options.is_empty() && required.len() > 1 { + return Err(invalid( + "joint unary price profile has multiple required inputs", + )); + } + for (index, option) in options.iter().enumerate() { + if required.is_empty() && !option.compatible_suppliers.is_empty() { + return Err(invalid("source price option has supplier bindings")); + } + if let Some(input) = required.first() { + let mut suppliers = Vec::new(); + for binding in &option.compatible_suppliers { + if !input.compatible_producers.contains(&binding.producer) { + return Err(invalid("price option references incompatible supplier")); + } + let supplier = projection + .options + .get(binding.producer.0) + .and_then(|options| options.get(binding.price_option)) + .ok_or_else(|| { + invalid("price option references missing supplier option") + })?; + if !suppliers.contains(supplier) { + suppliers.push(*supplier); + } + } + expanded.candidates[projection.options[id][index].0] + .producer_requirements + .push(ProducerRequirement { + compatible_producers: suppliers.into(), + }); + } + } + } + for root in &graph.query_roots { + expanded.query_roots.push(QueryRootCandidates { + compatible_candidates: root + .compatible_candidates + .iter() + .flat_map(|id| projection.options[id.0].iter().copied()) + .collect(), + }); + } + if expanded + .query_roots + .iter() + .any(|root| root.compatible_candidates.is_empty()) + { + return Ok(None); + } + for budget in &graph.resource_budgets { + expanded.resource_budgets.push(ResourceBudget { + name: budget.name.clone(), + capacity: budget.capacity, + candidate_usage: projection + .origins + .iter() + .map(|id| budget.candidate_usage[id.0]) + .collect(), + }); + } + for rule in &graph.reader_constraints { + let (candidate, mapped): (_, Box ReaderConstraint>) = + match rule { + ReaderConstraint::ActivationRequires { .. } => continue, + ReaderConstraint::MinimumQueryReaders { + candidate, + query_outputs, + minimum, + } => ( + *candidate, + Box::new(|candidate| ReaderConstraint::MinimumQueryReaders { + candidate, + query_outputs: query_outputs.clone(), + minimum: *minimum, + }), + ), + ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs, + } => ( + *candidate, + Box::new( + |candidate| ReaderConstraint::HighestSharedQueryMaterialization { + candidate, + query_outputs: query_outputs.clone(), + }, + ), + ), + ReaderConstraint::AllowedOutputs { candidate, outputs } => ( + *candidate, + Box::new(|candidate| ReaderConstraint::AllowedOutputs { + candidate, + outputs: outputs.clone(), + }), + ), + ReaderConstraint::IncompatibleOutputs { + candidate, + left, + right, + } => ( + *candidate, + Box::new(|candidate| ReaderConstraint::IncompatibleOutputs { + candidate, + left: *left, + right: *right, + }), + ), + }; + expanded + .reader_constraints + .extend(projection.options[candidate.0].iter().map(|id| mapped(*id))); + } + let fixed = DirectCostProfile { + nodes, + schedules: profile.schedules.clone(), + query_rate: profile.query_rate, + request_latency_seconds: profile.request_latency_seconds, + request_cost: profile.request_cost, + }; + metrics.model_construction_seconds += started.elapsed().as_secs_f64(); + let remaining = remaining_seconds.map(|s| (s - started.elapsed().as_secs_f64()).max(0.0)); + metrics.joint_price_solve_calls += 1; + let proposal = super::model::solve_with_objective( + &expanded, + &[], + remaining, + metrics, + Some(&fixed), + Some(&projection), + )?; + proposal + .map(|mut proposal| { + let selected = &mut proposal.selected; + selected.selected_candidates = selected + .selected_candidates + .iter() + .map(|id| projection.origins[id.0]) + .collect(); + selected.selected_candidates.sort(); + if selected + .selected_candidates + .windows(2) + .any(|pair| pair[0] == pair[1]) + { + return Err(invalid( + "multiple price options selected for one physical candidate", + )); + } + selected.connections = selected + .connections + .iter() + .map(|edge| projection.connection(*edge)) + .collect(); + selected.query_roots = selected + .query_roots + .iter() + .map(|root| projection.root(*root)) + .collect(); + selected.modeled_cost = selected + .selected_candidates + .iter() + .map(|id| graph.candidates[id.0].local_cost) + .sum(); + validate_selected_graph(graph, selected) + .map_err(CandidateSelectionError::InvalidGraph)?; + Ok(proposal) + }) + .transpose() +} + +#[cfg(test)] +mod tests { + use super::*; + use asap_logical_asap_optimization::memo::MemoGroupId; + + fn candidate(inputs: &[usize]) -> PhysicalCandidate { + PhysicalCandidate { + group: MemoGroupId(0), + local_cost: 0.0, + producer_requirements: if inputs.is_empty() { + vec![] + } else { + vec![ProducerRequirement { + compatible_producers: inputs + .iter() + .map(|id| PhysicalCandidateId(*id)) + .collect::>() + .into(), + }] + }, + } + } + fn option( + cost: f64, + materialized: bool, + producer: Option<(usize, usize)>, + ) -> CandidatePriceOption { + CandidatePriceOption { + profile: NodeCostProfile { + execution_cost: cost, + materialized, + ..Default::default() + }, + compatible_suppliers: producer + .into_iter() + .map(|(id, price_option)| SupplierPriceBinding { + producer: PhysicalCandidateId(id), + price_option, + }) + .collect(), + } + } + fn fixture() -> (CandidateDependencyGraph, JointPriceProfile) { + let graph = CandidateDependencyGraph { + candidates: vec![ + candidate(&[]), + candidate(&[]), + candidate(&[0, 1]), + candidate(&[2]), + candidate(&[2]), + ], + query_roots: vec![ + QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(3)], + }, + QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(4)], + }, + ], + ..Default::default() + }; + let profile = JointPriceProfile { + schedules: None, + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + candidates: vec![ + vec![option(1.0, false, None)], + vec![option(5.0, false, None)], + vec![ + option(0.0, true, Some((0, 0))), + option(0.0, true, Some((1, 0))), + ], + vec![ + option(1.0, false, Some((2, 0))), + option(100.0, false, Some((2, 1))), + ], + vec![ + option(100.0, false, Some((2, 0))), + option(1.0, false, Some((2, 1))), + ], + ], + }; + (graph, profile) + } + + // Readers of the same shared state must agree on its upstream price interface. + #[test] + fn shared_producer_has_one_joint_price_configuration() { + let (graph, profile) = fixture(); + let proposal = solve_joint_price_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert_eq!(proposal.direct_modeled_cost, Some(102.0)); + assert!(proposal + .selected + .selected_candidates + .contains(&PhysicalCandidateId(0))); + assert!(!proposal + .selected + .selected_candidates + .contains(&PhysicalCandidateId(1))); + assert_eq!( + proposal + .selected + .selected_candidates + .iter() + .filter(|id| id.0 == 2) + .count(), + 1 + ); + } + + // Rejecting an original connection assignment excludes all internal encodings. + #[test] + fn projected_rejection_selects_the_other_supplier() { + let (graph, profile) = fixture(); + let first = solve_joint_price_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + let observations = vec![ObservedAssignment { + selected: first.selected, + exact_cost: None, + rejected_queries: vec![], + }]; + let second = solve_joint_price_proposal_with_metrics( + &graph, + &observations, + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert_eq!(second.direct_modeled_cost, Some(106.0)); + assert!(second + .selected + .selected_candidates + .contains(&PhysicalCandidateId(1))); + } + // Resource and reader rules still refer to the physical candidate identity. + #[test] + fn original_capacity_and_reader_constraints_survive_option_expansion() { + let (mut graph, profile) = fixture(); + graph.resource_budgets.push(ResourceBudget { + name: "exclude first source".into(), + capacity: 0.0, + candidate_usage: vec![1.0, 0.0, 0.0, 0.0, 0.0], + }); + graph + .reader_constraints + .push(ReaderConstraint::MinimumQueryReaders { + candidate: PhysicalCandidateId(2), + query_outputs: vec![vec![0], vec![1]], + minimum: 2, + }); + let proposal = solve_joint_price_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert_eq!(proposal.direct_modeled_cost, Some(106.0)); + } +} diff --git a/crates/plan-selection/src/search/milp/mod.rs b/crates/plan-selection/src/search/milp/mod.rs index d459feae3..d50e12d21 100644 --- a/crates/plan-selection/src/search/milp/mod.rs +++ b/crates/plan-selection/src/search/milp/mod.rs @@ -2,3 +2,4 @@ pub(crate) mod model; pub(crate) mod direct_cost; +pub(crate) mod direct_price_options; diff --git a/crates/plan-selection/src/search/milp/model.rs b/crates/plan-selection/src/search/milp/model.rs index b4fe8f20d..f3d78fde8 100644 --- a/crates/plan-selection/src/search/milp/model.rs +++ b/crates/plan-selection/src/search/milp/model.rs @@ -171,6 +171,8 @@ pub(crate) struct CandidateSolveMetrics { pub solver_seconds: f64, pub solve_calls: usize, pub direct_cost_solve_calls: usize, + /// Direct solves jointly selecting supplier-dependent pricing options. + pub joint_price_solve_calls: usize, pub direct_cost_validated_plans: usize, pub direct_cost_best_validated_cost: Option, pub direct_cost_fallback_reasons: Vec, @@ -202,7 +204,7 @@ pub(crate) fn solve_candidate_proposal_with_metrics( remaining_seconds: Option, metrics: &mut CandidateSolveMetrics, ) -> Result, CandidateSelectionError> { - solve_with_objective(graph, observations, remaining_seconds, metrics, None) + solve_with_objective(graph, observations, remaining_seconds, metrics, None, None) } pub(crate) fn solve_direct_cost_proposal_with_metrics( @@ -218,15 +220,46 @@ pub(crate) fn solve_direct_cost_proposal_with_metrics( remaining_seconds, metrics, Some(profile), + None, ) } -fn solve_with_objective( +pub(super) fn validate_observations( + graph: &CandidateDependencyGraph, + observations: &[ObservedAssignment], +) -> Result<(), CandidateSelectionError> { + for observation in observations { + validate_selected_graph(graph, &observation.selected) + .map_err(CandidateSelectionError::InvalidGraph)?; + for witness in &observation.rejected_queries { + witness + .validate(&observation.selected) + .map_err(CandidateSelectionError::InvalidGraph)?; + } + if observation.exact_cost.is_some() && !observation.rejected_queries.is_empty() { + return Err(CandidateSelectionError::InvalidGraph( + "feasible observation cannot reject a query".into(), + )); + } + if observation + .exact_cost + .is_some_and(|cost| !cost.is_finite() || cost < observation.selected.modeled_cost) + { + return Err(CandidateSelectionError::InvalidGraph( + "observed cost must be finite and at least its certified local bound".into(), + )); + } + } + Ok(()) +} + +pub(super) fn solve_with_objective( graph: &CandidateDependencyGraph, observations: &[ObservedAssignment], remaining_seconds: Option, metrics: &mut CandidateSolveMetrics, direct_profile: Option<&super::direct_cost::DirectCostProfile>, + projection: Option<&super::direct_price_options::PriceOptionProjection<'_>>, ) -> Result, CandidateSelectionError> { let started = std::time::Instant::now(); graph.validate()?; @@ -247,28 +280,7 @@ fn solve_with_objective( Ok(()) } }; - for observation in observations { - validate_selected_graph(graph, &observation.selected) - .map_err(CandidateSelectionError::InvalidGraph)?; - for witness in &observation.rejected_queries { - witness - .validate(&observation.selected) - .map_err(CandidateSelectionError::InvalidGraph)?; - } - if observation.exact_cost.is_some() && !observation.rejected_queries.is_empty() { - return Err(CandidateSelectionError::InvalidGraph( - "feasible observation cannot reject a query".into(), - )); - } - if observation - .exact_cost - .is_some_and(|cost| !cost.is_finite() || cost < observation.selected.modeled_cost) - { - return Err(CandidateSelectionError::InvalidGraph( - "observed cost must be finite and at least its certified local bound".into(), - )); - } - } + validate_observations(graph, observations)?; if graph.query_roots.is_empty() { if observations .iter() @@ -332,6 +344,9 @@ fn solve_with_objective( let mut root_assignments = Vec::new(); let mut constraints = Vec::new(); let mut support = vec![Expression::from(0.0); count]; + if let Some(projection) = projection { + projection.constrain_activation(&candidate_selected, &mut constraints); + } for (consumer, candidate) in graph.candidates.iter().enumerate() { check_deadline()?; @@ -796,6 +811,14 @@ fn solve_with_objective( None => constraints.push(constraint!(mismatch >= 1.0)), } } + if let Some(projection) = projection { + projection.constrain_observations( + &candidate_selected, + &connections, + &root_assignments, + &mut constraints, + ); + } if warm_start.is_some() && variables.initial_solution_len() != variables.len() { return Err(CandidateSelectionError::Solver( "incomplete candidate warm start".into(), diff --git a/crates/plan-selection/src/search/refinement/components.rs b/crates/plan-selection/src/search/refinement/components.rs index d95fe8172..704de97af 100644 --- a/crates/plan-selection/src/search/refinement/components.rs +++ b/crates/plan-selection/src/search/refinement/components.rs @@ -453,6 +453,7 @@ pub(in crate::search) fn optimize_independent_components( summary.solve_metrics.solver_seconds += metrics.solver_seconds; summary.solve_metrics.solve_calls += metrics.solve_calls; summary.solve_metrics.direct_cost_solve_calls += metrics.direct_cost_solve_calls; + summary.solve_metrics.joint_price_solve_calls += metrics.joint_price_solve_calls; summary.solve_metrics.direct_cost_validated_plans += metrics.direct_cost_validated_plans; // Component proposal costs are not complete workload costs; do not // expose their minimum as the price of a validated whole-workload plan. diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index 6e02d228b..7d365e2b1 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -14,7 +14,7 @@ use crate::search::candidates::physical::{ }; use crate::search::refinement::search::{ search_candidates_with_direct_feedback, CandidateEvaluation, CandidateSearchLimits, - CandidateSearchResult, + CandidateSearchResult, DirectPriceModel, }; use crate::{CalibrationFit, CostModel, PlanSelector, PlanningModels, Selection, SelectionError}; @@ -122,33 +122,40 @@ pub(in crate::search) fn optimize_configured_candidates( } else { Vec::new() }; + let mut joint_fallback_reasons = Vec::new(); + let joint_context = CandidatePlanningContext { + deadline: earliest_deadline( + context.deadline, + std::time::Instant::now() + .checked_add((limits.wall_time / 6).min(std::time::Duration::from_secs(2))), + ), + ..*context + }; + let joint = joint_cost_seed( + configured, + &joint_context, + &exclusions, + &mut joint_fallback_reasons, + )?; let mut direct = Vec::new(); - if let Some(seed) = direct_cost_seed(configured, context, &exclusions, false)? { - direct.push(seed); - } - if configured.configurations.iter().any(|configuration| { - matches!( - configuration.materialization, - asap_types::physical::Materialization::MatAtQueryKeptAcrossEvaluations(_) - ) - }) || configured.inventory.templates.iter().flatten().any(|node| { - matches!( - node.non_asap(), - Some(asap_types::ir::NonASAPOp::TimeRange { - kind: asap_types::ir::TimeRangeKind::Instant, - .. - }) - ) - }) { + // Keep the older restricted profile only as a bounded fallback when the + // joint interface catalog cannot be certified within the planning budget. + if joint.is_none() { + if let Some(seed) = direct_cost_seed(configured, context, &exclusions, false)? { + direct.push(seed); + } if let Some(seed) = direct_cost_seed(configured, context, &exclusions, true)? { direct.push(seed); } } - let direct_refs: Vec<_> = direct + let mut direct_refs: Vec<_> = direct .iter() - .map(|(graph, profile)| (graph, profile)) + .map(|(graph, profile)| (graph, DirectPriceModel::Fixed(profile))) .collect(); - let search = search_candidates_with_direct_feedback( + if let Some((graph, profile)) = &joint { + direct_refs.push((graph, DirectPriceModel::Joint(profile))); + } + let mut search = search_candidates_with_direct_feedback( &configured.inventory.graph, CandidateSearchLimits { wall_time: limits.wall_time.saturating_sub(weighting_started.elapsed()), @@ -196,6 +203,10 @@ pub(in crate::search) fn optimize_configured_candidates( Ok(Some(cost).into()) }, )?; + search + .solve_metrics + .direct_cost_fallback_reasons + .extend(joint_fallback_reasons); match (&search.incumbent, &best) { (Some(incumbent), Some((cost, selected, ..))) if incumbent.actual_cost == *cost && incumbent.selected == *selected => {} @@ -210,6 +221,61 @@ pub(in crate::search) fn optimize_configured_candidates( }) } +/// Every supported local price interface participates in one joint solve. +fn joint_cost_seed( + configured: &CandidateConfigurationInventory, + context: &CandidatePlanningContext<'_>, + exclusions: &[crate::search::candidates::graph::PhysicalCandidateId], + fallback_reasons: &mut Vec, +) -> Result< + Option<( + crate::search::candidates::graph::CandidateDependencyGraph, + crate::search::milp::direct_price_options::JointPriceProfile, + )>, + String, +> { + use crate::search::costing::direct_profile::{extract_price_options, DirectProfileError}; + let mut profile = match extract_price_options(configured, context) { + Ok(Some(profile)) => profile, + Ok(None) => return Ok(None), + Err(DirectProfileError::DeadlineExceeded(_)) => { + fallback_reasons + .push("joint pricing: interface generation exhausted its reserved budget".into()); + return Ok(None); + } + Err(DirectProfileError::Unsupported(reason)) => { + fallback_reasons.push(format!("joint pricing: {reason}")); + return Ok(None); + } + Err(error) => return Err(error.to_string()), + }; + // As with the fixed-price seed, avoid permutations of interchangeable + // execution copies in this bounded proposal. Every supplier price option + // remains available on the first copy; the full search retains all copies. + for constraint in &configured.inventory.graph.reader_constraints { + if let crate::search::candidates::graph::ReaderConstraint::ActivationRequires { + candidate, + .. + } = constraint + { + profile.candidates[candidate.0].clear(); + } + } + for excluded in exclusions { + profile.candidates[excluded.0].clear(); + } + // Remove bindings to excluded options without renumbering other options. + let sizes: Vec<_> = profile.candidates.iter().map(Vec::len).collect(); + for options in &mut profile.candidates { + for option in options { + option + .compatible_suppliers + .retain(|binding| binding.price_option < sizes[binding.producer.0]); + } + } + Ok(Some((configured.inventory.graph.clone(), profile))) +} + /// Restrict only the direct-price proposal. Unsupported configurations and /// supplier interfaces remain available in the full evaluator-refined search. fn direct_cost_seed( diff --git a/crates/plan-selection/src/search/refinement/search.rs b/crates/plan-selection/src/search/refinement/search.rs index ce24c36c6..f1ab2c5d7 100644 --- a/crates/plan-selection/src/search/refinement/search.rs +++ b/crates/plan-selection/src/search/refinement/search.rs @@ -3,6 +3,15 @@ //! failures; full-assignment cuts remain a correctness fallback; //! they make no claim of scalable demand modeling or support for every cost model. use crate::search::milp::direct_cost::DirectCostProfile; +use crate::search::milp::direct_price_options::{ + solve_joint_price_proposal_with_metrics, JointPriceProfile, +}; + +#[derive(Clone, Copy)] +pub(in crate::search) enum DirectPriceModel<'a> { + Fixed(&'a DirectCostProfile), + Joint(&'a JointPriceProfile), +} use std::time::{Duration, Instant}; use crate::search::candidates::graph::{ @@ -77,7 +86,7 @@ pub(in crate::search) struct CandidateSearchResult { struct SeedRestrictions<'a> { excluded: &'a [PhysicalCandidateId], materialization_changes: &'a [(PhysicalCandidateId, PhysicalCandidateId)], - direct: &'a [(&'a CandidateDependencyGraph, &'a DirectCostProfile)], + direct: &'a [(&'a CandidateDependencyGraph, DirectPriceModel<'a>)], } /// The callback returns an actual total cost for a validated feasible plan, or @@ -171,7 +180,7 @@ pub(in crate::search) fn search_candidates_with_direct_feedback( weights: &[f64], excluded_from_seed: &[PhysicalCandidateId], materialization_changes: &[(PhysicalCandidateId, PhysicalCandidateId)], - direct: &[(&CandidateDependencyGraph, &DirectCostProfile)], + direct: &[(&CandidateDependencyGraph, DirectPriceModel<'_>)], evaluate: impl FnMut(&SelectedCandidateGraph) -> Result, ) -> Result { search_with_structural_baseline( @@ -452,13 +461,22 @@ fn search_with_solver( continue; } let proposed = if use_direct { - solve_direct_cost_proposal_with_metrics( - proposal_graph, - &proposal_observations, - Some(solve_budget.as_secs_f64()), - &mut solve_metrics, - restrictions.direct[direct_index].1, - ) + match restrictions.direct[direct_index].1 { + DirectPriceModel::Fixed(profile) => solve_direct_cost_proposal_with_metrics( + proposal_graph, + &proposal_observations, + Some(solve_budget.as_secs_f64()), + &mut solve_metrics, + profile, + ), + DirectPriceModel::Joint(profile) => solve_joint_price_proposal_with_metrics( + proposal_graph, + &proposal_observations, + Some(solve_budget.as_secs_f64()), + &mut solve_metrics, + profile, + ), + } } else { propose( proposal_graph, @@ -737,7 +755,7 @@ mod tests { &[1.0, 1.0], &[], &[], - &[(&restricted, &profile)], + &[(&restricted, DirectPriceModel::Fixed(&profile))], |selected| Ok(Some(profile.selected_cost(&restricted, selected)).into()), ) .unwrap(); @@ -778,7 +796,10 @@ mod tests { &[1.0, 1.0], &[], &[], - &[(&first, &profile), (&second, &profile)], + &[ + (&first, DirectPriceModel::Fixed(&profile)), + (&second, DirectPriceModel::Fixed(&profile)), + ], |selected| Ok(Some(profile.selected_cost(&graph, selected)).into()), ) .unwrap(); diff --git a/crates/plan-selection/src/search/route.rs b/crates/plan-selection/src/search/route.rs index 06b2ca0ab..ae7e14554 100644 --- a/crates/plan-selection/src/search/route.rs +++ b/crates/plan-selection/src/search/route.rs @@ -596,6 +596,7 @@ fn add_phase_metrics( } report.solve_calls += phase.solve_calls; report.direct_cost_solve_calls += phase.direct_cost_solve_calls; + report.joint_price_solve_calls += phase.joint_price_solve_calls; report.direct_cost_validated_plans += phase.direct_cost_validated_plans; if let Some(cost) = phase.direct_cost_best_validated_cost { report.direct_cost_best_validated_cost = Some( @@ -1068,6 +1069,7 @@ fn search_domain( validation_ms: result.validation_seconds * 1000.0, solve_calls: result.metrics.solve_calls, direct_cost_solve_calls: result.metrics.direct_cost_solve_calls, + joint_price_solve_calls: result.metrics.joint_price_solve_calls, direct_cost_validated_plans: result.metrics.direct_cost_validated_plans, direct_cost_best_validated_cost: result.metrics.direct_cost_best_validated_cost, direct_cost_fallback_reasons: result.metrics.direct_cost_fallback_reasons.clone(), diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 850e13aed..9de9784c2 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -204,7 +204,13 @@ The direct-cost objective computes the total modeled cost of the selected execut Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Whole-result targets therefore add conservatively sized capacity candidates before physical costing, preserving the original candidates. Matching schedules use a compact demand model. Larger sets of fixed-interval schedules use phase activation and bounded integer rate totals, avoiding enumeration of every reader subset; rates follow the evaluator's accounting policy, including unknown phases. The normalized rate total is bounded at 4,096; small heterogeneous workloads can also use evaluator-derived subset tables. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. -Direct pane pricing covers query-time rebuilding, eligible state retained across evaluations, and fixed-retention ingestion maintenance when every admitted unary supplier binding certifies the same ingest-worker count. Source-certified HLL panes use the same population and failure-probability checks as HLL whole-window builds. Rebuilt state pays for its execution demand and temporary lifetime; retained query-time state pays for resident panes and required refreshes. Ingestion updates run at ingestion demand, resident state is charged once, and ingestion update work is excluded from query latency. Ordinary full-window scans and latest-pane refresh scans use separate restricted pricing classes: a consumer can connect only to suppliers with the matching scan context. These classes share the planning budget and preserve the best validated plan; they do not enumerate every mixed-context DAG. For multiple ingest workers, the model includes the maximum compaction delay among the ingestion states reached by each query; this affects query-time state holding cost. Worker counts crossing a configuration boundary remain unsupported in that pricing class. +Direct pane pricing covers query-time rebuilding, eligible state retained across evaluations, and fixed-retention ingestion maintenance. Source-certified HLL panes use the same population and failure-probability checks as HLL whole-window builds. Rebuilt state pays for execution demand and temporary lifetime; retained state pays for resident panes and refreshes. Ingestion work is excluded from query latency; the maximum compaction delay among reached ingestion states contributes to query latency and temporary holding cost. + +**Supplier-dependent prices are joint decisions.** For each supported unary candidate, costing derives finite local pricing options from its compatible suppliers' priced output interfaces. An option records local CPU/state costs, output statistics, upstream work and the resulting certified worker count. For example, two supplier configurations requiring 0.9 and 2.4 CPU-seconds per ingestion second produce one-worker and three-worker options; their compaction delays need not agree. A downstream reader can likewise have different merge/extraction costs for different state capacities. The optimizer retains these options rather than keeping only the first price class. + +Let `price[c,k]` select pricing option `k` of candidate `c`. Require `Σ_k price[c,k] = selected[c]`, and permit a selected consumer/producer connection only with a compatible pair of pricing options. Thus all consumers of one shared candidate observe the **same** chosen interface. The implementation lifts these options into an internal MILP graph with mutual exclusion per original candidate, reuses the demand/lifetime cost formulation, and projects the result back to the original candidate DAG. Pricing options are not extra executions or extra copies of a shared state. Rejected assignments and reader/resource constraints retain their original physical identities. + +Local interface generation deduplicates identical outputs, costs and upstream-work certificates; it does not enumerate complete workload DAGs. It currently covers the supported unary operators. Acyclic catalogs use dependency order; catalogs with potential cycles propagate local interfaces for at most the candidate count, sufficient for every legal unary path. Query-time upstream work is not part of an ingestion worker certificate, preventing irrelevant path histories from multiplying equivalent interfaces. Final DAG ordering and one pricing option per physical candidate exclude cyclic executions. All legal scan-lookback classes participate jointly, with exact context matching on each scan connection; sharing one wider scan across narrower consumers requires an additional maximum-demand witness and remains outside this encoding. Multi-input pricing or domains exceeding the finite interface/time budget use the existing validated fallback. Generation reserves time for fallback rather than consuming the entire planning budget. A worker count is derived from the selected upstream work, not an independently tunable cluster size. The bounded production seed uses the first interchangeable execution copy while retaining all its pricing options; the full search retains every copy. This is a restricted proposal, not a proof that additional copies are unnecessary. Reports distinguish joint price-option solves from fixed-profile direct solves. Matching range requests on the same fixed schedule are directly priced for rebuilt and ingestion-maintained state: evaluation frequency includes every range step, request overhead remains per request, and maintained panes retain the required historical span. Query-kept range state, differing range contexts, uncertified supplier interfaces, and schedules outside the bounded encodings still require evaluator refinement. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. From 3b8243f6262b407edb67f80e90e173968a5776e2 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 06:43:40 +0000 Subject: [PATCH 21/29] fix(planner): archive only completed candidate evaluations --- .../src/search/refinement/objectives.rs | 36 +++++++++++++++---- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/crates/plan-selection/src/search/refinement/objectives.rs b/crates/plan-selection/src/search/refinement/objectives.rs index 8fe3e7b80..997ba2256 100644 --- a/crates/plan-selection/src/search/refinement/objectives.rs +++ b/crates/plan-selection/src/search/refinement/objectives.rs @@ -1,7 +1,7 @@ //! Exact evaluator ranking over a bounded traversal of the candidate graph. //! Traversal priorities are not an objective scalarization: PlanSelector owns //! cluster lexicographic priorities, Pareto dominance, constraints and tie policy. -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::time::Instant; use asap_physical_asap_optimization::PhysicalWorkloadPlan; @@ -135,6 +135,7 @@ pub(in crate::search) fn optimize_objective_candidates_with_options( let mut observations = Vec::new(); let mut physical_catalog = Vec::new(); let mut evaluated_plans = Vec::new(); + let mut completed_offers = BTreeSet::new(); let mut stage2_rejected_assignments = Vec::new(); let mut selected_by_id = BTreeMap::new(); let mut metrics = CandidateSolveMetrics::default(); @@ -224,9 +225,11 @@ pub(in crate::search) fn optimize_objective_candidates_with_options( Err(error) => Err(error), } } - CandidateCalibration::Frozen(_) => { - selector.offer_before(&physical, context.deadline) - } + CandidateCalibration::Frozen(_) => selector + .offer_before(&physical, context.deadline) + .map(|()| { + completed_offers.insert(physical.id.clone()); + }), } } Ok(None) => Ok(()), @@ -284,7 +287,9 @@ pub(in crate::search) fn optimize_objective_candidates_with_options( break; } match selector.offer_before(physical, context.deadline) { - Ok(()) => {} + Ok(()) => { + completed_offers.insert(physical.id.clone()); + } Err(SelectionError::EvaluationDeadlineExceeded) => { status = CandidateSearchStatus::TimeLimit; evaluation_complete = false; @@ -315,6 +320,10 @@ pub(in crate::search) fn optimize_objective_candidates_with_options( } CandidateCalibration::Frozen(_) => "not_required", }; + // Reconstruction and calibration do not complete evaluation. In particular, + // a compiler deadline must not leave an archived plan with neither a cost + // nor a rejection. Keep every completed offer, including rejected plans. + evaluated_plans.retain(|(_, physical)| completed_offers.contains(&physical.id)); let mut evaluation_rejections = Vec::new(); let best = match selector.finish() { Ok(best) => { @@ -896,17 +905,25 @@ mod tests { models, calibration: CalibrationFit::default(), }; - let result = optimize_objective_candidates( + let result = optimize_objective_candidates_with_options( &configured, &context, limits(10), CandidateCalibration::Frozen(CalibrationFit::default()), + ObjectiveSearchOptions { + keep_every_candidate: true, + exhaustive: true, + }, ) .unwrap(); assert_eq!(result.status, CandidateSearchStatus::TimeLimit); assert_eq!(result.refinements, 1); assert!(!result.catalog_exhausted); assert!(result.best.is_none()); + assert!( + result.evaluated_plans.is_empty(), + "interrupted offers are not evaluated archive entries" + ); assert!(result.evaluation_rejections.is_empty()); assert!(result.rejections.counts.is_empty()); assert_eq!(result.certified_query_cuts, 0); @@ -942,7 +959,7 @@ mod tests { models, calibration: CalibrationFit::default(), }; - let result = optimize_objective_candidates( + let result = optimize_objective_candidates_with_options( &configured, &context, CandidateSearchLimits { @@ -950,6 +967,10 @@ mod tests { wall_time: Duration::from_secs(2), }, CandidateCalibration::CompleteCatalog, + ObjectiveSearchOptions { + keep_every_candidate: true, + exhaustive: true, + }, ) .unwrap(); assert!(result.catalog_exhausted); @@ -958,6 +979,7 @@ mod tests { assert_eq!(result.calibration_scope, "complete_catalog"); assert!(result.calibration.factor() > 1.0); assert!(result.best.is_none()); + assert!(result.evaluated_plans.is_empty()); assert!(result.evaluation_rejections.is_empty()); assert_eq!(check.0.get(), 1); } From 8c64d6d55aab36537dec9bb84e22b461b97cecc9 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 06:43:40 +0000 Subject: [PATCH 22/29] fix(planner): archive only completed candidate evaluations --- .../src/search/refinement/objectives.rs | 36 +++++++++++++++---- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/crates/plan-selection/src/search/refinement/objectives.rs b/crates/plan-selection/src/search/refinement/objectives.rs index 8fe3e7b80..997ba2256 100644 --- a/crates/plan-selection/src/search/refinement/objectives.rs +++ b/crates/plan-selection/src/search/refinement/objectives.rs @@ -1,7 +1,7 @@ //! Exact evaluator ranking over a bounded traversal of the candidate graph. //! Traversal priorities are not an objective scalarization: PlanSelector owns //! cluster lexicographic priorities, Pareto dominance, constraints and tie policy. -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::time::Instant; use asap_physical_asap_optimization::PhysicalWorkloadPlan; @@ -135,6 +135,7 @@ pub(in crate::search) fn optimize_objective_candidates_with_options( let mut observations = Vec::new(); let mut physical_catalog = Vec::new(); let mut evaluated_plans = Vec::new(); + let mut completed_offers = BTreeSet::new(); let mut stage2_rejected_assignments = Vec::new(); let mut selected_by_id = BTreeMap::new(); let mut metrics = CandidateSolveMetrics::default(); @@ -224,9 +225,11 @@ pub(in crate::search) fn optimize_objective_candidates_with_options( Err(error) => Err(error), } } - CandidateCalibration::Frozen(_) => { - selector.offer_before(&physical, context.deadline) - } + CandidateCalibration::Frozen(_) => selector + .offer_before(&physical, context.deadline) + .map(|()| { + completed_offers.insert(physical.id.clone()); + }), } } Ok(None) => Ok(()), @@ -284,7 +287,9 @@ pub(in crate::search) fn optimize_objective_candidates_with_options( break; } match selector.offer_before(physical, context.deadline) { - Ok(()) => {} + Ok(()) => { + completed_offers.insert(physical.id.clone()); + } Err(SelectionError::EvaluationDeadlineExceeded) => { status = CandidateSearchStatus::TimeLimit; evaluation_complete = false; @@ -315,6 +320,10 @@ pub(in crate::search) fn optimize_objective_candidates_with_options( } CandidateCalibration::Frozen(_) => "not_required", }; + // Reconstruction and calibration do not complete evaluation. In particular, + // a compiler deadline must not leave an archived plan with neither a cost + // nor a rejection. Keep every completed offer, including rejected plans. + evaluated_plans.retain(|(_, physical)| completed_offers.contains(&physical.id)); let mut evaluation_rejections = Vec::new(); let best = match selector.finish() { Ok(best) => { @@ -896,17 +905,25 @@ mod tests { models, calibration: CalibrationFit::default(), }; - let result = optimize_objective_candidates( + let result = optimize_objective_candidates_with_options( &configured, &context, limits(10), CandidateCalibration::Frozen(CalibrationFit::default()), + ObjectiveSearchOptions { + keep_every_candidate: true, + exhaustive: true, + }, ) .unwrap(); assert_eq!(result.status, CandidateSearchStatus::TimeLimit); assert_eq!(result.refinements, 1); assert!(!result.catalog_exhausted); assert!(result.best.is_none()); + assert!( + result.evaluated_plans.is_empty(), + "interrupted offers are not evaluated archive entries" + ); assert!(result.evaluation_rejections.is_empty()); assert!(result.rejections.counts.is_empty()); assert_eq!(result.certified_query_cuts, 0); @@ -942,7 +959,7 @@ mod tests { models, calibration: CalibrationFit::default(), }; - let result = optimize_objective_candidates( + let result = optimize_objective_candidates_with_options( &configured, &context, CandidateSearchLimits { @@ -950,6 +967,10 @@ mod tests { wall_time: Duration::from_secs(2), }, CandidateCalibration::CompleteCatalog, + ObjectiveSearchOptions { + keep_every_candidate: true, + exhaustive: true, + }, ) .unwrap(); assert!(result.catalog_exhausted); @@ -958,6 +979,7 @@ mod tests { assert_eq!(result.calibration_scope, "complete_catalog"); assert!(result.calibration.factor() > 1.0); assert!(result.best.is_none()); + assert!(result.evaluated_plans.is_empty()); assert!(result.evaluation_rejections.is_empty()); assert_eq!(check.0.get(), 1); } From e597a48001bcd2a2ce3664010ca3952352063e11 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 06:46:27 +0000 Subject: [PATCH 23/29] test(executor): use explicit frequency operator candidates --- .../tests/frequency_execution.rs | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/crates/integration-tests/tests/frequency_execution.rs b/crates/integration-tests/tests/frequency_execution.rs index a6f9071c0..f69a1681b 100644 --- a/crates/integration-tests/tests/frequency_execution.rs +++ b/crates/integration-tests/tests/frequency_execution.rs @@ -12,7 +12,7 @@ use asap_executor::physical_planner::{ use asap_executor::runtime::Scope; use asap_executor::values::{Batch, Value}; use asap_integration_tests::fixtures::lower_promql; -use asap_integration_tests::planning::logical_candidates; +use asap_integration_tests::planning::explicit_local_candidates; use asap_types::ir::physical_export::{compile_physical_asap_dag, PhysicalASAPOperatorPayload}; use asap_types::ir::properties::AccuracyTarget; use asap_types::ir::schema::{FieldDataType, SketchAlgorithm}; @@ -128,7 +128,9 @@ fn standin_exact_and_univmon_candidates_execute() { let root = lower_promql(query, accuracy.clone()).unwrap(); let root = promql_rows::with_series_identity(&root).unwrap(); let mut executed = Vec::new(); - for candidate in logical_candidates(root, accuracy) { + // This is an operator execution test. Enumerate each local rule once, + // not an arbitrary sized configuration visited by a budgeted search. + for candidate in explicit_local_candidates(root) { let family = built_family(&candidate); let name = match &family { None => "exact".to_string(), @@ -139,6 +141,18 @@ fn standin_exact_and_univmon_candidates_execute() { } _ => continue, }; + if query.starts_with("distinct_over_time") { + if let Some(FieldDataType::Sketch(kind, _)) = &family { + let asap_types::ir::schema::SketchParams::UnivMon { + distinct_failure_probability, + .. + } = kind.params() + else { + unreachable!("family filtered above") + }; + assert_eq!(*distinct_failure_probability, Some(delta)); + } + } let values = execute(&candidate); assert!( matches!(values.as_slice(), [value] if (value - expected).abs() < 1e-9), From cb01c460b5e95f154bd67624bb04743a6f098507 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 06:46:27 +0000 Subject: [PATCH 24/29] test(executor): use explicit frequency operator candidates --- .../tests/frequency_execution.rs | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/crates/integration-tests/tests/frequency_execution.rs b/crates/integration-tests/tests/frequency_execution.rs index a6f9071c0..f69a1681b 100644 --- a/crates/integration-tests/tests/frequency_execution.rs +++ b/crates/integration-tests/tests/frequency_execution.rs @@ -12,7 +12,7 @@ use asap_executor::physical_planner::{ use asap_executor::runtime::Scope; use asap_executor::values::{Batch, Value}; use asap_integration_tests::fixtures::lower_promql; -use asap_integration_tests::planning::logical_candidates; +use asap_integration_tests::planning::explicit_local_candidates; use asap_types::ir::physical_export::{compile_physical_asap_dag, PhysicalASAPOperatorPayload}; use asap_types::ir::properties::AccuracyTarget; use asap_types::ir::schema::{FieldDataType, SketchAlgorithm}; @@ -128,7 +128,9 @@ fn standin_exact_and_univmon_candidates_execute() { let root = lower_promql(query, accuracy.clone()).unwrap(); let root = promql_rows::with_series_identity(&root).unwrap(); let mut executed = Vec::new(); - for candidate in logical_candidates(root, accuracy) { + // This is an operator execution test. Enumerate each local rule once, + // not an arbitrary sized configuration visited by a budgeted search. + for candidate in explicit_local_candidates(root) { let family = built_family(&candidate); let name = match &family { None => "exact".to_string(), @@ -139,6 +141,18 @@ fn standin_exact_and_univmon_candidates_execute() { } _ => continue, }; + if query.starts_with("distinct_over_time") { + if let Some(FieldDataType::Sketch(kind, _)) = &family { + let asap_types::ir::schema::SketchParams::UnivMon { + distinct_failure_probability, + .. + } = kind.params() + else { + unreachable!("family filtered above") + }; + assert_eq!(*distinct_failure_probability, Some(delta)); + } + } let values = execute(&candidate); assert!( matches!(values.as_slice(), [value] if (value - expected).abs() < 1e-9), From b22ef89cbd0e264d6edb94fd03c5481ce666118b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 17:30:36 +0000 Subject: [PATCH 25/29] fix(planner): exclude unresolved confidence from feasibility seeds --- .../plan-selection/src/search/refinement/planner.rs | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index 6e02d228b..e206c71e7 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -419,6 +419,10 @@ fn accuracy_baseline( if output_queries.len() != graph.query_roots.len() { return Err("baseline output references do not match original query roots".into()); } + let confidence_required = context + .entries + .iter() + .any(|entry| entry.requirements.accuracy.failure_probability.is_some()); let dds_budgets = seed_dds_budgets(configured, context, &output_queries); let mut rank_requested = vec![false; graph.candidates.len()]; let mut pending: Vec<_> = graph @@ -563,7 +567,12 @@ fn accuracy_baseline( crate::series_read(&root, context.data), context.data, &context.models, - ).is_none_or(|guarantee| guarantee.spec.error.evaluate().is_none()) + ).is_none_or(|guarantee| { + guarantee.spec.error.evaluate().is_none() + || (confidence_required + && !guarantee.is_exact() + && guarantee.spec.failure_probability.evaluate().is_none()) + }) }; let contract_risk = !context.models.plan_selection.ignore_accuracy_guarantees && (unresolved_estimator_error || node.guarantee.as_ref().is_some_and(|guarantee| { From 6a1da3587b623753baefbfc932403f16976ab447 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 18:13:22 +0000 Subject: [PATCH 26/29] fix(planner): continue past unpriceable candidates without false certificates --- .../src/search/refinement/planner.rs | 14 ++++ .../src/search/refinement/search.rs | 72 ++++++++++++++++++- .../proposals/joint-planning-search.md | 2 + 3 files changed, 87 insertions(+), 1 deletion(-) diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index e206c71e7..0ccb878d1 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -177,12 +177,26 @@ pub(in crate::search) fn optimize_configured_candidates( ..Default::default() }); } + Err( + error @ SelectionError::Cost { + error: crate::AnalyticalCostError::Overflow, + .. + }, + ) => { + let reason = error.to_string(); + rejections.record("cost_not_representable".into(), &reason); + return Ok(CandidateEvaluation { + unpriced_reason: Some(reason), + ..Default::default() + }); + } Err(error) => return Err(error.to_string()), }) else { return Ok(CandidateEvaluation { actual_cost: None, rejected_queries, budget_exhausted: false, + unpriced_reason: None, }); }; let cost = selection diff --git a/crates/plan-selection/src/search/refinement/search.rs b/crates/plan-selection/src/search/refinement/search.rs index ce24c36c6..817eac5f3 100644 --- a/crates/plan-selection/src/search/refinement/search.rs +++ b/crates/plan-selection/src/search/refinement/search.rs @@ -17,6 +17,8 @@ use crate::search::milp::model::{ #[derive(Debug, Default)] pub(in crate::search) struct CandidateEvaluation { pub actual_cost: Option, + /// Temporarily skip an unpriceable assignment without proving it infeasible. + pub unpriced_reason: Option, /// A stopped evaluator must not emit any rejection/cost cut. pub budget_exhausted: bool, pub rejected_queries: Vec, @@ -27,6 +29,7 @@ impl From> for CandidateEvaluation { Self { actual_cost, budget_exhausted: false, + unpriced_reason: None, rejected_queries: Vec::new(), } } @@ -81,7 +84,8 @@ struct SeedRestrictions<'a> { } /// The callback returns an actual total cost for a validated feasible plan, or -/// `None` for a proven infeasible assignment. Errors stop search, preserving the +/// `None` for a proven infeasible assignment. Explicit unpriced feedback skips +/// one assignment and invalidates full-domain certificates. Errors stop search, preserving the /// incumbent. Its cost must include every configured cost component and must be /// deterministic for an assignment. Local graph costs are certified lower bounds, /// not estimates. A violated certification is an error, never silently repaired. @@ -224,6 +228,7 @@ fn search_with_solver( graph.validate().map_err(|error| error.to_string())?; let started = Instant::now(); let mut observations = Vec::new(); + let mut unpriced_reason = None; let mut incumbent: Option = None; let mut lower_bound: f64 = 0.0; let mut refinements = 0; @@ -574,6 +579,14 @@ fn search_with_solver( if feedback.budget_exhausted { break CandidateSearchStatus::TimeLimit; } + if let Some(reason) = feedback.unpriced_reason { + if feedback.actual_cost.is_some() || !feedback.rejected_queries.is_empty() { + return Err( + "unpriceable feedback cannot contain a cost or certified query cut".into(), + ); + } + unpriced_reason.get_or_insert(reason); + } let actual_cost = feedback.actual_cost; if actual_cost.is_some() && !feedback.rejected_queries.is_empty() { return Err("feasible evaluator result cannot reject query subgraphs".into()); @@ -631,6 +644,21 @@ fn search_with_solver( break CandidateSearchStatus::Optimal; } }; + // Excluding an unpriceable assignment helps find a usable incumbent but + // does not prove infeasibility or a bound over the original candidate set. + let status = if let Some(reason) = unpriced_reason { + lower_bound = 0.0; + match status { + CandidateSearchStatus::Optimal | CandidateSearchStatus::Infeasible => { + CandidateSearchStatus::EvaluatorStopped(format!( + "unpriceable assignments excluded: {reason}" + )) + } + other => other, + } + } else { + status + }; let relative_gap = incumbent.as_ref().map(|best| { if best.actual_cost == 0.0 { 0.0 @@ -704,6 +732,47 @@ mod tests { } } + // An unpriceable candidate must not prevent a later validated plan, nor + // allow exclusions of unknown costs to certify a full-domain optimum. + #[test] + fn unpriceable_assignments_preserve_search_without_an_optimality_claim() { + for costs in [vec![1.0, 2.0], vec![1.0]] { + let graph = choices(&costs); + let result = search_candidates_with_direct_feedback( + &graph, + limits(8), + &costs, + &[], + &[], + &[], + |selected| { + if selected + .selected_candidates + .contains(&PhysicalCandidateId(0)) + { + Ok(CandidateEvaluation { + unpriced_reason: Some("analytical arithmetic overflowed".into()), + ..Default::default() + }) + } else { + Ok(Some(2.0).into()) + } + }, + ) + .unwrap(); + assert_eq!( + result.incumbent.as_ref().map(|best| best.actual_cost), + (costs.len() == 2).then_some(2.0) + ); + assert_eq!(result.lower_bound, 0.0); + assert!(matches!( + result.status, + CandidateSearchStatus::EvaluatorStopped(_) + )); + assert_eq!(result.relative_gap, (costs.len() == 2).then_some(1.0)); + } + } + // A restricted direct-cost seed supplies an incumbent, never a full-domain bound. #[test] fn direct_cost_seed_validates_price_without_claiming_global_optimum() { @@ -863,6 +932,7 @@ mod tests { bad_evaluations += 1; Ok(CandidateEvaluation { budget_exhausted: false, + unpriced_reason: None, actual_cost: None, rejected_queries: vec![RejectedQuerySubgraph::from_query_outputs( selected, diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 850e13aed..367f767ef 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -208,6 +208,8 @@ Direct pane pricing covers query-time rebuilding, eligible state retained across Matching range requests on the same fixed schedule are directly priced for rebuilt and ingestion-maintained state: evaluation frequency includes every range step, request overhead remains per request, and maintained panes retain the required historical span. Query-kept range state, differing range contexts, uncertified supplier interfaces, and schedules outside the bounded encodings still require evaluator refinement. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. +A candidate whose evaluator cost overflows the representable range is skipped with a `cost_not_representable` diagnostic so scalar-price search can still find a validated plan. This is an unknown cost, not proof of infeasibility: if such assignments were excluded, the search reports a conservative zero lower bound and cannot claim a full-domain optimum or infeasibility. Invalid settings and other evaluator errors remain errors. + Numerically ill-conditioned cost profiles, rate tables, or normalized latency terms also fall back explicitly rather than silently dropping charges. An optimal direct solve must additionally agree with an independently decoded selected-DAG cost before its restricted-domain certificate is accepted. Fixed query overhead can be omitted during minimization but must be restored in reported totals. From 180456660e7c6ddc7ab045c1038ed21f0b354437 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 18:41:14 +0000 Subject: [PATCH 27/29] test(planner): validate TopK contracts across visited capacities --- .../tests/promql_to_post_asap.rs | 94 ++++++++++--------- 1 file changed, 51 insertions(+), 43 deletions(-) diff --git a/crates/integration-tests/tests/promql_to_post_asap.rs b/crates/integration-tests/tests/promql_to_post_asap.rs index 5af7acd4f..821974e8d 100644 --- a/crates/integration-tests/tests/promql_to_post_asap.rs +++ b/crates/integration-tests/tests/promql_to_post_asap.rs @@ -255,54 +255,62 @@ fn planner_only_e2e_temporal_topk_preserves_query_update_and_evaluation_contract AccuracyTarget::max_error(0.01).failure_probability(0.01), ) .expect("lower temporal Top-K"); - let candidate = candidates_using( + let candidates = candidates_using( &pre, AccuracyTarget::max_error(0.01).failure_probability(0.01), expected_family.clone(), - ) - .into_iter() - .next() - .expect("heap-backed temporal Top-K candidate"); - let Some(ASAPOp::SummaryEstimate { - summary_input, - query: SketchStatistic::TopK { k, .. }, - }) = candidate.asap() - else { - panic!("expected Top-K estimate, got {:?}", candidate.operator) - }; - assert_eq!( - *k, 5, - "the requested Top-K cardinality must survive binding" ); - let Some(ASAPOp::SummaryAgg { - input: state_input, - family, - child, - .. - }) = summary_input.asap() - else { - panic!("expected structured Top-K state input") - }; - let FieldDataType::Sketch(kind, _) = family else { - panic!("expected a heap-backed sketch family, got {family:?}") - }; - assert_eq!(kind.algorithm(), &expected_family); - let heap_size = match kind.params() { - SketchParams::CmsWithHeap { heap_size, .. } - | SketchParams::CountSketchWithHeap { heap_size, .. } => *heap_size, - params => panic!("expected heap-bearing Top-K parameters, got {params:?}"), - }; - assert_eq!(heap_size, 100u32.max(*k as u32)); - assert_eq!( - state_input.item.as_ref(), - Some(&SummaryInputExpr::EntityIdentity( - EntityIdentity::PromqlLabelSet { - excluding: excluded_labels - } - )) + assert!( + !candidates.is_empty(), + "heap-backed temporal Top-K candidate" ); - assert_eq!(state_input.weight, expected_update); - assert!(!child.contains_asap()); + // The search archive may visit tighter local error allocations first. + // Every visited configuration must preserve the same update contract. + for candidate in candidates { + let Some(ASAPOp::SummaryEstimate { + summary_input, + query: SketchStatistic::TopK { k, .. }, + }) = candidate.asap() + else { + panic!("expected Top-K estimate, got {:?}", candidate.operator) + }; + assert_eq!( + *k, 5, + "the requested Top-K cardinality must survive binding" + ); + let Some(ASAPOp::SummaryAgg { + input: state_input, + family, + child, + .. + }) = summary_input.asap() + else { + panic!("expected structured Top-K state input") + }; + let FieldDataType::Sketch(kind, _) = family else { + panic!("expected a heap-backed sketch family, got {family:?}") + }; + assert_eq!(kind.algorithm(), &expected_family); + let heap_size = match kind.params() { + SketchParams::CmsWithHeap { heap_size, .. } + | SketchParams::CountSketchWithHeap { heap_size, .. } => *heap_size, + params => panic!("expected heap-bearing Top-K parameters, got {params:?}"), + }; + assert!( + heap_size >= 100u32.max(*k as u32), + "a tighter local error budget may increase, never reduce, capacity" + ); + assert_eq!( + state_input.item.as_ref(), + Some(&SummaryInputExpr::EntityIdentity( + EntityIdentity::PromqlLabelSet { + excluding: excluded_labels.clone() + } + )) + ); + assert_eq!(state_input.weight, expected_update); + assert!(!child.contains_asap()); + } } } From 86e3dafbbca231aeba8e8c39cf06978f94bc4a8e Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 18:59:01 +0000 Subject: [PATCH 28/29] test(planner): decouple sketch execution fixtures from search archives --- .../integration-tests/tests/sketch_rollup.rs | 11 ++++---- .../tests/summary_coverage_e2e.rs | 25 ++++++++----------- 2 files changed, 17 insertions(+), 19 deletions(-) diff --git a/crates/integration-tests/tests/sketch_rollup.rs b/crates/integration-tests/tests/sketch_rollup.rs index bc2a20737..3fb0da5e9 100644 --- a/crates/integration-tests/tests/sketch_rollup.rs +++ b/crates/integration-tests/tests/sketch_rollup.rs @@ -17,7 +17,7 @@ use asap_executor::{ values::{Batch, Value}, }; use asap_frontend_sql::{lower_sql, SqlCatalog}; -use asap_integration_tests::planning::logical_candidates; +use asap_integration_tests::planning::explicit_local_candidates; use asap_types::ir::operator::operator_properties::Reduction; use asap_types::ir::physical_export::PhysicalASAPOperatorPayload; use asap_types::ir::properties::summary_coverage::CoverageError; @@ -241,16 +241,17 @@ fn rolled_up(coarse: &OperatorNode) -> Rc { .unwrap() } -/// Every sketch candidate of `sql` (one build by `job`), each with its +/// Every local sketch rule for `sql` (one build by `job`), each with its /// rollup from `(region, job)`: the family, the direct root and the -/// rolled-up root. +/// rolled-up root. Explicit rule fixtures keep this execution test independent +/// of the search archive and its tighter, potentially much larger capacities. async fn direct_and_rolled_up( sql: &str, accuracy: AccuracyTarget, ) -> Vec<(SketchAlgorithm, Rc, Rc)> { let root = lower_sql(sql, &catalog(), accuracy.clone()).await.unwrap(); let mut out = Vec::new(); - for candidate in logical_candidates(root, accuracy) { + for candidate in explicit_local_candidates(root) { let Some(build) = build(&candidate) else { continue; }; @@ -355,7 +356,7 @@ async fn invalid_rollups_are_rejected() { ) .await .unwrap(); - let candidate = logical_candidates(root, accuracy) + let candidate = explicit_local_candidates(root) .into_iter() .find(|candidate| { build(candidate).is_some_and(|build| { diff --git a/crates/integration-tests/tests/summary_coverage_e2e.rs b/crates/integration-tests/tests/summary_coverage_e2e.rs index 937447ad7..0191b84ee 100644 --- a/crates/integration-tests/tests/summary_coverage_e2e.rs +++ b/crates/integration-tests/tests/summary_coverage_e2e.rs @@ -11,7 +11,7 @@ use std::rc::Rc; use asap_frontend_sql::{lower_sql, SqlCatalog}; use asap_integration_tests::fixtures::lower_promql; -use asap_integration_tests::planning::logical_candidates; +use asap_integration_tests::planning::explicit_summary_candidate; use asap_types::ir::properties::summary_coverage::{ColumnIdentity, Constraint, SelectionBox}; use asap_types::ir::properties::AccuracyTarget; use asap_types::ir::scalar::expr_ir::ScalarValue; @@ -23,20 +23,17 @@ fn accuracy() -> AccuracyTarget { AccuracyTarget::max_error(0.01).failure_probability(0.01) } -/// The single KLL `SummaryAgg` among the logical candidates for `pre`. +/// The single KLL `SummaryAgg` in an explicit KLL rule fixture for `pre`. +/// Coverage semantics do not depend on the capacities visited by search. fn kll_state(pre: Rc) -> Rc { - let mut states: Vec> = Vec::new(); - for dag in logical_candidates(pre, accuracy()) { - for node in OperatorNode::reachable(&dag) { - let is_kll = matches!(node.asap(), Some(ASAPOp::SummaryAgg { family: FieldDataType::Sketch(kind, _), .. }) - if kind.algorithm() == &SketchAlgorithm::Kll); - // Each candidate is composed on its own: the same state is - // structurally equal across candidates. - if is_kll && !states.iter().any(|s| **s == *node) { - states.push(node); - } - } - } + let dag = explicit_summary_candidate(pre, SketchAlgorithm::Kll); + let mut states: Vec<_> = OperatorNode::reachable(&dag) + .into_iter() + .filter(|node| { + matches!(node.asap(), Some(ASAPOp::SummaryAgg { family: FieldDataType::Sketch(kind, _), .. }) + if kind.algorithm() == &SketchAlgorithm::Kll) + }) + .collect(); assert_eq!( states.len(), 1, From ab8645a606ebcc1d21769f820427d794ba9d2564 Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Mon, 26 Oct 2026 15:53:44 -0400 Subject: [PATCH 29/29] feat: jointly price multi-input plans and mixed schedules (#940) * feat: jointly price multi-input plans and mixed schedules * fix(planner): preserve search after unsupported direct pricing * refactor(milp): keep one mixed-schedule representation and encoder --- .../tests/production_rollup_workloads.rs | 56 ++ .../src/cost/execution_demand.rs | 110 +++ .../costing/direct_price_options_tests.rs | 176 +++- .../src/search/costing/direct_profile.rs | 374 +++++---- .../search/costing/multi_input_price_tests.rs | 292 +++++++ .../costing/range_price_options_tests.rs | 266 ++++++ .../search/costing/supplier_price_options.rs | 415 +++++++--- .../src/search/milp/direct_cost.rs | 137 +--- .../src/search/milp/direct_fixed_schedule.rs | 567 ------------- .../src/search/milp/direct_price_options.rs | 216 ++++- .../src/search/milp/direct_schedule.rs | 528 ------------ .../src/search/milp/general_schedule.rs | 178 ++++ .../src/search/milp/general_schedule_rates.rs | 771 ++++++++++++++++++ .../src/search/milp/general_schedule_tests.rs | 254 ++++++ .../plan-selection/src/search/milp/model.rs | 13 + .../search/milp/schedule_selection_tests.rs | 355 ++++++++ .../src/search/refinement/planner.rs | 60 +- .../proposals/joint-planning-search.md | 14 +- 18 files changed, 3276 insertions(+), 1506 deletions(-) create mode 100644 crates/plan-selection/src/search/costing/multi_input_price_tests.rs create mode 100644 crates/plan-selection/src/search/costing/range_price_options_tests.rs delete mode 100644 crates/plan-selection/src/search/milp/direct_fixed_schedule.rs delete mode 100644 crates/plan-selection/src/search/milp/direct_schedule.rs create mode 100644 crates/plan-selection/src/search/milp/general_schedule.rs create mode 100644 crates/plan-selection/src/search/milp/general_schedule_rates.rs create mode 100644 crates/plan-selection/src/search/milp/general_schedule_tests.rs create mode 100644 crates/plan-selection/src/search/milp/schedule_selection_tests.rs diff --git a/crates/integration-tests/tests/production_rollup_workloads.rs b/crates/integration-tests/tests/production_rollup_workloads.rs index e00b2ba7c..e22d027fc 100644 --- a/crates/integration-tests/tests/production_rollup_workloads.rs +++ b/crates/integration-tests/tests/production_rollup_workloads.rs @@ -292,3 +292,59 @@ fn independent_kll_cost( let result = asap_plan_selection::select_plan(&[plan], entries, data, models).unwrap(); result.costs[&result.selected].cost_per_second.unwrap() } + +// Exercise multi-input operators through the production route under more than +// three heterogeneous query schedules, not just the local pricing kernel. +#[test] +fn production_binary_queries_use_joint_prices_with_mixed_schedules() { + use asap_types::workload::{DemandEstimate, Rate}; + let accuracy = AccuracyTarget::exact(); + let queries = ["m + n", "m + m", "m or n", "m * n", "m - n"]; + let roots = queries + .iter() + .enumerate() + .map(|(id, query)| { + let root = lower_promql(query, accuracy.clone()).unwrap(); + ( + id, + QueryRoot::Operator(promql_rows::with_series_identity(&root).unwrap()), + ) + }) + .collect(); + let mut entries: Vec<_> = [59_000, 61_000, 67_000, 71_000, 73_000] + .into_iter() + .map(|interval| entry(&accuracy, interval)) + .collect(); + entries[3].recurrence = + QueryRecurrence::Repeated(RepeatedDemand::EstimatedRate(DemandEstimate { + expected_rate: Rate(0.137), + observed_at: None, + valid_for: None, + })); + entries[4].recurrence = QueryRecurrence::Unknown; + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival":{"ingesting":{"rows_per_second":16.0}}, + "sources":{"m":{"rows_held":80,"series_count":8,"sample_interval":1000}, + "n":{"rows_held":80,"series_count":8,"sample_interval":1000}} + })) + .unwrap(); + let run = production(roots, &entries, &data); + let selection = run + .stage3_plan_selection + .as_ref() + .expect("validated binary workload"); + let report = selection.search.candidate_connections.as_ref().unwrap(); + println!("mixed binary workload report={report:?}"); + assert!( + report.joint_price_solve_calls > 0 && report.direct_cost_validated_plans > 0, + "{report:?}" + ); + let accepted = &selection.accepted[&selection.selected]; + assert_eq!(accepted.len(), queries.len()); + assert!(accepted + .iter() + .all(|item| !item.accepted_regardless_of_guarantee)); + assert!(selection.costs[&selection.selected] + .cost_per_second + .is_some_and(|cost| cost.is_finite() && cost >= 0.0)); +} diff --git a/crates/plan-selection/src/cost/execution_demand.rs b/crates/plan-selection/src/cost/execution_demand.rs index 5b912765f..fc930fcc6 100644 --- a/crates/plan-selection/src/cost/execution_demand.rs +++ b/crates/plan-selection/src/cost/execution_demand.rs @@ -120,6 +120,33 @@ pub(crate) fn evaluation_rate<'r>( Ok(repeated + estimated + invocations as f64 / non_repeating_query_horizon_seconds) } +/// Number of offsets common to nonempty arithmetic progressions +/// `{k * step | k * step <= span}`. A checked LCM beyond the shortest span +/// means only offset zero is shared. Kept in costing so both evaluator and +/// MILP demand atoms use the same integer arithmetic without layer inversion. +pub(crate) fn offset_intersection_count(geometries: &[(u64, u64)]) -> i128 { + fn gcd(mut a: u64, mut b: u64) -> u64 { + while b != 0 { + let remainder = a % b; + a = b; + b = remainder; + } + a + } + let Some(span) = geometries.iter().map(|(_, span)| *span).min() else { + return 0; + }; + let mut lcm = 1u64; + for &(step, _) in geometries { + let step = step.max(1); + match (lcm / gcd(lcm, step)).checked_mul(step) { + Some(next) if next <= span => lcm = next, + _ => return 1, + } + } + i128::from(span / lcm) + 1 +} + /// The distinct instants one request evaluates at, for steps in ms each with /// the longest span in ms at that step: the union of the offsets /// `k · step_ms ≤ span_ms` back from the request's end. One step needs no @@ -129,6 +156,30 @@ fn distinct_instants_per_request(longest_span_ms_by_step_ms: &BTreeMap let (&step_ms, &span_ms) = longest_span_ms_by_step_ms.iter().next().expect("one step"); return span_ms / step_ms.max(1) + 1; } + // Inclusion-exclusion is bounded by 4095 intersections for up to twelve + // distinct steps, regardless of the number of evaluations in each range. + if longest_span_ms_by_step_ms.len() <= 12 { + let geometries: Vec<_> = longest_span_ms_by_step_ms + .iter() + .map(|(&step, &span)| (step, span)) + .collect(); + let mut total = 0i128; + for mask in 1usize..(1usize << geometries.len()) { + let intersection: Vec<_> = geometries + .iter() + .enumerate() + .filter(|(bit, _)| mask & (1 << bit) != 0) + .map(|(_, geometry)| *geometry) + .collect(); + let count = offset_intersection_count(&intersection); + if mask.count_ones() % 2 == 1 { + total += count; + } else { + total -= count; + } + } + return u64::try_from(total).expect("distinct evaluation offset count fits u64"); + } let mut offsets_ms = BTreeSet::new(); for (&step_ms, &span_ms) in longest_span_ms_by_step_ms { offsets_ms.extend((0..=span_ms / step_ms.max(1)).map(|k| k * step_ms)); @@ -373,6 +424,65 @@ mod tests { ); } + #[test] + fn billion_offset_mixed_ranges_are_counted_arithmetically() { + let recurrence = fixed(1000, None); + let rate = evaluation_rate( + [ + ( + &recurrence, + Some(RangeEvaluation { + span_ms: 1_000_000_000, + step_ms: 2, + }), + ), + ( + &recurrence, + Some(RangeEvaluation { + span_ms: 1_000_000_000, + step_ms: 3, + }), + ), + ], + 60.0, + ) + .unwrap(); + assert_eq!(rate, 666_666_668.0); + } + + #[test] + fn arithmetic_union_matches_small_explicit_offset_sets() { + for left_step in 1..=7 { + for right_step in 1..=7 { + for left_span in [0, 5, 17, 30] { + for right_span in [0, 9, 23, 30] { + let mut ranges = BTreeMap::new(); + for (step, span) in + [(left_step, left_span), (right_step, right_span), (11, 29)] + { + let longest = ranges.entry(step).or_insert(0u64); + *longest = (*longest).max(span); + } + let mut oracle = BTreeSet::new(); + for (&step, &span) in &ranges { + oracle.extend((0..=span / step).map(|k| k * step)); + } + assert_eq!( + distinct_instants_per_request(&ranges), + oracle.len() as u64, + "{ranges:?}" + ); + } + } + } + } + let large = 1u64 << 63; + assert_eq!( + offset_intersection_count(&[(large - 1, large), (large - 3, large)]), + 1 + ); + } + // Invalid fixed intervals and estimated rates still produce the evaluator's error. #[test] fn invalid_recurrence_is_rejected() { diff --git a/crates/plan-selection/src/search/costing/direct_price_options_tests.rs b/crates/plan-selection/src/search/costing/direct_price_options_tests.rs index 70bca171b..2370ea24b 100644 --- a/crates/plan-selection/src/search/costing/direct_price_options_tests.rs +++ b/crates/plan-selection/src/search/costing/direct_price_options_tests.rs @@ -76,7 +76,7 @@ fn differing_supplier_statistics_keep_both_price_interfaces() { .collect(), }; let roots = vec![QueryRoot::Operator(project)]; - let entries = vec![QueryWorkloadEntry { + let entries = [QueryWorkloadEntry { query: Query("m".into()), requirements: QueryRequirements { accuracy: AccuracyTarget::exact(), @@ -118,21 +118,22 @@ fn differing_supplier_statistics_keep_both_price_interfaces() { .iter() .find(|option| { option - .compatible_suppliers + .compatible_input_tuples .iter() + .flatten() .any(|binding| binding.producer == producer) }) .expect("missing compatible supplier price option"); - assert_eq!(option.compatible_suppliers.len(), 1); - assert_eq!(option.compatible_suppliers[0].price_option, 0); + assert_eq!(option.compatible_input_tuples.len(), 1); + assert_eq!(option.compatible_input_tuples[0][0].price_option, 0); } let low = reader_options .iter() - .find(|option| option.compatible_suppliers[0].producer == PhysicalCandidateId(0)) + .find(|option| option.compatible_input_tuples[0][0].producer == PhysicalCandidateId(0)) .unwrap(); let high = reader_options .iter() - .find(|option| option.compatible_suppliers[0].producer == PhysicalCandidateId(1)) + .find(|option| option.compatible_input_tuples[0][0].producer == PhysicalCandidateId(1)) .unwrap(); assert!( high.profile.work_seconds > low.profile.work_seconds, @@ -299,7 +300,7 @@ fn downstream_prices_follow_the_selected_supplier_interface() { }); configured.inventory.graph.query_roots[0].compatible_candidates = vec![PhysicalCandidateId(3)]; let roots = vec![QueryRoot::Operator(downstream)]; - let entries = vec![QueryWorkloadEntry { + let entries = [QueryWorkloadEntry { query: Query("m".into()), requirements: QueryRequirements { accuracy: AccuracyTarget::exact(), @@ -336,8 +337,8 @@ fn downstream_prices_follow_the_selected_supplier_interface() { ); let mut referenced = Vec::new(); for option in &extracted.candidates[3] { - assert_eq!(option.compatible_suppliers.len(), 1); - let binding = &option.compatible_suppliers[0]; + assert_eq!(option.compatible_input_tuples.len(), 1); + let binding = &option.compatible_input_tuples[0][0]; assert_eq!(binding.producer, PhysicalCandidateId(2)); referenced.push(binding.price_option); let supplier = &extracted.candidates[2][binding.price_option]; @@ -378,7 +379,7 @@ fn downstream_prices_follow_the_selected_supplier_interface() { assert_eq!(cyclic.candidates[3].len(), 2); assert!(cyclic.candidates[2] .iter() - .flat_map(|option| &option.compatible_suppliers) + .flat_map(|option| option.compatible_input_tuples.iter().flatten()) .all(|binding| binding.producer != PhysicalCandidateId(2))); let chosen = solve(&configured, &cyclic); assert!( @@ -635,8 +636,9 @@ fn supplier_worker_buckets_remain_joint_choices_and_match_evaluator() { .iter() .find(|option| { option - .compatible_suppliers + .compatible_input_tuples .iter() + .flatten() .any(|binding| binding.producer == raw_supplier) }) .unwrap(); @@ -644,8 +646,9 @@ fn supplier_worker_buckets_remain_joint_choices_and_match_evaluator() { .iter() .find(|option| { option - .compatible_suppliers + .compatible_input_tuples .iter() + .flatten() .any(|binding| binding.producer == projected_supplier) }) .unwrap(); @@ -698,3 +701,152 @@ fn supplier_worker_buckets_remain_joint_choices_and_match_evaluator() { ); } } + +#[test] +fn mixed_schedule_joint_prices_match_the_real_evaluator() { + let scan = |metric| { + OperatorNode::reachable(&lower_promql(metric, AccuracyTarget::exact())) + .into_iter() + .find(|node| matches!(node.non_asap(), Some(NonASAPOp::Scan { .. }))) + .unwrap() + }; + let m = scan("m"); + let n = scan("n"); + let project = OperatorNode::new_shared(Operator::NonASAP(NonASAPOp::Project { + cols: m + .schema + .fields + .iter() + .enumerate() + .map(|(index, field)| ProjectItem { + alias: Some(field.name.clone()), + expr: ScalarExpr::Column(index), + }) + .collect(), + qualifier: None, + child: m.clone(), + })) + .unwrap(); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph { + candidates: (0..3) + .map(|id| PhysicalCandidate { + group: MemoGroupId(id as u32), + local_cost: 0.0, + producer_requirements: if id == 2 { + vec![ProducerRequirement { + compatible_producers: vec![ + PhysicalCandidateId(0), + PhysicalCandidateId(1), + ] + .into(), + }] + } else { + vec![] + }, + }) + .collect(), + query_roots: vec![QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(2)], + }], + ..Default::default() + }, + templates: vec![Some(m), Some(n), Some(project.clone())], + group_candidates: vec![], + }, + configurations: (0..3) + .map(|id| CandidateConfiguration { + source_candidate: PhysicalCandidateId(id), + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::QueryTime, + }) + .collect(), + }; + let roots = vec![QueryRoot::Operator(project); 5]; + let entries = [QueryWorkloadEntry { + query: Query("m".into()), + requirements: QueryRequirements { + accuracy: AccuracyTarget::exact(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval( + 60_000, + ))), + range_evaluation: None, + }]; + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival": {"ingesting": {"rows_per_second": 100.0}}, + "sources": {"m": {"rows_held":10,"series_count":1},"n":{"rows_held":100,"series_count":1}} + })) + .unwrap(); + use asap_types::workload::{DemandEstimate, Rate, TimestampMs}; + let mut entries = vec![entries[0].clone(); 5]; + entries[0].recurrence = + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(59_000))); + entries[1].recurrence = + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(61_000))); + entries[2].recurrence = + QueryRecurrence::Repeated(RepeatedDemand::EstimatedRate(DemandEstimate { + expected_rate: Rate(0.137), + observed_at: None, + valid_for: None, + })); + entries[3].recurrence = QueryRecurrence::Repeated(RepeatedDemand::Scheduled(vec![ + TimestampMs(10), + TimestampMs(20), + ])); + entries[4].recurrence = QueryRecurrence::OneTime { + invocations: 3, + execute_at: None, + }; + configured.inventory.graph.query_roots = + vec![configured.inventory.graph.query_roots[0].clone(); 5]; + // All roots use the original source; this parity check has no synthetic + // equivalence between different source names. + configured.inventory.graph.candidates[2].producer_requirements[0].compatible_producers = + vec![PhysicalCandidateId(0)].into(); + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &roots, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let profile = extract_price_options(&configured, &context) + .unwrap() + .unwrap(); + assert!(profile.schedules.is_some()); + let proposal = + crate::search::milp::direct_price_options::solve_joint_price_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &profile, + ) + .unwrap() + .unwrap(); + let mut rejections = crate::search::refinement::planner::CandidateRejectionSummary::default(); + let evaluated = crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("{rejections:?}")); + let actual = evaluated.0.costs[&evaluated.0.selected] + .cost_per_second + .unwrap(); + let modeled = proposal.direct_modeled_cost.unwrap(); + assert!( + (actual - modeled).abs() <= 1e-9 * actual.max(1e-15), + "mixed schedules: {modeled} vs {actual}" + ); +} diff --git a/crates/plan-selection/src/search/costing/direct_profile.rs b/crates/plan-selection/src/search/costing/direct_profile.rs index f431af283..428235f63 100644 --- a/crates/plan-selection/src/search/costing/direct_profile.rs +++ b/crates/plan-selection/src/search/costing/direct_profile.rs @@ -44,6 +44,8 @@ fn supported_node(node: &asap_types::ir::OperatorNode) -> bool { | NonASAPOp::Filter { .. } | NonASAPOp::Project { .. } | NonASAPOp::Aggregate { .. } + | NonASAPOp::BinaryOp { .. } + | NonASAPOp::Join { .. } ) ) || matches!( node.asap(), @@ -57,9 +59,9 @@ fn supported_node(node: &asap_types::ir::OperatorNode) -> bool { ) } -/// Identical fixed schedules or up to three heterogeneous schedules, no range requests and one operator -/// root per query are supported. Every returned profile is priced at one -/// execution; the MILP supplies demand, held lifetime, retention and overhead. +/// Fixed-interface compatibility adapter. Local profiles are priced at one +/// execution; the MILP supplies schedule demand, lifetime and shared charges. +/// Joint extraction additionally carries input tuples and retention tiers. pub(in crate::search) fn extract( configured: &CandidateConfigurationInventory, context: &CandidatePlanningContext<'_>, @@ -120,15 +122,6 @@ fn prepare_profile( .entries .iter() .any(|entry| entry.range_evaluation.is_some()); - if has_range - && (!equal_fixed - || context - .entries - .iter() - .any(|entry| entry.range_evaluation != first.range_evaluation)) - { - return Ok(None); - } let rates: Vec<_> = context .entries .iter() @@ -140,10 +133,15 @@ fn prepare_profile( .map_err(|error| DirectProfileError::Invalid(error.to_string())) }) .collect::>()?; - if rates.iter().any(|rate| !rate.is_finite() || *rate <= 0.0) { + if rates.iter().any(|rate| !rate.is_finite() || *rate < 0.0) { return Ok(None); } - let rate = rates[0]; + let rate = rates + .iter() + .copied() + .filter(|rate| *rate > 0.0) + .reduce(f64::max) + .unwrap_or(1.0); let request_rate: f64 = context .entries .iter() @@ -157,61 +155,18 @@ fn prepare_profile( .collect::, _>>()? .iter() .sum(); - let schedules = if equal_fixed { + let schedules = if equal_fixed && !has_range { None - } else if context.entries.len() > 3 { - use crate::search::milp::direct_cost::FixedQuerySchedule; - let fixed: Option> = context - .entries - .iter() - .map(|entry| match &entry.recurrence { - QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(interval)) => { - Some(FixedQuerySchedule { - interval_ms: interval.0, - phase_ms: None, - }) - } - QueryRecurrence::Repeated(RepeatedDemand::FixedIntervalAt { - interval, - evaluation_phase, - }) => Some(FixedQuerySchedule { - interval_ms: interval.0, - phase_ms: Some(evaluation_phase.0), - }), - _ => None, - }) - .collect(); - let Some(fixed) = fixed else { return Ok(None) }; - match QueryScheduleRates::from_fixed_schedules(fixed) { - Ok(schedule) => Some(schedule), - Err( - crate::search::candidates::graph::CandidateSelectionError::UnsupportedDirectCost(_), - ) => return Ok(None), - Err(error) => return Err(DirectProfileError::Invalid(error.to_string())), - } } else { - let mut materialized_subset_rates = Vec::with_capacity(1 << context.entries.len()); - for mask in 0..(1usize << context.entries.len()) { - materialized_subset_rates.push( - crate::cost::execution_demand::evaluation_rate( - context - .entries - .iter() - .enumerate() - .filter(|(query, _)| mask & (1 << query) != 0) - .map(|(_, entry)| (&entry.recurrence, None)), - settings.non_repeating_query_horizon_seconds, - ) - .map_err(|error| DirectProfileError::Invalid(error.to_string()))?, - ); - } - Some(QueryScheduleRates { - compact_fixed: None, - per_query_rates: rates, - materialized_subset_rates, - }) + let general = crate::search::milp::direct_cost::QueryScheduleRates::new( + context.entries, + settings.non_repeating_query_horizon_seconds, + context.deadline, + ) + .map_err(|error| DirectProfileError::Unsupported(error.to_string()))?; + Some(general) }; - let delta = match failure_budget(context) { + let delta = match failure_budget(context)? { Some(delta) => delta, None => return Ok(None), }; @@ -233,7 +188,7 @@ fn prepare_profile( && match configuration.timing { ExecutionTiming::QueryTime => { matches!(configuration.materialization, Materialization::NotMaterialized | Materialization::MatAtQuery(Medium::Memory)) - || (!has_range && configuration.materialization == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) + || (configuration.materialization == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) && matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) && context.models.deployment_capabilities.query_time_results_retained && (0..context.entries.len()).any(|reader| asap_physical_asap_optimization::summary_materialization_eligibility(node, &[reader], context.entries, context.data, context.models.deployment_capabilities).is_ok_and(|eligibility| eligibility.may_keep_across_evaluations))) @@ -257,10 +212,7 @@ fn prepare_profile( } }, } - && configured.inventory.graph.candidates[id] - .producer_requirements - .len() - <= 1 + }) .map(|node| { let mut position = 0; @@ -369,6 +321,23 @@ fn extract_scan_class( context: &CandidatePlanningContext<'_>, prefer_latest_pane: bool, ) -> Result, DirectProfileError> { + // The fixed adapter has no selected-reader retention tiers. The joint + // option extractor carries those decisions instead. + if context + .entries + .iter() + .any(|entry| entry.range_evaluation != context.entries[0].range_evaluation) + || context + .entries + .iter() + .any(|entry| entry.range_evaluation.is_some()) + && configured.configurations.iter().any(|c| { + c.materialization + == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) + }) + { + return Ok(None); + } let Some(prepared) = prepare_profile(configured, context, prefer_latest_pane)? else { return Ok(None); }; @@ -381,6 +350,16 @@ fn extract_scan_class( request_rate, .. } = prepared; + let mut nodes = nodes; + for (id, node) in nodes.iter_mut().enumerate() { + if configured.inventory.graph.candidates[id] + .producer_requirements + .len() + > 1 + { + *node = None; + } + } let count = nodes.len(); let settings = context.models.plan_selection; let CostModel::UsageOverTime(prices) = &settings.cost_model else { @@ -418,6 +397,7 @@ fn extract_scan_class( let supported: Vec<_> = cache.iter().map(Option::is_some).collect(); let zero = || NodeCostProfile { + per_request: false, execution_cost: 0.0, work_seconds: 0.0, compaction_seconds: 0.0, @@ -478,6 +458,8 @@ fn profile_for_priced( }) .transpose()?; Ok(NodeCostProfile { + per_request: node.materialization + == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory), execution_cost: priced.cpu_ops_per_second * prices.cost_per_cpu_op + priced.scanned_bytes_per_second * prices.cost_per_scanned_byte, ingestion: node.output_state.timing == ExecutionTiming::IngestionTime, @@ -493,6 +475,13 @@ fn profile_for_priced( }) } +struct PriceInputs { + suppliers: Vec, + upstream_work: WorkBounds, + upstream_sketches: usize, + range_span_ms: Option, +} + #[allow(clippy::too_many_arguments)] fn price<'a>( id: usize, @@ -510,7 +499,7 @@ fn price<'a>( worker_counts: &mut [Option], compaction_seconds: &mut [f64], delta: Option, - supplier_override: Option, + input_override: Option<&PriceInputs>, ) -> Result<(), DirectProfileError> { check_deadline(context.deadline)?; if visited[id] || visiting[id] { @@ -522,27 +511,26 @@ fn price<'a>( }; visiting[id] = true; let requirements = &configured.inventory.graph.candidates[id].producer_requirements; - let options: Vec<_> = supplier_override.map_or_else( + let options: Vec> = input_override.map_or_else( || { requirements.first().map_or_else( - || vec![None], + || vec![vec![]], |input| { input .compatible_producers .iter() - .copied() - .map(Some) + .map(|id| vec![*id]) .collect() }, ) }, - |supplier| vec![Some(supplier)], + |inputs| vec![inputs.suppliers.clone()], ); - for supplier in options { + 'bindings: for input_tuple in options { let mut views = Vec::new(); - if let Some(supplier) = supplier { + for &supplier in &input_tuple { if visiting[supplier.0] { - continue; + continue 'bindings; } price( supplier.0, @@ -563,7 +551,7 @@ fn price<'a>( None, )?; let Some(priced) = &cache[supplier.0] else { - continue; + continue 'bindings; }; let producer = nodes[supplier.0].as_ref().unwrap(); if producer.output_state.timing == ExecutionTiming::QueryTime @@ -573,7 +561,7 @@ fn price<'a>( ) && lookbacks[supplier.0] != scan_consumer_lookback(node) { - continue; + continue 'bindings; } let consumer_ingests = node.output_state.timing == ExecutionTiming::IngestionTime; @@ -584,7 +572,7 @@ fn price<'a>( && producer_ingests && producer.materialization != Materialization::MatAtIngestion(Medium::Memory)) { - continue; + continue 'bindings; } views.push(ProducerOutput { payload: &producer.payload, @@ -595,6 +583,7 @@ fn price<'a>( unchanged_columns: priced.unchanged_columns.clone(), }); } + let supplier = input_tuple.first().copied(); if let asap_types::ir::Operator::ASAP( ASAPOp::SummaryAgg { family: FieldDataType::Sketch(kind, _), @@ -640,22 +629,24 @@ fn price<'a>( } _ => None, }; - let sketches = supplier.map_or(0, |supplier| sketch_counts[supplier.0]) - + usize::from(matches!( - &node.payload, - asap_types::ir::Operator::ASAP( - ASAPOp::SummaryAgg { - family: FieldDataType::Sketch(..), - .. - } | ASAPOp::PaneChain { - family: FieldDataType::Sketch(..), - .. - } - ) - )); + let sketches = input_override.map_or_else( + || supplier.map_or(0, |supplier| sketch_counts[supplier.0]), + |inputs| inputs.upstream_sketches, + ) + usize::from(matches!( + &node.payload, + asap_types::ir::Operator::ASAP( + ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(..), + .. + } | ASAPOp::PaneChain { + family: FieldDataType::Sketch(..), + .. + } + ) + )); // Failure-budget sizing splits delta across approximate inputs. This // slice admits at most one sketch build along each unary output path. - if sketches > 1 { + if sketches > 1 && input_override.is_none() { continue; } let (retained_panes, ingestion_window) = match &node.payload { @@ -665,11 +656,15 @@ fn price<'a>( .. }) => { let window = pane_width_ms.checked_mul(u64::from(*pane_count)); - let span = context - .entries - .first() - .and_then(|entry| entry.range_evaluation) - .map_or(0, |range| range.span_ms); + let span = input_override + .and_then(|inputs| inputs.range_span_ms) + .unwrap_or_else(|| { + context + .entries + .first() + .and_then(|entry| entry.range_evaluation) + .map_or(0, |range| range.span_ms) + }); let Some(retention) = window.and_then(|window| window.checked_add(span)) else { continue; }; @@ -691,9 +686,14 @@ fn price<'a>( ) else { continue; }; - let prior = supplier.map_or(WorkBounds::default(), |supplier| { - closure_work_ms[supplier.0] - }); + let prior = input_override.map_or_else( + || { + supplier.map_or(WorkBounds::default(), |supplier| { + closure_work_ms[supplier.0] + }) + }, + |inputs| inputs.upstream_work, + ); let closure_work = WorkBounds { minimum: (prior.minimum + priced.work_ms).next_down().max(0.0), maximum: (prior.maximum + priced.work_ms).next_up(), @@ -771,53 +771,111 @@ fn price<'a>( /// A conservative delta valid for every selected unary output path, including /// rollups whose finer grouping the current evaluator uses for its row bound. /// Unknown population bounds reject this direct proposal, not the full search. -fn failure_budget(context: &CandidatePlanningContext<'_>) -> Option> { - let population = || { - let mut sources = HashSet::new(); - for root in context.original_roots { - let QueryRoot::Operator(root) = root else { - return None; - }; - for node in asap_types::ir::OperatorNode::reachable(root) { - if let Some(NonASAPOp::Scan { source, .. }) = node.non_asap() { - sources.insert(crate::source_name(source).to_string()); +fn failure_budget( + context: &CandidatePlanningContext<'_>, +) -> Result>, DirectProfileError> { + check_deadline(context.deadline)?; + let result = (|| { + let population = || { + let mut sources = HashSet::new(); + for root in context.original_roots { + let QueryRoot::Operator(root) = root else { + return None; + }; + for node in asap_types::ir::OperatorNode::reachable(root) { + if let Some(NonASAPOp::Scan { source, .. }) = node.non_asap() { + sources.insert(crate::source_name(source).to_string()); + } } } + let counts: Option> = sources + .iter() + .map(|source| { + context + .data + .sources + .get(source)? + .series_count + .filter(|count| *count > 0) + }) + .collect(); + let maximum_source = counts?.into_iter().max()?; + fn bound( + node: &std::rc::Rc, + maximum_source: u64, + deadline: Option, + memo: &mut std::collections::HashMap, + ) -> Option { + if deadline.is_some_and(|deadline| std::time::Instant::now() >= deadline) { + return None; + } + let identity = std::rc::Rc::as_ptr(node) as usize; + if let Some(bound) = memo.get(&identity) { + return Some(*bound); + } + let mut children = Vec::new(); + node.operator.map_children(|child| { + children.push(bound(child, maximum_source, deadline, memo)) + }); + let children: Vec<_> = children.into_iter().collect::>()?; + let value = if children.is_empty() { + maximum_source + } else if children.len() == 1 { + children[0] + } else { + // Branching operators can concatenate rows or pair inputs. + // Both union and cross-product bounds cover supported joins, + // set operations and vector matching; sharing does not reduce + // the number of logical input positions in this calculation. + let sum = children + .iter() + .try_fold(0u64, |sum, child| sum.checked_add(*child))?; + let product = children + .iter() + .try_fold(1u64, |product, child| product.checked_mul(*child))?; + sum.max(product) + }; + memo.insert(identity, value); + Some(value) + } + let mut memo = std::collections::HashMap::new(); + context + .original_roots + .iter() + .map(|root| match root { + QueryRoot::Operator(node) => { + bound(node, maximum_source, context.deadline, &mut memo) + } + _ => None, + }) + .collect::>>()? + .into_iter() + .max() + }; + let mut delta: Option = None; + for entry in context.entries { + let target = &entry.requirements.accuracy; + if target.is_exact() { + continue; + } + let requested = target + .failure_probability + .filter(|delta| delta.is_finite() && *delta > 0.0)?; + let values = match target.accuracy_scope { + AccuracyScope::WholeResult => population()?, + scope => scope.max_values_covered(None)?, + } + .max(1); + let required = requested / values as f64; + if !required.is_finite() || required <= 0.0 { + return None; + } + delta = Some(delta.map_or(required, |prior| prior.min(required))); } - let counts: Option> = sources - .iter() - .map(|source| { - context - .data - .sources - .get(source)? - .series_count - .filter(|count| *count > 0) - }) - .collect(); - counts?.into_iter().max() - }; - let mut delta: Option = None; - for entry in context.entries { - let target = &entry.requirements.accuracy; - if target.is_exact() { - continue; - } - let requested = target - .failure_probability - .filter(|delta| delta.is_finite() && *delta > 0.0)?; - let values = match target.accuracy_scope { - AccuracyScope::WholeResult => population()?, - scope => scope.max_values_covered(None)?, - } - .max(1); - let required = requested / values as f64; - if !required.is_finite() || required <= 0.0 { - return None; - } - delta = Some(delta.map_or(required, |prior| prior.min(required))); - } - Some(delta) + Some(delta) + })(); + check_deadline(context.deadline)?; + Ok(result) } fn requires_source_sizing(node: &asap_types::ir::OperatorNode) -> bool { @@ -2094,17 +2152,11 @@ mod tests { .schedules .as_ref() .unwrap() - .materialized_subset_rates[3], + .union_rate([true, true].into_iter()), if phases { 2.0 } else { 3.0 } ); } else { - assert!(extracted - .profile - .schedules - .as_ref() - .unwrap() - .compact_fixed - .is_some()); + assert!(extracted.profile.schedules.is_some()); } let proposal = crate::search::milp::model::solve_direct_cost_proposal_with_metrics( &configured.inventory.graph, @@ -2261,3 +2313,11 @@ pub(in crate::search) use supplier_price_options::extract_price_options; #[cfg(test)] #[path = "direct_price_options_tests.rs"] mod price_options_tests; + +#[cfg(test)] +#[path = "multi_input_price_tests.rs"] +mod multi_input_price_tests; + +#[cfg(test)] +#[path = "range_price_options_tests.rs"] +mod range_price_options_tests; diff --git a/crates/plan-selection/src/search/costing/multi_input_price_tests.rs b/crates/plan-selection/src/search/costing/multi_input_price_tests.rs new file mode 100644 index 000000000..768897b10 --- /dev/null +++ b/crates/plan-selection/src/search/costing/multi_input_price_tests.rs @@ -0,0 +1,292 @@ +//! End-to-end pricing uses the evaluator's binary operators and shared DAGs. +use super::*; +use crate::search::candidates::configurations::CandidateConfiguration; +use crate::search::candidates::graph::*; +use crate::search::candidates::inventory::CandidateInventory; +use asap_logical_asap_optimization::{memo::MemoGroupId, test_support::lower_promql}; +use asap_types::ir::{properties::AccuracyTarget, OperatorNode}; +use asap_types::workload::{ + DataWorkload, Predictability, Query, QueryRequirements, QueryWorkloadEntry, RepetitionInterval, +}; +use std::rc::Rc; + +fn append( + node: &Rc, + inventory: &mut CandidateConfigurationInventory, +) -> PhysicalCandidateId { + // Exact fixture expressions can be interned safely; repeated operands must + // share one producer while keeping their two ordered input positions. + if let Some(id) = inventory.inventory.templates.iter().position(|known| { + known + .as_ref() + .is_some_and(|known| known.as_ref() == node.as_ref()) + }) { + return PhysicalCandidateId(id); + } + let mut inputs = Vec::new(); + node.operator + .map_children(|child| inputs.push(append(child, inventory))); + let id = PhysicalCandidateId(inventory.configurations.len()); + inventory + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: MemoGroupId(id.0 as u32), + local_cost: 0.0, + producer_requirements: inputs + .into_iter() + .map(|producer| ProducerRequirement { + compatible_producers: vec![producer].into(), + }) + .collect(), + }); + inventory.inventory.templates.push(Some(node.clone())); + inventory.configurations.push(CandidateConfiguration { + source_candidate: id, + materialization: Materialization::NotMaterialized, + timing: ExecutionTiming::QueryTime, + }); + id +} + +#[test] +fn binary_prices_match_evaluator_with_distinct_and_shared_inputs() { + for query in ["m + n", "m + m", "m or n"] { + let logical = lower_promql(query, AccuracyTarget::exact()); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: Default::default(), + templates: vec![], + group_candidates: vec![], + }, + configurations: vec![], + }; + let root = append(&logical, &mut configured); + configured + .inventory + .graph + .query_roots + .push(QueryRootCandidates { + compatible_candidates: vec![root], + }); + assert!(configured + .inventory + .graph + .candidates + .iter() + .any(|candidate| candidate.producer_requirements.len() == 2)); + let roots = vec![QueryRoot::Operator(logical)]; + let entries = vec![QueryWorkloadEntry { + query: Query(query.into()), + requirements: QueryRequirements { + accuracy: AccuracyTarget::exact(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval( + RepetitionInterval(60_000), + )), + range_evaluation: None, + }]; + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival":{"ingesting":{"rows_per_second":100.0}}, + "sources":{"m":{"rows_held":10,"series_count":1},"n":{"rows_held":100,"series_count":1}} + })) + .unwrap(); + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &roots, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let profile = extract_price_options(&configured, &context) + .unwrap() + .unwrap(); + assert!( + !profile.candidates[root.0].is_empty(), + "unsupported binary fixture {query}" + ); + let proposal = + crate::search::milp::direct_price_options::solve_joint_price_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &profile, + ) + .unwrap() + .unwrap(); + let mut rejections = + crate::search::refinement::planner::CandidateRejectionSummary::default(); + let evaluated = crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("{query}: {rejections:?}")); + let actual = evaluated.0.costs[&evaluated.0.selected] + .cost_per_second + .unwrap(); + let direct = proposal.direct_modeled_cost.unwrap(); + assert!( + (actual - direct).abs() <= 1e-9 * actual.max(1e-15), + "{query}: actual={actual}, direct={direct}" + ); + if query == "m + m" { + let binary = configured + .inventory + .graph + .candidates + .iter() + .enumerate() + .find(|(_, candidate)| candidate.producer_requirements.len() == 2) + .unwrap() + .0; + let edges: Vec<_> = proposal + .selected + .connections + .iter() + .filter(|edge| edge.consumer.0 == binary) + .collect(); + assert_eq!(edges.len(), 2); + assert_eq!(edges[0].producer, edges[1].producer); + assert_ne!(edges[0].requirement, edges[1].requirement); + } + } +} + +// Two independently approximate inputs must retain their divided confidence +// budgets; pricing may not silently resize either state after selection. +#[test] +fn two_sketch_binary_prices_match_evaluator_after_budget_sizing() { + use asap_logical_asap_optimization::per_computation_candidates::{ + compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, + }; + use asap_types::ir::properties::{AccuracyMetric, AccuracyScope}; + use asap_types::ir::schema::SketchAlgorithm; + let query = "quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, n[5m])"; + let target = AccuracyTarget::max_error(0.05) + .accuracy_metric(AccuracyMetric::Relative) + .failure_probability(0.01) + .accuracy_scope(AccuracyScope::PerValue); + let planned = AccuracyTarget::max_error(0.01) + .accuracy_metric(AccuracyMetric::Relative) + .failure_probability(0.01) + .accuracy_scope(AccuracyScope::PerValue); + let original = lower_promql(query, planned); + let generated = + generate_per_computation_candidates(vec![(0, QueryRoot::Operator(original.clone()))]) + .unwrap(); + let choices: Vec<_> = generated + .targets + .iter() + .map(|target| { + target + .choices + .candidates + .iter() + .position(|choice| { + matches!( + choice, + ComputationCandidate::Summary(SketchAlgorithm::DDSketch) + ) + }) + .unwrap() + }) + .collect(); + assert_eq!(choices.len(), 2); + let roots: Vec<_> = compose_logical_candidate(&generated, &choices) + .unwrap() + .into_iter() + .map(|(_, root)| root) + .collect(); + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival":{"ingesting":{"rows_per_second":100.0}}, + "sources":{"m":{"rows_held":3000,"series_count":1,"values_non_negative":true}, + "n":{"rows_held":3000,"series_count":1,"values_non_negative":true}} + })) + .unwrap(); + let sized = crate::resize_for_failure_probability(&roots, std::slice::from_ref(&target), &data) + .unwrap(); + let QueryRoot::Operator(root) = &sized[0] else { + panic!("operator root") + }; + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: Default::default(), + templates: vec![], + group_candidates: vec![], + }, + configurations: vec![], + }; + let id = append(root, &mut configured); + configured + .inventory + .graph + .query_roots + .push(QueryRootCandidates { + compatible_candidates: vec![id], + }); + let original = vec![QueryRoot::Operator(original)]; + let entries = vec![QueryWorkloadEntry { + query: Query(query.into()), + requirements: QueryRequirements { + accuracy: target, + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval( + 60_000, + ))), + range_evaluation: None, + }]; + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &original, + entries: &entries, + data: &data, + models: crate::PlanningModels::builtin(), + calibration: crate::CalibrationFit::default(), + }; + let profile = extract_price_options(&configured, &context) + .unwrap() + .unwrap(); + assert!( + !profile.candidates[id.0].is_empty(), + "two certified sketches must reach root" + ); + let proposal = + crate::search::milp::direct_price_options::solve_joint_price_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &profile, + ) + .unwrap() + .unwrap(); + let mut rejections = crate::search::refinement::planner::CandidateRejectionSummary::default(); + let evaluated = crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("two-sketch plan rejected: {rejections:?}")); + let actual = evaluated.0.costs[&evaluated.0.selected] + .cost_per_second + .unwrap(); + assert!((actual - proposal.direct_modeled_cost.unwrap()).abs() <= 1e-9 * actual.max(1e-15)); +} diff --git a/crates/plan-selection/src/search/costing/range_price_options_tests.rs b/crates/plan-selection/src/search/costing/range_price_options_tests.rs new file mode 100644 index 000000000..93a3c7f87 --- /dev/null +++ b/crates/plan-selection/src/search/costing/range_price_options_tests.rs @@ -0,0 +1,266 @@ +//! Whole-evaluator parity for selected consumer range-retention tiers. +use super::*; +use crate::search::candidates::configurations::CandidateConfiguration; +use crate::search::candidates::graph::{ + CandidateDependencyGraph, PhysicalCandidate, ProducerRequirement, QueryRootCandidates, +}; +use crate::search::candidates::inventory::CandidateInventory; +use asap_logical_asap_optimization::memo::MemoGroupId; +use asap_logical_asap_optimization::per_computation_candidates::{ + compose_logical_candidate, generate_per_computation_candidates, ComputationCandidate, +}; +use asap_logical_asap_optimization::test_support::lower_promql; +use asap_types::ir::properties::AccuracyTarget; +use asap_types::ir::schema::SketchAlgorithm; +use asap_types::ir::OperatorNode; +use asap_types::workload::{ + DataWorkload, Predictability, Query, QueryRequirements, QueryWorkloadEntry, RangeEvaluation, + RepetitionInterval, +}; +use std::rc::Rc; + +#[test] +fn shared_pane_range_tiers_match_evaluator_for_ingestion_and_kept_refresh() { + fn append( + node: Rc, + configured: &mut CandidateConfigurationInventory, + ingestion: bool, + mode: Materialization, + ) -> PhysicalCandidateId { + let ingestion = ingestion + || (matches!(node.asap(), Some(ASAPOp::PaneChain { .. })) + && mode == Materialization::MatAtIngestion(Medium::Memory)); + let pane = matches!(node.asap(), Some(ASAPOp::PaneChain { .. })); + let mut inputs = Vec::new(); + node.operator + .map_children(|child| inputs.push(append(child.clone(), configured, ingestion, mode))); + let id = PhysicalCandidateId(configured.configurations.len()); + configured.configurations.push(CandidateConfiguration { + source_candidate: id, + materialization: if pane { + mode + } else { + Materialization::NotMaterialized + }, + timing: if ingestion { + ExecutionTiming::IngestionTime + } else { + ExecutionTiming::QueryTime + }, + }); + configured.inventory.templates.push(Some(node)); + configured + .inventory + .graph + .candidates + .push(PhysicalCandidate { + group: MemoGroupId(id.0 as u32), + local_cost: 0.0, + producer_requirements: inputs + .into_iter() + .map(|supplier| ProducerRequirement { + compatible_producers: vec![supplier].into(), + }) + .collect(), + }); + id + } + + for mode in [ + Materialization::MatAtIngestion(Medium::Memory), + Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory), + ] { + let target = AccuracyTarget::max_error(0.05) + .failure_probability(0.01) + .accuracy_scope(AccuracyScope::PerValue); + let original = lower_promql("quantile_over_time(0.95, latency[5m])", target.clone()); + let generated = + generate_per_computation_candidates(vec![(0, QueryRoot::Operator(original.clone()))]) + .unwrap(); + let choice = generated.targets[0] + .choices + .candidates + .iter() + .position(|candidate| { + matches!( + candidate, + ComputationCandidate::Summary(SketchAlgorithm::Kll) + ) + }) + .unwrap(); + let roots: Vec<_> = compose_logical_candidate(&generated, &[choice]) + .unwrap() + .into_iter() + .map(|(_, root)| root) + .collect(); + let rules = asap_logical_asap_optimization::asap_aware_cse::pane_candidates( + &roots, + &[Some(10_000)], + ); + let roots = + asap_logical_asap_optimization::asap_aware_cse::window_composition_pass2_candidates( + &roots, &rules, + ) + .unwrap() + .pop() + .unwrap() + .roots + .unwrap(); + let data: DataWorkload = serde_json::from_value(serde_json::json!({ + "arrival":{"ingesting":{"rows_per_second":100.0}}, + "sources":{"latency":{"series_count":2,"rows_held":30000,"values_non_negative":true}} + })) + .unwrap(); + let sized = + crate::resize_for_failure_probability(&roots, std::slice::from_ref(&target), &data) + .unwrap(); + let mut configured = CandidateConfigurationInventory { + inventory: CandidateInventory { + graph: CandidateDependencyGraph::default(), + templates: vec![], + group_candidates: vec![], + }, + configurations: vec![], + }; + let QueryRoot::Operator(root) = &sized[0] else { + panic!("operator") + }; + let root_id = append(root.clone(), &mut configured, false, mode); + configured.inventory.graph.query_roots = vec![ + QueryRootCandidates { + compatible_candidates: vec![root_id] + }; + 2 + ]; + let originals = vec![ + QueryRoot::Operator(original.clone()), + QueryRoot::Operator(original), + ]; + let entries: Vec<_> = [(60_000, 120_000, 10_000), (90_000, 600_000, 30_000)] + .into_iter() + .map(|(interval, span_ms, step_ms)| QueryWorkloadEntry { + query: Query("quantile_over_time(0.95, latency[5m])".into()), + requirements: QueryRequirements { + accuracy: target.clone(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedInterval( + RepetitionInterval( + if mode == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory) + { + 10_000 + } else { + interval + }, + ), + )), + range_evaluation: Some(RangeEvaluation { span_ms, step_ms }), + }) + .collect(); + let mut settings = crate::PlanSelectionSettings::default(); + settings.resource_speed.ms_per_cpu_op *= 1000.0; + settings.resource_speed.ms_per_scanned_byte *= 1000.0; + settings.measured_operation_costs.ns_per_evaluation = 1_000_000.0; + let deployment = asap_logical_asap_optimization::DeploymentCapabilities { + query_time_results_retained: true, + ..Default::default() + }; + let context = CandidatePlanningContext { + deadline: None, + refine_materialization: false, + original_roots: &originals, + entries: &entries, + data: &data, + models: crate::PlanningModels { + plan_selection: &settings, + deployment_capabilities: &deployment, + ..crate::PlanningModels::builtin() + }, + calibration: crate::CalibrationFit::default(), + }; + let profile = extract_price_options(&configured, &context) + .unwrap() + .unwrap(); + let chain = configured + .inventory + .templates + .iter() + .position(|node| { + node.as_ref() + .is_some_and(|node| matches!(node.asap(), Some(ASAPOp::PaneChain { .. }))) + }) + .unwrap(); + let tiers: Vec<_> = profile.candidates[chain] + .iter() + .map(|option| option.required_range_span_ms) + .collect(); + assert!( + tiers.contains(&Some(120_000)) && tiers.contains(&Some(600_000)), + "{mode:?}: {tiers:?}" + ); + let general = profile.schedules.as_ref().unwrap(); + let kept = mode == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory); + assert!( + (general.union_rate([true, true].into_iter()) - if kept { 2.9 } else { 0.45 }).abs() + < 1e-12 + ); + assert!( + (general.request_union_rate([true, true].into_iter()) + - if kept { 0.1 } else { 1.0 / 60.0 + 1.0 / 90.0 }) + .abs() + < 1e-12 + ); + assert!(profile.candidates[chain] + .iter() + .all(|option| option.profile.per_request + == (mode == Materialization::MatAtQueryKeptAcrossEvaluations(Medium::Memory)))); + let proposal = + crate::search::milp::direct_price_options::solve_joint_price_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &profile, + ) + .unwrap() + .unwrap(); + // Keep option indices/bindings unchanged, but deny every option a long + // enough retention witness. Both outputs must then be infeasible. + let mut insufficient = profile.clone(); + for option in &mut insufficient.candidates[chain] { + option.required_range_span_ms = Some(120_000); + } + assert!( + crate::search::milp::direct_price_options::solve_joint_price_proposal_with_metrics( + &configured.inventory.graph, + &[], + None, + &mut Default::default(), + &insufficient + ) + .unwrap() + .is_none() + ); + let mut rejections = + crate::search::refinement::planner::CandidateRejectionSummary::default(); + let evaluated = crate::search::refinement::planner::evaluate_configured_candidate( + &configured, + &context, + &proposal.selected, + 0, + &mut rejections, + &mut Vec::new(), + ) + .unwrap() + .unwrap_or_else(|| panic!("{mode:?}: {rejections:?}")); + let actual = evaluated.0.costs[&evaluated.0.selected] + .cost_per_second + .unwrap(); + let modeled = proposal.direct_modeled_cost.unwrap(); + assert!( + (actual - modeled).abs() <= 1e-9 * actual.max(1e-15), + "{mode:?}: direct {modeled}, evaluator {actual}" + ); + } +} diff --git a/crates/plan-selection/src/search/costing/supplier_price_options.rs b/crates/plan-selection/src/search/costing/supplier_price_options.rs index 322e9f4c6..cf1e30748 100644 --- a/crates/plan-selection/src/search/costing/supplier_price_options.rs +++ b/crates/plan-selection/src/search/costing/supplier_price_options.rs @@ -17,12 +17,20 @@ struct PriceClass<'a> { work: WorkBounds, workers: Option, compaction: f64, - bindings: Vec, + bindings: Vec>, + ancestor_work: std::collections::BTreeMap, + sketch_ids: std::collections::BTreeSet, + sketch_sources: std::collections::BTreeMap>, + range_span_ms: Option, } impl PriceClass<'_> { fn same_interface(&self, other: &Self) -> bool { - same_price(&self.priced, &other.priced) + self.ancestor_work == other.ancestor_work + && self.sketch_ids == other.sketch_ids + && self.sketch_sources == other.sketch_sources + && self.range_span_ms == other.range_span_ms + && same_price(&self.priced, &other.priced) && self.lookback == other.lookback && self.sketches == other.sketches && self.raw_source == other.raw_source @@ -36,6 +44,45 @@ impl PriceClass<'_> { /// Exact scan-lookback equality remains required: max-width sharing needs an /// explicit selected-consumer witness. Cyclic catalogs use bounded local /// interface propagation; selected physical graphs must still be acyclic. +struct InputTuples { + choices: Vec>, + indices: Vec, + done: bool, +} +impl InputTuples { + fn new(choices: Vec>) -> Self { + Self { + indices: vec![0; choices.len()], + done: choices.iter().any(Vec::is_empty), + choices, + } + } +} +impl Iterator for InputTuples { + type Item = Vec; + fn next(&mut self) -> Option { + if self.done { + return None; + } + let tuple = self + .choices + .iter() + .zip(&self.indices) + .map(|(choices, index)| choices[*index]) + .collect(); + self.done = true; + for position in (0..self.indices.len()).rev() { + self.indices[position] += 1; + if self.indices[position] < self.choices[position].len() { + self.done = false; + break; + } + self.indices[position] = 0; + } + Some(tuple) + } +} + pub(in crate::search) fn extract_price_options( configured: &CandidateConfigurationInventory, context: &CandidatePlanningContext<'_>, @@ -104,6 +151,7 @@ pub(in crate::search) fn extract_price_options( let mut compactions = vec![0.0; count]; let mut classes: Vec>> = vec![Vec::new(); count]; let mut total = 0; + let mut total_bindings = 0usize; // Every legal unary DAG path has at most N candidates. N monotone passes // therefore cover every supported simple path even in a cyclic catalog. // Walk-derived bindings are harmless: the solver enforces original-node @@ -115,110 +163,275 @@ pub(in crate::search) fn extract_price_options( if nodes[id].is_none() { continue; } - let mut supplier_counts = Vec::new(); - if let Some(input) = configured.inventory.graph.candidates[id] - .producer_requirements - .first() - { + let mut inputs = Vec::new(); + for input in &configured.inventory.graph.candidates[id].producer_requirements { + let mut choices = Vec::new(); for producer in &input.compatible_producers { check_deadline(context.deadline)?; - if producer.0 != id { - supplier_counts.push((Some(*producer), classes[producer.0].len())); + if producer.0 == id { + continue; + } + for price_option in 0..classes[producer.0].len() { + choices.push(SupplierPriceBinding { + producer: *producer, + price_option, + }); } } - } else { - supplier_counts.push((None, 1)); + inputs.push(choices); } - // Stream local bindings instead of allocating a potentially enormous - // supplier-by-option product before the deadline can be checked. - let inputs = supplier_counts.into_iter().flat_map(|(producer, count)| { - (0..count).map(move |price_option| { - producer.map(|producer| SupplierPriceBinding { - producer, - price_option, - }) - }) - }); - for binding in inputs { - for &lookback in &all_lookbacks[id] { - check_deadline(context.deadline)?; - if let Some(binding) = binding { - let supplier = binding.producer.0; - let source = &classes[supplier][binding.price_option]; - cache[supplier] = Some(source.priced.clone()); - lookbacks[supplier] = source.lookback; - sketch_counts[supplier] = source.sketches; - raw_sources[supplier] = source.raw_source.clone(); - closure_work[supplier] = source.work; - worker_counts[supplier] = source.workers; - compactions[supplier] = source.compaction; - } - lookbacks[id] = lookback; - cache[id] = None; - visited[id] = false; - suppliers[id].clear(); - sketch_counts[id] = 0; - raw_sources[id] = None; - closure_work[id] = WorkBounds::default(); - worker_counts[id] = None; - compactions[id] = 0.0; - price( - id, - configured, - context, - &nodes, - &lookbacks, - &mut cache, - &mut visited, - &mut visiting, - &mut suppliers, - &mut sketch_counts, - &mut raw_sources, - &mut closure_work, - &mut worker_counts, - &mut compactions, - delta, - binding.map(|binding| binding.producer), - )?; - let priced = cache[id].take(); - if let Some(binding) = binding { - cache[binding.producer.0] = None; - } - let Some(priced) = priced else { continue }; - let candidate = PriceClass { - priced, - lookback, - sketches: sketch_counts[id], - raw_source: raw_sources[id].clone(), - // Query-time closure work never determines an ingestion - // worker count. Keeping it would distinguish every traversal - // of an identity cycle despite identical local interfaces. - work: if nodes[id].as_ref().unwrap().output_state.timing - == ExecutionTiming::IngestionTime + let node = nodes[id].as_ref().unwrap(); + let spans: Vec> = if matches!( + node.payload, + asap_types::ir::Operator::ASAP(ASAPOp::PaneChain { .. }) + ) { + let mut spans: Vec<_> = context + .entries + .iter() + .map(|entry| Some(entry.range_evaluation.map_or(0, |range| range.span_ms))) + .collect(); + spans.sort(); + spans.dedup(); + spans + } else { + vec![None] + }; + for binding in InputTuples::new(inputs) { + for range_span_ms in &spans { + for &lookback in &all_lookbacks[id] { + check_deadline(context.deadline)?; + let mut selected_sources = std::collections::BTreeMap::new(); + let mut ancestor_work = std::collections::BTreeMap::new(); + let mut sketch_ids = std::collections::BTreeSet::new(); + let mut sketch_sources = std::collections::BTreeMap::new(); + let mut compatible = true; + for binding in &binding { + if selected_sources + .insert(binding.producer.0, binding.price_option) + .is_some_and(|prior| prior != binding.price_option) + { + compatible = false; + break; + } + let source = &classes[binding.producer.0][binding.price_option]; + for (ancestor, work) in &source.ancestor_work { + if ancestor_work + .insert(*ancestor, *work) + .is_some_and(|prior| prior != *work) + { + compatible = false; + } + } + sketch_ids.extend(source.sketch_ids.iter().copied()); + for (sketch, proof) in &source.sketch_sources { + if sketch_sources + .insert(*sketch, proof.clone()) + .is_some_and(|prior| prior != *proof) + { + compatible = false; + } + } + } + if !compatible || ancestor_work.contains_key(&id) { + continue; + } + let upstream_sketches = sketch_ids.len(); + if matches!( + &node.payload, + asap_types::ir::Operator::ASAP( + ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(..), + .. + } | ASAPOp::PaneChain { + family: FieldDataType::Sketch(..), + .. + } + ) + ) { + sketch_ids.insert(id); + sketch_sources.insert( + id, + binding.first().and_then(|input| { + classes[input.producer.0][input.price_option] + .raw_source + .clone() + }), + ); + } + if !sketch_ids.is_empty() { + let Some(delta) = delta else { continue }; + let divided = delta / sketch_ids.len() as f64; + if !sketch_ids.iter().all(|sketch| { + let template = + configured.inventory.templates[*sketch].as_ref().unwrap(); + if requires_source_sizing(template) { + let Some(source) = + sketch_sources.get(sketch).and_then(|s| s.as_ref()) + else { + return false; + }; + let Some(bound) = context + .data + .sources + .get(source) + .and_then(|facts| facts.max_distinct_per_evaluation) + else { + return false; + }; + let Some( + ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(kind, _), + .. + } + | ASAPOp::PaneChain { + family: FieldDataType::Sketch(kind, _), + .. + }, + ) = template.asap() + else { + return false; + }; + let SketchParams::Hll { + precision, + relative_error, + } = kind.params() + else { + return false; + }; + hll_size_certified(*precision, *relative_error, divided, bound) + } else { + stable_size(template, Some(divided)) + } + }) { + continue; + } + } + let mut upstream_work = WorkBounds::default(); + for work in ancestor_work.values() { + upstream_work.minimum = (upstream_work.minimum + f64::from_bits(*work)) + .next_down() + .max(0.0); + upstream_work.maximum = + (upstream_work.maximum + f64::from_bits(*work)).next_up(); + } + for binding in &binding { + let supplier = binding.producer.0; + let source = &classes[supplier][binding.price_option]; + cache[supplier] = Some(source.priced.clone()); + lookbacks[supplier] = source.lookback; + sketch_counts[supplier] = source.sketches; + raw_sources[supplier] = source.raw_source.clone(); + closure_work[supplier] = source.work; + worker_counts[supplier] = source.workers; + compactions[supplier] = source.compaction; + } + lookbacks[id] = lookback; + cache[id] = None; + visited[id] = false; + suppliers[id].clear(); + sketch_counts[id] = 0; + raw_sources[id] = None; + closure_work[id] = WorkBounds::default(); + worker_counts[id] = None; + compactions[id] = 0.0; + let input_context = PriceInputs { + suppliers: binding.iter().map(|binding| binding.producer).collect(), + upstream_work, + upstream_sketches, + range_span_ms: *range_span_ms, + }; + price( + id, + configured, + context, + &nodes, + &lookbacks, + &mut cache, + &mut visited, + &mut visiting, + &mut suppliers, + &mut sketch_counts, + &mut raw_sources, + &mut closure_work, + &mut worker_counts, + &mut compactions, + delta, + Some(&input_context), + )?; + let priced = cache[id].take(); + for binding in &binding { + cache[binding.producer.0] = None; + } + let Some(priced) = priced else { continue }; + if matches!( + &node.payload, + asap_types::ir::Operator::ASAP( + ASAPOp::SummaryAgg { + family: FieldDataType::Sketch(..), + .. + } | ASAPOp::PaneChain { + family: FieldDataType::Sketch(..), + .. + } + ) + ) { + sketch_ids.insert(id); + } + if node.output_state.timing == ExecutionTiming::IngestionTime { + ancestor_work.insert(id, priced.work_ms.to_bits()); + } else { + ancestor_work.clear(); + } + let candidate = PriceClass { + priced, + lookback, + sketches: sketch_counts[id], + raw_source: raw_sources[id].clone(), + // Query-time closure work never determines an ingestion + // worker count. Keeping it would distinguish every traversal + // of an identity cycle despite identical local interfaces. + work: if nodes[id].as_ref().unwrap().output_state.timing + == ExecutionTiming::IngestionTime + { + closure_work[id] + } else { + WorkBounds::default() + }, + workers: worker_counts[id], + compaction: compactions[id], + bindings: vec![binding.clone()], + ancestor_work, + sketch_ids, + sketch_sources, + range_span_ms: *range_span_ms, + }; + if let Some(existing) = classes[id] + .iter_mut() + .find(|known| known.same_interface(&candidate)) { - closure_work[id] + for binding in candidate.bindings { + if !existing.bindings.contains(&binding) { + total_bindings += 1; + if total_bindings > 200_000 { + return Err(DirectProfileError::Unsupported("local pricing domain exceeds 200000 correlated input tuples".into())); + } + existing.bindings.push(binding); + } + } } else { - WorkBounds::default() - }, - workers: worker_counts[id], - compaction: compactions[id], - bindings: binding.into_iter().collect(), - }; - if let Some(existing) = classes[id] - .iter_mut() - .find(|known| known.same_interface(&candidate)) - { - for binding in candidate.bindings { - if !existing.bindings.contains(&binding) { - existing.bindings.push(binding); + total_bindings += candidate.bindings.len(); + if total_bindings > 200_000 { + return Err(DirectProfileError::Unsupported( + "local pricing domain exceeds 200000 correlated input tuples" + .into(), + )); } + total += 1; + if total > MAXIMUM_PRICE_OPTIONS { + return Err(DirectProfileError::Unsupported(format!("local pricing domain exceeds {MAXIMUM_PRICE_OPTIONS} interface options"))); + } + classes[id].push(candidate); } - } else { - total += 1; - if total > MAXIMUM_PRICE_OPTIONS { - return Err(DirectProfileError::Unsupported(format!("local pricing domain exceeds {MAXIMUM_PRICE_OPTIONS} interface options"))); - } - classes[id].push(candidate); } } } @@ -241,7 +454,8 @@ pub(in crate::search) fn extract_price_options( class.compaction, context, )?, - compatible_suppliers: class.bindings.clone(), + compatible_input_tuples: class.bindings.clone(), + required_range_span_ms: class.range_span_ms, }) }) .collect::, DirectProfileError>>() @@ -253,6 +467,11 @@ pub(in crate::search) fn extract_price_options( }; let request_ms = settings.measured_operation_costs.ns_per_evaluation * 1e-6; Ok(Some(JointPriceProfile { + query_range_spans_ms: context + .entries + .iter() + .map(|entry| entry.range_evaluation.map_or(0, |range| range.span_ms)) + .collect(), schedules, query_rate: rate, request_latency_seconds: request_ms / 1000.0, diff --git a/crates/plan-selection/src/search/milp/direct_cost.rs b/crates/plan-selection/src/search/milp/direct_cost.rs index 5e4970fc6..b9a2fd125 100644 --- a/crates/plan-selection/src/search/milp/direct_cost.rs +++ b/crates/plan-selection/src/search/milp/direct_cost.rs @@ -1,16 +1,19 @@ //! Exact objective encoding for certified query-time and ingestion profiles. -//! Fixed evaluation schedules may be equal or use a bounded union-rate table; -//! maintained and retained states carry separate standing resource charges. +//! Equal schedules have a compact fast path; mixed recurrence/range demand +//! uses exact evaluator atoms. Retained state carries separate standing charges. //! Profiles are supplied by physical costing; this module never infers prices //! from a logical template or treats accuracy as a monetary penalty. use good_lp::{constraint, variable, Constraint, Expression, ProblemVariables, Variable}; use std::collections::BTreeMap; -#[path = "direct_fixed_schedule.rs"] -mod fixed_schedules; -#[path = "direct_schedule.rs"] -mod schedules; -pub(crate) use fixed_schedules::{FixedQuerySchedule, FixedScheduleLattice}; +#[path = "general_schedule.rs"] +mod general_schedule; +#[path = "general_schedule_rates.rs"] +mod general_schedule_rates; +#[cfg(test)] +#[path = "schedule_selection_tests.rs"] +mod schedule_selection_tests; +pub(crate) use general_schedule_rates::QueryScheduleRates; use crate::search::candidates::graph::*; @@ -28,6 +31,7 @@ const MINIMUM_NORMALIZED_LATENCY_TERM: f64 = 1e-6; #[derive(Debug, Clone, Default)] pub(crate) struct NodeCostProfile { + pub per_request: bool, pub execution_cost: f64, pub ingestion: bool, pub standing_cost_per_second: f64, @@ -49,15 +53,6 @@ impl NodeCostProfile { } } -/// Exact evaluator demand for each nonempty selected query subset. Only small -/// heterogeneous domains are admitted; this is not a workload-plan product. -#[derive(Debug, Clone)] -pub(crate) struct QueryScheduleRates { - pub per_query_rates: Vec, - pub materialized_subset_rates: Vec, - pub compact_fixed: Option, -} - #[derive(Debug, Clone)] pub(crate) struct DirectCostProfile { pub nodes: Vec, @@ -81,6 +76,7 @@ impl DirectCostProfile { || !valid(self.request_cost) || self.nodes.iter().any(|n| { !valid(n.execution_cost) + || (n.per_request && (!n.materialized || n.ingestion)) || !valid(n.standing_cost_per_second) || (n.ingestion && n.held_memory_cost_per_second != 0.0) || !valid(n.compaction_seconds) @@ -126,47 +122,14 @@ impl DirectCostProfile { )); } if let Some(schedule) = &self.schedules { - let queries = graph.query_roots.len(); - if let Some(lattice) = &schedule.compact_fixed { - lattice.validate(schedule, queries)?; - } else { - if queries > 3 { - return Err(CandidateSelectionError::UnsupportedDirectCost( - "heterogeneous query schedule table supports at most three output queries" - .into(), - )); - } - if !schedule.per_query_rates.iter().sum::().is_finite() - || schedule.per_query_rates.len() != queries - || schedule.materialized_subset_rates.len() != (1usize << queries) - || schedule.materialized_subset_rates[0] != 0.0 - || schedule - .per_query_rates - .iter() - .any(|rate| !rate.is_finite() || *rate <= 0.0) - || schedule - .materialized_subset_rates - .iter() - .skip(1) - .any(|rate| !rate.is_finite() || *rate <= 0.0) - || schedule - .per_query_rates - .iter() - .enumerate() - .any(|(q, rate)| *rate != schedule.materialized_subset_rates[1 << q]) - { - return Err(CandidateSelectionError::InvalidGraph( - "invalid direct query-schedule rates".into(), - )); - } - } + schedule.validate(graph.query_roots.len())?; } if let Some(schedule) = &self.schedules { - let total_rate: f64 = schedule.per_query_rates.iter().sum(); + let total_rate: f64 = schedule.per_query_rates().iter().sum(); if schedule - .per_query_rates + .per_query_rates() .iter() - .any(|rate| rate / total_rate < MINIMUM_QUERY_RATE_WEIGHT) + .any(|rate| *rate > 0.0 && rate / total_rate < MINIMUM_QUERY_RATE_WEIGHT) { return Err(CandidateSelectionError::UnsupportedDirectCost( "query-rate weights are below the supported weighted-lifetime numerical range" @@ -208,10 +171,11 @@ impl DirectCostProfile { .filter(|term| *term > 0.0) .reduce(f64::min); let minimum_rate_weight = self.schedules.as_ref().map_or(1.0, |schedule| { - let total: f64 = schedule.per_query_rates.iter().sum(); + let total: f64 = schedule.per_query_rates().iter().sum(); schedule - .per_query_rates + .per_query_rates() .iter() + .filter(|rate| **rate > 0.0) .map(|rate| rate / total) .fold(1.0, f64::min) }); @@ -268,7 +232,7 @@ impl DirectCostProfile { .collect(); let rates = self.schedules.as_ref().map_or_else( || vec![self.query_rate; graph.query_roots.len()], - |schedule| schedule.per_query_rates.clone(), + |schedule| schedule.per_query_rates().to_vec(), ); let union_rate = |id: usize| { if self.nodes[id].ingestion { @@ -277,15 +241,11 @@ impl DirectCostProfile { let Some(schedule) = &self.schedules else { return self.query_rate; }; - if let Some(lattice) = &schedule.compact_fixed { - return lattice.union_rate(reached.iter().map(|row| row[id])); + if self.nodes[id].per_request { + schedule.request_union_rate(reached.iter().map(|row| row[id])) + } else { + schedule.union_rate(reached.iter().map(|row| row[id])) } - let mask = reached - .iter() - .enumerate() - .filter(|(_, row)| row[id]) - .fold(0usize, |mask, (q, _)| mask | (1 << q)); - schedule.materialized_subset_rates[mask] }; let mut execution_rates = vec![0.0; self.nodes.len()]; let boundaries = selected @@ -335,7 +295,7 @@ impl DirectCostProfile { .sum::() / denominator } else { - 0.0 + readers.iter().copied().fold(0.0, f64::max) } }; total += node.standing_cost_per_second @@ -427,23 +387,8 @@ impl DirectCostProfile { let latency_scale = self.maximum_latency(graph).max(f64::MIN_POSITIVE); let rates = self.schedules.as_ref().map_or_else( || vec![self.query_rate], - |schedule| { - if schedule.compact_fixed.is_some() { - return schedule.per_query_rates.clone(); - } - schedule - .materialized_subset_rates - .iter() - .copied() - .filter(|rate| *rate > 0.0) - .collect() - }, + |schedule| schedule.coefficient_rates(), ); - let denominator_scale = self - .schedules - .as_ref() - .and_then(|s| s.compact_fixed.as_ref()) - .map_or(1.0, FixedScheduleLattice::maximum_denominator_scale); self.nodes.iter().flat_map(move |node| { let mut coefficients = Vec::with_capacity(rates.len() * 2 + 1); let node_rates = if node.ingestion { @@ -456,13 +401,6 @@ impl DirectCostProfile { node.execution_cost * rate, node.held_memory_cost_per_second * rate * latency_scale, ]); - // Compact mean-lifetime selectors multiply by total weight / - // selected weight; include that largest possible coefficient. - if !node.materialized && denominator_scale > 1.0 { - coefficients.push( - node.held_memory_cost_per_second * rate * latency_scale * denominator_scale, - ); - } } coefficients.push(node.standing_cost_per_second); coefficients.push(node.raw_retention.as_ref().map_or(0.0, |(_, cost)| *cost)); @@ -647,7 +585,7 @@ pub(super) fn encode( check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, ) -> Result { if let Some(schedule) = &profile.schedules { - return schedules::encode( + return general_schedule::encode( graph, profile, schedule, @@ -863,6 +801,7 @@ mod tests { request_cost: 7.0, nodes: vec![ NodeCostProfile { + per_request: false, ingestion: false, standing_cost_per_second: 0.0, execution_cost: 2.0, @@ -922,6 +861,7 @@ mod tests { ..Default::default() }, NodeCostProfile { + per_request: false, ingestion: false, standing_cost_per_second: 0.0, execution_cost: 1.0, @@ -970,6 +910,7 @@ mod tests { request_cost: 7.0, nodes: vec![ NodeCostProfile { + per_request: false, ingestion: false, standing_cost_per_second: 0.0, execution_cost: materialized_build_cost, @@ -1162,11 +1103,7 @@ mod tests { fn two_ingestion_boundaries_recompute_their_shared_input() { let (graph, mut profile) = ingestion_profile(true, 1.0); assert!((solve(&graph, &profile) - 29.4).abs() < 1e-6); - profile.schedules = Some(QueryScheduleRates { - compact_fixed: None, - per_query_rates: vec![1.0, 2.0], - materialized_subset_rates: vec![0.0, 1.0, 2.0, 3.0], - }); + profile.schedules = Some(schedule_selection_tests::rate_schedule(&[1.0, 2.0])); let proposal = solve_direct_cost_proposal_with_metrics( &graph, &[], @@ -1327,11 +1264,7 @@ mod tests { // refresh; distinct fixed intervals add under the evaluator's policy. for (rates, union, expected) in [(vec![2.0, 2.0], 2.0, 74.4), (vec![1.0, 2.0], 3.0, 83.0)] { if rates[0] != rates[1] { - profile.schedules = Some(QueryScheduleRates { - compact_fixed: None, - per_query_rates: rates.clone(), - materialized_subset_rates: vec![0.0, rates[0], rates[1], union], - }); + profile.schedules = Some(schedule_selection_tests::rate_schedule(&rates)); } let weighted_latency = rates[0] * 3.6 + rates[1] * 4.6; let rate_sum: f64 = rates.iter().sum(); @@ -1458,11 +1391,7 @@ mod tests { }; for rates in [[1.0, 1.0], [1.0, 2.0]] { if rates[0] != rates[1] { - profile.schedules = Some(QueryScheduleRates { - compact_fixed: None, - per_query_rates: rates.to_vec(), - materialized_subset_rates: vec![0.0, rates[0], rates[1], rates[0] + rates[1]], - }); + profile.schedules = Some(schedule_selection_tests::rate_schedule(&rates)); } let expected = (0..4) .map(|mask| { diff --git a/crates/plan-selection/src/search/milp/direct_fixed_schedule.rs b/crates/plan-selection/src/search/milp/direct_fixed_schedule.rs deleted file mode 100644 index 4174e57d0..000000000 --- a/crates/plan-selection/src/search/milp/direct_fixed_schedule.rs +++ /dev/null @@ -1,567 +0,0 @@ -//! Compact fixed-interval accounting follows the evaluator's policy: distinct -//! intervals add, distinct known phases add, unknown phase joins a known phase. -//! Weighted mean lifetimes use reachable integer rate totals, not reader subsets. -use super::*; - -const MAXIMUM_RATE_TICKS: u64 = 4096; - -#[derive(Debug, Clone, Copy)] -pub(crate) struct FixedQuerySchedule { - pub interval_ms: u32, - pub phase_ms: Option, -} - -#[derive(Debug, Clone)] -pub(crate) struct FixedScheduleLattice { - queries: Vec, - weights: Vec, -} - -fn gcd(mut left: u64, mut right: u64) -> u64 { - while right != 0 { - (left, right) = (right, left % right); - } - left -} - -impl QueryScheduleRates { - pub(crate) fn from_fixed_schedules( - queries: Vec, - ) -> Result { - if queries.is_empty() || queries.iter().any(|q| q.interval_ms == 0) { - return Err(CandidateSelectionError::InvalidGraph( - "empty or zero-interval fixed schedule".into(), - )); - } - let mut common = 1u64; - for query in &queries { - let interval = u64::from(query.interval_ms); - common = (common / gcd(common, interval)) - .checked_mul(interval) - .ok_or_else(|| { - CandidateSelectionError::UnsupportedDirectCost( - "fixed schedule lattice arithmetic overflow".into(), - ) - })?; - } - let mut weights: Vec<_> = queries - .iter() - .map(|q| common / u64::from(q.interval_ms)) - .collect(); - let divisor = weights.iter().copied().reduce(gcd).unwrap(); - for weight in &mut weights { - *weight /= divisor; - } - let total = weights - .iter() - .try_fold(0u64, |sum, weight| sum.checked_add(*weight)); - if total.is_none_or(|total| total > MAXIMUM_RATE_TICKS) { - return Err(CandidateSelectionError::UnsupportedDirectCost(format!( - "fixed schedule rate lattice exceeds {MAXIMUM_RATE_TICKS} total ticks" - ))); - } - Ok(Self { - per_query_rates: queries - .iter() - .map(|q| 1000.0 / f64::from(q.interval_ms)) - .collect(), - materialized_subset_rates: Vec::new(), - compact_fixed: Some(FixedScheduleLattice { - queries, - weights: weights.into_iter().map(|w| w as usize).collect(), - }), - }) - } -} - -impl FixedScheduleLattice { - pub(super) fn maximum_denominator_scale(&self) -> f64 { - self.weights.iter().sum::() as f64 - } - - pub(super) fn validate( - &self, - schedule: &QueryScheduleRates, - queries: usize, - ) -> Result<(), CandidateSelectionError> { - let expected = QueryScheduleRates::from_fixed_schedules(self.queries.clone())?; - if queries != self.queries.len() - || expected.per_query_rates != schedule.per_query_rates - || !schedule.materialized_subset_rates.is_empty() - || expected.compact_fixed.as_ref().unwrap().weights != self.weights - { - return Err(CandidateSelectionError::InvalidGraph( - "inconsistent compact fixed schedule".into(), - )); - } - Ok(()) - } - - pub(super) fn union_rate(&self, selected: impl Iterator) -> f64 { - let mut intervals = BTreeMap::)>::new(); - for (query, selected) in self.queries.iter().zip(selected) { - if !selected { - continue; - } - let (unknown, phases) = intervals.entry(query.interval_ms).or_default(); - if let Some(phase) = query.phase_ms { - phases.insert(phase % u64::from(query.interval_ms)); - } else { - *unknown = true; - } - } - intervals - .into_iter() - .map(|(interval, (unknown, phases))| { - phases.len().max(usize::from(unknown)) as f64 * 1000.0 / f64::from(interval) - }) - .sum() - } -} - -fn logical_or( - variables: &mut ProblemVariables, - constraints: &mut Vec, - inputs: &[Variable], -) -> Option { - if inputs.is_empty() { - return None; - } - if inputs.len() == 1 { - return Some(inputs[0]); - } - let value = variables.add(variable().min(0).max(1)); - for input in inputs { - constraints.push(constraint!(value >= *input)); - } - constraints.push(constraint!( - value <= inputs.iter().copied().sum::() - )); - Some(value) -} - -fn logical_and( - variables: &mut ProblemVariables, - constraints: &mut Vec, - left: Variable, - right: Variable, -) -> Variable { - let value = variables.add(variable().min(0).max(1)); - constraints.push(constraint!(value <= left)); - constraints.push(constraint!(value <= right)); - constraints.push(constraint!(value >= left + right - 1)); - value -} - -fn union_terms( - lattice: &FixedScheduleLattice, - node: usize, - reach: &[Vec>], - variables: &mut ProblemVariables, - constraints: &mut Vec, -) -> Vec<(Variable, f64)> { - let mut intervals = BTreeMap::, Vec>>::new(); - for (q, schedule) in lattice.queries.iter().enumerate() { - if let Some(reached) = reach[q][node] { - intervals - .entry(schedule.interval_ms) - .or_default() - .entry( - schedule - .phase_ms - .map(|p| p % u64::from(schedule.interval_ms)), - ) - .or_default() - .push(reached); - } - } - let mut terms = Vec::new(); - for (interval, mut phases) in intervals { - let rate = 1000.0 / f64::from(interval); - let unknown = logical_or( - variables, - constraints, - &phases.remove(&None).unwrap_or_default(), - ); - let known: Vec<_> = phases - .values() - .filter_map(|readers| logical_or(variables, constraints, readers)) - .collect(); - terms.extend(known.iter().map(|phase| (*phase, rate))); - if let Some(unknown) = unknown { - if let Some(any_known) = logical_or(variables, constraints, &known) { - let extra = variables.add(variable().min(0).max(1)); - constraints.push(constraint!(extra <= unknown)); - constraints.push(constraint!(extra <= 1.0 - any_known)); - constraints.push(constraint!(extra >= unknown - any_known)); - terms.push((extra, rate)); - } else { - terms.push((unknown, rate)); - } - } - } - terms -} - -#[allow(clippy::too_many_arguments)] -pub(super) fn encode( - graph: &CandidateDependencyGraph, - profile: &DirectCostProfile, - schedule: &QueryScheduleRates, - lattice: &FixedScheduleLattice, - variables: &mut ProblemVariables, - constraints: &mut Vec, - active: &[Variable], - edges: &[(ProducerConsumerConnection, Variable)], - roots: &[(QueryRootAssignment, Variable)], - reach: &[Vec>], - divisor: f64, - check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, -) -> Result { - let count = graph.candidates.len(); - let queries = graph.query_roots.len(); - let scale = profile.maximum_latency(graph).max(f64::MIN_POSITIVE); - let latency = query_latency_expressions( - graph, - profile, - variables, - constraints, - reach, - &check_deadline, - )?; - let mut unions = Vec::with_capacity(count); - for id in 0..count { - check_deadline()?; - unions.push(union_terms(lattice, id, reach, variables, constraints)); - } - let mut demands = vec![Vec::<(Variable, f64)>::new(); count]; - for flow in boundary_flows( - graph, - profile, - variables, - constraints, - active, - edges, - roots, - &check_deadline, - )? { - for (id, used) in flow.reached.into_iter().enumerate() { - check_deadline()?; - let Some(used) = used else { continue }; - if flow.boundary < queries { - demands[id].push((used, schedule.per_query_rates[flow.boundary])); - } else if profile.nodes[flow.boundary - queries].ingestion { - demands[id].push((used, 1.0)); - } else { - for (phase, rate) in &unions[flow.boundary - queries] { - demands[id].push((logical_and(variables, constraints, used, *phase), *rate)); - } - } - } - } - let mut objective = Expression::from(profile.request_cost / divisor); - let mut raw = BTreeMap::<&str, Vec<(usize, f64)>>::new(); - for (id, node) in profile.nodes.iter().enumerate() { - check_deadline()?; - let terms = if node.materialized { - if node.ingestion { - vec![(active[id], 1.0)] - } else { - unions[id].clone() - } - } else { - demands[id].clone() - }; - objective += node.standing_cost_per_second / divisor * active[id]; - objective += - node.execution_cost / divisor * terms.iter().map(|(v, r)| *r * *v).sum::(); - if let Some((source, cost)) = &node.raw_retention { - raw.entry(source).or_default().push((id, *cost)); - } - if node.held_memory_cost_per_second == 0.0 { - continue; - } - let mut lifetime = Expression::from(0.0); - if node.materialized { - let longest = variables.add(variable().min(0).max(1)); - for (q, row) in reach.iter().enumerate() { - if let Some(reached) = row[id] { - constraints.push(constraint!(longest >= latency[q].clone() - (1.0 - reached))); - } - } - for (used, rate) in terms { - lifetime += rate * gated(variables, constraints, longest.into(), used, 1.0); - } - } else { - let possible: Vec<_> = reach - .iter() - .enumerate() - .filter_map(|(q, row)| row[id].map(|r| (q, r))) - .collect(); - let maximum: usize = possible.iter().map(|(q, _)| lattice.weights[*q]).sum(); - if maximum == 0 { - continue; - } - let mut totals = vec![false; maximum + 1]; - totals[0] = true; - let mut accumulated = 0; - for (q, _) in &possible { - let weight = lattice.weights[*q]; - for sum in (0..=accumulated).rev() { - if totals[sum] { - totals[sum + weight] = true; - } - } - accumulated += weight; - } - // Denominator totals need binary selectors: unlike full reader - // subsets, their weighted-sum equality alone permits mixtures. - let selectors: Vec<_> = totals - .iter() - .enumerate() - .filter(|(_, yes)| **yes) - .map(|(total, _)| (total, variables.add(variable().binary()))) - .collect(); - constraints.push(constraint!( - selectors.iter().map(|(_, v)| *v).sum::() == 1 - )); - constraints.push(constraint!( - selectors - .iter() - .map(|(d, v)| *d as f64 * *v) - .sum::() - == possible - .iter() - .map(|(q, r)| lattice.weights[*q] as f64 * *r) - .sum::() - )); - let numerator: Expression = possible - .iter() - .map(|(q, r)| { - lattice.weights[*q] as f64 / maximum as f64 - * gated(variables, constraints, latency[*q].clone(), *r, 1.0) - }) - .sum(); - for (denominator, selector) in selectors.into_iter().filter(|(d, _)| *d > 0) { - check_deadline()?; - let chosen = gated(variables, constraints, numerator.clone(), selector, 1.0); - for (used, rate) in &terms { - lifetime += *rate * maximum as f64 / denominator as f64 - * gated(variables, constraints, chosen.into(), *used, 1.0); - } - } - } - objective += node.held_memory_cost_per_second * scale / divisor * lifetime; - } - for entries in raw.into_values() { - let held = variables.add(variable().min(0)); - for (id, cost) in entries { - constraints.push(constraint!(held >= cost / divisor * active[id])); - } - objective += held; - } - Ok(objective) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::search::milp::model::{ - solve_direct_cost_proposal_with_metrics, CandidateSolveMetrics, - }; - use asap_logical_asap_optimization::memo::MemoGroupId; - use asap_types::workload::{QueryRecurrence, RepeatedDemand, RepetitionInterval}; - - fn node(input: Option) -> PhysicalCandidate { - PhysicalCandidate { - group: MemoGroupId(0), - local_cost: 0.0, - producer_requirements: input - .into_iter() - .map(|id| ProducerRequirement { - compatible_producers: vec![PhysicalCandidateId(id)].into(), - }) - .collect(), - } - } - - // All reader subsets agree with the existing interval accounting policy; - // the compact representation never constructs a 2^Q rate table. - #[test] - fn compact_fixed_rates_match_evaluator_subsets() { - let intervals = [1000, 2000, 1000, 4000, 2000, 1000]; - let fixed: Vec<_> = intervals - .iter() - .map(|ms| FixedQuerySchedule { - interval_ms: *ms, - phase_ms: None, - }) - .collect(); - let schedule = QueryScheduleRates::from_fixed_schedules(fixed).unwrap(); - let recurrences: Vec<_> = intervals - .iter() - .map(|ms| { - QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(*ms))) - }) - .collect(); - for mask in 0..1usize << intervals.len() { - let expected = crate::cost::execution_demand::evaluation_rate( - recurrences - .iter() - .enumerate() - .filter(|(q, _)| mask & (1 << q) != 0) - .map(|(_, r)| (r, None)), - 3600.0, - ) - .unwrap(); - let actual = schedule - .compact_fixed - .as_ref() - .unwrap() - .union_rate((0..intervals.len()).map(|q| mask & (1 << q) != 0)); - assert_eq!(actual, expected); - } - assert!(schedule.materialized_subset_rates.is_empty()); - let large = QueryScheduleRates::from_fixed_schedules(vec![ - FixedQuerySchedule { - interval_ms: 1000, - phase_ms: None - }; - 128 - ]) - .unwrap(); - assert_eq!(large.compact_fixed.unwrap().weights.len(), 128); - assert!(matches!( - QueryScheduleRates::from_fixed_schedules(vec![ - FixedQuerySchedule { - interval_ms: 1, - phase_ms: None - }, - FixedQuerySchedule { - interval_ms: 100000, - phase_ms: None - }, - ]), - Err(CandidateSelectionError::UnsupportedDirectCost(_)) - )); - } - - // Known phases add; unknown phase adds only when no known phase is active. - #[test] - fn compact_phase_activation_is_exact_for_every_subset() { - let schedule = QueryScheduleRates::from_fixed_schedules(vec![ - FixedQuerySchedule { - interval_ms: 1000, - phase_ms: None, - }, - FixedQuerySchedule { - interval_ms: 1000, - phase_ms: Some(0), - }, - FixedQuerySchedule { - interval_ms: 1000, - phase_ms: Some(500), - }, - FixedQuerySchedule { - interval_ms: 1000, - phase_ms: Some(1500), - }, - ]) - .unwrap(); - for mask in 0..16 { - let mut variables = ProblemVariables::new(); - let mut constraints = Vec::new(); - let reach: Vec<_> = (0..4) - .map(|q| { - let value = variables.add(variable().binary()); - constraints.push(constraint!(value == f64::from(mask & (1 << q) != 0))); - vec![Some(value)] - }) - .collect(); - let lattice = schedule.compact_fixed.as_ref().unwrap(); - let terms = union_terms(lattice, 0, &reach, &mut variables, &mut constraints); - let expression: Expression = terms.iter().map(|(v, r)| *r * *v).sum(); - use good_lp::{highs, Solution, SolverModel}; - let solution = variables - .minimise(-expression.clone()) - .using(highs) - .with_all(constraints) - .solve() - .unwrap(); - let expected = lattice.union_rate((0..4).map(|q| mask & (1 << q) != 0)); - assert_eq!(solution.eval(&expression), expected); - } - } - - // Five readers share a query-materialized build. Upstream lifetime uses - // their weighted mean; the build uses their maximum and union refresh rate. - #[test] - fn compact_schedule_prices_shared_refresh_and_weighted_memory() { - let intervals = [1000, 2000, 1000, 4000, 2000]; - let schedule = QueryScheduleRates::from_fixed_schedules( - intervals - .iter() - .map(|ms| FixedQuerySchedule { - interval_ms: *ms, - phase_ms: None, - }) - .collect(), - ) - .unwrap(); - let mut candidates = vec![node(None), node(Some(0))]; - candidates.extend((0..5).map(|_| node(Some(1)))); - let graph = CandidateDependencyGraph { - candidates, - query_roots: (2..7) - .map(|id| QueryRootCandidates { - compatible_candidates: vec![PhysicalCandidateId(id)], - }) - .collect(), - ..Default::default() - }; - let mut nodes = vec![ - NodeCostProfile { - execution_cost: 2.0, - work_seconds: 1.0, - held_memory_cost_per_second: 3.0, - ..Default::default() - }, - NodeCostProfile { - execution_cost: 4.0, - work_seconds: 2.0, - held_memory_cost_per_second: 2.0, - materialized: true, - ..Default::default() - }, - ]; - nodes.extend((0..5).map(|q| NodeCostProfile { - execution_cost: 1.0, - work_seconds: q as f64 + 1.0, - ..Default::default() - })); - let rates = schedule.per_query_rates.clone(); - let union = 1.75; - let weighted: f64 = rates - .iter() - .enumerate() - .map(|(q, r)| r * (q as f64 + 4.0)) - .sum(); - let sum: f64 = rates.iter().sum(); - let expected = union * (6.0 + 3.0 * weighted / sum + 2.0 * 8.0) + sum; - let profile = DirectCostProfile { - nodes, - schedules: Some(schedule), - query_rate: 1.0, - request_latency_seconds: 0.0, - request_cost: 0.0, - }; - let mut metrics = CandidateSolveMetrics::default(); - let proposal = - solve_direct_cost_proposal_with_metrics(&graph, &[], None, &mut metrics, &profile) - .unwrap() - .unwrap(); - assert!((proposal.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); - assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-6); - assert!(metrics.peak_variables < 5000); - } -} diff --git a/crates/plan-selection/src/search/milp/direct_price_options.rs b/crates/plan-selection/src/search/milp/direct_price_options.rs index e734d49d2..8d2a90752 100644 --- a/crates/plan-selection/src/search/milp/direct_price_options.rs +++ b/crates/plan-selection/src/search/milp/direct_price_options.rs @@ -12,7 +12,8 @@ pub(crate) struct SupplierPriceBinding { #[derive(Debug, Clone)] pub(crate) struct CandidatePriceOption { pub profile: NodeCostProfile, - pub compatible_suppliers: Vec, + pub compatible_input_tuples: Vec>, + pub required_range_span_ms: Option, } #[derive(Debug, Clone)] @@ -22,12 +23,13 @@ pub(crate) struct JointPriceProfile { pub request_latency_seconds: f64, pub request_cost: f64, pub candidates: Vec>, + pub query_range_spans_ms: Vec, } use super::direct_cost::DirectCostProfile; use super::model::{CandidateProposal, CandidateSolveMetrics, ObservedAssignment}; use crate::search::candidates::graph::*; -use good_lp::{constraint, Constraint, Expression, Variable}; +use good_lp::{constraint, variable, Constraint, Expression, ProblemVariables, Variable}; /// Internal option nodes share an original physical identity. At most one may /// activate, so two readers cannot obtain incompatible views of one producer. @@ -36,6 +38,9 @@ pub(super) struct PriceOptionProjection<'a> { origins: Vec, options: Vec>, observations: &'a [ObservedAssignment], + input_tuples: Vec<(PhysicalCandidateId, Vec>)>, + range_contexts: Vec<(PhysicalCandidateId, u64)>, + query_spans: &'a [u64], } impl PriceOptionProjection<'_> { @@ -81,6 +86,77 @@ impl PriceOptionProjection<'_> { } } + pub(super) fn constrain_input_tuples( + &self, + variables: &mut ProblemVariables, + active: &[Variable], + edges: &[(ProducerConsumerConnection, Variable)], + constraints: &mut Vec, + check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, + ) -> Result<(), CandidateSelectionError> { + let mut by_consumer = std::collections::BTreeMap::<_, Vec<_>>::new(); + for (edge, selected) in edges { + check_deadline()?; + by_consumer + .entry(edge.consumer) + .or_default() + .push((edge, selected)); + } + for (consumer, tuples) in &self.input_tuples { + check_deadline()?; + let mut selectors = Vec::new(); + let mut support = std::collections::BTreeMap::<_, Expression>::new(); + for tuple in tuples { + check_deadline()?; + let selector = variables.add(variable().binary()); + selectors.push(selector); + for (position, producer) in tuple.iter().enumerate() { + *support + .entry((position, *producer)) + .or_insert_with(|| Expression::from(0.0)) += selector; + } + } + constraints.push(constraint!( + selectors.iter().copied().sum::() == active[consumer.0] + )); + for (edge, selected) in by_consumer.get(consumer).into_iter().flatten() { + check_deadline()?; + let matches = support + .get(&(edge.requirement, edge.producer)) + .cloned() + .unwrap_or_else(|| Expression::from(0.0)); + constraints.push(constraint!(**selected == matches)); + } + } + Ok(()) + } + + pub(super) fn constrain_reader_contexts( + &self, + active: &[Variable], + reach: &[Vec>], + constraints: &mut Vec, + check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, + ) -> Result<(), CandidateSelectionError> { + for (candidate, span) in &self.range_contexts { + check_deadline()?; + let mut witnesses = Expression::from(0.0); + for (query, row) in reach.iter().enumerate() { + check_deadline()?; + if let Some(reached) = row[candidate.0] { + if self.query_spans[query] > *span { + constraints.push(constraint!(reached == 0)); + } + if self.query_spans[query] == *span { + witnesses += reached; + } + } + } + constraints.push(constraint!(active[candidate.0] <= witnesses)); + } + Ok(()) + } + /// A rejected physical DAG remains rejected across all of its price-option /// encodings. Cuts refer to sums over the original activation/edge identity. pub(super) fn constrain_observations( @@ -163,6 +239,9 @@ pub(crate) fn solve_joint_price_proposal_with_metrics( origins: Vec::new(), options: Vec::new(), observations, + input_tuples: Vec::new(), + range_contexts: Vec::new(), + query_spans: &profile.query_range_spans_ms, }; let mut expanded = CandidateDependencyGraph::default(); let mut nodes = Vec::new(); @@ -186,38 +265,65 @@ pub(crate) fn solve_joint_price_proposal_with_metrics( )); } let required = &graph.candidates[id].producer_requirements; - if !options.is_empty() && required.len() > 1 { - return Err(invalid( - "joint unary price profile has multiple required inputs", - )); - } for (index, option) in options.iter().enumerate() { - if required.is_empty() && !option.compatible_suppliers.is_empty() { - return Err(invalid("source price option has supplier bindings")); + let consumer = projection.options[id][index]; + if let Some(span) = option.required_range_span_ms { + if profile.query_range_spans_ms.len() != graph.query_roots.len() { + return Err(invalid("range contexts do not match query roots")); + } + projection.range_contexts.push((consumer, span)); } - if let Some(input) = required.first() { - let mut suppliers = Vec::new(); - for binding in &option.compatible_suppliers { - if !input.compatible_producers.contains(&binding.producer) { - return Err(invalid("price option references incompatible supplier")); + let mut tuples = Vec::new(); + for tuple in &option.compatible_input_tuples { + if tuple.len() != required.len() { + return Err(invalid("price input tuple arity mismatch")); + } + let mut mapped = Vec::new(); + for (position, binding) in tuple.iter().enumerate() { + if !required[position] + .compatible_producers + .contains(&binding.producer) + { + return Err(invalid("price tuple references incompatible supplier")); } - let supplier = projection - .options - .get(binding.producer.0) - .and_then(|options| options.get(binding.price_option)) - .ok_or_else(|| { - invalid("price option references missing supplier option") - })?; - if !suppliers.contains(supplier) { - suppliers.push(*supplier); + mapped.push( + *projection + .options + .get(binding.producer.0) + .and_then(|options| options.get(binding.price_option)) + .ok_or_else(|| { + invalid("price tuple references missing supplier option") + })?, + ); + } + if !tuples.contains(&mapped) { + tuples.push(mapped); + } + } + for position in 0..required.len() { + let mut suppliers = Vec::new(); + for tuple in &tuples { + if !suppliers.contains(&tuple[position]) { + suppliers.push(tuple[position]); } } - expanded.candidates[projection.options[id][index].0] + expanded.candidates[consumer.0] .producer_requirements .push(ProducerRequirement { compatible_producers: suppliers.into(), }); } + // Unary choices and single tuples are already exact under the + // existing one-producer-per-input constraints; no extra binaries. + let cartesian_size = expanded.candidates[consumer.0] + .producer_requirements + .iter() + .try_fold(1usize, |size, input| { + size.checked_mul(input.compatible_producers.len()) + }); + if required.len() > 1 && tuples.len() > 1 && cartesian_size != Some(tuples.len()) { + projection.input_tuples.push((consumer, tuples)); + } } } for root in &graph.query_roots { @@ -390,11 +496,14 @@ mod tests { materialized, ..Default::default() }, - compatible_suppliers: producer + required_range_span_ms: None, + compatible_input_tuples: producer .into_iter() - .map(|(id, price_option)| SupplierPriceBinding { - producer: PhysicalCandidateId(id), - price_option, + .map(|(id, price_option)| { + vec![SupplierPriceBinding { + producer: PhysicalCandidateId(id), + price_option, + }] }) .collect(), } @@ -419,6 +528,7 @@ mod tests { ..Default::default() }; let profile = JointPriceProfile { + query_range_spans_ms: vec![], schedules: None, query_rate: 1.0, request_latency_seconds: 0.0, @@ -536,4 +646,54 @@ mod tests { .unwrap(); assert_eq!(proposal.direct_modeled_cost, Some(106.0)); } + // Tuple correlation forbids mixing the cheap first input from one tuple + // with the cheap second input of another tuple. + #[test] + fn correlated_inputs_cannot_form_an_unpriced_cartesian_combination() { + let mut consumer = candidate(&[0, 1]); + consumer + .producer_requirements + .push(consumer.producer_requirements[0].clone()); + let graph = CandidateDependencyGraph { + candidates: vec![candidate(&[]), candidate(&[]), consumer], + query_roots: vec![QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(2)], + }], + ..Default::default() + }; + let binding = |id| SupplierPriceBinding { + producer: PhysicalCandidateId(id), + price_option: 0, + }; + let mut reader = option(1.0, false, None); + reader.compatible_input_tuples = + vec![vec![binding(0), binding(1)], vec![binding(1), binding(0)]]; + let profile = JointPriceProfile { + candidates: vec![ + vec![option(0.0, true, None)], + vec![option(10.0, true, None)], + vec![reader], + ], + schedules: None, + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + query_range_spans_ms: vec![0], + }; + let solved = solve_joint_price_proposal_with_metrics( + &graph, + &[], + None, + &mut Default::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((solved.direct_modeled_cost.unwrap() - 11.0).abs() < 1e-9); + assert_eq!(solved.selected.connections.len(), 2); + assert_ne!( + solved.selected.connections[0].producer, + solved.selected.connections[1].producer + ); + } } diff --git a/crates/plan-selection/src/search/milp/direct_schedule.rs b/crates/plan-selection/src/search/milp/direct_schedule.rs deleted file mode 100644 index 2c55dc2d1..000000000 --- a/crates/plan-selection/src/search/milp/direct_schedule.rs +++ /dev/null @@ -1,528 +0,0 @@ -//! Heterogeneous execution demand for a small query-subset domain. -//! Subset rates come from the evaluator, not a second scheduling policy. -use super::*; - -#[allow(clippy::too_many_arguments)] -pub(super) fn encode( - graph: &CandidateDependencyGraph, - profile: &DirectCostProfile, - schedule: &QueryScheduleRates, - variables: &mut ProblemVariables, - constraints: &mut Vec, - active: &[Variable], - edges: &[(ProducerConsumerConnection, Variable)], - roots: &[(QueryRootAssignment, Variable)], - reach: &[Vec>], - divisor: f64, - check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, -) -> Result { - if let Some(lattice) = &schedule.compact_fixed { - return fixed_schedules::encode( - graph, - profile, - schedule, - lattice, - variables, - constraints, - active, - edges, - roots, - reach, - divisor, - check_deadline, - ); - } - let count = graph.candidates.len(); - let queries = graph.query_roots.len(); - let subsets = 1usize << queries; - let latency_scale = profile.maximum_latency(graph).max(f64::MIN_POSITIVE); - let latency = query_latency_expressions( - graph, - profile, - variables, - constraints, - reach, - &check_deadline, - )?; - let mut masks = Vec::with_capacity(count); - for id in 0..count { - check_deadline()?; - let allowed = reach - .iter() - .enumerate() - .filter(|(_, row)| row[id].is_some()) - .fold(0usize, |mask, (q, _)| mask | (1 << q)); - let row: Vec<_> = (0..subsets) - .map(|mask| (mask & !allowed == 0).then(|| variables.add(variable().min(0).max(1)))) - .collect(); - // Integral query reach fixes exactly one subset, so these selector - // auxiliaries need no additional integer branching. - constraints.push(constraint!( - row.iter().flatten().copied().sum::() == 1 - )); - constraints.push(constraint!( - row.iter().skip(1).flatten().copied().sum::() == active[id] - )); - for (query, reachable) in reach.iter().enumerate() { - let included: Expression = row - .iter() - .enumerate() - .filter(|(mask, _)| mask & (1 << query) != 0) - .filter_map(|(_, variable)| *variable) - .sum(); - constraints.push(constraint!( - included == reachable[id].map_or_else(|| Expression::from(0.0), Expression::from) - )); - } - masks.push(row); - } - // Each pair is (binary demand indicator, executions per second). A query - // boundary has its own rate; a materialized boundary uses its selected - // downstream query subset's union rate and stops upstream propagation. - let mut demands = vec![Vec::<(Variable, f64)>::new(); count]; - for flow in boundary_flows( - graph, - profile, - variables, - constraints, - active, - edges, - roots, - &check_deadline, - )? { - let boundary = flow.boundary; - for (id, reaches) in flow.reached.into_iter().enumerate() { - check_deadline()?; - let Some(reaches) = reaches else { continue }; - if boundary < queries { - demands[id].push((reaches, schedule.per_query_rates[boundary])); - } else if profile.nodes[boundary - queries].ingestion { - demands[id].push((reaches, 1.0)); - } else { - for (mask, selector) in masks[boundary - queries].iter().enumerate().skip(1) { - let Some(selector) = selector else { continue }; - let used = variables.add(variable().min(0).max(1)); - constraints.push(constraint!(used <= reaches)); - constraints.push(constraint!(used <= *selector)); - constraints.push(constraint!(used >= reaches + *selector - 1)); - demands[id].push((used, schedule.materialized_subset_rates[mask])); - } - } - } - } - let mut objective = Expression::from(profile.request_cost / divisor); - let mut retained = BTreeMap::<&str, Vec<(usize, f64)>>::new(); - for (id, node) in profile.nodes.iter().enumerate() { - check_deadline()?; - let executions: Expression = if node.materialized && node.ingestion { - active[id].into() - } else if node.materialized { - masks[id] - .iter() - .enumerate() - .skip(1) - .filter_map(|(mask, selector)| { - selector.map(|s| schedule.materialized_subset_rates[mask] * s) - }) - .sum() - } else { - demands[id].iter().map(|(used, rate)| *rate * *used).sum() - }; - objective += node.standing_cost_per_second / divisor * active[id]; - objective += node.execution_cost / divisor * executions; - if let Some((source, cost)) = &node.raw_retention { - retained.entry(source).or_default().push((id, *cost)); - } - if node.held_memory_cost_per_second == 0.0 { - continue; - } - let mut lifetime_demand = Expression::from(0.0); - if node.materialized { - let longest = variables.add(variable().min(0).max(1)); - for (query, row) in reach.iter().enumerate() { - if let Some(reached) = row[id] { - constraints.push(constraint!( - longest >= latency[query].clone() - (1.0 - reached) - )); - } - } - for (mask, selector) in masks[id].iter().enumerate().skip(1) { - if let Some(selector) = selector { - lifetime_demand += schedule.materialized_subset_rates[mask] - * gated(variables, constraints, longest.into(), *selector, 1.0); - } - } - } else { - for (mask, selector) in masks[id].iter().enumerate().skip(1) { - check_deadline()?; - let Some(selector) = selector else { continue }; - let total_rate: f64 = schedule - .per_query_rates - .iter() - .enumerate() - .filter(|(q, _)| mask & (1 << q) != 0) - .map(|(_, rate)| *rate) - .sum(); - let mean: Expression = latency - .iter() - .enumerate() - .filter(|(q, _)| mask & (1 << q) != 0) - .map(|(q, latency)| schedule.per_query_rates[q] / total_rate * latency.clone()) - .sum(); - let selected_mean = gated(variables, constraints, mean, *selector, 1.0); - for (demand, rate) in &demands[id] { - check_deadline()?; - lifetime_demand += - *rate * gated(variables, constraints, selected_mean.into(), *demand, 1.0); - } - } - } - objective += node.held_memory_cost_per_second * latency_scale / divisor * lifetime_demand; - } - for entries in retained.into_values() { - let retained_cost = variables.add(variable().min(0)); - for (id, cost) in entries { - constraints.push(constraint!(retained_cost >= cost / divisor * active[id])); - } - objective += retained_cost; - } - Ok(objective) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::search::milp::model::{ - solve_direct_cost_proposal_with_metrics, CandidateSolveMetrics, - }; - use asap_logical_asap_optimization::memo::MemoGroupId; - use asap_types::workload::{QueryRecurrence, RepeatedDemand, RepetitionInterval}; - - fn schedules(queries: &[QueryRecurrence]) -> QueryScheduleRates { - let union = |mask: usize| { - crate::cost::execution_demand::evaluation_rate( - queries - .iter() - .enumerate() - .filter(|(q, _)| mask & (1 << q) != 0) - .map(|(_, query)| (query, None)), - 3600.0, - ) - .unwrap() - }; - QueryScheduleRates { - compact_fixed: None, - per_query_rates: (0..queries.len()).map(|q| union(1 << q)).collect(), - materialized_subset_rates: (0..1 << queries.len()).map(union).collect(), - } - } - fn node(input: Option) -> PhysicalCandidate { - PhysicalCandidate { - group: MemoGroupId(0), - local_cost: 0.0, - producer_requirements: input - .into_iter() - .map(|id| ProducerRequirement { - compatible_producers: vec![PhysicalCandidateId(id)].into(), - }) - .collect(), - } - } - fn root(id: usize) -> QueryRootCandidates { - QueryRootCandidates { - compatible_candidates: vec![PhysicalCandidateId(id)], - } - } - - // Boundary demand is the evaluator's union rate; upstream held memory uses - // the rate-weighted mean, not the arithmetic mean or the sum of root rates. - #[test] - fn heterogeneous_intervals_price_boundary_rates_and_weighted_lifetimes() { - let queries = [500, 1000].map(|ms| { - QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms))) - }); - let graph = CandidateDependencyGraph { - candidates: vec![node(None), node(Some(0)), node(Some(1)), node(Some(1))], - query_roots: vec![root(2), root(3)], - ..Default::default() - }; - let profile = DirectCostProfile { - schedules: Some(schedules(&queries)), - query_rate: 1.0, - request_latency_seconds: 0.5, - request_cost: 7.0, - nodes: vec![ - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 2.0, - work_seconds: 1.0, - compaction_seconds: 0.0, - held_memory_cost_per_second: 3.0, - ..Default::default() - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 1.0, - work_seconds: 2.0, - compaction_seconds: 0.0, - held_memory_cost_per_second: 4.0, - materialized: true, - raw_retention: None, - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - work_seconds: 3.0, - compaction_seconds: 0.0, - ..Default::default() - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - work_seconds: 4.0, - compaction_seconds: 0.0, - ..Default::default() - }, - ], - }; - assert_eq!( - profile - .schedules - .as_ref() - .unwrap() - .materialized_subset_rates[3], - 3.0 - ); - let result = solve_direct_cost_proposal_with_metrics( - &graph, - &[], - None, - &mut CandidateSolveMetrics::default(), - &profile, - ) - .unwrap() - .unwrap(); - assert!((result.direct_modeled_cost.unwrap() - 167.5).abs() < 1e-8); - assert!((result.model_lower_bound.unwrap() - 167.5).abs() < 1e-8); - } - - // Same-interval readers can share one evaluation only at compatible phases. - // Use the actual evaluator's table rather than assuming all equal rates merge. - #[test] - fn aligned_and_offset_phases_use_evaluator_union_policy() { - use asap_types::workload::TimestampMs; - let graph = CandidateDependencyGraph { - candidates: vec![node(None)], - query_roots: vec![root(0), root(0)], - ..Default::default() - }; - for (second_phase, expected) in [(0, 7.0), (500, 14.0)] { - let queries = [0, second_phase].map(|phase| { - QueryRecurrence::Repeated(RepeatedDemand::FixedIntervalAt { - interval: RepetitionInterval(1000), - evaluation_phase: TimestampMs(phase), - }) - }); - let profile = DirectCostProfile { - schedules: Some(schedules(&queries)), - query_rate: 1.0, - request_latency_seconds: 0.0, - request_cost: 0.0, - nodes: vec![NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 3.0, - work_seconds: 2.0, - compaction_seconds: 0.0, - held_memory_cost_per_second: 2.0, - materialized: true, - raw_retention: None, - }], - }; - let result = solve_direct_cost_proposal_with_metrics( - &graph, - &[], - None, - &mut CandidateSolveMetrics::default(), - &profile, - ) - .unwrap() - .unwrap(); - assert!((result.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); - // HiGHS epigraph feasibility is tolerance-based; the decoded selected - // cost above is recomputed independently and remains exact here. - assert!( - (result.model_lower_bound.unwrap() - expected).abs() <= 1e-7 * expected, - "bound={:?}, expected={expected}", - result.model_lower_bound - ); - } - } - - // Every reader can independently choose either materialized producer. - // Compare the joint optimum with four independently priced assignments. - #[test] - fn heterogeneous_selection_matches_exhaustive_reader_assignments() { - let queries = [500, 1000].map(|ms| { - QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms))) - }); - let schedule = schedules(&queries); - let mut graph = CandidateDependencyGraph { - candidates: vec![node(None), node(None), node(Some(0)), node(Some(0))], - query_roots: vec![root(2), root(3)], - ..Default::default() - }; - for id in [2, 3] { - graph.candidates[id].producer_requirements[0].compatible_producers = - vec![PhysicalCandidateId(0), PhysicalCandidateId(1)].into(); - } - let profile = DirectCostProfile { - schedules: Some(schedule.clone()), - query_rate: 1.0, - request_latency_seconds: 0.5, - request_cost: 7.0, - nodes: vec![ - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 10.0, - work_seconds: 1.0, - compaction_seconds: 0.0, - held_memory_cost_per_second: 3.0, - materialized: true, - raw_retention: None, - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 2.0, - work_seconds: 0.5, - compaction_seconds: 0.0, - held_memory_cost_per_second: 1.0, - materialized: true, - raw_retention: None, - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 1.0, - work_seconds: 2.0, - compaction_seconds: 0.0, - ..Default::default() - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 1.0, - work_seconds: 3.0, - compaction_seconds: 0.0, - ..Default::default() - }, - ], - }; - let expected = [[0, 0], [0, 1], [1, 0], [1, 1]] - .into_iter() - .map(|assignment| { - let latency = [ - 2.5 + [1.0, 0.5][assignment[0]], - 3.5 + [1.0, 0.5][assignment[1]], - ]; - let mut cost = 7.0 + 3.0; - for supplier in [0, 1] { - let mask = assignment - .iter() - .enumerate() - .filter(|(_, chosen)| **chosen == supplier) - .fold(0, |mask, (q, _)| mask | (1 << q)); - let longest = latency - .iter() - .enumerate() - .filter(|(q, _)| mask & (1 << q) != 0) - .map(|(_, time)| *time) - .fold(0.0, f64::max); - cost += schedule.materialized_subset_rates[mask] - * ([10.0, 2.0][supplier] + [3.0, 1.0][supplier] * longest); - } - cost - }) - .fold(f64::INFINITY, f64::min); - let result = solve_direct_cost_proposal_with_metrics( - &graph, - &[], - None, - &mut CandidateSolveMetrics::default(), - &profile, - ) - .unwrap() - .unwrap(); - assert!((result.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); - // HiGHS epigraph feasibility is tolerance-based; the decoded selected - // cost above is recomputed independently and remains exact here. - assert!( - (result.model_lower_bound.unwrap() - expected).abs() <= 1e-7 * expected, - "bound={:?}, expected={expected}", - result.model_lower_bound - ); - } - // A rare slow reader's weighted held lifetime can dominate cost despite a - // tiny rate. Reject that numerically unsupported mean before optimization, - // rather than erase its contribution in a relaxed equality. - #[test] - fn extreme_reader_rates_are_explicitly_unsupported() { - let queries = [1, u32::MAX].map(|ms| { - QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms))) - }); - let graph = CandidateDependencyGraph { - candidates: vec![node(None), node(None), node(Some(0)), node(Some(0))], - query_roots: vec![root(2), root(3)], - ..Default::default() - }; - let profile = DirectCostProfile { - schedules: Some(schedules(&queries)), - query_rate: 1.0, - request_latency_seconds: 0.0, - request_cost: 0.0, - nodes: vec![ - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 1.0, - held_memory_cost_per_second: 1e6, - ..Default::default() - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - execution_cost: 2.0, - held_memory_cost_per_second: 1e6, - ..Default::default() - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - work_seconds: 1.0, - compaction_seconds: 0.0, - ..Default::default() - }, - NodeCostProfile { - ingestion: false, - standing_cost_per_second: 0.0, - work_seconds: 1e9, - compaction_seconds: 0.0, - ..Default::default() - }, - ], - }; - let rates = &profile.schedules.as_ref().unwrap().per_query_rates; - assert!( - rates[1] * 1e9 * 1e6 > 1e8, - "the rare reader has meaningful memory cost" - ); - assert!(matches!( - profile.validate(&graph), - Err(CandidateSelectionError::UnsupportedDirectCost(_)) - )); - } -} diff --git a/crates/plan-selection/src/search/milp/general_schedule.rs b/crates/plan-selection/src/search/milp/general_schedule.rs new file mode 100644 index 000000000..dd29c5c1e --- /dev/null +++ b/crates/plan-selection/src/search/milp/general_schedule.rs @@ -0,0 +1,178 @@ +//! Exact mixed-schedule demand and weighted lifetimes without subset tables. +use super::*; + +fn both( + variables: &mut ProblemVariables, + constraints: &mut Vec, + left: Variable, + right: Variable, +) -> Variable { + let value = variables.add(variable().min(0).max(1)); + constraints.push(constraint!(value <= left)); + constraints.push(constraint!(value <= right)); + constraints.push(constraint!(value >= left + right - 1)); + value +} + +#[allow(clippy::too_many_arguments)] +pub(super) fn encode( + graph: &CandidateDependencyGraph, + profile: &DirectCostProfile, + schedule: &QueryScheduleRates, + variables: &mut ProblemVariables, + constraints: &mut Vec, + active: &[Variable], + edges: &[(ProducerConsumerConnection, Variable)], + roots: &[(QueryRootAssignment, Variable)], + reach: &[Vec>], + divisor: f64, + check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, +) -> Result { + let count = graph.candidates.len(); + let queries = graph.query_roots.len(); + let latency_scale = profile.maximum_latency(graph).max(f64::MIN_POSITIVE); + let latency = query_latency_expressions( + graph, + profile, + variables, + constraints, + reach, + &check_deadline, + )?; + let mut materialized = vec![Vec::new(); count]; + for (id, node) in profile.nodes.iter().enumerate() { + check_deadline()?; + if node.materialized { + materialized[id] = if node.ingestion { + vec![(active[id], 1.0)] + } else { + schedule.encode_terms( + id, + reach, + variables, + constraints, + &check_deadline, + node.per_request, + )? + }; + } + } + let mut demands = materialized.clone(); + for flow in boundary_flows( + graph, + profile, + variables, + constraints, + active, + edges, + roots, + &check_deadline, + )? { + for (id, reached) in flow.reached.into_iter().enumerate() { + check_deadline()?; + let Some(reached) = reached else { continue }; + if flow.boundary < queries { + if schedule.per_query_rates()[flow.boundary] > 0.0 { + demands[id].push((reached, schedule.per_query_rates()[flow.boundary])); + } + } else { + for (used, rate) in &materialized[flow.boundary - queries] { + demands[id].push((both(variables, constraints, reached, *used), *rate)); + } + } + } + } + let mut objective = Expression::from(profile.request_cost / divisor); + let mut retained = BTreeMap::<&str, Vec<(usize, f64)>>::new(); + for (id, node) in profile.nodes.iter().enumerate() { + check_deadline()?; + objective += node.standing_cost_per_second / divisor * active[id]; + for (used, rate) in &demands[id] { + objective += node.execution_cost * *rate / divisor * *used; + } + if let Some((source, cost)) = &node.raw_retention { + retained.entry(source).or_default().push((id, *cost)); + } + if node.held_memory_cost_per_second == 0.0 || demands[id].is_empty() { + continue; + } + let lifetime = variables.add(variable().min(0).max(1)); + if node.materialized { + for (q, row) in reach.iter().enumerate() { + if let Some(reached) = row[id] { + constraints.push(constraint!( + lifetime >= latency[q].clone() - (1.0 - reached) + )); + } + } + } else { + // D*A=N is linear after gating A and each query latency by binary + // reach: sum(w_q * reached_q * A) = sum(w_q * reached_q * L_q). + // No reciprocal, LCM, or enumeration of selected reader subsets. + let total: f64 = schedule.per_query_rates().iter().sum(); + let mut left = Expression::from(0.0); + let mut right = Expression::from(0.0); + if total > 0.0 { + for (q, row) in reach.iter().enumerate() { + check_deadline()?; + if let Some(reached) = row[id] { + let weight = schedule.per_query_rates()[q] / total; + if weight == 0.0 { + continue; + } + left += + weight * gated(variables, constraints, lifetime.into(), reached, 1.0); + right += weight + * gated(variables, constraints, latency[q].clone(), reached, 1.0); + } + } + } + constraints.push(constraint!(left == right)); + if schedule.per_query_rates().contains(&0.0) { + let positive: Vec<_> = reach + .iter() + .enumerate() + .filter(|(q, _)| schedule.per_query_rates()[*q] > 0.0) + .filter_map(|(_, row)| row[id]) + .collect(); + let has_positive = variables.add(variable().min(0).max(1)); + for reached in &positive { + constraints.push(constraint!(has_positive >= *reached)); + } + constraints.push(constraint!( + has_positive <= positive.iter().copied().sum::() + )); + // Match the evaluator's longest-reader fallback when every reached + // reader has zero expected demand; no division by zero is needed. + for (q, row) in reach.iter().enumerate() { + if let Some(reached) = row[id] { + constraints.push(constraint!( + lifetime >= latency[q].clone() - (1.0 - reached) - has_positive + )); + } + } + } + } + for (used, rate) in &demands[id] { + objective += node.held_memory_cost_per_second * latency_scale * *rate / divisor + * gated(variables, constraints, lifetime.into(), *used, 1.0); + } + } + for costs in retained.values() { + check_deadline()?; + let maximum = costs + .iter() + .map(|(_, cost)| *cost / divisor) + .fold(0.0, f64::max); + let cost = variables.add(variable().min(0).max(maximum)); + for (id, charge) in costs { + constraints.push(constraint!(cost >= *charge / divisor * active[*id])); + } + objective += cost; + } + Ok(objective) +} + +#[cfg(test)] +#[path = "general_schedule_tests.rs"] +mod tests; diff --git a/crates/plan-selection/src/search/milp/general_schedule_rates.rs b/crates/plan-selection/src/search/milp/general_schedule_rates.rs new file mode 100644 index 000000000..f76ad6bce --- /dev/null +++ b/crates/plan-selection/src/search/milp/general_schedule_rates.rs @@ -0,0 +1,771 @@ +//! Exact finite demand atoms for the evaluator's recurrence accounting policy. +//! Different fixed intervals add; an unknown phase joins the first selected +//! known phase, estimated rates add, and nonrepeating demand takes a maximum. +use crate::cost::execution_demand::evaluation_rate; +use crate::search::candidates::graph::CandidateSelectionError; +use asap_types::workload::{QueryRecurrence, QueryWorkloadEntry, RangeEvaluation, RepeatedDemand}; +use good_lp::{constraint, variable, Constraint, ProblemVariables, Variable}; +use std::collections::BTreeMap; +use std::time::Instant; + +const MAXIMUM_ENUMERATED_OFFSETS: usize = 100_000; +const MAXIMUM_OFFSET_VISITS: usize = 500_000; + +#[derive(Debug, Clone)] +struct Atom { + readers: Vec, + count: u64, +} +#[derive(Debug, Clone)] +struct FixedPhase { + readers: Vec, + atoms: Vec, +} +#[derive(Debug, Clone)] +struct FixedInterval { + rate: f64, + unknown: Vec, + unknown_atoms: Vec, + // Ascending phase order is semantically significant. + known: Vec, +} +#[derive(Debug, Clone, Default)] +struct DemandAtoms { + fixed: Vec, + estimated: Vec<(usize, f64)>, + batch: Vec<(u64, Vec)>, + horizon: f64, +} +#[derive(Debug, Clone)] +pub(crate) struct QueryScheduleRates { + queries: Vec<(QueryRecurrence, Option)>, + rates: Vec, + evaluations: DemandAtoms, + requests: DemandAtoms, +} +fn unsupported(reason: &str) -> CandidateSelectionError { + CandidateSelectionError::UnsupportedDirectCost(reason.into()) +} +fn check(deadline: Option) -> Result<(), CandidateSelectionError> { + if deadline.is_some_and(|deadline| Instant::now() >= deadline) { + Err(CandidateSelectionError::Solver( + "time budget exhausted constructing schedule atoms".into(), + )) + } else { + Ok(()) + } +} + +// With a common step, k-ranges are nested and can be counted without expanding +// even a billion individual timestamps. Mixed steps retain exact coverage sets. +fn coverage( + readers: &[(usize, u64, u64)], + deadline: Option, +) -> Result, CandidateSelectionError> { + if readers.is_empty() { + return Ok(Vec::new()); + } + if readers.iter().all(|(_, step, _)| *step == readers[0].1) { + let mut endpoints: Vec<_> = readers + .iter() + .map(|(_, step, span)| span / step + 1) + .collect(); + endpoints.sort_unstable(); + endpoints.dedup(); + let mut previous = 0; + let mut atoms = Vec::new(); + for end in endpoints { + check(deadline)?; + let active = readers + .iter() + .filter(|(_, step, span)| span / step + 1 >= end) + .map(|(id, _, _)| *id) + .collect(); + atoms.push(Atom { + readers: active, + count: end - previous, + }); + previous = end; + } + return Ok(atoms); + } + // A few distinct arithmetic progressions can be counted exactly without + // enumerating their offsets. Duplicate geometries share all their readers. + let mut geometries = BTreeMap::<(u64, u64), Vec>::new(); + for &(id, step, span) in readers { + check(deadline)?; + geometries.entry((step, span)).or_default().push(id); + } + if geometries.len() <= 12 { + return arithmetic_coverage(&geometries.into_iter().collect::>(), deadline); + } + let mut offsets: BTreeMap> = BTreeMap::new(); + let mut visits = 0usize; + for &(id, step, span) in readers { + let count = span / step + 1; + if count > MAXIMUM_OFFSET_VISITS as u64 + || visits.saturating_add(count as usize) > MAXIMUM_OFFSET_VISITS + { + return Err(unsupported( + "mixed-step range demand exceeds the exact offset construction budget", + )); + } + visits += count as usize; + for k in 0..count { + check(deadline)?; + offsets.entry(k * step).or_default().push(id); + if offsets.len() > MAXIMUM_ENUMERATED_OFFSETS { + return Err(unsupported( + "mixed-step range demand exceeds the exact offset domain", + )); + } + } + } + let mut classes = BTreeMap::, u64>::new(); + for (_, mut readers) in offsets { + check(deadline)?; + readers.sort_unstable(); + readers.dedup(); + *classes.entry(readers).or_default() += 1; + } + Ok(classes + .into_iter() + .map(|(readers, count)| Atom { readers, count }) + .collect()) +} + +fn arithmetic_coverage( + geometries: &[((u64, u64), Vec)], + deadline: Option, +) -> Result, CandidateSelectionError> { + let size = 1usize << geometries.len(); + let mut counts = vec![0i128; size]; + for (mask, count) in counts.iter_mut().enumerate().skip(1) { + check(deadline)?; + let intersection: Vec<_> = geometries + .iter() + .enumerate() + .filter(|(bit, _)| mask & (1 << bit) != 0) + .map(|(_, (geometry, _))| *geometry) + .collect(); + *count = crate::cost::execution_demand::offset_intersection_count(&intersection); + } + // Superset Möbius inversion changes intersection counts into counts of + // offsets covered by exactly each geometry subset. + for bit in 0..geometries.len() { + for mask in 1..size { + check(deadline)?; + if mask & (1 << bit) == 0 { + counts[mask] -= counts[mask | (1 << bit)]; + } + } + } + let mut atoms = Vec::new(); + for (mask, count) in counts.into_iter().enumerate().skip(1) { + check(deadline)?; + if count == 0 { + continue; + } + let count = u64::try_from(count) + .map_err(|_| unsupported("arithmetic range coverage count overflow"))?; + let mut readers: Vec<_> = geometries + .iter() + .enumerate() + .filter(|(bit, _)| mask & (1 << bit) != 0) + .flat_map(|(_, (_, readers))| readers.iter().copied()) + .collect(); + readers.sort_unstable(); + readers.dedup(); + atoms.push(Atom { readers, count }); + } + Ok(atoms) +} + +impl DemandAtoms { + fn new( + queries: &[(QueryRecurrence, Option)], + horizon: f64, + per_request: bool, + deadline: Option, + ) -> Result { + let mut fixed = BTreeMap::, Vec<(usize, u64, u64)>>>::new(); + let mut estimated = Vec::new(); + let mut batch = BTreeMap::>::new(); + for (id, (recurrence, range)) in queries.iter().enumerate() { + check(deadline)?; + let range = if per_request { None } else { *range }; + let steps = range.map_or(1, |range| range.evaluations_per_request()); + if steps > (1u64 << 53) { + return Err(unsupported( + "range evaluation count exceeds exact floating-point integer precision", + )); + } + match recurrence { + QueryRecurrence::Repeated( + RepeatedDemand::FixedInterval(interval) + | RepeatedDemand::FixedIntervalAt { interval, .. }, + ) => { + let phase = match recurrence { + QueryRecurrence::Repeated(RepeatedDemand::FixedIntervalAt { + evaluation_phase, + .. + }) => Some(evaluation_phase.0 % u64::from(interval.0)), + _ => None, + }; + let (step, span) = range + .filter(|_| steps > 1) + .map_or((u64::from(interval.0), 0), |r| (r.step_ms, r.span_ms)); + fixed + .entry(interval.0) + .or_default() + .entry(phase) + .or_default() + .push((id, step, span)); + } + QueryRecurrence::Repeated(RepeatedDemand::EstimatedRate(estimate)) => { + estimated.push((id, estimate.expected_rate.0 * steps as f64)) + } + _ => { + let requests = match recurrence { + QueryRecurrence::Repeated(RepeatedDemand::Scheduled(times)) => { + times.len() as u64 + } + QueryRecurrence::OneTime { invocations, .. } => *invocations, + _ => 1, + }; + let count = requests + .checked_mul(steps) + .ok_or_else(|| unsupported("batch invocation count overflow"))?; + if count > (1u64 << 53) { + return Err(unsupported( + "batch invocation count exceeds exact floating-point integer precision", + )); + } + if count > 0 { + batch.entry(count).or_default().push(id); + } + } + } + } + let mut intervals = Vec::new(); + for (interval, mut phases) in fixed { + let unknown = phases.remove(&None).unwrap_or_default(); + let unknown_atoms = coverage(&unknown, deadline)?; + let mut known = Vec::new(); + for (_, readers) in phases { + let members = readers.iter().map(|(id, _, _)| *id).collect(); + let mut together = readers; + together.extend_from_slice(&unknown); + known.push(FixedPhase { + readers: members, + atoms: coverage(&together, deadline)?, + }); + } + intervals.push(FixedInterval { + rate: 1000.0 / f64::from(interval), + unknown: unknown.iter().map(|(id, _, _)| *id).collect(), + unknown_atoms, + known, + }); + } + Ok(Self { + fixed: intervals, + estimated, + batch: batch.into_iter().collect(), + horizon, + }) + } +} + +impl QueryScheduleRates { + pub(crate) fn new( + entries: &[QueryWorkloadEntry], + horizon: f64, + deadline: Option, + ) -> Result { + check(deadline)?; + if entries.is_empty() || !horizon.is_finite() || horizon <= 0.0 { + return Err(CandidateSelectionError::InvalidGraph( + "invalid general schedule horizon or empty workload".into(), + )); + } + let queries: Vec<_> = entries + .iter() + .map(|entry| (entry.recurrence.clone(), entry.range_evaluation)) + .collect(); + for (recurrence, range) in &queries { + check(deadline)?; + if range.is_some_and(|r| r.step_ms == 0 || r.span_ms == u64::MAX) { + return Err(unsupported("invalid or overflowing range schedule")); + } + if matches!(recurrence,QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(interval) | RepeatedDemand::FixedIntervalAt{interval,..}) if interval.0==0) + { + return Err(CandidateSelectionError::InvalidGraph( + "zero fixed interval".into(), + )); + } + } + // Build checked integer products before calling the evaluator oracle. + let evaluations = DemandAtoms::new(&queries, horizon, false, deadline)?; + let requests = DemandAtoms::new(&queries, horizon, true, deadline)?; + let rates = queries + .iter() + .map(|(r, range)| { + evaluation_rate([(r, *range)], horizon) + .map_err(|e| CandidateSelectionError::InvalidGraph(e.to_string())) + }) + .collect::, _>>()?; + if rates.iter().any(|rate| !rate.is_finite() || *rate < 0.0) { + return Err(unsupported("nonfinite general schedule rate")); + } + Ok(Self { + queries, + rates, + evaluations, + requests, + }) + } + pub(crate) fn validate(&self, queries: usize) -> Result<(), CandidateSelectionError> { + if queries != self.queries.len() { + return Err(CandidateSelectionError::InvalidGraph( + "inconsistent general schedule rates".into(), + )); + } + Ok(()) + } + pub(crate) fn coefficient_rates(&self) -> Vec { + let mut result = Vec::new(); + for atoms in [&self.evaluations, &self.requests] { + for interval in &atoms.fixed { + result.extend( + interval + .known + .iter() + .flat_map(|phase| &phase.atoms) + .chain(&interval.unknown_atoms) + .map(|atom| interval.rate * atom.count as f64), + ); + } + result.extend(atoms.estimated.iter().map(|(_, rate)| *rate)); + let mut previous = 0; + for (count, _) in &atoms.batch { + result.push((*count - previous) as f64 / atoms.horizon); + previous = *count; + } + } + result.retain(|rate| *rate > 0.0); + result + } + pub(crate) fn per_query_rates(&self) -> &[f64] { + &self.rates + } + pub(crate) fn union_rate(&self, selected: impl Iterator) -> f64 { + self.rate(selected, false) + } + pub(crate) fn request_union_rate(&self, selected: impl Iterator) -> f64 { + self.rate(selected, true) + } + fn rate(&self, selected: impl Iterator, per_request: bool) -> f64 { + let selected: Vec<_> = selected.collect(); + let atoms = if per_request { + &self.requests + } else { + &self.evaluations + }; + let any = |readers: &[usize]| { + readers + .iter() + .any(|q| selected.get(*q).copied().unwrap_or(false)) + }; + let mut total = 0.0; + for interval in &atoms.fixed { + let first = interval.known.iter().position(|phase| any(&phase.readers)); + for (index, phase) in interval.known.iter().enumerate() { + for atom in &phase.atoms { + if atom.readers.iter().any(|q| { + selected.get(*q).copied().unwrap_or(false) + && (!interval.unknown.contains(q) || first == Some(index)) + }) { + total += interval.rate * atom.count as f64; + } + } + } + if first.is_none() { + for atom in &interval.unknown_atoms { + if any(&atom.readers) { + total += interval.rate * atom.count as f64; + } + } + } + } + total += atoms + .estimated + .iter() + .filter(|(q, _)| selected.get(*q).copied().unwrap_or(false)) + .map(|(_, rate)| *rate) + .sum::(); + total += atoms + .batch + .iter() + .filter(|(_, readers)| any(readers)) + .map(|(count, _)| *count) + .max() + .unwrap_or(0) as f64 + / atoms.horizon; + total + } + pub(crate) fn encode_terms( + &self, + node: usize, + reach: &[Vec>], + variables: &mut ProblemVariables, + constraints: &mut Vec, + check_deadline: impl Fn() -> Result<(), CandidateSelectionError>, + per_request: bool, + ) -> Result, CandidateSelectionError> { + let atoms = if per_request { + &self.requests + } else { + &self.evaluations + }; + let mut terms = Vec::new(); + let readers = |ids: &[usize]| { + ids.iter() + .filter_map(|q| reach[*q][node]) + .collect::>() + }; + for interval in &atoms.fixed { + check_deadline()?; + let mut prior = None; + for phase in &interval.known { + let present = or(variables, constraints, &readers(&phase.readers)); + let first = present.map(|present| without(variables, constraints, present, prior)); + for atom in &phase.atoms { + check_deadline()?; + let (unknown, known): (Vec<&usize>, Vec<&usize>) = atom + .readers + .iter() + .partition(|q| interval.unknown.contains(q)); + let known: Vec<_> = known.into_iter().filter_map(|q| reach[*q][node]).collect(); + let unknown: Vec<_> = unknown + .into_iter() + .filter_map(|q| reach[*q][node]) + .collect(); + let mut occupied = or(variables, constraints, &known) + .into_iter() + .collect::>(); + if let (Some(first), Some(unknown)) = + (first, or(variables, constraints, &unknown)) + { + occupied.push(and(variables, constraints, first, unknown)); + } + if let Some(value) = or(variables, constraints, &occupied) { + terms.push((value, interval.rate * atom.count as f64)); + } + } + prior = or( + variables, + constraints, + &prior.into_iter().chain(present).collect::>(), + ); + } + for atom in &interval.unknown_atoms { + check_deadline()?; + if let Some(unknown) = or(variables, constraints, &readers(&atom.readers)) { + terms.push(( + without(variables, constraints, unknown, prior), + interval.rate * atom.count as f64, + )); + } + } + } + for &(q, rate) in &atoms.estimated { + check_deadline()?; + if rate > 0.0 { + if let Some(reach) = reach[q][node] { + terms.push((reach, rate)); + } + } + } + let mut previous = 0; + for (count, _) in &atoms.batch { + check_deadline()?; + let members: Vec<_> = atoms + .batch + .iter() + .filter(|(other, _)| other >= count) + .flat_map(|(_, ids)| readers(ids)) + .collect(); + if let Some(value) = or(variables, constraints, &members) { + terms.push((value, (*count - previous) as f64 / atoms.horizon)); + } + previous = *count; + } + Ok(terms) + } +} +fn or( + variables: &mut ProblemVariables, + constraints: &mut Vec, + inputs: &[Variable], +) -> Option { + if inputs.is_empty() { + return None; + } + if inputs.len() == 1 { + return Some(inputs[0]); + } + let value = variables.add(variable().min(0).max(1)); + for input in inputs { + constraints.push(constraint!(value >= *input)); + } + constraints.push(constraint!( + value <= inputs.iter().copied().sum::() + )); + Some(value) +} +fn and( + variables: &mut ProblemVariables, + constraints: &mut Vec, + left: Variable, + right: Variable, +) -> Variable { + let value = variables.add(variable().min(0).max(1)); + constraints.push(constraint!(value <= left)); + constraints.push(constraint!(value <= right)); + constraints.push(constraint!(value >= left + right - 1)); + value +} +fn without( + variables: &mut ProblemVariables, + constraints: &mut Vec, + value: Variable, + excluded: Option, +) -> Variable { + let Some(excluded) = excluded else { + return value; + }; + let result = variables.add(variable().min(0).max(1)); + constraints.push(constraint!(result <= value)); + constraints.push(constraint!(result <= 1.0 - excluded)); + constraints.push(constraint!(result >= value - excluded)); + result +} + +#[cfg(test)] +mod tests { + use super::*; + use asap_types::ir::properties::AccuracyTarget; + use asap_types::workload::{ + DemandEstimate, Predictability, Query, QueryRequirements, Rate, RepetitionInterval, + TimestampMs, + }; + use good_lp::{highs, Expression, Solution, SolverModel}; + + fn entry(recurrence: QueryRecurrence, range: Option<(u64, u64)>) -> QueryWorkloadEntry { + QueryWorkloadEntry { + query: Query("sum(m)".into()), + requirements: QueryRequirements { + accuracy: AccuracyTarget::exact(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence, + range_evaluation: range.map(|(span_ms, step_ms)| RangeEvaluation { span_ms, step_ms }), + } + } + fn fixed(interval: u32, phase: Option, range: Option<(u64, u64)>) -> QueryWorkloadEntry { + entry( + QueryRecurrence::Repeated(match phase { + None => RepeatedDemand::FixedInterval(RepetitionInterval(interval)), + Some(phase) => RepeatedDemand::FixedIntervalAt { + interval: RepetitionInterval(interval), + evaluation_phase: TimestampMs(phase), + }, + }), + range, + ) + } + fn compare_all_subsets(entries: &[QueryWorkloadEntry]) { + let schedule = QueryScheduleRates::new(entries, 60.0, None).unwrap(); + for per_request in [false, true] { + for mask in 0..(1usize << entries.len()) { + let mut variables = ProblemVariables::new(); + let mut constraints = Vec::new(); + let reach: Vec<_> = (0..entries.len()) + .map(|q| { + let v = variables.add(variable().binary()); + constraints.push(constraint!(v == f64::from(mask & (1 << q) != 0))); + vec![Some(v)] + }) + .collect(); + let terms = schedule + .encode_terms( + 0, + &reach, + &mut variables, + &mut constraints, + || Ok(()), + per_request, + ) + .unwrap(); + let expression: Expression = terms.iter().map(|(v, rate)| *rate * *v).sum(); + // Maximise to prove that an auxiliary rate cannot be inflated; + // all OR/first-phase/max relations are exact equalities in use. + let solution = variables + .minimise(-expression.clone()) + .using(highs) + .with_all(constraints) + .solve() + .unwrap(); + let expected = evaluation_rate( + entries + .iter() + .enumerate() + .filter(|(q, _)| mask & (1 << q) != 0) + .map(|(_, entry)| { + ( + &entry.recurrence, + if per_request { + None + } else { + entry.range_evaluation + }, + ) + }), + 60.0, + ) + .unwrap(); + let actual = solution.eval(&expression); + let decoded = schedule.rate( + (0..entries.len()).map(|q| mask & (1 << q) != 0), + per_request, + ); + assert!( + (decoded - expected).abs() <= 1e-10 * expected.max(1.0), + "decoded {decoded} != oracle {expected}" + ); + assert!( + (actual - expected).abs() <= 1e-10 * expected.max(1.0), + "mask={mask}, requests={per_request}: {actual} != {expected}" + ); + } + } + } + #[test] + fn fixed_ranges_join_unknown_to_first_selected_known_phase() { + compare_all_subsets(&[ + fixed(1000, None, Some((6, 2))), + fixed(1000, Some(0), Some((6, 3))), + fixed(1000, Some(500), Some((6, 2))), + fixed(2000, Some(0), Some((6, 2))), + ]); + } + #[test] + fn mixed_estimates_scheduled_batches_and_unknown_match_evaluator() { + let estimate = |rate| { + QueryRecurrence::Repeated(RepeatedDemand::EstimatedRate(DemandEstimate { + expected_rate: Rate(rate), + observed_at: None, + valid_for: None, + })) + }; + compare_all_subsets(&[ + fixed(997, None, Some((10, 3))), + entry(estimate(0.137), Some((8, 2))), + entry(estimate(0.193), None), + entry( + QueryRecurrence::Repeated(RepeatedDemand::Scheduled(vec![ + TimestampMs(10), + TimestampMs(20), + ])), + Some((6, 3)), + ), + entry( + QueryRecurrence::OneTime { + invocations: 3, + execute_at: None, + }, + Some((8, 2)), + ), + entry(QueryRecurrence::Unknown, Some((10, 1))), + ]); + } + #[test] + fn same_step_large_ranges_are_counted_without_timestamp_enumeration() { + let entries = [ + fixed(1000, None, Some((1_000_000_000, 1))), + fixed(1000, None, Some((500_000_000, 1))), + ]; + let schedule = QueryScheduleRates::new(&entries, 60.0, None).unwrap(); + assert_eq!(schedule.evaluations.fixed[0].unknown_atoms.len(), 2); + assert_eq!( + schedule.union_rate([true, true].into_iter()), + 1_000_000_001.0 + ); + assert_eq!(schedule.request_union_rate([true, true].into_iter()), 1.0); + } + #[test] + fn large_coprime_ranges_use_exact_arithmetic_coverage() { + let entries = [ + fixed(1000, None, Some((1_000_000_000, 2))), + fixed(1000, None, Some((1_000_000_000, 3))), + ]; + let schedule = QueryScheduleRates::new(&entries, 60.0, None).unwrap(); + assert_eq!(schedule.evaluations.fixed[0].unknown_atoms.len(), 3); + assert_eq!(schedule.union_rate([true, true].into_iter()), 666_666_668.0); + assert_eq!( + schedule.union_rate([true, false].into_iter()), + 500_000_001.0 + ); + assert_eq!( + schedule.union_rate([false, true].into_iter()), + 333_333_334.0 + ); + compare_all_subsets(&[ + fixed(1000, None, Some((60, 2))), + fixed(1000, None, Some((60, 3))), + ]); + } + #[test] + fn arithmetic_atoms_match_offset_coverage_with_duplicates_and_lcm_overflow() { + let readers = [(0, 2, 60), (1, 3, 60), (2, 2, 60), (3, 5, 37), (4, 7, 0)]; + let atoms = coverage(&readers, None).unwrap(); + let actual: BTreeMap<_, _> = atoms + .into_iter() + .map(|atom| (atom.readers, atom.count)) + .collect(); + let mut expected = BTreeMap::, u64>::new(); + for offset in 0..=60 { + let members: Vec<_> = readers + .iter() + .filter(|(_, step, span)| offset <= *span && offset % step == 0) + .map(|(id, _, _)| *id) + .collect(); + if !members.is_empty() { + *expected.entry(members).or_default() += 1; + } + } + assert_eq!(actual, expected); + let large = 1u64 << 63; + let atoms = coverage(&[(0, large - 1, large), (1, large - 3, large)], None).unwrap(); + assert_eq!(atoms.iter().map(|atom| atom.count).sum::(), 3); + assert!(atoms + .iter() + .any(|atom| atom.readers == vec![0, 1] && atom.count == 1)); + } + #[test] + fn excessive_mixed_offset_domains_and_deadlines_are_explicit() { + let entries: Vec<_> = (1..=13) + .map(|step| fixed(1000, None, Some((10_000_000, step)))) + .collect(); + assert!(matches!( + QueryScheduleRates::new(&entries, 60.0, None), + Err(CandidateSelectionError::UnsupportedDirectCost(_)) + )); + assert!(matches!( + QueryScheduleRates::new(&entries, 60.0, Some(Instant::now())), + Err(CandidateSelectionError::Solver(_)) + )); + } +} diff --git a/crates/plan-selection/src/search/milp/general_schedule_tests.rs b/crates/plan-selection/src/search/milp/general_schedule_tests.rs new file mode 100644 index 000000000..0305324ad --- /dev/null +++ b/crates/plan-selection/src/search/milp/general_schedule_tests.rs @@ -0,0 +1,254 @@ +use super::*; +use crate::search::milp::model::{solve_direct_cost_proposal_with_metrics, CandidateSolveMetrics}; +use asap_types::ir::properties::AccuracyTarget; +use asap_types::workload::{ + DemandEstimate, Predictability, Query, QueryRecurrence, QueryRequirements, QueryWorkloadEntry, + RangeEvaluation, Rate, RepeatedDemand, RepetitionInterval, +}; + +fn entry(recurrence: QueryRecurrence, evaluations: u64) -> QueryWorkloadEntry { + QueryWorkloadEntry { + query: Query("sum(m)".into()), + requirements: QueryRequirements { + accuracy: AccuracyTarget::exact(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence, + range_evaluation: (evaluations > 1).then_some(RangeEvaluation { + span_ms: (evaluations - 1) * 1000, + step_ms: 1000, + }), + } +} +fn fixed(interval: u32, evaluations: u64) -> QueryWorkloadEntry { + entry( + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(interval))), + evaluations, + ) +} +fn node(id: usize, input: Option) -> PhysicalCandidate { + PhysicalCandidate { + group: asap_logical_asap_optimization::memo::MemoGroupId(id as u32), + local_cost: 0.0, + producer_requirements: input + .into_iter() + .map(|input| ProducerRequirement { + compatible_producers: vec![PhysicalCandidateId(input)].into(), + }) + .collect(), + } +} +fn check_cost( + entries: &[QueryWorkloadEntry], + expected_rates: &[f64], + expected_union: f64, + per_request: bool, +) { + let general = QueryScheduleRates::new(entries, 60.0, None).unwrap(); + for (actual, expected) in general.per_query_rates().iter().zip(expected_rates) { + assert!((actual - expected).abs() < 1e-12); + } + let schedule = general; + let mut candidates = vec![node(0, None), node(1, Some(0))]; + candidates.extend((0..entries.len()).map(|q| node(q + 2, Some(1)))); + let graph = CandidateDependencyGraph { + candidates, + query_roots: (0..entries.len()) + .map(|q| QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(q + 2)], + }) + .collect(), + ..Default::default() + }; + let mut nodes = vec![ + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 4.0, + work_seconds: 2.0, + held_memory_cost_per_second: if per_request { 0.0 } else { 2.0 }, + standing_cost_per_second: if per_request { 13.0 } else { 0.0 }, + materialized: true, + per_request, + ..Default::default() + }, + ]; + nodes.extend((0..entries.len()).map(|q| NodeCostProfile { + execution_cost: 1.0, + work_seconds: q as f64 + 1.0, + ..Default::default() + })); + let sum: f64 = expected_rates.iter().sum(); + let weighted: f64 = expected_rates + .iter() + .enumerate() + .map(|(q, rate)| rate * (q as f64 + 4.0)) + .sum(); + let longest = entries.len() as f64 + 3.0; + let expected = expected_union + * (6.0 + 3.0 * weighted / sum + if per_request { 0.0 } else { 2.0 * longest }) + + sum + + if per_request { 13.0 } else { 0.0 }; + let profile = DirectCostProfile { + nodes, + schedules: Some(schedule), + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + let mut metrics = CandidateSolveMetrics::default(); + let proposal = + solve_direct_cost_proposal_with_metrics(&graph, &[], None, &mut metrics, &profile) + .unwrap() + .unwrap(); + let direct = proposal.direct_modeled_cost.unwrap(); + assert!( + (direct - expected).abs() < 1e-8 * expected.max(1.0), + "direct {direct}, independent expected {expected}" + ); + assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-6 * expected.max(1.0)); + assert!( + metrics.peak_variables < 5000, + "a rate lattice or reader subset product is unnecessary" + ); +} + +#[test] +fn noncommensurate_fixed_intervals_price_weighted_memory_without_integer_lattice() { + let intervals = [59_000, 61_000, 67_000, 71_000]; + let entries: Vec<_> = intervals + .iter() + .map(|interval| fixed(*interval, 1)) + .collect(); + let rates: Vec<_> = intervals + .iter() + .map(|interval| 1000.0 / f64::from(*interval)) + .collect(); + check_cost(&entries, &rates, rates.iter().sum(), false); +} + +#[test] +fn mixed_schedules_price_materialized_union_and_unmaterialized_mean() { + let estimate = QueryRecurrence::Repeated(RepeatedDemand::EstimatedRate(DemandEstimate { + expected_rate: Rate(0.137), + observed_at: None, + valid_for: None, + })); + let entries = vec![ + fixed(1000, 2), + fixed(2000, 3), + entry(estimate, 4), + entry( + QueryRecurrence::OneTime { + invocations: 3, + execute_at: None, + }, + 5, + ), + entry(QueryRecurrence::Unknown, 6), + ]; + let rates = [2.0, 1.5, 0.548, 0.25, 0.1]; + // Fixed intervals and estimates add; the two batch demands use max(15,6)/60. + check_cost(&entries, &rates, 2.0 + 1.5 + 0.548 + 0.25, false); + // Retained refreshes use request rates; readers and weighted lifetime still + // use evaluation rates including each query's different range length. + check_cost(&entries, &rates, 1.0 + 0.5 + 0.137 + 3.0 / 60.0, true); +} + +#[test] +fn zero_rate_readers_preserve_materialized_and_unmaterialized_lifetimes() { + // With all rates zero the execution cost is zero. With one positive reader, + // unmaterialized lifetime ignores the zero-rate reader, while a shared + // query-time materialization still uses its longest reached latency. + for rates in [[0.0, 0.0], [0.0, 0.5], [0.5, 0.0]] { + let entries: Vec<_> = rates + .iter() + .map(|rate| { + entry( + QueryRecurrence::Repeated(RepeatedDemand::EstimatedRate(DemandEstimate { + expected_rate: Rate(*rate), + observed_at: None, + valid_for: None, + })), + 1, + ) + }) + .collect(); + let general = QueryScheduleRates::new(&entries, 60.0, None).unwrap(); + let graph = CandidateDependencyGraph { + candidates: vec![ + node(0, None), + node(1, Some(0)), + node(2, Some(1)), + node(3, Some(1)), + ], + query_roots: vec![ + QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(2)], + }, + QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(3)], + }, + ], + ..Default::default() + }; + let profile = DirectCostProfile { + nodes: vec![ + NodeCostProfile { + execution_cost: 2.0, + work_seconds: 1.0, + held_memory_cost_per_second: 3.0, + ..Default::default() + }, + NodeCostProfile { + materialized: true, + execution_cost: 4.0, + work_seconds: 2.0, + held_memory_cost_per_second: 2.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 1.0, + ..Default::default() + }, + NodeCostProfile { + execution_cost: 1.0, + work_seconds: 2.0, + ..Default::default() + }, + ], + schedules: Some(general), + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + }; + let sum: f64 = rates.iter().sum(); + let lifetime = if sum > 0.0 { + (rates[0] * 4.0 + rates[1] * 5.0) / sum + } else { + 5.0 + }; + let expected = sum * (2.0 + 4.0 + 3.0 * lifetime + 2.0 * 5.0) + sum; + let proposal = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut Default::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!( + (proposal.direct_modeled_cost.unwrap() - expected).abs() < 1e-8, + "rates {rates:?}: expected {expected}, got {:?}", + proposal.direct_modeled_cost + ); + assert!((proposal.model_lower_bound.unwrap() - expected).abs() < 1e-6); + } +} diff --git a/crates/plan-selection/src/search/milp/model.rs b/crates/plan-selection/src/search/milp/model.rs index f3d78fde8..ca4aa32a4 100644 --- a/crates/plan-selection/src/search/milp/model.rs +++ b/crates/plan-selection/src/search/milp/model.rs @@ -812,6 +812,19 @@ pub(super) fn solve_with_objective( } } if let Some(projection) = projection { + projection.constrain_input_tuples( + &mut variables, + &candidate_selected, + &connections, + &mut constraints, + check_deadline, + )?; + projection.constrain_reader_contexts( + &candidate_selected, + &objective_reach, + &mut constraints, + check_deadline, + )?; projection.constrain_observations( &candidate_selected, &connections, diff --git a/crates/plan-selection/src/search/milp/schedule_selection_tests.rs b/crates/plan-selection/src/search/milp/schedule_selection_tests.rs new file mode 100644 index 000000000..16767b398 --- /dev/null +++ b/crates/plan-selection/src/search/milp/schedule_selection_tests.rs @@ -0,0 +1,355 @@ +use super::*; +use crate::search::milp::model::{solve_direct_cost_proposal_with_metrics, CandidateSolveMetrics}; +use asap_logical_asap_optimization::memo::MemoGroupId; +use asap_types::workload::{QueryRecurrence, RepeatedDemand, RepetitionInterval}; + +fn union(queries: &[QueryRecurrence], mask: usize) -> f64 { + crate::cost::execution_demand::evaluation_rate( + queries + .iter() + .enumerate() + .filter(|(q, _)| mask & (1 << q) != 0) + .map(|(_, query)| (query, None)), + 3600.0, + ) + .unwrap() +} +fn schedules(queries: &[QueryRecurrence]) -> QueryScheduleRates { + use asap_types::workload::{Predictability, Query, QueryRequirements, QueryWorkloadEntry}; + let entries: Vec<_> = queries + .iter() + .map(|recurrence| QueryWorkloadEntry { + query: Query("m".into()), + requirements: QueryRequirements { + accuracy: asap_types::ir::properties::AccuracyTarget::exact(), + max_latency_ms: None, + }, + predictability: Predictability::Predictable { known_at: None }, + recurrence: recurrence.clone(), + range_evaluation: None, + }) + .collect(); + QueryScheduleRates::new(&entries, 3600.0, None).unwrap() +} +pub(super) fn rate_schedule(rates: &[f64]) -> QueryScheduleRates { + schedules( + &rates + .iter() + .map(|rate| { + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval( + (1000.0 / rate) as u32, + ))) + }) + .collect::>(), + ) +} +fn node(input: Option) -> PhysicalCandidate { + PhysicalCandidate { + group: MemoGroupId(0), + local_cost: 0.0, + producer_requirements: input + .into_iter() + .map(|id| ProducerRequirement { + compatible_producers: vec![PhysicalCandidateId(id)].into(), + }) + .collect(), + } +} +fn root(id: usize) -> QueryRootCandidates { + QueryRootCandidates { + compatible_candidates: vec![PhysicalCandidateId(id)], + } +} + +// Boundary demand is the evaluator's union rate; upstream held memory uses +// the rate-weighted mean, not the arithmetic mean or the sum of root rates. +#[test] +fn heterogeneous_intervals_price_boundary_rates_and_weighted_lifetimes() { + let queries = [500, 1000] + .map(|ms| QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms)))); + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(Some(0)), node(Some(1)), node(Some(1))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + let profile = DirectCostProfile { + schedules: Some(schedules(&queries)), + query_rate: 1.0, + request_latency_seconds: 0.5, + request_cost: 7.0, + nodes: vec![ + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 2.0, + work_seconds: 1.0, + compaction_seconds: 0.0, + held_memory_cost_per_second: 3.0, + ..Default::default() + }, + NodeCostProfile { + per_request: false, + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 1.0, + work_seconds: 2.0, + compaction_seconds: 0.0, + held_memory_cost_per_second: 4.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + work_seconds: 3.0, + compaction_seconds: 0.0, + ..Default::default() + }, + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + work_seconds: 4.0, + compaction_seconds: 0.0, + ..Default::default() + }, + ], + }; + assert_eq!( + profile + .schedules + .as_ref() + .unwrap() + .union_rate([true, true].into_iter()), + 3.0 + ); + let result = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((result.direct_modeled_cost.unwrap() - 167.5).abs() < 1e-8); + assert!((result.model_lower_bound.unwrap() - 167.5).abs() < 1e-8); +} + +// Same-interval readers can share one evaluation only at compatible phases. +// Compare with the evaluator's policy rather than assuming equal rates merge. +#[test] +fn aligned_and_offset_phases_use_evaluator_union_policy() { + use asap_types::workload::TimestampMs; + let graph = CandidateDependencyGraph { + candidates: vec![node(None)], + query_roots: vec![root(0), root(0)], + ..Default::default() + }; + for (second_phase, expected) in [(0, 7.0), (500, 14.0)] { + let queries = [0, second_phase].map(|phase| { + QueryRecurrence::Repeated(RepeatedDemand::FixedIntervalAt { + interval: RepetitionInterval(1000), + evaluation_phase: TimestampMs(phase), + }) + }); + let profile = DirectCostProfile { + schedules: Some(schedules(&queries)), + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + nodes: vec![NodeCostProfile { + per_request: false, + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 3.0, + work_seconds: 2.0, + compaction_seconds: 0.0, + held_memory_cost_per_second: 2.0, + materialized: true, + raw_retention: None, + }], + }; + let result = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((result.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + // HiGHS epigraph feasibility is tolerance-based; the decoded selected + // cost above is recomputed independently and remains exact here. + assert!( + (result.model_lower_bound.unwrap() - expected).abs() <= 1e-7 * expected, + "bound={:?}, expected={expected}", + result.model_lower_bound + ); + } +} + +// Every reader can independently choose either materialized producer. +// Compare the joint optimum with four independently priced assignments. +#[test] +fn heterogeneous_selection_matches_exhaustive_reader_assignments() { + let queries = [500, 1000] + .map(|ms| QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms)))); + let schedule = schedules(&queries); + let mut graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None), node(Some(0)), node(Some(0))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + for id in [2, 3] { + graph.candidates[id].producer_requirements[0].compatible_producers = + vec![PhysicalCandidateId(0), PhysicalCandidateId(1)].into(); + } + let profile = DirectCostProfile { + schedules: Some(schedule.clone()), + query_rate: 1.0, + request_latency_seconds: 0.5, + request_cost: 7.0, + nodes: vec![ + NodeCostProfile { + per_request: false, + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 10.0, + work_seconds: 1.0, + compaction_seconds: 0.0, + held_memory_cost_per_second: 3.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + per_request: false, + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 2.0, + work_seconds: 0.5, + compaction_seconds: 0.0, + held_memory_cost_per_second: 1.0, + materialized: true, + raw_retention: None, + }, + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 1.0, + work_seconds: 2.0, + compaction_seconds: 0.0, + ..Default::default() + }, + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 1.0, + work_seconds: 3.0, + compaction_seconds: 0.0, + ..Default::default() + }, + ], + }; + let expected = [[0, 0], [0, 1], [1, 0], [1, 1]] + .into_iter() + .map(|assignment| { + let latency = [ + 2.5 + [1.0, 0.5][assignment[0]], + 3.5 + [1.0, 0.5][assignment[1]], + ]; + let mut cost = 7.0 + 3.0; + for supplier in [0, 1] { + let mask = assignment + .iter() + .enumerate() + .filter(|(_, chosen)| **chosen == supplier) + .fold(0, |mask, (q, _)| mask | (1 << q)); + let longest = latency + .iter() + .enumerate() + .filter(|(q, _)| mask & (1 << q) != 0) + .map(|(_, time)| *time) + .fold(0.0, f64::max); + cost += union(&queries, mask) + * ([10.0, 2.0][supplier] + [3.0, 1.0][supplier] * longest); + } + cost + }) + .fold(f64::INFINITY, f64::min); + let result = solve_direct_cost_proposal_with_metrics( + &graph, + &[], + None, + &mut CandidateSolveMetrics::default(), + &profile, + ) + .unwrap() + .unwrap(); + assert!((result.direct_modeled_cost.unwrap() - expected).abs() < 1e-8); + // HiGHS epigraph feasibility is tolerance-based; the decoded selected + // cost above is recomputed independently and remains exact here. + assert!( + (result.model_lower_bound.unwrap() - expected).abs() <= 1e-7 * expected, + "bound={:?}, expected={expected}", + result.model_lower_bound + ); +} +// A rare slow reader's weighted held lifetime can dominate cost despite a +// tiny rate. Reject that numerically unsupported mean before optimization, +// rather than erase its contribution in a relaxed equality. +#[test] +fn extreme_reader_rates_are_explicitly_unsupported() { + let queries = [1, u32::MAX] + .map(|ms| QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(ms)))); + let graph = CandidateDependencyGraph { + candidates: vec![node(None), node(None), node(Some(0)), node(Some(0))], + query_roots: vec![root(2), root(3)], + ..Default::default() + }; + let profile = DirectCostProfile { + schedules: Some(schedules(&queries)), + query_rate: 1.0, + request_latency_seconds: 0.0, + request_cost: 0.0, + nodes: vec![ + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 1.0, + held_memory_cost_per_second: 1e6, + ..Default::default() + }, + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + execution_cost: 2.0, + held_memory_cost_per_second: 1e6, + ..Default::default() + }, + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + work_seconds: 1.0, + compaction_seconds: 0.0, + ..Default::default() + }, + NodeCostProfile { + ingestion: false, + standing_cost_per_second: 0.0, + work_seconds: 1e9, + compaction_seconds: 0.0, + ..Default::default() + }, + ], + }; + let rates = &profile.schedules.as_ref().unwrap().per_query_rates(); + assert!( + rates[1] * 1e9 * 1e6 > 1e8, + "the rare reader has meaningful memory cost" + ); + assert!(matches!( + profile.validate(&graph), + Err(CandidateSelectionError::UnsupportedDirectCost(_)) + )); +} diff --git a/crates/plan-selection/src/search/refinement/planner.rs b/crates/plan-selection/src/search/refinement/planner.rs index 76cf52c9f..34c21a1af 100644 --- a/crates/plan-selection/src/search/refinement/planner.rs +++ b/crates/plan-selection/src/search/refinement/planner.rs @@ -138,13 +138,15 @@ pub(in crate::search) fn optimize_configured_candidates( &mut joint_fallback_reasons, )?; let mut direct = Vec::new(); - // Keep the older restricted profile only as a bounded fallback when the + // Use a fixed-interface profile as a bounded fallback when the // joint interface catalog cannot be certified within the planning budget. + // Both adapters share the same construction deadline, preserving time for + // the feasibility search even when direct pricing is unsupported. if joint.is_none() { - if let Some(seed) = direct_cost_seed(configured, context, &exclusions, false)? { + if let Some(seed) = direct_cost_seed(configured, &joint_context, &exclusions, false)? { direct.push(seed); } - if let Some(seed) = direct_cost_seed(configured, context, &exclusions, true)? { + if let Some(seed) = direct_cost_seed(configured, &joint_context, &exclusions, true)? { direct.push(seed); } } @@ -282,9 +284,11 @@ fn joint_cost_seed( let sizes: Vec<_> = profile.candidates.iter().map(Vec::len).collect(); for options in &mut profile.candidates { for option in options { - option - .compatible_suppliers - .retain(|binding| binding.price_option < sizes[binding.producer.0]); + option.compatible_input_tuples.retain(|tuple| { + tuple + .iter() + .all(|binding| binding.price_option < sizes[binding.producer.0]) + }); } } Ok(Some((configured.inventory.graph.clone(), profile))) @@ -315,7 +319,9 @@ fn direct_cost_seed( }; let mut extracted = match extracted { Ok(Some(extracted)) => extracted, - Ok(None) | Err(DirectProfileError::DeadlineExceeded(_)) => return Ok(None), + Ok(None) + | Err(DirectProfileError::DeadlineExceeded(_)) + | Err(DirectProfileError::Unsupported(_)) => return Ok(None), Err(error) => return Err(error.to_string()), }; if context @@ -1113,6 +1119,46 @@ mod tests { ) } + // An optional direct-price proposal must not abort planning when its + // schedule construction budget expires; the ordinary search remains usable. + #[test] + fn unsupported_schedule_profiles_leave_both_seed_fallbacks_available() { + let (configured, roots, mut entries, data) = fixture(); + entries[1].recurrence = QueryRecurrence::Unknown; + let context = CandidatePlanningContext { + deadline: Some(std::time::Instant::now()), + refine_materialization: false, + original_roots: &roots, + entries: &entries, + data: &data, + models: PlanningModels::builtin(), + calibration: CalibrationFit::default(), + }; + let mut reasons = Vec::new(); + assert!(joint_cost_seed(&configured, &context, &[], &mut reasons) + .unwrap() + .is_none()); + assert!(!reasons.is_empty()); + for latest_panes in [false, true] { + assert!(direct_cost_seed(&configured, &context, &[], latest_panes) + .unwrap() + .is_none()); + } + // Malformed demand is a real input error, not an unsupported model. + let mut invalid_entries = entries.clone(); + invalid_entries[1].recurrence = + QueryRecurrence::Repeated(RepeatedDemand::FixedInterval(RepetitionInterval(0))); + let invalid_context = CandidatePlanningContext { + deadline: None, + entries: &invalid_entries, + ..context + }; + assert!(joint_cost_seed(&configured, &invalid_context, &[], &mut reasons).is_err()); + for latest_panes in [false, true] { + assert!(direct_cost_seed(&configured, &invalid_context, &[], latest_panes).is_err()); + } + } + // One logical group can serve a repeated reader through panes and an // unpredictable reader locally; only selected pane consumers need eligibility. #[test] diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index c5f7a8c20..2ea1befbe 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -202,17 +202,21 @@ Memory-time and capacity answer different questions. A state occupying `m` bytes The direct-cost objective computes the total modeled cost of the selected execution DAG. It does not precompute one fixed total cost for each candidate or enumerate every complete DAG. Before solving, the planner computes each supported candidate's local cost parameters from its configuration and compatible suppliers. Inside MILP, the selected nodes and connections determine execution frequency and repeated work, consumer-specific operations, state holding costs, shared raw retention, and query overhead. A shared producer owns its charge once, including every required execution; its consumers' work is charged separately. **Local candidate cost parameters are precomputed; the complete selected DAG's cost is determined jointly by MILP.** -Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Whole-result targets therefore add conservatively sized capacity candidates before physical costing, preserving the original candidates. Matching schedules use a compact demand model. Larger sets of fixed-interval schedules use phase activation and bounded integer rate totals, avoiding enumeration of every reader subset; rates follow the evaluator's accounting policy, including unknown phases. The normalized rate total is bounded at 4,096; small heterogeneous workloads can also use evaluator-derived subset tables. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. +Direct pricing currently covers certified query-time exact and fixed sketch configurations, including KMV, KLL, DDS, Count-Min, CountSketch, Theta, and source-certified HLL. Capacity must already satisfy the applicable failure-probability sizing: a later resize would invalidate precomputed prices. Whole-result targets therefore add conservatively sized capacity candidates before physical costing, preserving the original candidates. Matching schedules use a compact demand model. Mixed fixed intervals, known/unknown phases, estimated rates, scheduled requests, one-off requests and range evaluations use finite demand atoms following the evaluator's accounting policy. Different fixed intervals add; an unknown phase joins the first selected known phase on its interval; estimates add; nonrepeating demand uses the largest selected batch. These rules describe the configured cost model, not a new runtime scheduler. The production encoding needs neither a query-subset table nor an integer rate lattice. There is one production mixed-schedule encoding; the former subset and lattice encoders are removed. Behavioral selection tests use the current encoder and compare against separately computed evaluator costs. The model includes raw retention, request overhead, and latency-dependent temporary state, and checks the decoded total against the complete DAG evaluator. Accuracy still requires full output validation. Direct pane pricing covers query-time rebuilding, eligible state retained across evaluations, and fixed-retention ingestion maintenance. Source-certified HLL panes use the same population and failure-probability checks as HLL whole-window builds. Rebuilt state pays for execution demand and temporary lifetime; retained state pays for resident panes and refreshes. Ingestion work is excluded from query latency; the maximum compaction delay among reached ingestion states contributes to query latency and temporary holding cost. -**Supplier-dependent prices are joint decisions.** For each supported unary candidate, costing derives finite local pricing options from its compatible suppliers' priced output interfaces. An option records local CPU/state costs, output statistics, upstream work and the resulting certified worker count. For example, two supplier configurations requiring 0.9 and 2.4 CPU-seconds per ingestion second produce one-worker and three-worker options; their compaction delays need not agree. A downstream reader can likewise have different merge/extraction costs for different state capacities. The optimizer retains these options rather than keeping only the first price class. +**Supplier-dependent prices are joint decisions.** For each supported candidate, costing derives finite local pricing options from complete ordered tuples of its suppliers' priced output interfaces. Executable binary operations and relational joins preserve input positions even when two positions use the same producer. Shared ingestion ancestors contribute their work once to a worker-count certificate. An option records local CPU/state costs, output statistics, upstream work and the resulting certified worker count. For example, two supplier configurations requiring 0.9 and 2.4 CPU-seconds per ingestion second produce one-worker and three-worker options; their compaction delays need not agree. A downstream reader can likewise have different merge/extraction costs for different state capacities. The optimizer retains these options rather than keeping only the first price class. -Let `price[c,k]` select pricing option `k` of candidate `c`. Require `Σ_k price[c,k] = selected[c]`, and permit a selected consumer/producer connection only with a compatible pair of pricing options. Thus all consumers of one shared candidate observe the **same** chosen interface. The implementation lifts these options into an internal MILP graph with mutual exclusion per original candidate, reuses the demand/lifetime cost formulation, and projects the result back to the original candidate DAG. Pricing options are not extra executions or extra copies of a shared state. Rejected assignments and reader/resource constraints retain their original physical identities. +Let `price[c,k]` select pricing option `k` of candidate `c`. Require `Σ_k price[c,k] = selected[c]`, and permit a selected consumer/producer connection only with a compatible pair of pricing options. For multiple inputs, a tuple selector chooses one complete certified combination and determines every input connection; separately compatible suppliers cannot form an unpriced combination. Thus all consumers of one shared candidate observe the **same** chosen interface. The implementation lifts these options into an internal MILP graph with mutual exclusion per original candidate, reuses the demand/lifetime cost formulation, and projects the result back to the original candidate DAG. Pricing options are not extra executions or extra copies of a shared state. Rejected assignments and reader/resource constraints retain their original physical identities. -Local interface generation deduplicates identical outputs, costs and upstream-work certificates; it does not enumerate complete workload DAGs. It currently covers the supported unary operators. Acyclic catalogs use dependency order; catalogs with potential cycles propagate local interfaces for at most the candidate count, sufficient for every legal unary path. Query-time upstream work is not part of an ingestion worker certificate, preventing irrelevant path histories from multiplying equivalent interfaces. Final DAG ordering and one pricing option per physical candidate exclude cyclic executions. All legal scan-lookback classes participate jointly, with exact context matching on each scan connection; sharing one wider scan across narrower consumers requires an additional maximum-demand witness and remains outside this encoding. Multi-input pricing or domains exceeding the finite interface/time budget use the existing validated fallback. Generation reserves time for fallback rather than consuming the entire planning budget. A worker count is derived from the selected upstream work, not an independently tunable cluster size. The bounded production seed uses the first interchangeable execution copy while retaining all its pricing options; the full search retains every copy. This is a restricted proposal, not a proof that additional copies are unnecessary. Reports distinguish joint price-option solves from fixed-profile direct solves. +Local interface generation deduplicates identical outputs, costs and upstream-work certificates; it does not enumerate complete workload DAGs. It covers supported single-input and multi-input operators and timing configurations. Acyclic catalogs use dependency order; catalogs with potential cycles propagate local interfaces for at most the candidate count, sufficient for every legal acyclic dependency depth. Query-time upstream work is not part of an ingestion worker certificate, preventing irrelevant path histories from multiplying equivalent interfaces. Final DAG ordering and one pricing option per physical candidate exclude cyclic executions. All legal scan-lookback classes participate jointly, with exact context matching on each scan connection; sharing one wider scan across narrower consumers requires an additional maximum-demand witness and remains outside this encoding. Uncertified operators or domains exceeding the finite interface/tuple/time budget use the existing validated fallback. Generation reserves time for fallback rather than consuming the entire planning budget. Joint and fixed-price extraction share that reserved deadline. Unsupported profiles, including schedule-atom construction timeouts, skip the optional direct proposal and continue through validated search; invalid profile errors remain fatal. A worker count is derived from the selected upstream work, not an independently tunable cluster size. The bounded production seed uses the first interchangeable execution copy while retaining all its pricing options; the full search retains every copy. This is a restricted proposal, not a proof that additional copies are unnecessary. Reports distinguish joint price-option solves from fixed-profile direct solves. -Matching range requests on the same fixed schedule are directly priced for rebuilt and ingestion-maintained state: evaluation frequency includes every range step, request overhead remains per request, and maintained panes retain the required historical span. Query-kept range state, differing range contexts, uncertified supplier interfaces, and schedules outside the bounded encodings still require evaluator refinement. Unsupported configurations remain in the full candidate catalog and currently use evaluator-based refinement; these are implementation limitations to remove, not the intended final architecture. The restricted subspace's optimum is not a full-catalog optimum. Reports expose direct-cost solve counts, validated costs, and fallback reasons so a successful fallback cannot be mistaken for direct-model coverage. +Range demand distinguishes evaluations from requests. Query-local state follows the union of evaluation offsets, while eligible query-kept state refreshes once per request. Pane pricing options carry retention-span tiers: a selected tier must cover every selected reader and have a reader witnessing that maximum. This prevents both undercharging long ranges and charging retention demanded only by unselected readers. Existing pane alignment, freshness and materialization restrictions still apply. + +For a nonmaterialized node, let `A` be its normalized average holding time, `r_q` binary query reach, `e_q` the evaluation rate, and `L_q` query latency. Encode `Σ e_q × r_q × A = Σ e_q × r_q × L_q` by linearizing each binary/continuous product. Multiplying `A` by each selected execution-demand atom is linearized similarly. This models finite nonnegative rates without enumerating denominators or reader subsets. When all reached readers have zero expected demand, holding time falls back to the longest reader latency, matching the evaluator. Materialized state uses the longest reached-query latency. + +Common-step ranges compress to nested coverage counts, even for very long spans. Mixed-step ranges group identical reader-coverage sets. Small geometry domains use arithmetic intersections and exact inclusion/exclusion counts, so large spans need no timestamp expansion; larger domains use bounded offset construction. Excessive domains, ill-conditioned rates, or exhausted construction budgets report a fallback rather than coarsening the schedule. The evaluator uses the same exact intersection-count primitive when validating large mixed ranges. Unsupported contexts remain in the complete candidate catalog for validated evaluator refinement. Restricted-subspace bounds are not full-catalog optimum certificates. Reports expose direct-cost solve counts, validated costs and fallback reasons. A candidate whose evaluator cost overflows the representable range is skipped with a `cost_not_representable` diagnostic so scalar-price search can still find a validated plan. This is an unknown cost, not proof of infeasibility: if such assignments were excluded, the search reports a conservative zero lower bound and cannot claim a full-domain optimum or infeasibility. Invalid settings and other evaluator errors remain errors.