[zzylol]

Before this PR

Approximate candidates received an extra search surcharge, and the rollup-heavy viewer workload could return separate exact aggregates despite a cheaper valid shared KMV plan. Complete DAG costs depended on evaluator refinement. The document did not define safe local DP boundaries.

After this PR

Accuracy is feasibility without an approximate-candidate surcharge. The original 30-second viewer workload selects a shared KMV build and regrouping merge while retaining both accuracy targets. MILP directly prices certified complete DAGs using execution demand, temporary state lifetime, resident state, raw-retention maxima, and request overhead; decoded prices are checked against the complete evaluator.

Production acceptance now also covers SQL distinct rollup, three SQL KLL groupings, and PromQL host/job quantiles. The SQL cases select shared rollups. The PromQL fixture explicitly supplies held-row facts, beats independently built KLLs, and permits a cheaper validated exact plan instead of forcing approximation.

Implemented scope and remaining limits

• Certified exact/fixed-sketch pricing, including source-certified HLL; rebuilt, retained-query-time, and fixed-worker ingestion panes, including compaction latency.
• Matching range requests on an equal fixed schedule for rebuilt and ingestion-maintained state. Request overhead remains per request rather than per range step.
• Larger fixed-interval workloads use exact evaluator phase accounting and bounded integer rate totals (maximum 4,096), avoiding all reader subsets. Small heterogeneous workloads retain the subset encoding.
• Whole-result accuracy targets add larger capacity candidates before costing; original candidates remain available.
• Selected connections enforce Stage 2's highest-shared non-summary materialization rule, including graph validation, component projection, and warm starts.
• Numerical range guards and independent optimal-objective checks prevent false direct-cost certificates. Dependency-aware latency bounds preserve normal workload coverage.
• Remaining: differing range contexts, query-kept range state, general supplier-dependent price configurations, and schedules outside supported bounded encodings use evaluator refinement. Restricted optima never certify the full catalog.
• General conditional DP/Pareto candidate compression remains a design proposal. Dependency-bound computation and rate-total enumeration are not that planner algorithm.

Scenarios handled by this PR

Scenario	Before	After / acceptance
Viewer T11: fine (service, endpoint) distinct counts and coarse service counts	Timed search could miss cheaper shared KMV	Production search selects a validated shared KMV rollup under the 30-second search budget
SQL distinct counts by (region, service) and service	Controlled candidates were tested, but production search was not covered	Production acceptance requires one shared KMV build and one rollup
SQL P99 by (region, service), region, and service	Controlled candidates were tested, but production search was not covered	Production acceptance requires one shared KLL build and two rollups
PromQL quantile by (host, job) and job, with explicit held-row facts	Whole-result capacities and scan-price contexts could exclude useful directly priced candidates	Adds scoped capacities and scan contexts; production direct incumbent beats independent KLLs, while final selection may use a cheaper accurate exact plan
Rebuilt, retained-query-time, and fixed-worker ingestion panes	Complete objective coverage was restricted	Direct prices include execution frequency, residency, held lifetime, and fixed-worker compaction; checked against the complete evaluator
Matching range requests on the same fixed schedule	Range requests were excluded from direct pricing	Rebuilt and ingestion state include range-step demand and historical pane retention; fixed overhead remains per request
More than three commensurate fixed schedules	Heterogeneous direct pricing used a small reader-subset table	Phase activation plus bounded integer rate totals; five-reader evaluator parity and schedule-accounting tests
A shared ordinary node lies below another shared node	MILP could propose a materialization that Stage 2 later rejected	Selected-consumer constraints enforce highest-shared-node eligibility before evaluation

Not covered: differing range contexts, retained-query-time range refresh, arbitrary supplier-dependent output/worker configurations, large rate lattices, and large mixtures of non-fixed recurrences. These remain evaluator-refined; this PR does not claim full-catalog direct-cost optimality.

Key code interfaces added / modified

• Direct profiles: search/costing/direct_profile.rs.
• DAG objective and schedule encodings: search/milp/direct_cost.rs, direct_schedule.rs, direct_fixed_schedule.rs.
• Candidate capacities and selected-reader constraints: search/candidates/accuracy_configurations.rs, graph/configuration handling, MILP, and refinement integration.
• Production acceptance: rollup_controlled.rs, production_rollup_workloads.rs.

Major algorithm changes

Jointly price node activation, execution boundaries, consumer demand, state lifetime, and shared resources. Certified scan-price classes share one planning budget while preserving the full refinement catalog. Apply selected-consumer materialization constraints before evaluation, and generate scoped capacities before pricing.

Validation

• 351 plan-selection library tests pass, including complete-evaluator range and five-reader heterogeneous-schedule parity, exhaustive tiny-choice comparisons, and materialization/warm-start regressions.
• 22 focused integration tests pass, including the original viewer workload and three additional production workloads.
• Strict clippy passes for plan selection and the new integration test; formatting passes.
• Extreme-latency regression reproduces an incorrect solver bound before the fix and passes afterward. Missing raw facts are not silently invented; the PromQL fixture documents its explicit snapshot assumption.
• These are modeled costs and bounded-search results, not measured runtime savings or proof of full-catalog optimality. Remote CI must pass for the current head before merge.

CI presentation-test follow-up

The display-cap scenario now uses two instant quantiles in exhaustive mode, retaining assertions for a real cap, the selected plan, and archived DAG links without expanding window panes. The per-node cost scenario uses the normal bounded planner and retains its timing/CPU accounting checks. Both focused tests pass (0.30s and 30.12s). This changes test fixtures, not production planning or costs. Full CI is rerunning; the original CI artifacts did not establish the precise cause of the empty selections.

Interrupted evaluation archive consistency

A compiler deadline can interrupt evaluation after a physical plan is reconstructed. Such a plan now stays out of the evaluated archive until its selector offer completes; completed accepted and rejected plans remain archived. The same rule applies after complete-catalog calibration. This preserves the viewer contract that every archived physical plan has either a cost or a rejection.

A deterministic compiler-timeout regression fails before the fix and passes afterward. All 10 objective-search tests pass, including both frozen and complete-calibration timeout cases. The original example-1 archive accounting test passes unchanged (30.56s).

CI repair and current-main integration

Scenario: three repeated/different-window SQL COUNT DISTINCT outputs with an explicit confidence target and a 10-second planning budget. A feasibility seed previously kept choosing estimators with an unresolved confidence contract. The seed now excludes those proposals; the complete catalog and final accuracy validation remain unchanged, and there is no approximate-candidate price surcharge. Includes latest main (#938 family capability consistency). This preserves the original test budget and accuracy requirements.

Candidate-cost overflow recovery

Scenario: Example 2 can select a physical configuration whose analytical cost arithmetic overflows. Previously that stopped search before a usable incumbent. Scalar-price search now skips the unpriceable assignment, records cost_not_representable, and continues looking for a validated plan. Unknown cost is not infeasibility: exclusions of unpriceable assignments invalidate full-domain optimality/infeasibility claims and reset the reported bound conservatively. Other evaluator errors remain fatal. The new regression covers both a later feasible candidate and an entirely unpriceable domain; Example 2 retains its original requirements and budget.

TopK contract regression

Scenario: a temporal TopK search visits a tighter error allocation first, with heap capacity 200 instead of the default 100. The integration test now verifies every visited heap-backed candidate preserves the requested k, minimum capacity, entity identity and update expression, rather than requiring a fixed first-candidate capacity. Default sizing remains covered by its unit test. The original assertion failed locally; all 13 tests in promql_to_post_asap pass after this correction.

Stable sketch execution and coverage fixtures

Scenarios: a search archive may contain tightened, large sketch configurations or multiple KLL capacities. Rollup execution and coverage tests now use the existing explicit local-rule fixtures, so they test all intended sketch rules and coverage semantics without depending on solver visitation. The rollup memory limit and result assertions remain unchanged. The former fixtures failed locally; corrected rollup (3) and coverage (8) tests pass. The integration package was exercised in batches: 330 passed, 23 pre-existing ignored; affected tests also pass strict clippy. Full remote CI passed on this head before merge.