Skip to content

A partial measured cost table can still mis-rank plans: calibrate per operator kind #791

Open

Description

@zzylol

Problem, by example

Take a per-series p99 over 5 min, quantile_over_time(0.99, m[5m]), queried every minute over 10,000 rows/s and priced on CPU only. There are two candidates:

  • EXACT: an exact rebuild from raw samples at query time (Scan, TimeRange, Aggregate(quantile)).
  • B1: KLL panes maintained at ingestion time.

The deployment's measured cost table covers the KLL (PaneChain 150 ns/row, SummaryMerge 2,000 ns/state, SummaryEstimate 5,000 ns/group), TimeRange (2 ns/row) and Project (1 ns/row). It does not cover Scan or the quantile Aggregate.

Real cost of the unmeasured operators Stage 3 selects (partial table) Full table selects
Scan 40 ns/sample, quantile 20 ns/row B1 EXACT; B1 costs 1.42x more
Scan 100 ns/sample, quantile 50 ns/row B1 B1

These numbers come from the test a_partial_table_can_still_mis_rank_plans in crates/plan-selection/src/lib.rs (PR #789). The gap widens as the unmeasured operators get cheaper; review measured up to 2.6x.

Expected: a partial table should rank plans the way the full table does whenever the unmeasured operators' real costs follow the measured operators of the same kind.

Why it happens

#771 / PR #789 prices a node with no entry as its formula CPU times one calibration factor (CalibrationFit::factor). The factor is Σ measured ÷ Σ formula time over every measured node of every candidate. Here it is dominated by the KLL's 150 ns against the formula's 2 operations per row, so the unmeasured Scan and quantile Aggregate are priced at the sketch's ratio (about 100x) rather than at anything an exact operator was measured at. The factor is a heuristic, documented as such in the crate's "Measured operation costs" docs. Full tables are what make the choice right today.

Proposal

Fit one factor per operator class instead of one global factor, falling back up the hierarchy:

  1. the same operator kind at the same phase;
  2. the same operator kind at either phase;
  3. the same class: sketch summary operators, exact accumulators, relational operators (Scan, Filter, Aggregate, Sort, ...);
  4. the global factor.

An alternative is to refuse, or warn on, a table that misses an operator kind the candidates use, and leave coverage to the runtime's measurement tool. The two can be combined.

Scope and acceptance

  • In scope: the calibration in price_at_scale / select_plan, and reporting which factor priced each node in NodeCost::detail.
  • Out of scope: the runtime's measure_costs coverage.
  • Acceptance: a_partial_table_can_still_mis_rank_plans flips to EXACT once the table measures one exact relational operator near the real scan and aggregate cost (e.g. TimeRange or Filter at 30 ns), and calibration_keeps_unmeasured_nodes_on_the_measured_scale still selects B1.

Status and related

This is a follow-up of #771 and PR #789, which documents the limitation. Related: #761 (Stage 3 CPU under-prediction).

Metadata

Metadata

Assignees

No one assigned

    Labels

    cost-modelCost formulas, statistics, ranking, and selectionenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions