Repository navigation
SQL per-entity fold over an estimate has no guarantee: SELECT src, max(p) … GROUP BY src over a DDSketch p #751
Copy link
Copy link
Open
Labels
accuracyAccuracy requirements, guarantees, and propagationAccuracy requirements, guarantees, and propagationenhancementNew feature or requestNew feature or requestnested queriessqlSQL front-end lowering (DataFusion → L2)SQL front-end lowering (DataFusion → L2)
Description
Problem, by example
SELECT src, max(p)
FROM (SELECT src, approx_percentile_cont(bytes, 0.99) AS p FROM flows GROUP BY src) s
GROUP BY srcover flows(src Utf8, bytes Float64), with target AccuracyTarget::max_error(0.01). The inner query has one row per src, so each outer group holds exactly one p, and max(p) returns that value. The DDSketch candidate's p is within relative 0.01 (α = 0.01), so the outer result is within relative 0.01 too.
What Stage 1 composes today (run through asap_integration_tests::planning::logical_candidates at 20d2923, filtered to the candidates that read a DDSketch):
| Query | Root guarantee today | Sound guarantee |
|---|---|---|
inner alone: SELECT src, approx_percentile_cont(bytes, 0.99) AS p FROM flows GROUP BY src |
relative, error 0.01 | relative 0.01 |
global fold: SELECT max(p) FROM (…) s |
relative, error max(0.01), δ over group_row_count |
same |
per-src fold: SELECT src, max(p) FROM (…) s GROUP BY src |
None |
relative 0.01 |
So a per-entity fold over an estimate has no guarantee, and plan selection rejects every approximate candidate for it; only exact plans remain.
Why it happens
- The SQL frontend lowers the outer
GROUP BY srctoAggregate { reduction: Reduce(GroupKeys { keys: [0, 1] }), measures: [Max { col: Some(1) }] }over(src, p): the group keys includep(column 1), presumably added becausepis functionally dependent on the subquery's unique keysrc. groups_by_estimate(crates/logical-asap-optimization/src/per_computation_candidates.rs) sees a non-label, non-time group key (p, Float64) over approximate input and says the grouping may decide groups by an estimate, socomposed_guaranteereturnsNone.- Even with the grouping recognized as one row per entity (
Reduction::PerEntity),fold_statsgivesPropagationStatsforPerEntityonly over a PromQL subquery with aresolution(rows = range / resolution). A SQL per-entity fold has no resolution, so it also returnsNone.
Proposal
- Recognize a fold whose group keys contain a unique key of its input (
src, the inner aggregate's group key) as one row per group: keys functionally dependent on that key (herep) do not decide the grouping, sogroups_by_estimateisfalsefor them. - Give such a fold a row count:
fold_statsreturnsinput_row_count: Some(1)for a per-entity fold over a SQL subquery (each group holds one input row), as it returnsrange / resolutionfor a PromQL subquery. The relative fold then composesmax(0.01)with δ unchanged. - Alternative: have the SQL frontend lower a
GROUP BYover the input's unique key toReduction::PerEntity, and resolve the row count infold_statsonly for that.
Scope and acceptance
- In scope: SQL folds (
max,min,sum,avg) over a subquery grouped by the subquery's own group keys. - Out of scope: column-level guarantee tracking (Track exact vs approximate per column so key-only filters over approximate inputs keep their guarantee #689).
- Acceptance: the per-src query above has a DDSketch candidate whose guarantee is relative with error 0.01;
a_filter_or_group_key_over_an_estimate_has_no_guarantee(aGROUP BY pover the estimate) still givesNone.
Status and related work
Raised as a review note on the SQL fold composition change (#687). Related: #689 (column-level guarantees).
Metadata
Metadata
Assignees
Labels
accuracyAccuracy requirements, guarantees, and propagationAccuracy requirements, guarantees, and propagationenhancementNew feature or requestNew feature or requestnested queriessqlSQL front-end lowering (DataFusion → L2)SQL front-end lowering (DataFusion → L2)
Activity