Repository navigation
A filtered query (city = 'NYC') doesn't read its group from a grouped summary another query builds #902
Description
Workload
Three repeating SQL queries on 4535c957, every 10 s with ε=0.01, δ=0.001, each the p90 of value over the last 5 minutes:
- Q1: overall
- Q2:
… AND city = 'NYC' - Q3:
… GROUP BY city
Save as wl.json:
{"query_workload": {"language": {"sql": "datafusion_sql"}, "query_batch": null, "repeating_queries": [{"query": "SELECT percentile_cont(0.9) WITHIN GROUP (ORDER BY value) FROM metrics WHERE ts > now() - INTERVAL '5' MINUTE", "demand": {"fixed_interval": 10000}, "requirements": {"accuracy": {"max_error": 0.01, "failure_probability": 0.001}}, "predictability": {"predictable": {"known_at": null}}}, {"query": "SELECT percentile_cont(0.9) WITHIN GROUP (ORDER BY value) FROM metrics WHERE ts > now() - INTERVAL '5' MINUTE AND city = 'NYC'", "demand": {"fixed_interval": 10000}, "requirements": {"accuracy": {"max_error": 0.01, "failure_probability": 0.001}}, "predictability": {"predictable": {"known_at": null}}}, {"query": "SELECT city, percentile_cont(0.9) WITHIN GROUP (ORDER BY value) FROM metrics WHERE ts > now() - INTERVAL '5' MINUTE GROUP BY city", "demand": {"fixed_interval": 10000}, "requirements": {"accuracy": {"max_error": 0.01, "failure_probability": 0.001}}, "predictability": {"predictable": {"known_at": null}}}]}, "data_workload": {"arrival": {"ingesting": {"rows_per_second": 666.67}}, "sources": {"metrics": {"series_count": 100}}}}cargo run -p asap-devtools --bin stage_pipeline -- --workload wl.json --exhaustive --max-displayed 500 --out out.json \
--table '{"name": "metrics", "columns": [{"name": "ts", "type": "timestamp", "nullable": false}, {"name": "city", "type": "utf8"}, {"name": "host", "type": "utf8"}, {"name": "value", "type": "float64"}], "time_index": 0}'Observation
The selected plan is P103-m3:
Q3, Q1: Scan → TimeRange(5m) → SummaryAgg(KLL per city) [query time, shared]
├→ SummaryEstimate → Q3
└→ SummaryMerge (rollup to one) → SummaryEstimate → Q1
Q2: Scan(city = 'NYC') → PaneChain(KLL, 30 × 10s, ingestion time) → SummaryMerge → SummaryEstimate
Q3's per-city KLL already holds the NYC group, but no candidate answers Q2 by reading that group. Q2 always builds its own KLL, and shares neither the scan nor the build with Q1 or Q3.
Expected: a candidate in which Q2 reads the city = 'NYC' group of Q3's per-city summary, with no build of its own.
Activity
zzylol commentedon Oct 23, 2026
Deferred until the memo search (#879, #888) lands. This sharing/reuse rule will be added as a memo transformation rather than to the legacy search.
devtools: planner_fuzz, a metamorphic fuzz tester for missed reuse an…
zzylol commentedon Oct 27, 2026
Confirmed that the memo migration has not added the required filtered-group reuse transformation.
Using the issue's SQL table nullability (only ts explicitly non-nullable), the original three-query workload plans on the local current-engine diagnostic tree, but the filtered query still has its own raw scan/range/aggregate path. A 30 s enforced-accuracy run returned cost 0.3723209703 with exact aggregates; this cost is a bounded-search result, not an optimum.
The structural reason is upstream of MILP: source predicates remain part of scan/input identity, while existing state sharing/rollup requires the same input rows and compatible filters. The memo reuse rules have no transformation that turns the raw
city='NYC'predicate into a reader of the unfiltered per-city producer. Scan widening intentionally preserves source predicates. Also, a plain NonASAP Filter currently rejects a State input (NonASAPOpstructural validation), so inserting an ordinary Filter directly on a KLL state is not an available implementation.The fix needs an explicit group-selection/readout candidate, with preserved accuracy and SQL empty-input semantics. Merely filtering Q3's result is insufficient for the original ungrouped Q2: when NYC is absent, Q2's global percentile must produce its NULL aggregate result, whereas filtering grouped output produces zero rows.
A separate diagnostic variant with a non-nullable value column exposed another bug: a grouped-to-global quantile merge was marked non-nullable, causing the rollup output-contract check to abort planning.
quantile_may_be_emptydid not account for a merge with an empty group_by. A local correction removes that abort. This is an additional variant, NOT the root cause of the original nullable-column example.No fix for the original filtered-reader transformation is complete yet. Changes/tests are uncommitted; no PR or commit has been published.
zzylol commentedon Oct 27, 2026
Implemented in PR #943. A key-equality reader reuses the existing grouped quantile result, including a KLL readout, without its own summary build. A reduction over at most one selected group preserves the single NULL row required when the group is absent. Execution tests cover missing groups, empty input and NULL-only samples; a multi-group filter does not inherit a single quantile rank bound. Original workload cost falls from 0.3723209703/s to 0.1963089020/s.
Final local validation: 690 focused tests passed, plus workspace format and all-target/all-feature Clippy checks.