Skip to content

refactor(rqe-optimizer): move rqe-optimizer from sketch-bench - #915

Open
milindsrivastava1997 wants to merge 58 commits into
mainfrom
refactor/move-rqe-optimizer
Open

milindsrivastava1997 wants to merge 58 commits into
mainfrom
refactor/move-rqe-optimizer

Conversation

@milindsrivastava1997

Copy link
Copy Markdown
Contributor

Merge with "Create a merge commit". Squash or rebase would flatten or rewrite the 52 imported sketch-bench commits this PR carries on purpose.

What

Moves the RQE deployment optimizer from ProjectASAP/sketch-bench into this repo as the workspace crate rqe-optimizer/, together with its full history (from sketch-bench@c750fa6):

  • the crate itself (MILP, candidates, cost model, saturation accuracy, AutoSketch baseline), with its tests and examples, including the §6.3 runner autosketch_vs_asap;
  • the scripts that run, plot and summarize the eval, now under rqe-optimizer/scripts/;
  • the committed results (rqe-optimizer/results/, 41 MB on disk, about 20 MB in git history);
  • the design docs, now under rqe-optimizer/docs/.

asap-planner-rs now depends on it by path. Before, it used a git dependency on sketch-bench main whose lockfile pinned an old rev (ASAPQuery#836).

What stays in sketch-bench: the benchmarks and the analysis on top of them, including what turns measurements into solver input: the saturation study, the cost table, and the eval-table exporters. The crate's only sketch-bench dependency left is aqpbm-core, for the atomic-cost table's row types, pinned to c750fa6.

Commits

  1. Import with history: the sketch-bench history is filtered to the moved paths and merged with --allow-unrelated-histories. Commit messages' bare #N (N ≤ 231) became ProjectASAP/sketch-bench#N; larger numbers were already ASAPQuery's and stay bare.
  2. Workspace wiring: a new workspace member, a path dependency from the planner, and good_lp/csv as direct dependencies.
  3. Style: the moved Python had never run this repo's hooks. It's now black-formatted, with an E731 fix and a shellcheck directive. check-added-large-files skips rqe-optimizer/results/.
  4. References: paths from sketch-bench's root are rewritten.
    • Moved files are now rqe-optimizer/....
    • Files that stayed are written sketch-bench/..., and commands use $SKETCH_BENCH.
    • Bare sketch-bench numbers in the files are now sketch-bench#N.
  5. Planner docs: comments in asap-planner-rs no longer call it "sketch-bench's rqe-optimizer".

Verification

  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings and cargo fmt --check pass.
  • cargo test -p rqe-optimizer --all-targets and -p asap_planner pass. That includes the runner's tests, and the planner now builds against the current solver rather than the old pinned rev.
  • The moved runner reproduces the committed results exactly (all fields but timings) for anomaly-base-0, perf-base-0 and anomaly-drilldown_endpoints-50.
  • summarize_case_studies.py regenerates summary.md byte for byte after black.

The companion sketch-bench PR removes the moved files. Merge this one first.

🤖 Generated with Claude Code

milindsrivastava1997 and others added 30 commits October 4, 2026 15:53
…#129)

* Document sliding RQE deployment model

* Clarify RQE accuracy terminology

* Implement sliding RQE deployment candidates

* Add safe streaming RQE candidate search

* Add HiGHS MILP RQE solver

* Verify MILP against brute-force search

* Clarify RQE optimizer status

* Add bounded RQE mapping samples

* Warn before eager RQE enumeration
…-bench#137)

* rqe-optimizer: price plans per EC2 machine family

Score retained memory per active deployment, (x + max S) / y instances,
and add milp::minimize_cost, which minimizes the hourly price on one EC2
machine family in fractional instances (whichever of CPU or memory binds).
Prices are a committed on-demand snapshot from scripts/fetch_ec2_pricing.py.
Dominance pruning also compares retained memory so it cannot drop a
candidate that is cheaper under the new objective.

Part of #777.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: label small_problem MILP output for both objectives

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: scale the MILP so tiny costs and latencies solve exactly

HiGHS's absolute gap (1e-6) and feasibility tolerance (1e-7) exceed
real plan costs (~1e-6 $/hour) and query latencies (µs). Rows are now
scaled by the plan's demand without sharing, and per-RQE memory and
latency bounds become f64 exclusions of the choices over them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…aseline (ProjectASAP/sketch-bench#135)

* rqe-optimizer: add AutoSketch-Adapted per-RQE baseline

Port of AutoSketch Algorithm 4 (LHS seeds, feasibility-directed neighbor
search, pruning, stopping) from ASAPQuery-backend autosketch_comparison.rs,
generalized to each sketch variant's measured parameter axes. Each RQE is
searched independently and gets a dedicated x=S sliding deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: follow AutoSketch's EXAMINE rule and seeding more closely

- Evaluate and expand each configuration once (EXAMINE rule 1); an
  evaluated configuration is never re-queued.
- Seed the cheapest configuration only when LHS hits no measured point,
  as the comment already said.
- Compare resources as (memory, insert CPU) instead of adding bytes and
  seconds.
- Document the cross-variant search as an extension over the paper.
- Test the single-axis stopping rule, the cross-variant choice, a
  higher-is-better floor and LHS distinctness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ProjectASAP/sketch-bench#136)

* Export the AutoSketch vs ASAP evaluation table from saturation curves

Adds scripts/export_autosketch_eval_table.py and the generated
rqe-optimizer/data/autosketch-eval/table.json (W2: alibaba_v2022, boom,
google_2011). Every config lists AutoSketch's single-sketch error at
max_N and ASAP's error after merging m = 1/4/16/64 shards, with the
saturation flag ASAP requires. recommend_config.py's grid rounding moves
into grid_point() so both scripts share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Note that sketch-bench ProjectASAP/sketch-bench#131 has merged

ProjectASAP/sketch-bench#131 merged into main as bd644fe with content identical to the
merged-accuracy head the merge curves came from, so drop the
not-merged caveat. The table and its recorded revision are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…eap and UnivMon top-k (ProjectASAP/sketch-bench#144)

* rqe-optimizer: exact *_over_time accumulators, more top-k candidates

Replace the optimizer's Freq capability with SumOrCount, MinOrMax and
RateOrIncrease, served by exact multi-subpopulation accumulators that
match ASAPQuery's Multiple{Sum,MinMax,Increase}Accumulator. Add
CountSketch+heap and UnivMon as TopK candidates. Price grouped
accumulators per group so the optimizer's card(group-by) multiply
counts each group once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: move autosketch tests off the removed Freq capability

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: split MinOrMax into Min and Max

exact-max only tracked the maximum, so min_over_time queries had no
candidate. Min and Max are now separate capabilities served by
exact-min and exact-max (ASAPQuery's MinMax sub-types), so a min query
never lands on or shares a max accumulator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… objective (ProjectASAP/sketch-bench#145)

* rqe-optimizer: one MILP entry point with an Objective enum

Replace minimize_tco/minimize_cost with minimize(..., Objective), where
Objective::{Tco, Cost(&MachineFamily)} owns its scaling units, extra
variables, rows and goal. The shared assignment/activation/bound rows no
longer branch on the family, and TCO no longer adds an unused instances
variable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: drop the Tco MILP objective, rename Cost to AUCCost

The MILP now only minimizes hourly price on a machine family. AUCCost
names what it prices: mean CPU-sec/sec, the area under the load curve,
so bursts are not priced. small_problem --milp now requires
--machine-family. The former TCO tests run under AUCCost with the same
expected mappings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: per-phase cost model, workload facts, weighted AUCCost

- Costs split into ingest, merge, query and storage, each with CPU and
  memory. objectives.rs holds the one copy of the formulas; score() and
  the MILP share it.
- Storage counts closed windows as (L - x)/y + 1 for the longest lookback
  (ProjectASAP/sketch-bench#143 S7). Ingest memory is open windows only.
- MetricFacts { labels, scrape_interval_secs, cardinality } replaces
  LabelSetInfo (ProjectASAP/sketch-bench#143 S2). The arrival rate is derived as
  card(labels) / scrape interval. validate_facts reports every bad fact.
- Rqe and Deployment carry metric and grouping_labels.
- Candidate windows and slides are multiples of the scrape interval
  (ProjectASAP/sketch-bench#143 S6, half).
- Objective::AUCCost { w_cpu, w_mem } minimizes w_cpu * CPU +
  w_mem * memory GiB, default (1, 0). MachineFamily, the EC2 pricing
  snapshot and fetch_ec2_pricing.py are removed, as is the peak-memory
  bound (ProjectASAP/sketch-bench#143 S3).
- Objectives is per phase; the Pareto vector is (CPU, memory, latencies).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: state what each rqe-optimizer cost is counted per

Per deployment or per RQE, per group, per window and per key, plus what
is not shared between RQEs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: readability pass on the cost model

Descriptive names in place of single letters, named ingest helpers in
candidate pruning, and shorter comments without issue references.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: explain candidate slides with an example

State that gcd(x, T) is the largest slide serving an RQE, why each subset
of RQEs gets the gcd of theirs, and why one pass computes all of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: frame candidate slides as strictly better than finer ones

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: list the cost model's variables one per line

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: merge memory holds both operands of a pairwise merge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: merge memory is one accumulator, none for direct queries

Stored windows are live objects already counted in storage, so a merge
adds only the accumulator per group. A query with L == x merges nothing.
Candidate pruning now compares merge memory per served RQE, since it is
no longer the same instance size for every RQE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: add intuition for the two-step candidate construction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: one dominance comparison in candidate pruning

candidate_dominates and strictly_better walked the same costs twice. A
single dominance() collects (replacement, original) cost pairs once and
reports strictly better, equal or not dominating.

Docs: define open and closed instances, fix the candidate pseudocode
indentation, and reword card(G) and merge memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: explain instance shape and size law in the cost model

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: add a glossary, link the cost export, define keys

Defines family, configuration, instance and the other terms the cost
model uses, links scripts/export_rqe_optimizer_costs.sh, explains what a
key is and why quantile sketches are treated as fixed size, and drops
the per-cost breakdown table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: call RQEs repeating atomic query expressions (RAQEs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: flatten the candidate generation pseudocode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: add phase to the glossary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: rename objectives to analytical_cost_model, Objectives to PlanCost

The module holds the per-phase cost formulas and score(); its output is
a plan's cost, not an objective, and Objective already names what the
MILP minimizes. In milp.rs, name the storage variables and cost closures
for what they hold: deployment_storage_cost,
storage_cost_for_rqe_and_deployment and merge_and_query_cost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…RAQE latency SLA, deployable families (ProjectASAP/sketch-bench#148)

* rqe-optimizer: rename Rqe to Raqe and accuracy_tolerance to accuracy_sla

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: time inputs in milliseconds

Lookback, interval, window, slide and scrape interval are Millis. Rates
(CPU-sec/sec, samples/sec) stay per second via secs().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: spatial_filter and latency_sla_ms on Raqe, drop MilpBounds

Stream identity is now (capability, metric, spatial_filter, grouping_labels).
spatial_filter is opaque matcher text used for identity only. The latency
limit moves from MilpBounds onto each Raqe; the example's --latency-limit
RAQE_ID=SECONDS becomes --latency-sla RAQE_ID=MS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: only ASAPQuery-deployable families become candidates

DEPLOYABLE_FAMILIES lists the variants ASAPQuery's engine runs, each with
its AggregationType. Candidate generation and the AutoSketch baseline both
go through Capability::deployable_families(), so dd, univmon-* and
countsketch-heap are measured but never planned.

AutoSketch's chooses_the_cheaper_variant test is removed: no capability has
two deployable variants until hydra-kll is measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: address ProjectASAP/sketch-bench#148 review

- Latency is modeled in ms end to end (PlanCost::query_latency_ms), so the
  SLA compares without a unit conversion and the example prints ms.
- validate_facts rejects non-finite or non-positive latency SLAs, and
  non-empty spatial filters until facts carry per-filter cardinality.
- small_problem rejects --latency-sla without --milp.
- Deployable-family test checks each entry is in some families().
- Restore AutoSketch's cross-variant test via search_variants().
- Fix the families() and export-script comments; drop redundant
  spatial_filter fields in tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: --allow-undeployable-families

build_all_candidates(_unpruned) take allow_undeployable_families; when set,
every measured family in Capability::families() becomes a candidate. The
small_problem example exposes it as --allow-undeployable-families.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: address second ProjectASAP/sketch-bench#148 review

- Capability::candidate_families(allow_undeployable_families) is the one
  family rule; candidate generation and AutoSketch's plan/search both take
  the flag, so the baseline sees the same sketches as the MILP.
- validate_facts rejects zero lookback/interval and ones that aren't
  multiples of the scrape interval.
- Candidate windows come from divisors(L / scrape) × scrape.
- small_problem rejects --latency-sla with --candidates-only or
  --sample-mappings, and non-finite SLAs.
- Capability::ALL for the deployable-family test; spatial_filter in the
  pruning comment and the TODO doc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ll_problem (ProjectASAP/sketch-bench#150)

export_rqe_optimizer_costs.sh takes a required OUT_DIR and refuses to run
if it already exists. small_problem takes a required --cost-dir DIR and
reads DIR/rqe_atomic_costs.json.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ketch-bench#153)

Flags were looked up by name, so a typo like --latency_p99_1h=0.5 was
silently ignored and the run proceeded without the intended SLA.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ash-table overhead (ProjectASAP/sketch-bench#151)

* rqe cost export: count top-k heaps, repeat top-k queries, add exact hash-table overhead

- CMS+heap and CountSketch+heap memory now adds CMS_HEAP_TOP_K HHItems,
  as UnivMon counts its heaps, so the TopK rows compare like for like (ProjectASAP/sketch-bench#146).
- TopkGT asks its one question 1024 times per pass, as CardinalityGT does,
  and the CMS/CS/UnivMon top-k queries answer once per probe, so a pass sits
  above the timer floor and work = answers keeps per-op CPU per call.
- Exact accumulators' footprint adds an estimate of hashbrown's table:
  capacity x (key + cell + 1 control byte).

Closes ProjectASAP/sketch-bench#146. Part of ProjectASAP/sketch-bench#147.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Name getrusage as the query timing floor; note exact memory's load-factor dependence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ojectASAP/sketch-bench#152)

* rqe-optimizer: size DDSketch memory from the metric's value range

Add optional `MetricFacts::value_range` (lo, hi), validated by
`validate_facts` (0 < lo <= hi < inf). For DDSketch rows with a known
range, memory per instance is (floor(ln(hi/lo) / ln((1+a)/(1-a))) + 1)
buckets x 8 B, matching sketch-bench's `dd_footprint` for positive values
(one store). Without a range, or for other families, the measured
`mem_bytes_per_instance` is kept.

One helper, `instance_memory_bytes`, replaces the direct reads in the
ingest, merge and storage costs and in candidate dominance pruning, which
now takes `facts`.

Part of ProjectASAP/sketch-bench#147

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cap DDSketch buckets by values per instance; note the range is an upper bound

Buckets = min(range buckets, λ·x/card(G)), as dd_footprint caps them by count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Size DDSketch's merge accumulator by the lookback, not one window

The accumulator holds L/x windows' values, so its bucket cap is λ·L/card(G).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ProjectASAP/sketch-bench#155)

* atomic-costs: record each row's measurement conditions (measured_at)

Each AtomicCostEntry now carries the conditions it was measured under,
read off the record's workload: items per instance, keys per instance,
the value column's range and distribution, and the merge operand size.
Optional on read so tables written before it still load.

Part of ProjectASAP/sketch-bench#147.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* measured_at: no value range for a monotonic (counter) column

Its values are a running sum of the draws, so the draws' bounds don't bound them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* measured_at: saturate the key-population product

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…/sketch-bench#154)

* rqe-optimizer: accuracy at the query's merge count

Export measures merge accuracy at 4, 16 and 64 shards as one --flat row per
count; atomic-costs joins them by cell (--merge-accuracy) into a new
AtomicCostEntry.merge_accuracy map, omitted when empty. Eligibility checks
accuracy at L/x merged windows: the smallest measured count >= L/x, else the
largest; one window or no measurements reads query_accuracy.

Part of ProjectASAP/sketch-bench#147

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: check both bracketing merge counts; skip partial grids

Eligibility now requires the metric to pass at both measured merge counts
bracketing L/x (count 1 is query_accuracy), since error is not monotone in
the count. Export measures 4..1024 shards and documents the fixed-total
split. atomic-costs skips a row missing any merge count in the grid.

Part of ProjectASAP/sketch-bench#147

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Merge-accuracy review fixes: docs, one run per merge pass, higher-is-better bracket test

- docs: eligibility rule 3 checks both bracketing merge counts; replace the
  single-instance accuracy TODO.
- export: merge-accuracy passes score once, so --runs 1 --warmup-runs 0;
  note 1024 shards on 200k-row specs come out as 1021.
- same_cell: note it has no comparator.
- test: both brackets must clear a higher-is-better metric.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…d DeltaSet key tracker (ProjectASAP/sketch-bench#160)

* feat(rqe-optimizer): family properties, shared fixed-size sketches and DeltaSet key tracker

S4b of ProjectASAP/sketch-bench#143, including the cost-model part of ProjectASAP/sketch-bench#142.

- family_properties(variant): mergeable_across_windows,
  one_fixed_size_sketch_for_all_groups, needs_delta_set_key_tracker;
  unknown variants panic.
- hydra-kll joins Quantile families (undeployable). Its deployments carry
  the exact-delta-set row as key_tracker and are candidates only when
  card(G) <= the row's measured subpopulations.
- Cost phases sum over sketch and tracker; a shared fixed-size sketch is
  one instance, not card(G).
- Dominance pruning compares the cost model's full costs.
- AutoSketch skips shared fixed-size families (ProjectASAP/sketch-bench#159).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rqe-optimizer): assert tracker presence, deployable-only tracker, cost vectors computed once in pruning

- priced_parts asserts a deployment carries a key tracker iff its family
  needs one, so a hydra Deployment built elsewhere can't be underpriced.
- The key tracker row is used only when it is deployable or undeployable
  families are allowed.
- Dominance pruning computes each candidate's cost vector once instead of
  per pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rqe-optimizer): decide shared-sketch validity in is_eligible, one tracker row

- is_eligible takes facts and holds the group-count gate and the
  tracker-presence check, so the MILP, enumeration and pruning agree;
  facts is threaded through eligible_deployments_for and enumerate.
- The key tracker row is used whenever its family is a candidate (no
  separate deployability check); more than one exact-delta-set row panics.
- The pruning test zeroes the tracker's insert and memory, so only
  latency keeps Hydra.
- Export script comment and design doc updated (latency and query memory
  rows; ProjectASAP/sketch-bench#165, ProjectASAP/sketch-bench#166).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…y fields (ProjectASAP/sketch-bench#163)

* rqe-optimizer: split SumOrCount into Sum and Count; deny unknown AtomicCostEntry fields

Sum and count shared one capability, so the MILP could serve both items of
ASAPQuery's avg rewrite from one exact-sum accumulator, undercounting ingest
and memory ~2x. They are now separate capabilities on the same family.

AtomicCostEntry and MeasuredAt now reject unknown fields, so a misspelled or
newer field fails to load instead of dropping its data.

Closes ProjectASAP/sketch-bench#161

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: split TopK into TopKByValue and TopKByCount

Value-weighted and count-weighted top-k queries at the same cadence became
identical Raqes, so the MILP could serve both from one CMS-heap that
ASAPQuery deploys as two (different count_events). Separate capabilities
on the same families keep them apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject unknown distribution fields; test the plan, not eligibility

The distribution parameter structs nested in MeasuredAt now deny unknown
fields too, so a typo like `skewnes` fails instead of being dropped. All
checked-in distribution blocks still parse; the schema snapshot is
regenerated.

The sharing test now solves the MILP from a cost row: two identical queries
must share one deployment (sharing halves ingest), and paired capabilities
must not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… counts (ProjectASAP/sketch-bench#168)

* feat(rqe-optimizer): plan output with active deployments and instance counts

MilpSolution now lists only the active deployments, renumbered, each with
its retained instance count, and per RAQE the serving deployment index and
merged instance count. Drops the candidate-index mapping; adds the solved
objective value. sketch-bench#143 S8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(rqe-optimizer): dense plan renumbering, open windows counted once

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(rqe-optimizer): drop MilpSolution::objective_value; callers use objective.value

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ketch-bench#170)

hydra-kll is no longer a candidate, even with
--allow-undeployable-families. Its FamilyProperties and the DeltaSet
key-tracker path stay, so re-adding it to Capability::families is a
one-line change.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…AP/sketch-bench#171)

* rqe-optimizer: read sketch accuracy from saturation curves (ProjectASAP/sketch-bench#156)

Sketch accuracy now comes from the saturation study's error-vs-N curves,
read at the items a deployment's answer covers per group, at the metric's
fitted data shape. Exact accumulators keep the cost table's value.

- saturation.rs: SaturationCurves loads out_grid_1e7_cost/ and out_1e9/
  (1e9 wins) and applies the ProjectASAP/sketch-bench#156 rules: worse neighbour between
  checkpoints, worst bracketing grid point, curve metric must match,
  saturated panes for lossy merges (KLL, top-k).
- MetricFacts.data_shape: fitted (zipf θ, K, tail index) per grouping.
- An Accuracy oracle is threaded through eligibility, pruning,
  enumeration and minimize.
- small_problem takes --saturation-dir.
- Cost export sweeps grid configs: KLL k 200/800, CMS-heap cols 1024/4096.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: merged answers read the curve at the merged count only

Drop the saturated-pane requirement for KLL and top-k: every merged answer
reads the curve at n(T), whatever the pane size. Their merge penalty
stays ProjectASAP/sketch-bench#158. Resolves difference 2 on ProjectASAP/sketch-bench#171.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: check the cost table against the saturation curves

A cost-table sketch row and a saturation grid point measure the same
thing, error at (sketch, config, data shape, N). They were keyed
differently, so the optimizer could not tell that a row is one point on a
curve.

- aqpbm-core: MeasuredShape (Zipf θ and K, or Pareto a), the key both
  share, and MeasuredAt::data_shape().
- saturation.rs: grid points are keyed by MeasuredShape and keep seed_se.
  SaturationCurves::check_cost_table reads each sketch row's curve at
  measured_at. It reports a row as mismatched when the values disagree
  beyond 3 seed SEs + 5%, the config is not in the grid, or the table
  lacks the curve's metric. Rows with no point at their shape or N
  (KLL: Zipf ranks vs Pareto) are reported as unchecked.
- small_problem refuses to plan on a mismatch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: describe the saturation lookup in words, not symbols

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: exactness in the family registry; one SLA check

- FamilyProperties::exact marks the exact accumulators, so
  SaturationCurves::accuracy asks the registry instead of matching the
  "exact-" name prefix.
- Raqe::meets_sla(Option<f64>) is the one accuracy-vs-SLA check, used by
  MILP eligibility and the AutoSketch search alike.

Addresses review comments on ProjectASAP/sketch-bench#171.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zzylol <zeyingz@umd.edu>
…SAP/sketch-bench#173)

* rqe-optimizer: check each family by its own accuracy metric (ProjectASAP/sketch-bench#167)

Families declare their accuracy metric and direction in FamilyProperties;
a RAQE gives one accuracy_sla checked against each candidate's metric, so
DDSketch (mean_relative_value_error) and KLL (mean_rank_err) can both
serve one quantile RAQE. A cost-table row missing its family's metric, or
a saturation curve for another metric, panics. dd is deployable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: refuse curves for another metric at load

Address review on ProjectASAP/sketch-bench#173: SaturationCurves::load rejects a candidate
family's curve whose error_metric isn't its accuracy_key, so a stale study
(e.g. DD's max error before ProjectASAP/sketch-bench#175) fails before planning, whatever the
lookback, and check_cost_table compares against the family's metric.
Drops the per-call assert and a stale fixture comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: reject an accuracy SLA no metric can meet

validate_facts rejected only a non-finite accuracy_sla, so a negative SLA
or a precision floor above 1 made every candidate silently ineligible and
the RAQE read as unservable. Every metric is >= 0 and the only
higher-is-better one (precision_at_k) is a fraction, so the SLA must lie
in [0, 1] when the capability has such a family, else [0, inf).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…etch-bench#174) (ProjectASAP/sketch-bench#178)

* rqe-optimizer: one study for sketch cost and accuracy (ProjectASAP/sketch-bench#174)

The saturation study is now the only source of the optimizer's inputs.

- study_saturation.py --phase optimizer-cost: one serial cost pass (5 runs,
  3 warm-ups) and one accuracy pass per (sketch, config), plus the exact
  accumulators, at one shape: Zipf 1.1 over 1e4 keys (Pareto a = 2 for
  quantiles), 1e6 items -- the AutoSketch vs. ASAP evaluation's dataset.
  Reduced by atomic-costs to rqe_atomic_costs.json; fails on a skipped row.
- AtomicCostEntry: required accuracy_metric (the query_accuracy key that is
  the row's accuracy) and measured_at; merge_accuracy removed. atomic-costs
  takes --accuracy-metric VARIANT=METRIC in place of --merge-accuracy.
- rqe-optimizer: exact accumulators read the row's accuracy without merge
  brackets (accuracy_at removed); check_cost_table also refuses a row whose
  accuracy_metric isn't its family's or its curve's. small_problem reads the
  table from --saturation-dir (--cost-dir removed).
- Delete scripts/export_rqe_optimizer_costs.sh. export_atomic_costs.sh
  (ASAPQuery's grid) passes --accuracy-metric.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: read a row's declared accuracy_metric, and require it be its family's

row_accuracy read the family's key and ignored the row's accuracy_metric,
so the two were only compared in check_cost_table. It now reads
query_accuracy[accuracy_metric] and panics when that metric isn't the
family's: one version, checked wherever a row's accuracy is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* study_saturation: run the cost shape on the curve grid; fix export_atomic_costs pass order

- The accuracy grid also runs every config at the optimizer-cost shape
  (Zipf 1.1 over 1e4 keys, Pareto a = 2) unless --no-cost-shape, so each
  cost-table row is a point on a curve and check_cost_table checks every
  sketch row. The synthetic evaluation uses the same data and now reads an
  exact curve.
- --phase optimizer-cost shows atomic-costs' error when it fails, and
  requires its summary to match the row count exactly.
- export_atomic_costs.sh: cost pass first (flatten keeps the first pass's
  query timings, so accuracy-first skipped every row), accuracy pass once
  with no warm-ups (accuracy scores one run), and fail on skipped rows.
- small_problem's missing-table message names --out DIR/optimizer_cost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* study_saturation: optimizer-cost skips frequency; test exact accumulators merge losslessly

From review on ProjectASAP/sketch-bench#178:
- --phase optimizer-cost measures only the families some capability plans
  (top-k, cardinality, quantile): no capability reads the frequency rows.
- The optimizer takes exact accumulators to merge without loss, and the
  merge-accuracy sweep that covered them is gone, so a CLI test now checks
  that each one merged from 4 shards scores exactly like one (error 0).
- read_curve's exit names --no-cost-shape for older accuracy runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ojectASAP/sketch-bench#158) (ProjectASAP/sketch-bench#179)

* rqe-optimizer: merged KLL and top-k answers read the merge curves (ProjectASAP/sketch-bench#158)

Since ProjectASAP/sketch-bench#171 every sketch read its plain curve at the lookback's item count,
whatever the window, so a KLL or top-k answer merged from L/x windows got
the unmerged sketch's accuracy. SaturationCurves now loads
saturation_merge_curve.csv and reads it for those two families at
m = L/x: the worse of the measured shard counts either side, the largest
past it, and no accuracy without a merge curve. A later run's point keeps
an earlier run's merge curves when it has none (out_1e9 measures none).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: merge curves extrapolate only from a saturation they cover

From review on ProjectASAP/sketch-bench#179:
- Past a curve's last checkpoint, its plateau only if n_sat is at or before
  it: a merge curve stops at 1e7, while its point's n_sat may come from the
  1e9 run.
- A later run keeps an earlier run's merge curves at shard counts it didn't
  measure, moved rather than cloned.
- load refuses a candidate KLL or top-k point with no merge curve, rather
  than leaving every merged deployment silently without accuracy.
- shards must be a whole count >= 1; univmon-topk merges lossily too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: no merged accuracy past the measured shard counts

From the second review of ProjectASAP/sketch-bench#179:
- Past the largest measured shard count, a merged KLL or top-k answer has
  no accuracy, rather than the largest count's: merge error grows with the
  count, and the study measures whatever counts a workload needs.
- load refuses only a study with no merge curves at all (run without
  --merge-shards-list). A point without one, such as a 1e9-only point, just
  has no merged accuracy.
- agrees_at takes the plateau under the same rule as curve_error_at
  (n_sat at or before the last checkpoint).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: merged accuracy only where the merge curves measured it

From the third review of ProjectASAP/sketch-bench#179:
- The merge count is Deployment::query_instance_count, as in the cost
  model: no answer for a zero window or a lookback that isn't whole windows.
- Past a merge curve's last N the merged answer is unknown. The plain
  curve's saturation says nothing about the merged sketch. A curve's last
  checkpoint itself is read as measured.
- load refuses each lossy candidate sketch that has no merge curve at all,
  not only a study with none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…easured nothing (ProjectASAP/sketch-bench#180)

* rqe-optimizer: fall back to each sketch's guarantee where the study measured nothing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: test the guarantee fallback end to end

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: a top-k guarantee for every key at once; fallbacks no better than measured

From review on ProjectASAP/sketch-bench#180:
- CMS top-k: the per-key 1 − e^−rows wasn't a guarantee for the whole
  top k. Now Markov per row, independent rows and a union bound over the
  K − k other keys give each top key's failure probability; keys count
  from the most frequent while the summed failure stays within 5%.
- A fallback is never better than the last measurement it extends, and
  it applies only at or above the first N of the curves it would read,
  merge curves included.
- HLL uses the one-sided 95% normal quantile (1.645), as CONFIDENCE says.
- The Zipf normalizer is exact for the first 1000 terms and an
  Euler–Maclaurin tail beyond, so large K is O(1).
- accuracy_with_source reports the source of the worst value only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: fallback floors on both bracketing merge curves; two-sided rank and relative errors

From the second review of ProjectASAP/sketch-bench#180:
- A fallback's floor, and its first-N gate, read the measured shard counts
  either side of the merge count, the curves merged_error_at would have
  read, not only the smaller one or every merge curve.
- KLL rank error and HLL relative error are absolute, so their 95% bounds
  are two-sided (KLL scaled by sqrt(ln 40 / ln 200), HLL z = 1.96). The
  top-k bound stays one-sided: CMS only overestimates.
- Docs: the source is the returned value's; DDSketch's alpha isn't a hard
  bound on the measured metric, which the floor covers; the top-k bound
  assumes the heap ranks by final CMS estimates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…-table accuracy (ProjectASAP/sketch-bench#182)

* top-k: heap = m·k per deployment; heap a runtime knob; keep summary rows on --resume

sketch-bench:
- The top-k heap's capacity is a runtime param (`heap=` on cms-heap and the
  new countsketch-heap vocabulary, default and minimum TOPK_K = 32). The
  answered and graded k stays TOPK_K: a larger heap answers its heaviest k.
  Memory counts heap x HHItem. UnivMon and the CLI's TopkGT read TOPK_K.

rqe-optimizer:
- A heap top-k deployment answering from m merged windows keeps m·k.
  Candidates carry one heap per distinct m among a window's RAQEs, and a RAQE
  is eligible only where the heap holds its m·k.
- Costs come from the cost table's heap rows, linear in the heap between and
  past the measured sizes. One measured size serves any heap up to it.
- With heap >= m·k a merged answer reads as one sketch (plain curve, then the
  guarantee fallback), so top-k needs no merge curves; KLL still does. Curves
  match configs ignoring `heap`. check_cost_table skips heaps other than k.

study_saturation:
- --phase optimizer-cost measures top-k at heaps 32/128/512/2048
  (TOPK_HEAPS).
- --resume over a narrower grid keeps the saturation.csv rows of the other
  points, as the curve files keep theirs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* top-k heap: AutoSketch searches heap k only; one accuracy per config; flag single-heap pricing

From the first review of ProjectASAP/sketch-bench#182:
- AutoSketch never merges, so its search space holds heap-k rows only;
  larger heaps aren't an axis it tunes.
- Every heap row priced from a config carries the heap-k row's accuracy,
  measured hit or interpolated, since the curves are measured at k.
- check_cost_table flags a heap top-k config priced at one heap only: no
  merged candidate could be built from it.
- heap reads any JSON number (128 or 128.0).
- --resume keeps other points' saturation.csv rows in the accuracy phase
  only, tolerating older column sets.
- UnivMon's top-k cut reuses top_k.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* top-k heap: fail loudly on one-heap pricing and duplicate rows; heap_needed

Round-2 review of ProjectASAP/sketch-bench#182:
- at_heap panics, naming the optimizer-cost rerun, when a config is priced
  at one heap smaller than the heap a candidate needs; it no longer drops
  the candidate.
- heap_rows_by_shape panics on two rows at one heap (no heap and heap=k);
  AutoSketch's grid runs the same check.
- heap_needed(lookback, window) and Deployment::heap_needed give m · k for
  is_eligible, the per-window candidate heaps and SaturationCurves::accuracy.
- top_k selects the heaviest k with select_nth_unstable and sorts only
  those; the query cost is benchmarked.
- CountSketchParams no longer owns countsketch-heap-* variants (pinned in
  each_algorithm_owns_its_variants_and_no_neighbours).
- load_needs_no_merge_curves_for_heap_topk no longer splits another test
  from its doc comment; config_params' doc is one comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study_saturation: cost-table sketch accuracy is the curves' seed mean; CmsParams leaves cms-heap

- --phase optimizer-cost rewrites each sketch row's accuracy as the mean
  over seeds 1..--seeds of one accuracy run at COST_N: the run the curves
  make at that N. One seed-42 run against a three-seed mean failed
  check_cost_table on seed variance alone (top-k rows=3 cols=256, KLL
  k=200). Top-k heaps above k take the heap-k mean; exact rows keep 0.
- CmsParams no longer claims cms-heap-* variants, as CountSketchParams no
  longer claims countsketch-heap-*.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e a grid with holes (ProjectASAP/sketch-bench#186)

* rqe-optimizer: bracket the shape parameter within each K

The cost table's shape adds one (θ = 1.1, K = 1e4) point to the curve
grid (ProjectASAP/sketch-bench#178). bracketing_points bracketed θ over every point's θ and K over
every point's K, so a θ = 1.1 shape at another K asked for a (1.1, K) point
that was never measured and got no accuracy. small_problem's
top_endpoints_1h (θ = 1.1, K = 1e3) was unservable. Now θ is bracketed
within each K, and K among the K values that can bracket θ.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: one pass per K for bracketing; equal shapes all count

From review on ProjectASAP/sketch-bench#186:
- Gather each K's bracketing points in one pass; points with equal shapes
  (10000 vs 1e4 in two CSVs) all count instead of failing a length check.
- Quantile when every point is Pareto, not by the first point.
- Doc: the K bracket falls back to neighbouring Ks that have the
  parameter (conservative) rather than returning None.
- The test's extra point carries no merge curves of the point it copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study_saturation: the cost shape as a full grid row and column; keep the rectangular bracket

From the second review of ProjectASAP/sketch-bench#186: rather than bracketing a sparse grid, keep
the grid a full cross. --cost-shape now adds θ = 1.1 and K = 1e4 to --thetas
and --cardinalities (and a = 2 to --alphas), so θ = 1.1 is measured at every
K and K = 1e4 at every θ. bracketing_points goes back to the rectangular
bracket with its completeness check: a hole in the grid leaves the shape
without accuracy instead of bridging to farther points (accuracy needn't be
monotone in K). Quantile is decided from every point, not the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: refuse a curve grid that isn't a full cross, naming the holes

From the third review of ProjectASAP/sketch-bench#186:
- SaturationCurves::load refuses curves whose Zipf grid isn't a full cross
  of its θ and K values, naming the missing (θ, K). A hole used to leave
  every shape around it silently without accuracy.
- Quantile is decided by the first point again (the grids hold one
  distribution per config).
- The hole test removes the point by shape, not position; a study test
  checks a grid that already holds the cost shape adds nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* top-k: k is a per-query variable, not the constant 32 (ProjectASAP/sketch-bench#184)

- sketch-bench: heap top-k variants take topk_k= (default 32); the heap
  defaults to k and a heap below k is refused; the answer is the heaviest
  k and the comparator grades precision@k at it (answered_k).
- study_saturation: top-k curves run at each k in --topk-ks (10, 32, 100);
  k != 32 is keyed in the config string (topk_k=k), so k = 32 keeps its
  old keys and older files read unchanged.
- rqe-optimizer: Raqe.topk_k (default TOPK_K); heap needed m·k and
  eligibility heap >= (L/x)·k; accuracy reads the curve at k, else the
  next larger measured k, and past every measured k the theory bound at
  k; output cost is k entries; AutoSketch searches the smallest measured
  heap >= k.
- Docs: rqe_sketch_deployment_v1.md, saturation_study.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* top-k k: floor heaps, carry topk_k, share heap pricing with AutoSketch (ProjectASAP/sketch-bench#185 review)

From the first review of ProjectASAP/sketch-bench#185:
- A top-k need below the smallest measured heap prices at that heap,
  never extrapolated below; candidates are keyed by (heap, k) and carry
  topk_k when k != 32, so the deployed sketch answers k. A RAQE needs a
  deployment answering at least its k.
- univmon-topk answers TOPK_K only: ineligible for any other k.
- AutoSketch resolves heaps with the MILP's heap_row (same floor and
  interpolation/extension), index for index, instead of picking a
  measured heap >= k.
- Accuracy picks the next larger measured k whose curves bracket the
  shape; past every such k, the guarantee at k, floored by the largest
  bracketing k's measurement at this N. Params compare without cloning.
- One topk_k decoding (params_topk_k / config_topk_k): heap_capacity's
  missing heap is the row's k, as in sketch-bench.
- --topk-ks is deduplicated and must include 32.
- Top-k query CPU scales by k/32 (measured at 32), a first approximation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: the full-cross check is per top-k k

Rebased onto ProjectASAP/sketch-bench#186. Its load check groups grid points by their full
params, topk_k included, so each k's grid must be a full cross of θ and
K on its own and the k dimension doesn't trip it. The study crosses each
per-k top-k config with the cost-shape row and column like any config,
and the curve-at-k lookup keeps the rectangular bracket within the
chosen k. A test pins both: two k's on different grids load; a hole
within one k is named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* Address second ProjectASAP/sketch-bench#185 review: answered-k query cost, fixed-k families

- Query CPU and output bytes scale by the deployment's answered k.
- answered_k/serves_topk_k in lib.rs: a fixed-k family (univmon-topk)
  answers TOPK_K only; used by candidates::is_eligible and AutoSketch.
- heap_row's floor sets an explicit heap; one topk_k encoder
  (topk_k_param) shared with with_topk_k.
- priced_parts tags sketch vs tracker instead of pointer identity.
- AutoSketch resolves heaps once per (k, top-k) in plan, indexes heap
  rows once, and keeps heap/topk_k off its search axes.
- study_saturation: old k=32 curve files need --topk-ks 32 for cost and
  crossover; say so in the docstring and exit message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…g, type and SLA sweeps) (ProjectASAP/sketch-bench#177)

* rqe-optimizer: quantile_scenarios example sweeping lookback and interval

Each scenario is 10 quantile_over_time RAQEs on distinct metrics sharing
one (lookback, interval); the example solves each with the MILP and prints
one row per scenario.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: workload_scenarios example with grouping, type and SLA sweeps

Renames quantile_scenarios to workload_scenarios and adds sweeps over mixed
groupings, mixed query types, a latency SLA and an accuracy SLA. Each step
prints the deployments the MILP runs and the queries each one serves.
--sweep NAME runs one sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: workload_scenarios follows per-family accuracy and the study's cost table

Raqe no longer names an accuracy metric, so each query keeps only its SLA
and the output prints the chosen family's metric. The cost table is read
from the saturation directory, so --cost-dir is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…as an SLA) (ProjectASAP/sketch-bench#188)

* rqe-optimizer: batch latency SLA with provisioned vCPUs

minimize_batch_sla bounds when an aligned batch of every RAQE finishes:
the plan provisions V vCPUs shared by ingest (rho_ing) and the batch's
merge and query work W, a fluid queue, so V >= rho_ing + W / SLA, and
V >= the mean CPU. CPU is priced at V. A pair whose per-group work
exceeds the SLA is forbidden. With a loose SLA the plan is minimize's.
analytical_cost_model gains provisioned_vcpus and batch_makespan_ms to
price any plan at an SLA. The model is in docs/rqe_sketch_deployment_v1.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: cost models 1 and 2, batch latency and query placement (design for review)

Replaces the provisioned-V batch SLA write-up with the design under review:
precompute and query jobs per RAQE firing, each on at most one core; an
event-driven placement at a (CPU, memory) resource point gives batch
latency; cost model 1 bills AUC, cost model 2 the peak; the MILP uses a
linear surrogate and is verified by the placement. Four open questions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: precompute is ingest; queries merge then estimate; fractional capacity

Precompute is inserting samples into window instances (ingest), a steady
background load; each firing's query job merges the query window's
instances and estimates, at most one core. Capacity is fractional and
shared as rates. States sketch-bench's measurement assumption (single
threaded, CPU time = wall time). Two open questions remain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: parallel ingest split by sample, compaction at window close, memory in three parts

Ingest runs on ceil(rho_D) workers split by sample, each with its own open
windows; a compaction job merges the partials at window close, on the
query's critical path. Memory is ingest, storage and query, each counted
once. Cost model 1 bills CPU and memory by use (elastic CPU, latency is the
longest chain); cost model 2 provisions (C, M), billed at the peak, with the
placement giving latency. The resource-point question is resolved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: the MILP for cost models 1 and 2; the placement sizes (C, M)

Cost model 1 is exact and linear (elastic CPU: latency is the longest
chain). Cost model 2's batch row is a surrogate that chooses the mapping;
the placement then sizes C (smallest meeting the SLA) and M (its peak) for
every method alike, replacing the re-solve loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: cost models 1 and 2 with query placement

placement: a plan's load over time (ingest on ceil(rho) workers split by
sample, compaction at window close, one query job per firing, each at
most one core), cost model 1 (by use, elastic CPU, latency = longest
chain) and cost model 2 (a provisioned (C, M) billed at the peak, sized
by an event-driven placement: the smallest C meeting the SLA, M its peak).

milp::minimize_cost_model: cost model 1 is exact and linear; cost model 2
uses a linear surrogate (batch work over the capacity ingest leaves,
every query's memory at once), then corrects it with the placement's
error on the chosen mapping and solves again, keeping the cheapest placed
mapping. The earlier provisioned-V minimize_batch_sla is removed;
minimize is unchanged.

Tests: placement (one core per job, fractional capacity, shared batches,
compaction before queries, backlog, both cost models); cost model 1
matches brute force with and without an SLA; cost model 2 meets the SLA
and matches the brute-force placed optimum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: restructure the section as a design doc

Title 'MILP with latency SLA constraints: cost models, query placement and
batch latency'; in order: definitions, the two cost models, CPU and memory
parts and how each is computed, batch latency vs. the SLA, the job
placement algorithm, the MILP formulation, and the solution method
(cost model 1 exact; cost model 2 surrogate, placement sizing and
correction passes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: plain-English explanation for every formula and the placement algorithm

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs, placement: one job placement for both cost models

Cost model 1 is the placement with unlimited capacity (elastic CPU): every
job starts when ready on its own core, so the latency is the longest chain
and the closed form the MILP uses is exact; cost model 2 runs it with the
provisioned C. A test checks the closed form against the placement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs, placement: a query is issued, not released

Queries are issued at their firing time; compaction is triggered at a
window close. The doc section and placement.rs use these terms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: map cost models 1 and 2 to cloud billing; memory limit in the placement

Cost model 2 matches containers billed by allocation and VMs (queueing for
a core or for memory lowers the peaks); cost model 1 matches idealized
fine-grained autoscaling, where queueing saves nothing and the cost-latency
trade-off comes from the plan (precompute vs. query-time merging). The
placement makes a job wait when its memory does not fit in M, and cost
model 2's sizing searches (C, M) for the cheapest pair meeting L.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: memory does not limit the placement

Billing charges the memory allocated; whether an application queues jobs
to fit a smaller allocation is its own policy, not billing. So the
placement has no memory limit and M is the peak it reaches at C*.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer, docs: review fixes for cost models 1 and 2

- minimize_cost_model takes `allowed` candidates per RAQE, so PerQuery
  can rule out sharing: identical deployments as separate candidate copies
  each pay their own ingest (test).
- One chain function (analytical_cost_model::chain_ms) and one SLA slack
  (meets_sla, 1e-9 relative) for the MILP's chain filter and the
  placement, so a chain equal to L is never feasible in one and not the
  other.
- cost_model_2's doubling search stops after 64 steps.
- The placement keeps a pending list of issued, unfinished jobs instead of
  scanning every job at each event.
- A compaction holds the closed window's k partial copies while it runs:
  in AUC(memory), the placement's memory over time, and both MILPs'
  memory terms (test).
- Docs: instances vs. groups in the formulas; the compaction memory row;
  per-RAQE latency bounds are not used; the search bounds; PerQuery via
  `allowed`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer, docs: cost model 2 is evaluated with an SLA only

Without an SLA, cost model 2's capacity is not pinned down (at the mean
load, a placement that can't use every core keeps a growing backlog, so
its latency depends on how long it is simulated). cost_model_2 takes the
SLA, and minimize_cost_model asserts one for cost model 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer, docs: cost by use only, latency reported, not constrained

The paper's message is lower cost and lower latency, so the MILP drops
the peak-billed cost model and every latency constraint:

- usage.rs (replacing placement.rs): a plan's resource use (ingest on
  ceil(rho) workers split by sample, compaction at window close, one
  query job per firing), its cost w_cpu * AUC(CPU) + w_mem * AUC(memory),
  and its latency, the longest chain (CPU is elastic).
- milp::minimize_usage_cost (replacing minimize_cost_model): exact and
  linear, with `allowed` candidates per RAQE for PerQuery. Cost model 2,
  the placement, the SLA rows, the correction passes and meets_sla are
  removed; minimize is unchanged.
- Docs: "Cost by use and batch latency": definitions, the cost model,
  CPU and memory parts, batch latency (reported), the MILP, and how it
  is solved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer, docs: an optional latency bound to trace the cost-latency frontier

minimize_usage_cost takes latency_bound_ms: pairs whose chain exceeds it
are ruled out, so the plan is the cheapest at least that fast (exact,
since CPU is elastic). Sweeping it traces each method's Pareto frontier;
it is not an SLA. Test: matches brute force under four bounds, and looser
bounds never cost more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: two versions (no SLA; batch latency SLA) and the job placement algorithm

Version 1 minimizes cost by use and reports latency, with an optional bound
that traces the frontier; version 2 requires the batch latency, from the
job placement, to be at most an SLA. The placement algorithm is back: with
elastic CPU it has the closed form (longest chain) the code uses, which
keeps both versions exact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: answer ProjectASAP/sketch-bench#188's review on the CPU means

Every mean vCPU is an average rate over the same period (whole
hyperperiods), so the parts add; ingest is already a rate (lambda_D =
series / scrape interval), and only compaction and queries are divided by
their period. AUC(CPU) is written as three sums to make that plain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ontier and SLA versions) (ProjectASAP/sketch-bench#138)

* rqe-optimizer: evaluate AutoSketch vs ASAP on the synthetic and trace workloads

#777 (paper §6.3), consolidating ProjectASAP/sketch-bench#138, ProjectASAP/sketch-bench#139 and ProjectASAP/sketch-bench#141.

- rqe-optimizer/examples/autosketch_vs_asap.rs (traces and synthetic): ASAP
  (milp::minimize), PerQuery-CostAware and AutoSketch-Adapted. Costs and
  accuracy come from --saturation-dir, as the planner reads them:
  SaturationCurves::accuracy for ASAP (merge curves for KLL and top-k,
  ProjectASAP/sketch-bench#179), autosketch_accuracy for AutoSketch, and the one-study cost table
  (ProjectASAP/sketch-bench#178). Each run is scored by analytical_cost_model::score at #777's two
  weight settings and the SLA grid.
- scripts/run_autosketch_vs_asap.sh and plot_autosketch_vs_asap.py: the
  trace workloads alibaba_v2022 and google_2011. boom is left out for now.
- scripts/export_autosketch_eval_table.py --synthetic: one workload-only
  table per grid point, plus plan.tsv. The data constants come from
  study_saturation's COST_*. The traces path is unchanged from main.
- scripts/run_autosketch_vs_asap_synthetic.sh and
  plot_autosketch_vs_asap_synthetic.py: run the synthetic grid, then plot.
- scripts/profiling_time.py and the results README: ASAP's one-time
  profiling time, and how to reproduce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rqe-optimizer: trace key queries keep one accumulator per key; report the accuracy source

From review on ProjectASAP/sketch-bench#138:
- A traces key query (sum by key) keeps one exact accumulator per key: its
  groups are the window's keys, not the table's one sketch. A stream's
  card(g) is the largest over its RQEs.
- AutoSketch's benchmark lower bound uses one query of one instance, not
  the stream's q_r of them.
- --weights that names none of the weight settings is an error.
- The chosen report lists each RQE's accuracy source (ProjectASAP/sketch-bench#180's
  accuracy_with_source).
- Drop a duplicated BoomTest; capability_of's comment matches the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: one accuracy level, p95; plots join one RQE set and charge AutoSketch's benchmark

- One accuracy level for every workload, synthetic and traces: 95% in
  each family's metric (rank, relative value and relative error at most
  0.05; top-k precision at least 0.95). The strictness dimension is gone.
- fig_objective_vs_latency joins only points over one RQE set (solid for
  the full set, dashed for the subset tight SLAs keep).
- fig_planning_time adds AutoSketch's charged benchmark time (lower bound
  and 60 s per probe), as #777 section 7 counts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: summaries show Σ window/slide; synthetic summary to its own file

AutoSketch-Adapted's ingest cost is window / slide overlapping sketches per
deployment (window = the query's lookback), so the column explains its gap
to ASAP. The synthetic and trace scripts both wrote summary.md into one
results directory; the synthetic one is now summary_synthetic.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: RQEs carry their top-k k (ProjectASAP/sketch-bench#185)

Raqe now has topk_k. The tables name it only in the query id (topk32_...),
so the runner reads it there and stops on a top-k query that names none.
On the p95 synthetic and trace tables every top-k query is k = 32, and the
plans are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: p95 results for the synthetic grid and the traces

Runs on clnode155 over the asap_sketchlib 0.3.0 study (ProjectASAP/sketch-bench#174, ProjectASAP/sketch-bench#178–ProjectASAP/sketch-bench#186):
four synthetic workloads and alibaba_v2022 / google_2011, raw JSON,
summaries and figures. The README gives the no-SLA headline, including
Σ window/slide behind AutoSketch-Adapted's gap. profiling-time.json is
refilled from the study's raw records (0.77 h; the K = 1e4 column refill
of ProjectASAP/sketch-bench#186 is its own run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: figures and tables give each method's absolute cost

Cost figures label every bar and point with its value; the summaries and
the results README list each method's objective instead of its ratio to
ASAP. The by-dimension figure's legend moves above the panels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: metrics dimension, one dashboard per metric (21 · m RQEs)

Shared replicas on one metric stop adding RQEs past r = 8 (52 at r = 8,
57 at r = 64), so the grid keeps shared r ∈ {1, 8} and adds metrics
m ∈ {1, 8, 16}: copy i of the template set reads its own metric data_i,
so nothing is shared across copies and RQEs grow as 21 · m (168, 336).
The planning-time figure plots this dimension. The m = 8 and 16 runs are
added; the r = 64 result is removed. Existing tables are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: AutoSketch benchmarks each config once per metric

Synthetic tables name the metric each RQE reads (data, or data_i in the
metrics dimension), and AutoSketch-Adapted's benchmark time counts a
probed config once per metric, on that metric's data: 540, 4320 and
8640 s (60 s per probe) for m = 1, 8, 16. Trace tables name none and
read one metric per dataset, as before. All five synthetic workloads are
rerun with the current runner on four nodes; objectives are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: cost vs. total estimated latency for every workload

One column per workload (five synthetic, two traces), one row per weight
setting, one line per method across the SLA grid, each point labeled with
its cost. Total latency is the sum of the RQEs' estimated latencies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: finer SLA grid, {0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10} ms and none

Checks whether AutoSketch-Adapted's memory objective makes it miss an
SLA: it never does. Its plans never merge, so each RQE's latency is within
1.4% of the lowest any deployment reaches (equal for quantile, exact and
trace RQEs), and RQEs no method can meet are excluded for all methods.
All workloads rerun on four nodes; figures and summaries regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: AutoSketch probe list and measured benchmark time; traces with 6h/24h windows

- The runner lists AutoSketch-Adapted's distinct probed (metric, config)
  pairs with the data shape each is benchmarked on, and drops the
  benchmark-time lower bound. scripts/autosketch_benchmark_time.py runs
  each probe's approxbench accuracy benchmark (1e8 items, exact baseline
  and scoring) and times it, as AutoSketch benchmarks probed configs.
- Trace streams with more keys than samples per second are left out and
  listed (rounding their rate up to their key count inflated every
  method's cost).
- The trace table is regenerated from ASAPQuery #812's skew summary
  (6h and 24h windows, a day of Alibaba).
- Plots: one line per method over the full RQE set; AutoSketch's planning
  time with its measured benchmark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: cost by use (sketch-bench ProjectASAP/sketch-bench#188); Pareto frontier sweep; mixed set by default

- The runner prices every plan by use (usage::usage_cost): w_cpu * AUC(CPU)
  + w_mem * AUC(memory), CPU elastic, latency the longest chain, reported.
- ASAP is milp::minimize_usage_cost over every RQE; PerQuery the same over
  each RQE's own candidates (separate copies via `allowed`, no sharing);
  AutoSketch's plan is priced the same way.
- No SLA grid. Each method's cost-latency frontier: ASAP and PerQuery are
  solved unbounded and for 12 bounds log-spaced from the tightest feasible
  one to the unbounded latency, plus AutoSketch's latency.
- The synthetic grid is the mixed template set: shared r in {1, 8} and
  metrics m in {1, 8, 16} (50, 92, 400, 800 RQEs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: version 1 (no SLA) results on the mixed set, with AutoSketch's measured benchmark

- Four workloads of the mixed template set (50, 92 with r = 8, 400 with
  m = 8, 800 with m = 16 RQEs), cost by use (ProjectASAP/sketch-bench#188), each method's cheapest
  plan and ASAP's and PerQuery's cost-latency frontiers (12 bounds plus
  AutoSketch's latency), CPU-only and Fargate weights, --runs 3, one node
  per workload. No RQE dropped, no sanity violation.
- AutoSketch's benchmark time is measured: approxbench's accuracy run (1e8
  items, exact baseline, scoring) once per distinct probed config and
  shape, serially on idle clnode155 (21-37 s each), charged once per
  probed (metric, config): 212, 1697 and 3394 s for m = 1, 8, 16.
- plot_autosketch_vs_asap_synthetic.py: fig_frontier.png (cost vs. query
  latency, every point labeled), fig_planning_time.png (AutoSketch as
  search + measured benchmark; 60 s per probe only as a labeled
  reference), summary_synthetic.md. The SLA-era figures, the dashboard
  results and plot_autosketch_vs_asap_workloads.py are removed.
- README: the cost model, version 1, how to reproduce, and the headline
  numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: version 2, a batch latency SLA, on the synthetic mixed set

The runner adds version 2 of the evaluation beside version 1 (no latency
constraint and the frontiers, unchanged in `results`): at each SLA of a
grid (default 100, 300, 1000, 3000, 10000 ms; --slas-ms overrides), ASAP
and PerQuery plan the cheapest plan whose batch latency (the longest
chain) is at most the SLA, via minimize_usage_cost's latency bound, which
is exact with elastic CPU. An SLA below a method's tightest feasible
bound gives an infeasible record. AutoSketch's one plan is recorded at
each SLA with meets_sla. Records go to `sla_results`, with sanity checks
that every plan meets its SLA and that ASAP costs no more than PerQuery.

Results on clnode109 (--runs 3) for the mixed set, r = 8 and m = 8, 16:
no drops, no sanity violations; the tightest feasible SLA is 92 ms, and
AutoSketch (latency 1011 ms) meets only 3000 and 10000 ms. AutoSketch's
planning time is search plus its measured benchmark
(scripts/autosketch_benchmark_time.py: approxbench accuracy runs at 1e8
items, 20-33 s each, serial and alone on the node): 197 s at m = 1,
3159 s at m = 16. scripts/plot_autosketch_vs_asap_sla.py draws
fig_cost_vs_sla.png and summary_sla.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: remove the old trace results and figures

They came from the earlier model (per-RQE SLA over a µs grid, mean-CPU
cost); the traces are to be rerun on cost by use with both latency
versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: trim the PR to what it needs

- Result JSONs without per-RQE choices (`chosen`): the figures and
  summaries don't read them, and they were nearly all of the 2.9M lines;
  rerun the runner for them.
- Machine files folded into the READMEs (all Intel Xeon E5-2683 v3,
  56 cores); the synthetic run script no longer writes them.
- The SLA-era trace scripts (plot_autosketch_vs_asap.py,
  run_autosketch_vs_asap.sh) are removed; the traces are to be rerun on
  cost by use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: keep only what the three figures need

The PR's results are the frontier (version 1), cost vs. SLA (version 2)
and planning time on the synthetic mixed set. The trace table goes back
to the base version (the traces are to be rerun on cost by use, with
their new table), and the ASAP profiling-time script and data, which no
figure uses, are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: commit the inputs of the three figures

rqe-optimizer/results/autosketch-vs-asap-inputs/ holds what the figures
were computed from: the sketch-bench study on asap_sketchlib 0.3.0 (the
cost table, sketches and exact aggregations, and the accuracy curves the
runner reads as --saturation-dir) and the synthetic mixed-set tables with
plan.tsv. The READMEs point the reproduce steps at them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: the raw approxbench records behind the figures' study

rqe-optimizer/results/autosketch-vs-asap-inputs/raw/ holds the gzipped
approxbench records (920 KB) that ../saturation/ was reduced from, with a
README mapping each file to its machine, arguments, record count, run time
and output. Rerunning them is 0.77 h of machine time over 5 machines in
parallel (about 20-30 minutes of wall time).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…QEs (ProjectASAP/sketch-bench#190)

* rqe-optimizer: let mergeable finer-group deployments serve coarser RAQEs

A deployment grouped by G_d now serves a RAQE grouped by any subset G_r when
its family merges across groups (exact sum/min/max, HLL, UnivMon cardinality,
KLL, DD). The read is priced at G_r: card(G_d)·L/x − card(G_r) merges, with
card(G_r) accumulators, probes and output. Accuracy reads G_r's shape and
items, and KLL's merge curve at the average fan-out times L/x. Fine candidates
take windows from coarser RAQEs. validate_facts rejects a label set with more
groups than a superset.

Closes ProjectASAP/sketch-bench#187

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: split RQE optimizer design by concern

* docs: clarify RQE optimizer cost model status

* docs: one cost table for every phase, with roll-ups

Merge the cost-by-use CPU and memory tables into one table per phase
(ingest, compaction, query job, storage) in terms of I_d and I_r, so
roll-ups (G_r ⊂ G_d) are priced as the code prices them:
ℓ = card(G_r)·c_qry + (I_d·n − I_r)·c_mrg, accumulators at G_r.

The snapshot-AUC section now lists only how it differs (k = 1, query
memory always held, per-RAQE serial latency) instead of repeating the
table; the shape/size-law side-by-side gives each family's I_d, I_r and
tracker; the proposed Hydra cost model is §3's table with I_d = I_r = 1,
adding compaction and using the doc's symbols.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: KLL's memory is sketch-bench's nominal footprint, not measured at 1e6

The cost table's KLL m is kll_footprint = 4·k·sizeof(T), independent of
the item count (1600 B at k = 50 for 1e3, 1e6 and 1e8 items), so it is not
"the size measured at 1e6 values" and does not grow with n. Say so, and
how it compares with asap_sketchlib's preallocated capacity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: prune on every cost cost-by-use bills; fix stale comments

Dominance pruning compared ingest, query latency, merge memory and
storage. Cost by use also bills compaction (per window in every chain,
per slide in CPU and byte-seconds) and query output memory, which
differs between top-k configs answering different k. A direct RAQE
never merges at query time, so a candidate with cheaper ingest but
costlier merges could prune one that is cheaper by use. Compare those
too; the new test fails without it.

Also: workload_scenarios no longer says groupings never share; the
exact-increase exclusion states its real (conservative) reason;
merged_instance_count notes roll-ups; a circular doc comment; stale §N
citations now name the docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: consolidate the optimizer docs; add the Hydra admission design

- candidates: eligibility allows subset groupings for families that
  merge across groups; generation shows the roll-up RAQE sets; accuracy
  is read at G_r with KLL's fan-out merge count, and past the measured
  merge counts the guarantee applies, floored by the last measurement;
  pruning lists every compared cost; the MILP lives only in the cost
  doc; one roll-up section.
- cost model: top-k output and query CPU scale with k; k_D sums parts;
  mergeable_across_groups listed; KLL footprint points at ProjectASAP/sketch-bench#191/#192;
  Hydra moved out.
- v1 and TODO: phases include compaction and the query job; TODOs match
  cost by use.
- rqe_optimizer_hydra.md (new): Hydra in sketch-bench, accuracy bounds
  per combination (CMS, CountSketch, HLL, UnivMon, KLL) checked against
  the paper's Theorem 2 and the library (correlated inner seeds, fixed
  column hash, few rows), cost model, candidates and MILP edges,
  sketch-bench requirements and open decisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: univmon-cardinality merges lossily; increase's real reason

UnivMon's counters add exactly on merge, but each level's heavy-hitter
heap is rebuilt from the union of the two heaps, so a key heavy only in
the union is lost: a merged (or rolled-up) answer now reads merge
curves, as KLL's does.

exact-increase's accumulator merge joins two pieces of one counter in
time, so merging two groups' counters is wrong; say so instead of
"unconfirmed".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: check group counts only between label sets a plan uses

validate_facts compared every pair of label sets in a metric's
cardinality map, so an inconsistent entry no RAQE groups by rejected the
whole workload. The roll-up merge count only reads RAQE groupings and the
full label set, so check X ⊂ Y among those.

Docs: why rate/increase, top-k and Hydra don't roll up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs(hydra): hydra-kll is measured in the same Hydra rerun

Every Hydra measurement predates the admission requirements, so they
are all rerun; hydra-kll goes in that run with the other four variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: move the Hydra admission design out of this PR

Keep ProjectASAP/sketch-bench#190 to roll-ups. docs/rqe_optimizer_hydra.md and its links move
to a separate PR stacked on this one; the docs here only say Hydra is
future work and doesn't roll up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: tests for roll-up DD memory, facts and pruning; doc fixes

Tests that fail without their rule:
- a DDSketch roll-up sizes its accumulator at the coarse grouping;
- validate_facts rejects a RAQE grouping with more groups than series;
- pruning drops a heap whose larger answered k only adds output memory.

Docs: the KLL footprint paragraph now that ProjectASAP/sketch-bench#192 is in; univmon-cardinality
reads merge curves like KLL (and has no guarantee past them); the TODO's
done list covers roll-ups and the pruning costs; "temporal pre-merging";
enumerate.rs names the MILP variables z and u as the docs do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: name every lossy non-heap sketch that needs merge curves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zz_y <zeyingz@umd.edu>
…rojectASAP/sketch-bench#181)

* docs: end-to-end planning flow from traces to an rqe-optimizer plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: kind/weight/range and data_shape, fanning out, and ProjectASAP/sketch-bench#180/#182/#183 in the planning flow

From review on ProjectASAP/sketch-bench#181:
- skew_summary.csv's kind, weight and range columns, and how ranges map onto
  data_shape (keyed by grouping: the worst case over its RAQEs' ranges).
- A 'Fanning out over machines' section and scripts/merge_study_parts.py,
  which concatenates per-machine run directories, refusing a point in two
  parts.
- Brought up to date: top-k needs no merge curves (heap m·k, ProjectASAP/sketch-bench#182), the
  guarantee fallback is merged (ProjectASAP/sketch-bench#180), main is through ProjectASAP/sketch-bench#183.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 27 commits October 11, 2026 12:34
…h needs) (ProjectASAP/sketch-bench#193)

* docs: Hydra admission design (accuracy, cost, candidates, sketch-bench needs)

docs/rqe_optimizer_hydra.md: Hydra as sketch-bench runs it; accuracy
bounds per combination (CMS, CountSketch, HLL, UnivMon, KLL), checked
against the paper's Theorem 2 and the library; cost model as the cost
doc's §3 table with I_d = I_r = 1; candidates and MILP edges (MILP
unchanged, still linear); what sketch-bench must add, hydra-kll
included in the same rerun; open decisions. Links from the other
optimizer docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs(hydra): consolidate the Hydra notes already on main

- cost model: the Shared + Fixed bullet no longer presents the
  subpopulations gate as Hydra's rule or fan-out as a deviation; it is a
  necessary check only, and admission is the measured predicate of
  rqe_optimizer_hydra.md §2.4. The HydraKLL column of the side-by-side
  table moves to Hydra §3; the key tracker's grouping is open (§6.1).
- candidates.rs: measured_at_group_count's comment says the same.
- configs/datagen hydra yamls: since asap_sketchlib 0.3.0 the sketch keys
  subsets by column; only the polars baseline still aliases (ProjectASAP/sketch-bench#74).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ch-bench#138) (ProjectASAP/sketch-bench#194)

The eval's inputs README lists the saturation study's accuracy grid
(out_grid_1e7_cost) and targeted 1e9 points (out_1e9), but .gitignore's
*.csv kept their six CSVs out of ProjectASAP/sketch-bench#138, so the eval couldn't be rerun
from the repo. Add a narrow exception for that directory and commit
them (about 1 MB): the curves the committed results were planned with.


Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ASAP/sketch-bench#202)

* polars: tag Hydra baseline subset keys by column (ProjectASAP/sketch-bench#74)

The polars exact baselines keyed a label subset by its `;`-joined values,
asap_sketchlib 0.2.2's subkey. 0.3.0's Hydra tags each value with its
column (`label0:a;label1:b`), so on label columns that share a value
domain the baseline merged a key1 group with the same key2 group and
scored nonzero against itself. subset_key now builds the 0.3.0 key over
hydra_shared's `label{i}` schema (escaping `\`, `:` and `;`), empty
labels keep their column, and every lookup goes through it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* polars: write subset key tags without a temporary String; fix test comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: interleaved merge split; merge exactness and bound tests

`--merge-split contiguous|interleaved` (default contiguous, unchanged) picks
how merge runs split the stream into shards. Interleaved reorders the stream
so the existing contiguous `partition` deals it round-robin by record, at the
contiguous split's shard sizes, so every row's folds take it unchanged.

Tests: hydra-cms, hydra-cs and hydra-hll grids folded from 4 and 7 shards,
under both splits, answer every probed subpopulation exactly as the single
stream; and every scored group on a fixed 20k-record stream lands inside its
hydra doc §2.2 bound (CMS also never under).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* datagen: child_of columns for hierarchical labels

A `child_of: <index>` column with `fan_out: f` nests under an earlier
string column: its value is the parent row's value, a `.`, and a child
index in [0, f) drawn from the column's own distribution. Bad indices,
a fan_out that disagrees with the domain, and fields that would not mean
what they say on a child are refused before drawing.

Adds configs/datagen/hydra_hier.yaml (region 4 -> service 25 -> endpoint
25, status 4, user_id value) for the hydra rows, docs, and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: score any target grouping; per-group error output

Every hydra-* row (lib and polars, all five variants and hydra-univmon's
four statistics) takes `--group-columns J,K,...` (default `0`, today's
behavior) in place of the hardcoded SCORED_LABEL_COLUMN. A subpopulation
probe is now positioned (`Vec<Option<String>>`, Some at the grouped
columns), so `hydra_shared::query` asks any non-empty subset of the key
columns and the polars baselines look up `subset_key` with that subset's
mask (`group_key`, generalizing `prefix_key`).

Each grouped comparator reports its per-group error in its own metric
(relative error; mean rank error for hydra-kll) and summarises it as
err_mean, err_p50, err_p90, err_max, groups_scored, plus schema_width,
records and fanned_mass. `--per-group-out PATH` writes the rows as
`group_key,n_q,error`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* datagen: whole uniform bounds for child_of; 2- and 3-label hierarchy cuts

A uniform draw is continuous, so `uniform{0.0, 2.9}` with fan_out 2 drew
child indices 0, 1 and 2; child_of now requires whole uniform bounds.
hydra_hier_d2.yaml and hydra_hier_d3.yaml cut hydra_hier.yaml to its first
2 and 3 labels for the Hydra study's schema-width sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* datagen: the hydra_hier usage example passes --dtype

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: list every group in --per-group-out; escape keys; refuse no accuracy

- Group keys escape `\`, `:` and `;` as polars_shared::subset_key does.
- Groups that cannot be scored (zero true statistic, e.g. zero entropy) are
  written with an empty `error`, so n_q sums to the stream's records.
- --per-group-out with no accuracy measurement is refused instead of
  writing an empty file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: KLL CDF and UnivMon sum rows; KLL cell merge buffer

- hydra-kll-cdf (lib, polars): asks the KLL grid HydraQuery::Cdf(x) at each
  group's own values at phi = 0.01..0.99, scored by absolute CDF error
  (SubpopCdfErrorGT, comparator subpop-cdf-error). The polars baseline is the
  exact share at or below x over the same sorted groups.
- hydra-univmon-sum (lib, polars): each record inserted with count = value,
  so the UnivMon grid's L1 is the group's sum, scored by relative error
  against exact per-group sums (SubpopSumGT, comparator subpop-sum).
  Integer value dtypes only; f64/str and values outside [0, i32::MAX] are
  refused by name.
- memory_hydra_kll counts each cell as kll_lib_bytes does, merge buffer of
  cell_k items included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra-kll: the footprint's merge buffer is the steady state; tie tests

The grid clones its cells from one template, and a cloned Vec keeps no
capacity, so a cell's merge buffer is empty at build and grows to about
cell_k items at its first compaction. Say the footprint counts it at that
bound. The CDF test now asks a group of ties (at or below) and a mixed-tie
group on a grid wide enough that it shares no cell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: record merge_split; review fixes

- `merge_split` (contiguous|interleaved) is a record field next to
  `merge_shards` (BenchSection and MergeMetrics; schema regenerated).
  Merge records stamp it; a record without it split contiguously, so
  existing results parse, and flatten refuses two different splits in
  one row as it does two shard counts.
- `MergeSplit` moves to aqpbm-core (it is a record type) and is parsed by
  clap directly instead of matching strings.
- The interleaved copy of the stream is built only when a merge runs.
- README states the round-robin exactly (contiguous shard sizes, so a
  short last shard shifts the pattern); MergeMetrics docs name the split.
- hydra-cs bound test: note its margin (passes from beta ~4.8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study: --phase hydra on the eval's datasets; Hydra cost rows

configs/datagen/hydra_http.yaml, hydra_http_latency.yaml and hydra_flows.yaml
follow the §6.3 eval's http and flows schemas (labels in schema order,
cardinalities, child_of, skews; user_id Zipf 0.8, latency Pareto a = 2,
src_ip uniform). The flows DDoS burst is not generated: datagen draws each
column independently.

--phase hydra runs HYDRA_SWEEPS: the eval's variants on those datasets (every
grouping, W 1024/4096/16384, N 1e5..1e7, 1/4/16 interleaved shards, 3 seeds)
and every statistic on hydra_hier{_d2,_d3,}.yaml, one run per grouping with
--per-group-out, into hydra_saturation.csv with the coverage columns
err_max_cov_0.01/0.05 (share >= tau plus the largest group). --jobs under a
memory budget, --resume, and --hydra-shard i/n to split it across nodes.

--phase optimizer-cost adds Hydra rows (eval variants x W x dataset);
MeasuredAt gains dataset and schema_width, the reducer reading the width off
the row's scores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra-univmon-sum: the ground truth refuses negative values too

The lib row inserts each value as a UnivMon count and refuses negatives;
the truth (and so the polars baseline) accepted them, so a spec with
negatives ran on polars and failed on lib.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study: Hydra cost rows at 3 runs; their test; datagen path from the script

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: Hydra cost rows' run count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study_saturation: atomic hydra table rewrite, stricter resume, schema checks

--phase hydra --resume now writes the kept rows to hydra_saturation.csv.tmp
and os.replace()s it over the table, and keeps a row only when it ends in a
line end and every column parses (integers, numbers or "" for the error
columns, non-empty text). The CSV format is unchanged, so tables written by
003eb31 resume as before. The eval-spec test also checks each label's
cardinality, skew and child_of against PR ProjectASAP/sketch-bench#196's SCHEMAS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study_saturation: --hydra-variants/--hydra-ws narrow the Hydra cost rows too

A hydra-univmon-cardinality grid at W=16384 merged from 16 shards
outgrows a 251 GB node, and the cost phase can't resume; these flags let
the cost phase leave a variant or width out, as they do the hydra phase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* aqpbm-datagen: scale_by/scale_range; hydra_http_latency scales latency per service

A f64 column with `scale_by: <label column>` and `scale_range: [lo, hi]`
multiplies each row by a factor fixed per label value, log-uniform in
[lo, hi] from a hash of the value and the column's seed. Validation:
earlier string label column, f64 only, 0 < lo <= hi. Absent fields keep
every existing spec's output byte-identical.

hydra_http_latency.yaml scales latency by service in [1, 10], so groups
differ in latency scale and a Hydra-KLL mixture is no longer the queried
group's own CDF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…dra and roll-ups (ProjectASAP/sketch-bench#196)

* eval: multi-grouping anomaly-detection templates (http, flows)

The generator gets a metric-level ordered label schema (labels with
cardinality, child_of fan-out and skew; per-value data shape) and
per-RQE `grouping` and `covers_share`. The 10 templates are now the
"classic" set, byte-for-byte the committed tables; "all" adds templates
11, 12, 14, 15, 16, 17 on `http` and `flows`, whose streams each carry
several groupings so roll-ups (and later Hydra) can share. The plan runs
both sets over the grid.

The runner builds each schema stream's MetricFacts from the schema
(labels = schema + x; cardinality per used grouping and the full set),
takes Raqe.grouping_labels from the RQE's grouping, maps the
`cardinality` capability to HLL, and reports covers_share and roll-up
deployment groupings in each choice. Classic output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: skewed http labels, per-RQE group coverage, fresh-checkout test

Review of ProjectASAP/sketch-bench#196. http's labels are now Zipf-skewed (region 0.5, service
1.1, endpoint 1.1 within its service, status 2.0) and the flows DDoS
burst targets a tail subnet, so covers_share selects a few groups. The
generator computes each group's share (product of its labels' shares)
and, per RQE, the covered groups (share >= tau, plus always the largest)
and the smallest covered group's share and items per window. The runner
reads a covered RQE's accuracy (ASAP and AutoSketch) at that group's
items instead of the mean group's; covers_share = null is unchanged.

The schema test reads an inline HLL curve fixture instead of the
gitignored study curves. The synthetic plot keys runs by template set
too, and both plot scripts title classic "mixed" and all "mixed +
multi-grouping". README: the coverage rule, the N the runner uses, and
which schema fields are descriptive. label_set.keys_per_window is
commented as the universe K.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: read null-coverage RQEs at their smallest group; template 17 at {service, endpoint}

- covers_share = null covers every group, so the runner reads its accuracy
  at the smallest group (min_covered_share), like a tau RQE's smallest
  covered group. Classic RQEs (no covers_share) are unchanged.
- Template 17 (negative control) groups by {service, endpoint}: its
  smallest group holds 4.5e4 items in 5m, on the curves.
- The generator asserts every schema RQE's covered_min_N >= 1e3 (drops the
  vacuous `assert covered`); a test checks the old full-label-set template
  fails it.
- Covered facts go through validate_facts.
- README: null semantics, template 17, and the numbers' provenance (fresh
  `all` table, committed saturation inputs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: hold covered RQEs to the worse of their smallest and largest group

Per sketch-bench#189, KLL/DD/HLL error does not fall as N grows, so the
largest covered group can be harder than the smallest. The generator also
emits max_covered_share / covered_max_N (the largest group, always covered);
the runner reads each covered RQE's accuracy at both the smallest and the
largest covered group and keeps the worse (direction-aware), unknown if
either read is. Classic RQEs and Hydra's lookup are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: template 12 groups by (dst_subnet, proto), not (dst_subnet, dst_port)

(dst_subnet, dst_port) has 1e6 groups; answering each every minute takes
about 49 s on one core, which set every plan's batch latency above the
largest SLA (10 s) and collapsed the multi-grouping frontier. (dst_subnet,
proto) has 3e3 groups and keeps three groupings over one stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tes, with measured accuracy by coverage (ProjectASAP/sketch-bench#201)

* eval: multi-grouping anomaly-detection templates (http, flows)

The generator gets a metric-level ordered label schema (labels with
cardinality, child_of fan-out and skew; per-value data shape) and
per-RQE `grouping` and `covers_share`. The 10 templates are now the
"classic" set, byte-for-byte the committed tables; "all" adds templates
11, 12, 14, 15, 16, 17 on `http` and `flows`, whose streams each carry
several groupings so roll-ups (and later Hydra) can share. The plan runs
both sets over the grid.

The runner builds each schema stream's MetricFacts from the schema
(labels = schema + x; cardinality per used grouping and the full set),
takes Raqe.grouping_labels from the RQE's grouping, maps the
`cardinality` capability to HLL, and reports covers_share and roll-up
deployment groupings in each choice. Classic output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: skewed http labels, per-RQE group coverage, fresh-checkout test

Review of ProjectASAP/sketch-bench#196. http's labels are now Zipf-skewed (region 0.5, service
1.1, endpoint 1.1 within its service, status 2.0) and the flows DDoS
burst targets a tail subnet, so covers_share selects a few groups. The
generator computes each group's share (product of its labels' shares)
and, per RQE, the covered groups (share >= tau, plus always the largest)
and the smallest covered group's share and items per window. The runner
reads a covered RQE's accuracy (ASAP and AutoSketch) at that group's
items instead of the mean group's; covers_share = null is unchanged.

The schema test reads an inline HLL curve fixture instead of the
gitignored study curves. The synthetic plot keys runs by template set
too, and both plot scripts title classic "mixed" and all "mixed +
multi-grouping". README: the coverage rule, the N the runner uses, and
which schema fields are descriptive. label_set.keys_per_window is
commented as the universe K.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: read null-coverage RQEs at their smallest group; template 17 at {service, endpoint}

- covers_share = null covers every group, so the runner reads its accuracy
  at the smallest group (min_covered_share), like a tau RQE's smallest
  covered group. Classic RQEs (no covers_share) are unchanged.
- Template 17 (negative control) groups by {service, endpoint}: its
  smallest group holds 4.5e4 items in 5m, on the curves.
- The generator asserts every schema RQE's covered_min_N >= 1e3 (drops the
  vacuous `assert covered`); a test checks the old full-label-set template
  fails it.
- Covered facts go through validate_facts.
- README: null semantics, template 17, and the numbers' provenance (fresh
  `all` table, committed saturation inputs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: hold covered RQEs to the worse of their smallest and largest group

Per sketch-bench#189, KLL/DD/HLL error does not fall as N grows, so the
largest covered group can be harder than the smallest. The generator also
emits max_covered_share / covered_max_N (the largest group, always covered);
the runner reads each covered RQE's accuracy at both the smallest and the
largest covered group and keeps the worse (direction-aware), unknown if
either read is. Classic RQEs and Hydra's lookup are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: template 12 groups by (dst_subnet, proto), not (dst_subnet, dst_port)

(dst_subnet, dst_port) has 1e6 groups; answering each every minute takes
about 49 s on one core, which set every plan's batch latency above the
largest SLA (10 s) and collapsed the multi-grouping frontier. (dst_subnet,
proto) has 3e3 groups and keeps three groupings over one stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: admit Hydra (HLL, UnivMon cardinality, KLL) as candidates, with measured accuracy by coverage

- FamilyProperties for hydra-hll, hydra-univmon-cardinality and hydra-kll:
  one fixed-size grid, DeltaSet key tracker, no roll-up, and the new
  answers_any_subgrouping. Added to Capability::families, not to
  DEPLOYABLE_FAMILIES.
- Candidates: on a metric with a hydra_dataset, one grid per schema Λ (each
  RAQE grouping and the union, where card(Λ) is known); eligible for any
  RAQE with ∅ ≠ G_r ⊆ Λ. The subpopulations ceiling is gone.
- Accuracy: hydra_saturation.csv under the saturation dir (optional), keyed
  by variant, config, dataset and grouping; the column the RAQE's
  accuracy_covers_share selects; worst over records and seeds; lossy merges
  bracketed by merge_shards, unknown past the largest.
- Runner: ASAP and PerQuery plan with undeployable families; RQEs carry
  covers_share; http/flows streams name their Hydra dataset; results report
  hydra_rqes for workloads Hydra may serve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: Hydra grids at the full schema from their dataset's rows

Round-1 review of ProjectASAP/sketch-bench#201:
- One Hydra grid per (stream, variant, config, window, slide), over the
  metric's full schema only (MetricFacts::hydra_dataset now carries the
  dataset and the schema), the width the study measures.
- MeasuredAt gains optional `dataset` and `schema_width`; a Hydra row is a
  candidate on a metric only if measured on its dataset.
- Runner: card(full schema) = product of label fan-outs, capped at the
  stream's series; AutoSketch also allows undeployable families (it skips
  Hydra itself, ProjectASAP/sketch-bench#159).
- read_csv checks required columns and short rows; config params must be
  numbers.
- Docs: Hydra held to the worst case, built only at the full schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: validate the Hydra schema, skip empty Hydra cells, reject long CSV rows

validate_facts reports a Hydra schema outside the metric's labels or
without a nonzero cardinality, instead of a later panic. The Hydra
accuracy fold skips empty cells (no covered group scored in that run)
and is unknown only when every row left the column empty. read_csv
rejects rows longer than the header too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: test fixtures carry max_covered_share and label cardinalities

After rebasing onto ProjectASAP/sketch-bench#196's two-ended coverage read: ProjectASAP/sketch-bench#201's runner test RQEs
need max_covered_share, and ProjectASAP/sketch-bench#196's coverage test schema needs the label
cardinality the full-schema Hydra grid reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: roll-up ablation, ASAP without roll-ups

The runner adds the method `asap-norollup` ("ASAP (no roll-ups)"): ASAP's
MILP, candidates, frontier sweep and SLA grid, with each RQE allowed only
deployments at its own grouping or Hydra grids. Every result counts
`rolled_up_rqes` (RQEs served by a finer non-Hydra deployment), and sanity
checks asap <= asap-norollup <= perquery unbounded and at each SLA.

The plot scripts draw it as a dashed violet line, add
fig_rollup_ablation.png (unbounded cost with and without roll-ups, absolute,
CPU and Fargate panels) and a roll-up section in both summaries; results
without the method still plot. The run script passes --no-chosen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…n (§6.3) (ProjectASAP/sketch-bench#203)

* polars: tag Hydra baseline subset keys by column (ProjectASAP/sketch-bench#74)

The polars exact baselines keyed a label subset by its `;`-joined values,
asap_sketchlib 0.2.2's subkey. 0.3.0's Hydra tags each value with its
column (`label0:a;label1:b`), so on label columns that share a value
domain the baseline merged a key1 group with the same key2 group and
scored nonzero against itself. subset_key now builds the 0.3.0 key over
hydra_shared's `label{i}` schema (escaping `\`, `:` and `;`), empty
labels keep their column, and every lookup goes through it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: multi-grouping anomaly-detection templates (http, flows)

The generator gets a metric-level ordered label schema (labels with
cardinality, child_of fan-out and skew; per-value data shape) and
per-RQE `grouping` and `covers_share`. The 10 templates are now the
"classic" set, byte-for-byte the committed tables; "all" adds templates
11, 12, 14, 15, 16, 17 on `http` and `flows`, whose streams each carry
several groupings so roll-ups (and later Hydra) can share. The plan runs
both sets over the grid.

The runner builds each schema stream's MetricFacts from the schema
(labels = schema + x; cardinality per used grouping and the full set),
takes Raqe.grouping_labels from the RQE's grouping, maps the
`cardinality` capability to HLL, and reports covers_share and roll-up
deployment groupings in each choice. Classic output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* polars: write subset key tags without a temporary String; fix test comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: interleaved merge split; merge exactness and bound tests

`--merge-split contiguous|interleaved` (default contiguous, unchanged) picks
how merge runs split the stream into shards. Interleaved reorders the stream
so the existing contiguous `partition` deals it round-robin by record, at the
contiguous split's shard sizes, so every row's folds take it unchanged.

Tests: hydra-cms, hydra-cs and hydra-hll grids folded from 4 and 7 shards,
under both splits, answer every probed subpopulation exactly as the single
stream; and every scored group on a fixed 20k-record stream lands inside its
hydra doc §2.2 bound (CMS also never under).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* datagen: child_of columns for hierarchical labels

A `child_of: <index>` column with `fan_out: f` nests under an earlier
string column: its value is the parent row's value, a `.`, and a child
index in [0, f) drawn from the column's own distribution. Bad indices,
a fan_out that disagrees with the domain, and fields that would not mean
what they say on a child are refused before drawing.

Adds configs/datagen/hydra_hier.yaml (region 4 -> service 25 -> endpoint
25, status 4, user_id value) for the hydra rows, docs, and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: score any target grouping; per-group error output

Every hydra-* row (lib and polars, all five variants and hydra-univmon's
four statistics) takes `--group-columns J,K,...` (default `0`, today's
behavior) in place of the hardcoded SCORED_LABEL_COLUMN. A subpopulation
probe is now positioned (`Vec<Option<String>>`, Some at the grouped
columns), so `hydra_shared::query` asks any non-empty subset of the key
columns and the polars baselines look up `subset_key` with that subset's
mask (`group_key`, generalizing `prefix_key`).

Each grouped comparator reports its per-group error in its own metric
(relative error; mean rank error for hydra-kll) and summarises it as
err_mean, err_p50, err_p90, err_max, groups_scored, plus schema_width,
records and fanned_mass. `--per-group-out PATH` writes the rows as
`group_key,n_q,error`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* datagen: whole uniform bounds for child_of; 2- and 3-label hierarchy cuts

A uniform draw is continuous, so `uniform{0.0, 2.9}` with fan_out 2 drew
child indices 0, 1 and 2; child_of now requires whole uniform bounds.
hydra_hier_d2.yaml and hydra_hier_d3.yaml cut hydra_hier.yaml to its first
2 and 3 labels for the Hydra study's schema-width sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: skewed http labels, per-RQE group coverage, fresh-checkout test

Review of ProjectASAP/sketch-bench#196. http's labels are now Zipf-skewed (region 0.5, service
1.1, endpoint 1.1 within its service, status 2.0) and the flows DDoS
burst targets a tail subnet, so covers_share selects a few groups. The
generator computes each group's share (product of its labels' shares)
and, per RQE, the covered groups (share >= tau, plus always the largest)
and the smallest covered group's share and items per window. The runner
reads a covered RQE's accuracy (ASAP and AutoSketch) at that group's
items instead of the mean group's; covers_share = null is unchanged.

The schema test reads an inline HLL curve fixture instead of the
gitignored study curves. The synthetic plot keys runs by template set
too, and both plot scripts title classic "mixed" and all "mixed +
multi-grouping". README: the coverage rule, the N the runner uses, and
which schema fields are descriptive. label_set.keys_per_window is
commented as the universe K.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: read null-coverage RQEs at their smallest group; template 17 at {service, endpoint}

- covers_share = null covers every group, so the runner reads its accuracy
  at the smallest group (min_covered_share), like a tau RQE's smallest
  covered group. Classic RQEs (no covers_share) are unchanged.
- Template 17 (negative control) groups by {service, endpoint}: its
  smallest group holds 4.5e4 items in 5m, on the curves.
- The generator asserts every schema RQE's covered_min_N >= 1e3 (drops the
  vacuous `assert covered`); a test checks the old full-label-set template
  fails it.
- Covered facts go through validate_facts.
- README: null semantics, template 17, and the numbers' provenance (fresh
  `all` table, committed saturation inputs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* datagen: the hydra_hier usage example passes --dtype

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: list every group in --per-group-out; escape keys; refuse no accuracy

- Group keys escape `\`, `:` and `;` as polars_shared::subset_key does.
- Groups that cannot be scored (zero true statistic, e.g. zero entropy) are
  written with an empty `error`, so n_q sums to the stream's records.
- --per-group-out with no accuracy measurement is refused instead of
  writing an empty file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: KLL CDF and UnivMon sum rows; KLL cell merge buffer

- hydra-kll-cdf (lib, polars): asks the KLL grid HydraQuery::Cdf(x) at each
  group's own values at phi = 0.01..0.99, scored by absolute CDF error
  (SubpopCdfErrorGT, comparator subpop-cdf-error). The polars baseline is the
  exact share at or below x over the same sorted groups.
- hydra-univmon-sum (lib, polars): each record inserted with count = value,
  so the UnivMon grid's L1 is the group's sum, scored by relative error
  against exact per-group sums (SubpopSumGT, comparator subpop-sum).
  Integer value dtypes only; f64/str and values outside [0, i32::MAX] are
  refused by name.
- memory_hydra_kll counts each cell as kll_lib_bytes does, merge buffer of
  cell_k items included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra-kll: the footprint's merge buffer is the steady state; tie tests

The grid clones its cells from one template, and a cloned Vec keeps no
capacity, so a cell's merge buffer is empty at build and grows to about
cell_k items at its first compaction. Say the footprint counts it at that
bound. The CDF test now asks a group of ties (at or below) and a mixed-tie
group on a grid wide enough that it shares no cell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra: record merge_split; review fixes

- `merge_split` (contiguous|interleaved) is a record field next to
  `merge_shards` (BenchSection and MergeMetrics; schema regenerated).
  Merge records stamp it; a record without it split contiguously, so
  existing results parse, and flatten refuses two different splits in
  one row as it does two shard counts.
- `MergeSplit` moves to aqpbm-core (it is a record type) and is parsed by
  clap directly instead of matching strings.
- The interleaved copy of the stream is built only when a merge runs.
- README states the round-robin exactly (contiguous shard sizes, so a
  short last shard shifts the pattern); MergeMetrics docs name the split.
- hydra-cs bound test: note its margin (passes from beta ~4.8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study: --phase hydra on the eval's datasets; Hydra cost rows

configs/datagen/hydra_http.yaml, hydra_http_latency.yaml and hydra_flows.yaml
follow the §6.3 eval's http and flows schemas (labels in schema order,
cardinalities, child_of, skews; user_id Zipf 0.8, latency Pareto a = 2,
src_ip uniform). The flows DDoS burst is not generated: datagen draws each
column independently.

--phase hydra runs HYDRA_SWEEPS: the eval's variants on those datasets (every
grouping, W 1024/4096/16384, N 1e5..1e7, 1/4/16 interleaved shards, 3 seeds)
and every statistic on hydra_hier{_d2,_d3,}.yaml, one run per grouping with
--per-group-out, into hydra_saturation.csv with the coverage columns
err_max_cov_0.01/0.05 (share >= tau plus the largest group). --jobs under a
memory budget, --resume, and --hydra-shard i/n to split it across nodes.

--phase optimizer-cost adds Hydra rows (eval variants x W x dataset);
MeasuredAt gains dataset and schema_width, the reducer reading the width off
the row's scores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* hydra-univmon-sum: the ground truth refuses negative values too

The lib row inserts each value as a UnivMon count and refuses negatives;
the truth (and so the polars baseline) accepted them, so a spec with
negatives ran on polars and failed on lib.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study: Hydra cost rows at 3 runs; their test; datagen path from the script

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* docs: Hydra cost rows' run count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study_saturation: atomic hydra table rewrite, stricter resume, schema checks

--phase hydra --resume now writes the kept rows to hydra_saturation.csv.tmp
and os.replace()s it over the table, and keeps a row only when it ends in a
line end and every column parses (integers, numbers or "" for the error
columns, non-empty text). The CSV format is unchanged, so tables written by
003eb31 resume as before. The eval-spec test also checks each label's
cardinality, skew and child_of against PR ProjectASAP/sketch-bench#196's SCHEMAS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study_saturation: --hydra-variants/--hydra-ws narrow the Hydra cost rows too

A hydra-univmon-cardinality grid at W=16384 merged from 16 shards
outgrows a 251 GB node, and the cost phase can't resume; these flags let
the cost phase leave a variant or width out, as they do the hydra phase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: hold covered RQEs to the worse of their smallest and largest group

Per sketch-bench#189, KLL/DD/HLL error does not fall as N grows, so the
largest covered group can be harder than the smallest. The generator also
emits max_covered_share / covered_max_N (the largest group, always covered);
the runner reads each covered RQE's accuracy at both the smallest and the
largest covered group and keeps the worse (direction-aware), unknown if
either read is. Classic RQEs and Hydra's lookup are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* aqpbm-datagen: scale_by/scale_range; hydra_http_latency scales latency per service

A f64 column with `scale_by: <label column>` and `scale_range: [lo, hi]`
multiplies each row by a factor fixed per label value, log-uniform in
[lo, hi] from a hash of the value and the column's seed. Validation:
earlier string label column, f64 only, 0 < lo <= hi. Absent fields keep
every existing spec's output byte-identical.

hydra_http_latency.yaml scales latency by service in [1, 10], so groups
differ in latency scale and a Hydra-KLL mixture is no longer the queried
group's own CDF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: template 12 groups by (dst_subnet, proto), not (dst_subnet, dst_port)

(dst_subnet, dst_port) has 1e6 groups; answering each every minute takes
about 49 s on one core, which set every plan's batch latency above the
largest SLA (10 s) and collapsed the multi-grouping frontier. (dst_subnet,
proto) has 3e3 groups and keeps three groupings over one stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: admit Hydra (HLL, UnivMon cardinality, KLL) as candidates, with measured accuracy by coverage

- FamilyProperties for hydra-hll, hydra-univmon-cardinality and hydra-kll:
  one fixed-size grid, DeltaSet key tracker, no roll-up, and the new
  answers_any_subgrouping. Added to Capability::families, not to
  DEPLOYABLE_FAMILIES.
- Candidates: on a metric with a hydra_dataset, one grid per schema Λ (each
  RAQE grouping and the union, where card(Λ) is known); eligible for any
  RAQE with ∅ ≠ G_r ⊆ Λ. The subpopulations ceiling is gone.
- Accuracy: hydra_saturation.csv under the saturation dir (optional), keyed
  by variant, config, dataset and grouping; the column the RAQE's
  accuracy_covers_share selects; worst over records and seeds; lossy merges
  bracketed by merge_shards, unknown past the largest.
- Runner: ASAP and PerQuery plan with undeployable families; RQEs carry
  covers_share; http/flows streams name their Hydra dataset; results report
  hydra_rqes for workloads Hydra may serve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: Hydra grids at the full schema from their dataset's rows

Round-1 review of ProjectASAP/sketch-bench#201:
- One Hydra grid per (stream, variant, config, window, slide), over the
  metric's full schema only (MetricFacts::hydra_dataset now carries the
  dataset and the schema), the width the study measures.
- MeasuredAt gains optional `dataset` and `schema_width`; a Hydra row is a
  candidate on a metric only if measured on its dataset.
- Runner: card(full schema) = product of label fan-outs, capped at the
  stream's series; AutoSketch also allows undeployable families (it skips
  Hydra itself, ProjectASAP/sketch-bench#159).
- read_csv checks required columns and short rows; config params must be
  numbers.
- Docs: Hydra held to the worst case, built only at the full schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* rqe-optimizer: validate the Hydra schema, skip empty Hydra cells, reject long CSV rows

validate_facts reports a Hydra schema outside the metric's labels or
without a nonzero cardinality, instead of a later panic. The Hydra
accuracy fold skips empty cells (no covered group scored in that run)
and is unknown only when every row left the column empty. read_csv
rejects rows longer than the header too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: test fixtures carry max_covered_share and label cardinalities

After rebasing onto ProjectASAP/sketch-bench#196's two-ended coverage read: ProjectASAP/sketch-bench#201's runner test RQEs
need max_covered_share, and ProjectASAP/sketch-bench#196's coverage test schema needs the label
cardinality the full-schema Hydra grid reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: roll-up ablation, ASAP without roll-ups

The runner adds the method `asap-norollup` ("ASAP (no roll-ups)"): ASAP's
MILP, candidates, frontier sweep and SLA grid, with each RQE allowed only
deployments at its own grouping or Hydra grids. Every result counts
`rolled_up_rqes` (RQEs served by a finer non-Hydra deployment), and sanity
checks asap <= asap-norollup <= perquery unbounded and at each SLA.

The plot scripts draw it as a dashed violet line, add
fig_rollup_ablation.png (unbounded cost with and without roll-ups, absolute,
CPU and Fargate panels) and a roll-up section in both summaries; results
without the method still plot. The run script passes --no-chosen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: AutoSketch vs. ASAP with Hydra candidates and a roll-up ablation

Results of the §6.3 synthetic eval (#777) rerun with
roll-ups (ProjectASAP/sketch-bench#190) and their ablation, Hydra candidates measured on the eval's
own data (ProjectASAP/sketch-bench#195–ProjectASAP/sketch-bench#202), the multi-grouping template set (ProjectASAP/sketch-bench#196), ProjectASAP/sketch-bench#192's KLL
footprint and a re-measured cost table. 8 workloads, 0 RQEs dropped, 0
sanity violations.

Roll-ups serve 40% of the multi-grouping RQEs from a finer deployment and
halve ASAP's deployments, but save only 3.3–3.6% (CPU) / 4.4–4.8%
(Fargate). Hydra is eligible for many RQEs but never chosen: its insert
fans out to every label subset (1.9–5.1 µs per record vs. 3.85 ns for a
per-group HLL).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: cube template and a multi-grouping-only workload

Template 16 is now p99 latency GROUP BY CUBE(region, service, endpoint),
all 7 groupings, replacing {service}, {region, service}, {service, status}.
A new `multigroup` template set runs the multi-grouping templates alone, so
the roll-up ablation is not diluted by the classic set.

Roll-ups now save 60% on multi-grouping (0.551 -> 0.221 vCPU, 26 -> 5
deployments) and 8% on mixed + multi-grouping. Classic is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* eval: each workload's chosen plans, gzipped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ctASAP/sketch-bench#204 P2) (ProjectASAP/sketch-bench#208)

export_autosketch_eval_table.py --synthetic adds workloads[0].e2e to classic
tables: each PromQL query with its repeat interval and the RQEs it reads
(template 10's ratio reads two), and the metrics to generate (gauge data,
counter data_total; job and label_0 child_of job). The runner does not read
it, so the RQEs and the committed plans are unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…-bench#209)

* eval: e2e block for multi-grouping and all tables (ProjectASAP/sketch-bench#204)

export_autosketch_eval_table.py --synthetic now writes workloads[0].e2e to
every table. Schema templates have no PromQL (SQL is rendered downstream),
so each schema RQE gets one structured query entry: table, value,
capability, grouping, quantile (p99: 0.99), window_secs, interval_secs and
its rqes. Each schema metric is a `records` table (`http`, `flows`;
`data_i_http` under metric `data_i`) with SCHEMAS' labels, rate as
records_per_sec, and values (scaled_by, anomaly, burst as in SCHEMAS).
Classic entries are unchanged.

The multigroup and all tables in autosketch-vs-asap-hydra/inputs/tables are
regenerated; minus e2e they equal the committed ones byte for byte, and the
classic tables are unchanged. Their roll-up plans were already exported by
ProjectASAP/sketch-bench#203 (plans/*.plans.json.gz); a rerun of multigroup-shared1 and all-shared1
reproduces all 36 plans of each exactly, so the plans stand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: one e2e entry per distinct schema query (ProjectASAP/sketch-bench#204)

Schema query entries identical apart from `rqes` (templates 14, 15 and 17
share some groupings) are merged into one entry whose `rqes` lists every
RQE it reads, sorted, so each distinct SQL query appears once. Classic
entries are unchanged. The multigroup and all tables are regenerated;
minus e2e they still equal main's byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: wrap the e2e_block docstring

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: save ASAPQuery SQL in structured e2e queries

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Milind Srivastava <milindsrivastava1997@gmail.com>
…rojectASAP/sketch-bench#210)

* eval: e2e block for multi-grouping and all tables (ProjectASAP/sketch-bench#204)

export_autosketch_eval_table.py --synthetic now writes workloads[0].e2e to
every table. Schema templates have no PromQL (SQL is rendered downstream),
so each schema RQE gets one structured query entry: table, value,
capability, grouping, quantile (p99: 0.99), window_secs, interval_secs and
its rqes. Each schema metric is a `records` table (`http`, `flows`;
`data_i_http` under metric `data_i`) with SCHEMAS' labels, rate as
records_per_sec, and values (scaled_by, anomaly, burst as in SCHEMAS).
Classic entries are unchanged.

The multigroup and all tables in autosketch-vs-asap-hydra/inputs/tables are
regenerated; minus e2e they equal the committed ones byte for byte, and the
classic tables are unchanged. Their roll-up plans were already exported by
ProjectASAP/sketch-bench#203 (plans/*.plans.json.gz); a rerun of multigroup-shared1 and all-shared1
reproduces all 36 plans of each exactly, so the plans stand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: one e2e entry per distinct schema query (ProjectASAP/sketch-bench#204)

Schema query entries identical apart from `rqes` (templates 14, 15 and 17
share some groupings) are merged into one entry whose `rqes` lists every
RQE it reads, sorted, so each distinct SQL query appears once. Classic
entries are unchanged. The multigroup and all tables are regenerated;
minus e2e they still equal main's byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: wrap the e2e_block docstring

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: drop classic templates 7 and 9 (rates by job, top-k by rate)

`sum by (job) (rate(data_total[w]))` (template 7) and
`topk(32, sum by (label_0) (rate(data_total[w])))` (template 9) leave the
§6.3 evaluation (#777): ASAPQuery's engine would need
per-series counter deltas to serve them. Per-series `rate` (template 6)
stays, so `data_total` is still generated. The dashboard set drops the same
two (D3, D4). Templates keep their numbers.

Classic is now 8 templates, 42 RQEs (46 PromQL queries); all is 104 RQEs.
Multigroup is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: regenerate the classic and all tables without templates 7 and 9

export_autosketch_eval_table.py --synthetic, with each table's committed
--revision (classic 8daa0bc, all and multigroup b27934e). The multigroup
tables regenerate byte for byte. Classic: 42 RQEs (r = 8: 76; m = 8, 16:
336, 672); all: 104 (200; 832, 1664). The e2e blocks no longer hold
either query; rate(data_total[w]) remains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: render template 2's e2e PromQL as topk(32, sum_over_time(data[w]))

label_0 names one series (child_of job, unique values), so
`sum by (label_0)` only drops `job`: the ranking and values are the same.
ASAPQuery's planner supports the two-level shape today, not the
three-level one (#744, #772). The RQE (top-k keyed by
label_0, topk32_sum_by_label0_sum_over_time) is unchanged; only the
classic and all tables' e2e queries change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: rerun §6.3 without classic templates 7 and 9

The classic and all workloads, rerun exactly as ProjectASAP/sketch-bench#203 (`--runs 3
--no-chosen`, the same saturation inputs; plans from a `--runs 1` rerun
with chosen plans kept). Multigroup tables are unchanged and keep their
results; a rerun of multigroup-shared1 reproduces its costs and plans
exactly. AutoSketch's benchmark times are reused: every probe was already
measured, and each workload's benchmark time is unchanged.

Classic (42 RQEs): ASAP 2.38 vCPU (was 3.54), PerQuery 7.77 (10.5),
AutoSketch 2,739 (3,716); at AutoSketch's latency ASAP 4.22 (5.67).
All (104 RQEs): ASAP 2.60 (3.76), no roll-ups 2.93 (4.09), roll-ups save
11% (8%). Zero dropped RQEs, sanity violations or Hydra choices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: save ASAPQuery SQL in structured e2e queries

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Milind Srivastava <milindsrivastava1997@gmail.com>
ProjectASAP/sketch-bench#212)

* eval: e2e block for multi-grouping and all tables (ProjectASAP/sketch-bench#204)

export_autosketch_eval_table.py --synthetic now writes workloads[0].e2e to
every table. Schema templates have no PromQL (SQL is rendered downstream),
so each schema RQE gets one structured query entry: table, value,
capability, grouping, quantile (p99: 0.99), window_secs, interval_secs and
its rqes. Each schema metric is a `records` table (`http`, `flows`;
`data_i_http` under metric `data_i`) with SCHEMAS' labels, rate as
records_per_sec, and values (scaled_by, anomaly, burst as in SCHEMAS).
Classic entries are unchanged.

The multigroup and all tables in autosketch-vs-asap-hydra/inputs/tables are
regenerated; minus e2e they equal the committed ones byte for byte, and the
classic tables are unchanged. Their roll-up plans were already exported by
ProjectASAP/sketch-bench#203 (plans/*.plans.json.gz); a rerun of multigroup-shared1 and all-shared1
reproduces all 36 plans of each exactly, so the plans stand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: one e2e entry per distinct schema query (ProjectASAP/sketch-bench#204)

Schema query entries identical apart from `rqes` (templates 14, 15 and 17
share some groupings) are merged into one entry whose `rqes` lists every
RQE it reads, sorted, so each distinct SQL query appears once. Classic
entries are unchanged. The multigroup and all tables are regenerated;
minus e2e they still equal main's byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: wrap the e2e_block docstring

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: drop classic templates 7 and 9 (rates by job, top-k by rate)

`sum by (job) (rate(data_total[w]))` (template 7) and
`topk(32, sum by (label_0) (rate(data_total[w])))` (template 9) leave the
§6.3 evaluation (#777): ASAPQuery's engine would need
per-series counter deltas to serve them. Per-series `rate` (template 6)
stays, so `data_total` is still generated. The dashboard set drops the same
two (D3, D4). Templates keep their numbers.

Classic is now 8 templates, 42 RQEs (46 PromQL queries); all is 104 RQEs.
Multigroup is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: regenerate the classic and all tables without templates 7 and 9

export_autosketch_eval_table.py --synthetic, with each table's committed
--revision (classic 8daa0bc, all and multigroup b27934e). The multigroup
tables regenerate byte for byte. Classic: 42 RQEs (r = 8: 76; m = 8, 16:
336, 672); all: 104 (200; 832, 1664). The e2e blocks no longer hold
either query; rate(data_total[w]) remains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: render template 2's e2e PromQL as topk(32, sum_over_time(data[w]))

label_0 names one series (child_of job, unique values), so
`sum by (label_0)` only drops `job`: the ranking and values are the same.
ASAPQuery's planner supports the two-level shape today, not the
three-level one (#744, #772). The RQE (top-k keyed by
label_0, topk32_sum_by_label0_sum_over_time) is unchanged; only the
classic and all tables' e2e queries change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: rerun §6.3 without classic templates 7 and 9

The classic and all workloads, rerun exactly as ProjectASAP/sketch-bench#203 (`--runs 3
--no-chosen`, the same saturation inputs; plans from a `--runs 1` rerun
with chosen plans kept). Multigroup tables are unchanged and keep their
results; a rerun of multigroup-shared1 reproduces its costs and plans
exactly. AutoSketch's benchmark times are reused: every probe was already
measured, and each workload's benchmark time is unchanged.

Classic (42 RQEs): ASAP 2.38 vCPU (was 3.54), PerQuery 7.77 (10.5),
AutoSketch 2,739 (3,716); at AutoSketch's latency ASAP 4.22 (5.67).
All (104 RQEs): ASAP 2.60 (3.76), no roll-ups 2.93 (4.09), roll-ups save
11% (8%). Zero dropped RQEs, sanity violations or Hydra choices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

* eval: save ASAPQuery SQL in structured e2e queries

* eval: §6.3 workloads re-planned at 10 samples/s for the e2e run

export_autosketch_eval_table.py --synthetic gains --samples-per-sec (default
200): it sets SYNTHETIC_SAMPLES_PER_SEC and scales the schema metrics'
rates, and leaves out (logged) a schema RQE whose smallest covered group
falls below the curves' first N, with its e2e query, where the default
asserts. Without the flag the export is unchanged (the committed 200/s
tables, byte for byte but the final newline).

rqe-optimizer/results/e2e-rate10/: the classic, all and multigroup
tables (shared = 1) at 10 samples/s, the runner's results (p95, --runs 3
--no-chosen) and each method's plans (--runs 1, extracted by the new
scripts/extract_autosketch_plans.py, which reproduces the committed -hydra
plans). Two template-16 RQEs (p99 by (region, endpoint) and (region,
service, endpoint) over 5m, 403 items in the smallest group) are left
out; the runner would drop them as unservable and picks the same plans
either way. Every other RQE is served. Cheapest plan, CPU weights,
200/s -> 10/s: classic ASAP 2.38 -> 0.309 vCPU, PerQuery 7.77 -> 0.725,
AutoSketch 2,739 -> 137; all ASAP 2.60 -> 0.404; multigroup ASAP
0.221 -> 0.095.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Milind Srivastava <milindsrivastava1997@gmail.com>
…#206)

The runner's --target takes acc90, acc95 (p95, as in the committed runs) or
acc99: NN% accuracy in each family's own metric. The 12 synthetic workloads
rerun at 90% and 99%; scripts/plot_autosketch_vs_asap_targets.py plots cost
vs. target.

90% costs the same as 95%: the 95% plans already use the cheapest measured
configs. 99% raises ASAP's cost 9% on mixed and 2.8x on multi-grouping;
ASAP stays the cheapest at every target.


Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tASAP/sketch-bench#213)

* study: KLL k 25-800 (2x steps), DDSketch alpha 0.1, Hydra W 65536; --sketches/--configs to add configs to an existing study

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

* study: Hydra cost rows past W 16384 merge 4 shards (16 grids of 3.2 GB do not fit)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* eval: §6.3 rerun on the denser grid (KLL k 25-800, DDSketch alpha 0.1, Hydra W 65536)

Inputs: the new configs' curves appended to autosketch-vs-asap-inputs (raw
records in raw/dense_*), 7 cost rows (63 in all), and 999 Hydra runs at
W = 65536 (7,778 in all). Results: the 12 workloads at 95% (with plans) and
at 90% and 99%, on ProjectASAP/sketch-bench#210's tables.

CPU cost falls 0-1.4%, Fargate cost up to 13% (smaller KLL/DDSketch nearer
the target). 90% still equals 95%. Hydra is never chosen, even at W = 65536.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* review: one filter helper with name checks and tests; README provenance and claims

- filtered_configs()/check_filters(): --sketches or --configs skips the
  exact rows; a misspelled name exits; tests for both.
- The wide-merge comment says per fold, with operands 4x larger.
- rqe-optimizer README: the -hydra row's 2026-10-14 rerun, the one
  solver-tolerance violation, e2e-rate10 planned on the pre-ProjectASAP/sketch-bench#213 grid.
- -hydra README: rerun provenance moved to Reproduce; hydra-kll's figure
  marked unmerged, with the all-merge-counts value.
- raw README: measurement dates, clnode130.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* review round 2: --sketches help names the Hydra flags; planning-time comparability; rewrap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#215)

* eval: instant RQEs read one scrape, not 1 s

A PromQL instant query reads one sample per series, but the synthetic
spatial templates (sum by (job), quantile by (job)) looked back 1 s, so
their N grew with the data rate (2e5 per job at 200/s, 1e4 at 10/s). They
now look back one scrape interval (1 / samples_per_sec_per_series s), so
N = series per job = 1e3 at any rate.

autosketch_vs_asap scrapes a metric with samples_per_sec_per_series at that
rate (record metrics keep 1 s), so a sub-second lookback holds whole
scrapes, and writes sub-second windows and slides as fractional seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* eval: reject samples/s rates that don't divide 1000

An instant RQE's lookback is one scrape, 1000 / rate ms, and the runner keeps
lookbacks and scrape intervals in whole ms. A rate that doesn't divide 1000
(3/s gives 333 ms) truncates, and every synthetic RQE fails validation; above
1000/s the scrape is 0 ms, and 0 divides by zero. The exporter rejects such a
--samples-per-sec, and the runner such a samples_per_sec_per_series.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* eval: fail loudly when an RQE's lookback isn't whole scrapes

A metric missing from the e2e block falls back to a 1 s scrape, so a 5 ms
instant lookback had no candidate window and was dropped as unservable, with
only the dropped count logged. The runner now asserts that each RQE's
lookback and interval are multiples of its stream's scrape, and rounds them
to ms instead of truncating.

The e2e_block docstring and the byte-for-byte test no longer claim the runner
ignores the e2e block; the test checks every RQE metric's
samples_per_sec_per_series before dropping the block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…/sketch-bench#216)

* eval: one RQE per schema query, the strictest template's (ProjectASAP/sketch-bench#214)

Templates 14, 15 and 17 share some groupings, so one query string carried
several RQEs with different accuracy scopes. The MILP could serve them
from different deployments, which the engine can't do for one query
string. By default only the strictest template's RQE is kept (every group,
then 1%, then 5%), and an infeasible strictest copy is an error.
--keep-duplicate-queries keeps every copy; the §6.3 tables reproduce with
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* review: stale RQE counts, e2e-rate10 command and help text

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ry (ProjectASAP/sketch-bench#217)

Regenerates rqe-optimizer/results/e2e-rate10 (classic, all, multigroup at
shared = 1, 10 samples/s) on main: the denser grid's cost table and curves
(ProjectASAP/sketch-bench#213), instant RQEs reading one scrape (ProjectASAP/sketch-bench#215), and one RQE per schema query
(ProjectASAP/sketch-bench#216). Tables, --runs 3 results and --runs 1 plans; the README gives the
RQE counts, costs, classic deployments and ASAPQuery's
--data-ingestion-interval-ms 100.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* eval: RQE expression trees in e2e tables

Every e2e query gets `expr`, its RQE expression tree: a leaf per RQE it
reads (id, metric or table, capability, lookback_ms, grouping, phi/k/value),
and a `/` node for template 10. Rate queries, and schema queries merged
across several RQEs (--keep-duplicate-queries), get none.

Regenerate the e2e-rate10 tables: only `expr` is added, so the RQE ids and
plans are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1

* eval: leaf capabilities as a set, renaming only topk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ch#220)

Re-exports the 8 classic and all tables of autosketch-vs-asap-hydra
(--keep-duplicate-queries; only the instant RQEs change, now 5 ms) and
reruns them: --runs 3 results, --runs 1 plans, and the 90%/99% runs of
autosketch-vs-asap-targets. The multi-grouping tables are unchanged, so
their results stand. AutoSketch's probes are unchanged, so benchmark times
are reused. Figures, summaries and READMEs regenerated.

ASAP's cheapest CPU rises ~15% on mixed and mixed + multi-grouping (one more
exact-sum deployment per metric); PerQuery, AutoSketch and latency bounds
are unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…h-bench#229)

* eval: §6.3.1 case studies swept one knob at a time

Rebuilds the performance-monitoring and anomaly-detection case studies as
runner tables and sweeps latency SLA, refresh interval, sample rate, series,
cardinality and a drill-down query to find where the plan changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* eval: report only ASAP's plans in the case studies

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* eval: case-study summary reads each MV's grouping from the runner

Review of ProjectASAP/sketch-bench#229:
- summarize_case_studies.py guessed an MV's grouping as the finest of the
  queries it serves, so an MV grouped (region, service) that serves only
  `by (service)` (anomaly statuses=2, SLAs 0.885-1.41 ms) printed as
  `by (service)`. It now reads `deployment_grouping`, which the runner
  writes wherever the grouping differs from the RQE's. Regenerated
  summary.md: that label is fixed; every other change is label order (the
  runner's groupings are sorted).
- case_study_tables.anomaly() zipped queries with `intervals or [None] * 7`,
  so a short `intervals` silently dropped the drill-down. It now requires
  one interval per query. The regenerated tables are byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e type) (ProjectASAP/sketch-bench#221)

* eval: §6.3 95% results of the 8 changed tables rerun on CloudLab

The 8 classic and all tables' 95% runs, rerun on clnode109/155/167/178 (the
multi-grouping runs' nodes, each running its share serially), so every
planning time comes from one node type. Every cost, latency, frontier point
and SLA result matches ProjectASAP/sketch-bench#220's desktop run exactly; only planning times
change (CloudLab is 2-2.7x slower). Planning-time figure and table updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* eval: §6.3 paper figures and the §6.3.1 case studies

- scripts/plot_sec63_paper.py: the paper's cost bars (replacing Table 5),
  frontier (without the no-roll-up line) and roll-up ablation, from the
  committed results, into autosketch-vs-asap-hydra/paper/.
- scripts/export_case_studies.py and results/case-studies-6.3.1/: the
  performance-monitoring and anomaly-detection workloads, every method's
  plan (chosen kept), and a README of what each method deploys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plot_sec63_paper: cost bars drawn at print size (figure* 7 in, and a stacked 3.4 in column version), larger fonts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plot_sec63_paper: larger fonts in the figure* cost bars; one label per bar (ASAP's cost, each baseline's multiple), no overlaps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plot_sec63_paper: figure* cost bars trimmed to the drawing; larger axis titles; ASAP's cost on its bar, baseline multiples inside theirs, darker bar colors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plot_sec63_paper: cost bars on one set of axes at one column's width; LiteOlly, LiteOllyPerQuery, AutoSketch baseline; W-RangeHeavy/W-RollupHeavy/W-Mixed (the three-panel version kept as sec63_cost_bars_wide)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plot_sec63_paper: the single-axes cost bars under Fargate prices too (sec63_cost_bars_fargate)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…te) (ProjectASAP/sketch-bench#231)

The 38 sweep tables rerun without --weights cpu. CPU-only costs match the
committed results exactly; the Fargate results are added, and the README
notes the three plans that differ under Fargate prices.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…its history

The RQE deployment optimizer (MILP, candidates, cost model, saturation
accuracy), its AutoSketch-vs-ASAP evaluation runner, the scripts that
run, plot and summarize it, its committed results, and its design docs,
from ProjectASAP/sketch-bench@c750fa6. History is filtered to these
paths; sketch-bench issue references in commit messages are rewritten
to ProjectASAP/sketch-bench#N. The eval-table exporters stay in
sketch-bench, next to the measurements they read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r uses it by path

rqe-optimizer joins the workspace, and asap-planner-rs depends on it by
path instead of on sketch-bench's main by git (whose lock pinned an old
rev, ASAPQuery#836). Its one sketch-bench dependency left is aqpbm-core,
the atomic-cost table's row types, pinned to the rev the crate was
imported from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The imported scripts never ran ASAPQuery's hooks: black-format them,
end the files with a newline, and fix the two lint errors left (a lambda
assigned to a name; shellcheck on the intentional single quotes of an
`sh -c` script). check-added-large-files skips rqe-optimizer/results/,
whose result JSON reaches ~2 MB. No behavior change: summary.md
regenerates byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paths in the imported files were relative to sketch-bench's root:
- the moved scripts and docs are now under rqe-optimizer/scripts and
  rqe-optimizer/docs;
- what stayed in sketch-bench (the saturation study, the eval-table
  exporters, approxbench) is written sketch-bench/..., and commands
  run it from $SKETCH_BENCH, a sketch-bench checkout; relative links to
  it are GitHub links;
- bare sketch-bench issue numbers are now sketch-bench#N (numbers past
  sketch-bench's range were already ASAPQuery's and stay bare).

The README says what moved and what stayed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Dockerfiles copy rqe-optimizer, which the planner now depends on by
path; .dockerignore drops its results and data. Rust, Docker and
accuracy workflows trigger on rqe-optimizer changes. Planner comments
credit rqe-optimizer for DEPLOYABLE_FAMILIES, Raqe and is_eligible.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants