Repository navigation
refactor(rqe-optimizer): move rqe-optimizer from sketch-bench - #915
Open
milindsrivastava1997 wants to merge 58 commits into
Open
milindsrivastava1997 wants to merge 58 commits into
milindsrivastava1997 wants to merge 58 commits into
Conversation
…#129) * Document sliding RQE deployment model * Clarify RQE accuracy terminology * Implement sliding RQE deployment candidates * Add safe streaming RQE candidate search * Add HiGHS MILP RQE solver * Verify MILP against brute-force search * Clarify RQE optimizer status * Add bounded RQE mapping samples * Warn before eager RQE enumeration
…-bench#137) * rqe-optimizer: price plans per EC2 machine family Score retained memory per active deployment, (x + max S) / y instances, and add milp::minimize_cost, which minimizes the hourly price on one EC2 machine family in fractional instances (whichever of CPU or memory binds). Prices are a committed on-demand snapshot from scripts/fetch_ec2_pricing.py. Dominance pruning also compares retained memory so it cannot drop a candidate that is cheaper under the new objective. Part of #777. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: label small_problem MILP output for both objectives Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: scale the MILP so tiny costs and latencies solve exactly HiGHS's absolute gap (1e-6) and feasibility tolerance (1e-7) exceed real plan costs (~1e-6 $/hour) and query latencies (µs). Rows are now scaled by the plan's demand without sharing, and per-RQE memory and latency bounds become f64 exclusions of the choices over them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…aseline (ProjectASAP/sketch-bench#135) * rqe-optimizer: add AutoSketch-Adapted per-RQE baseline Port of AutoSketch Algorithm 4 (LHS seeds, feasibility-directed neighbor search, pruning, stopping) from ASAPQuery-backend autosketch_comparison.rs, generalized to each sketch variant's measured parameter axes. Each RQE is searched independently and gets a dedicated x=S sliding deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: follow AutoSketch's EXAMINE rule and seeding more closely - Evaluate and expand each configuration once (EXAMINE rule 1); an evaluated configuration is never re-queued. - Seed the cheapest configuration only when LHS hits no measured point, as the comment already said. - Compare resources as (memory, insert CPU) instead of adding bytes and seconds. - Document the cross-variant search as an extension over the paper. - Test the single-axis stopping rule, the cross-variant choice, a higher-is-better floor and LHS distinctness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ProjectASAP/sketch-bench#136) * Export the AutoSketch vs ASAP evaluation table from saturation curves Adds scripts/export_autosketch_eval_table.py and the generated rqe-optimizer/data/autosketch-eval/table.json (W2: alibaba_v2022, boom, google_2011). Every config lists AutoSketch's single-sketch error at max_N and ASAP's error after merging m = 1/4/16/64 shards, with the saturation flag ASAP requires. recommend_config.py's grid rounding moves into grid_point() so both scripts share it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Note that sketch-bench ProjectASAP/sketch-bench#131 has merged ProjectASAP/sketch-bench#131 merged into main as bd644fe with content identical to the merged-accuracy head the merge curves came from, so drop the not-merged caveat. The table and its recorded revision are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…eap and UnivMon top-k (ProjectASAP/sketch-bench#144) * rqe-optimizer: exact *_over_time accumulators, more top-k candidates Replace the optimizer's Freq capability with SumOrCount, MinOrMax and RateOrIncrease, served by exact multi-subpopulation accumulators that match ASAPQuery's Multiple{Sum,MinMax,Increase}Accumulator. Add CountSketch+heap and UnivMon as TopK candidates. Price grouped accumulators per group so the optimizer's card(group-by) multiply counts each group once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: move autosketch tests off the removed Freq capability Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: split MinOrMax into Min and Max exact-max only tracked the maximum, so min_over_time queries had no candidate. Min and Max are now separate capabilities served by exact-min and exact-max (ASAPQuery's MinMax sub-types), so a min query never lands on or shares a max accumulator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… objective (ProjectASAP/sketch-bench#145) * rqe-optimizer: one MILP entry point with an Objective enum Replace minimize_tco/minimize_cost with minimize(..., Objective), where Objective::{Tco, Cost(&MachineFamily)} owns its scaling units, extra variables, rows and goal. The shared assignment/activation/bound rows no longer branch on the family, and TCO no longer adds an unused instances variable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: drop the Tco MILP objective, rename Cost to AUCCost The MILP now only minimizes hourly price on a machine family. AUCCost names what it prices: mean CPU-sec/sec, the area under the load curve, so bursts are not priced. small_problem --milp now requires --machine-family. The former TCO tests run under AUCCost with the same expected mappings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: per-phase cost model, workload facts, weighted AUCCost - Costs split into ingest, merge, query and storage, each with CPU and memory. objectives.rs holds the one copy of the formulas; score() and the MILP share it. - Storage counts closed windows as (L - x)/y + 1 for the longest lookback (ProjectASAP/sketch-bench#143 S7). Ingest memory is open windows only. - MetricFacts { labels, scrape_interval_secs, cardinality } replaces LabelSetInfo (ProjectASAP/sketch-bench#143 S2). The arrival rate is derived as card(labels) / scrape interval. validate_facts reports every bad fact. - Rqe and Deployment carry metric and grouping_labels. - Candidate windows and slides are multiples of the scrape interval (ProjectASAP/sketch-bench#143 S6, half). - Objective::AUCCost { w_cpu, w_mem } minimizes w_cpu * CPU + w_mem * memory GiB, default (1, 0). MachineFamily, the EC2 pricing snapshot and fetch_ec2_pricing.py are removed, as is the peak-memory bound (ProjectASAP/sketch-bench#143 S3). - Objectives is per phase; the Pareto vector is (CPU, memory, latencies). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: state what each rqe-optimizer cost is counted per Per deployment or per RQE, per group, per window and per key, plus what is not shared between RQEs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: readability pass on the cost model Descriptive names in place of single letters, named ingest helpers in candidate pruning, and shorter comments without issue references. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: explain candidate slides with an example State that gcd(x, T) is the largest slide serving an RQE, why each subset of RQEs gets the gcd of theirs, and why one pass computes all of them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: frame candidate slides as strictly better than finer ones Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: list the cost model's variables one per line Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: merge memory holds both operands of a pairwise merge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: merge memory is one accumulator, none for direct queries Stored windows are live objects already counted in storage, so a merge adds only the accumulator per group. A query with L == x merges nothing. Candidate pruning now compares merge memory per served RQE, since it is no longer the same instance size for every RQE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: add intuition for the two-step candidate construction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: one dominance comparison in candidate pruning candidate_dominates and strictly_better walked the same costs twice. A single dominance() collects (replacement, original) cost pairs once and reports strictly better, equal or not dominating. Docs: define open and closed instances, fix the candidate pseudocode indentation, and reword card(G) and merge memory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: explain instance shape and size law in the cost model Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: add a glossary, link the cost export, define keys Defines family, configuration, instance and the other terms the cost model uses, links scripts/export_rqe_optimizer_costs.sh, explains what a key is and why quantile sketches are treated as fixed size, and drops the per-cost breakdown table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: call RQEs repeating atomic query expressions (RAQEs) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: flatten the candidate generation pseudocode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: add phase to the glossary Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: rename objectives to analytical_cost_model, Objectives to PlanCost The module holds the per-phase cost formulas and score(); its output is a plan's cost, not an objective, and Objective already names what the MILP minimizes. In milp.rs, name the storage variables and cost closures for what they hold: deployment_storage_cost, storage_cost_for_rqe_and_deployment and merge_and_query_cost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…RAQE latency SLA, deployable families (ProjectASAP/sketch-bench#148) * rqe-optimizer: rename Rqe to Raqe and accuracy_tolerance to accuracy_sla Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: time inputs in milliseconds Lookback, interval, window, slide and scrape interval are Millis. Rates (CPU-sec/sec, samples/sec) stay per second via secs(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: spatial_filter and latency_sla_ms on Raqe, drop MilpBounds Stream identity is now (capability, metric, spatial_filter, grouping_labels). spatial_filter is opaque matcher text used for identity only. The latency limit moves from MilpBounds onto each Raqe; the example's --latency-limit RAQE_ID=SECONDS becomes --latency-sla RAQE_ID=MS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: only ASAPQuery-deployable families become candidates DEPLOYABLE_FAMILIES lists the variants ASAPQuery's engine runs, each with its AggregationType. Candidate generation and the AutoSketch baseline both go through Capability::deployable_families(), so dd, univmon-* and countsketch-heap are measured but never planned. AutoSketch's chooses_the_cheaper_variant test is removed: no capability has two deployable variants until hydra-kll is measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: address ProjectASAP/sketch-bench#148 review - Latency is modeled in ms end to end (PlanCost::query_latency_ms), so the SLA compares without a unit conversion and the example prints ms. - validate_facts rejects non-finite or non-positive latency SLAs, and non-empty spatial filters until facts carry per-filter cardinality. - small_problem rejects --latency-sla without --milp. - Deployable-family test checks each entry is in some families(). - Restore AutoSketch's cross-variant test via search_variants(). - Fix the families() and export-script comments; drop redundant spatial_filter fields in tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: --allow-undeployable-families build_all_candidates(_unpruned) take allow_undeployable_families; when set, every measured family in Capability::families() becomes a candidate. The small_problem example exposes it as --allow-undeployable-families. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: address second ProjectASAP/sketch-bench#148 review - Capability::candidate_families(allow_undeployable_families) is the one family rule; candidate generation and AutoSketch's plan/search both take the flag, so the baseline sees the same sketches as the MILP. - validate_facts rejects zero lookback/interval and ones that aren't multiples of the scrape interval. - Candidate windows come from divisors(L / scrape) × scrape. - small_problem rejects --latency-sla with --candidates-only or --sample-mappings, and non-finite SLAs. - Capability::ALL for the deployable-family test; spatial_filter in the pruning comment and the TODO doc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ll_problem (ProjectASAP/sketch-bench#150) export_rqe_optimizer_costs.sh takes a required OUT_DIR and refuses to run if it already exists. small_problem takes a required --cost-dir DIR and reads DIR/rqe_atomic_costs.json. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ketch-bench#153) Flags were looked up by name, so a typo like --latency_p99_1h=0.5 was silently ignored and the run proceeded without the intended SLA. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ash-table overhead (ProjectASAP/sketch-bench#151) * rqe cost export: count top-k heaps, repeat top-k queries, add exact hash-table overhead - CMS+heap and CountSketch+heap memory now adds CMS_HEAP_TOP_K HHItems, as UnivMon counts its heaps, so the TopK rows compare like for like (ProjectASAP/sketch-bench#146). - TopkGT asks its one question 1024 times per pass, as CardinalityGT does, and the CMS/CS/UnivMon top-k queries answer once per probe, so a pass sits above the timer floor and work = answers keeps per-op CPU per call. - Exact accumulators' footprint adds an estimate of hashbrown's table: capacity x (key + cell + 1 control byte). Closes ProjectASAP/sketch-bench#146. Part of ProjectASAP/sketch-bench#147. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name getrusage as the query timing floor; note exact memory's load-factor dependence Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ojectASAP/sketch-bench#152) * rqe-optimizer: size DDSketch memory from the metric's value range Add optional `MetricFacts::value_range` (lo, hi), validated by `validate_facts` (0 < lo <= hi < inf). For DDSketch rows with a known range, memory per instance is (floor(ln(hi/lo) / ln((1+a)/(1-a))) + 1) buckets x 8 B, matching sketch-bench's `dd_footprint` for positive values (one store). Without a range, or for other families, the measured `mem_bytes_per_instance` is kept. One helper, `instance_memory_bytes`, replaces the direct reads in the ingest, merge and storage costs and in candidate dominance pruning, which now takes `facts`. Part of ProjectASAP/sketch-bench#147 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cap DDSketch buckets by values per instance; note the range is an upper bound Buckets = min(range buckets, λ·x/card(G)), as dd_footprint caps them by count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Size DDSketch's merge accumulator by the lookback, not one window The accumulator holds L/x windows' values, so its bucket cap is λ·L/card(G). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ProjectASAP/sketch-bench#155) * atomic-costs: record each row's measurement conditions (measured_at) Each AtomicCostEntry now carries the conditions it was measured under, read off the record's workload: items per instance, keys per instance, the value column's range and distribution, and the merge operand size. Optional on read so tables written before it still load. Part of ProjectASAP/sketch-bench#147. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * measured_at: no value range for a monotonic (counter) column Its values are a running sum of the draws, so the draws' bounds don't bound them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * measured_at: saturate the key-population product Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…/sketch-bench#154) * rqe-optimizer: accuracy at the query's merge count Export measures merge accuracy at 4, 16 and 64 shards as one --flat row per count; atomic-costs joins them by cell (--merge-accuracy) into a new AtomicCostEntry.merge_accuracy map, omitted when empty. Eligibility checks accuracy at L/x merged windows: the smallest measured count >= L/x, else the largest; one window or no measurements reads query_accuracy. Part of ProjectASAP/sketch-bench#147 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: check both bracketing merge counts; skip partial grids Eligibility now requires the metric to pass at both measured merge counts bracketing L/x (count 1 is query_accuracy), since error is not monotone in the count. Export measures 4..1024 shards and documents the fixed-total split. atomic-costs skips a row missing any merge count in the grid. Part of ProjectASAP/sketch-bench#147 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Merge-accuracy review fixes: docs, one run per merge pass, higher-is-better bracket test - docs: eligibility rule 3 checks both bracketing merge counts; replace the single-instance accuracy TODO. - export: merge-accuracy passes score once, so --runs 1 --warmup-runs 0; note 1024 shards on 200k-row specs come out as 1021. - same_cell: note it has no comparator. - test: both brackets must clear a higher-is-better metric. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…d DeltaSet key tracker (ProjectASAP/sketch-bench#160) * feat(rqe-optimizer): family properties, shared fixed-size sketches and DeltaSet key tracker S4b of ProjectASAP/sketch-bench#143, including the cost-model part of ProjectASAP/sketch-bench#142. - family_properties(variant): mergeable_across_windows, one_fixed_size_sketch_for_all_groups, needs_delta_set_key_tracker; unknown variants panic. - hydra-kll joins Quantile families (undeployable). Its deployments carry the exact-delta-set row as key_tracker and are candidates only when card(G) <= the row's measured subpopulations. - Cost phases sum over sketch and tracker; a shared fixed-size sketch is one instance, not card(G). - Dominance pruning compares the cost model's full costs. - AutoSketch skips shared fixed-size families (ProjectASAP/sketch-bench#159). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(rqe-optimizer): assert tracker presence, deployable-only tracker, cost vectors computed once in pruning - priced_parts asserts a deployment carries a key tracker iff its family needs one, so a hydra Deployment built elsewhere can't be underpriced. - The key tracker row is used only when it is deployable or undeployable families are allowed. - Dominance pruning computes each candidate's cost vector once instead of per pair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(rqe-optimizer): decide shared-sketch validity in is_eligible, one tracker row - is_eligible takes facts and holds the group-count gate and the tracker-presence check, so the MILP, enumeration and pruning agree; facts is threaded through eligible_deployments_for and enumerate. - The key tracker row is used whenever its family is a candidate (no separate deployability check); more than one exact-delta-set row panics. - The pruning test zeroes the tracker's insert and memory, so only latency keeps Hydra. - Export script comment and design doc updated (latency and query memory rows; ProjectASAP/sketch-bench#165, ProjectASAP/sketch-bench#166). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…y fields (ProjectASAP/sketch-bench#163) * rqe-optimizer: split SumOrCount into Sum and Count; deny unknown AtomicCostEntry fields Sum and count shared one capability, so the MILP could serve both items of ASAPQuery's avg rewrite from one exact-sum accumulator, undercounting ingest and memory ~2x. They are now separate capabilities on the same family. AtomicCostEntry and MeasuredAt now reject unknown fields, so a misspelled or newer field fails to load instead of dropping its data. Closes ProjectASAP/sketch-bench#161 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: split TopK into TopKByValue and TopKByCount Value-weighted and count-weighted top-k queries at the same cadence became identical Raqes, so the MILP could serve both from one CMS-heap that ASAPQuery deploys as two (different count_events). Separate capabilities on the same families keep them apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject unknown distribution fields; test the plan, not eligibility The distribution parameter structs nested in MeasuredAt now deny unknown fields too, so a typo like `skewnes` fails instead of being dropped. All checked-in distribution blocks still parse; the schema snapshot is regenerated. The sharing test now solves the MILP from a cost row: two identical queries must share one deployment (sharing halves ingest), and paired capabilities must not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… counts (ProjectASAP/sketch-bench#168) * feat(rqe-optimizer): plan output with active deployments and instance counts MilpSolution now lists only the active deployments, renumbered, each with its retained instance count, and per RAQE the serving deployment index and merged instance count. Drops the candidate-index mapping; adds the solved objective value. sketch-bench#143 S8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rqe-optimizer): dense plan renumbering, open windows counted once Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(rqe-optimizer): drop MilpSolution::objective_value; callers use objective.value Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ketch-bench#170) hydra-kll is no longer a candidate, even with --allow-undeployable-families. Its FamilyProperties and the DeltaSet key-tracker path stay, so re-adding it to Capability::families is a one-line change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…AP/sketch-bench#171) * rqe-optimizer: read sketch accuracy from saturation curves (ProjectASAP/sketch-bench#156) Sketch accuracy now comes from the saturation study's error-vs-N curves, read at the items a deployment's answer covers per group, at the metric's fitted data shape. Exact accumulators keep the cost table's value. - saturation.rs: SaturationCurves loads out_grid_1e7_cost/ and out_1e9/ (1e9 wins) and applies the ProjectASAP/sketch-bench#156 rules: worse neighbour between checkpoints, worst bracketing grid point, curve metric must match, saturated panes for lossy merges (KLL, top-k). - MetricFacts.data_shape: fitted (zipf θ, K, tail index) per grouping. - An Accuracy oracle is threaded through eligibility, pruning, enumeration and minimize. - small_problem takes --saturation-dir. - Cost export sweeps grid configs: KLL k 200/800, CMS-heap cols 1024/4096. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: merged answers read the curve at the merged count only Drop the saturated-pane requirement for KLL and top-k: every merged answer reads the curve at n(T), whatever the pane size. Their merge penalty stays ProjectASAP/sketch-bench#158. Resolves difference 2 on ProjectASAP/sketch-bench#171. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: check the cost table against the saturation curves A cost-table sketch row and a saturation grid point measure the same thing, error at (sketch, config, data shape, N). They were keyed differently, so the optimizer could not tell that a row is one point on a curve. - aqpbm-core: MeasuredShape (Zipf θ and K, or Pareto a), the key both share, and MeasuredAt::data_shape(). - saturation.rs: grid points are keyed by MeasuredShape and keep seed_se. SaturationCurves::check_cost_table reads each sketch row's curve at measured_at. It reports a row as mismatched when the values disagree beyond 3 seed SEs + 5%, the config is not in the grid, or the table lacks the curve's metric. Rows with no point at their shape or N (KLL: Zipf ranks vs Pareto) are reported as unchecked. - small_problem refuses to plan on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: describe the saturation lookup in words, not symbols Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: exactness in the family registry; one SLA check - FamilyProperties::exact marks the exact accumulators, so SaturationCurves::accuracy asks the registry instead of matching the "exact-" name prefix. - Raqe::meets_sla(Option<f64>) is the one accuracy-vs-SLA check, used by MILP eligibility and the AutoSketch search alike. Addresses review comments on ProjectASAP/sketch-bench#171. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zzylol <zeyingz@umd.edu>
…SAP/sketch-bench#173) * rqe-optimizer: check each family by its own accuracy metric (ProjectASAP/sketch-bench#167) Families declare their accuracy metric and direction in FamilyProperties; a RAQE gives one accuracy_sla checked against each candidate's metric, so DDSketch (mean_relative_value_error) and KLL (mean_rank_err) can both serve one quantile RAQE. A cost-table row missing its family's metric, or a saturation curve for another metric, panics. dd is deployable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: refuse curves for another metric at load Address review on ProjectASAP/sketch-bench#173: SaturationCurves::load rejects a candidate family's curve whose error_metric isn't its accuracy_key, so a stale study (e.g. DD's max error before ProjectASAP/sketch-bench#175) fails before planning, whatever the lookback, and check_cost_table compares against the family's metric. Drops the per-call assert and a stale fixture comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: reject an accuracy SLA no metric can meet validate_facts rejected only a non-finite accuracy_sla, so a negative SLA or a precision floor above 1 made every candidate silently ineligible and the RAQE read as unservable. Every metric is >= 0 and the only higher-is-better one (precision_at_k) is a fraction, so the SLA must lie in [0, 1] when the capability has such a family, else [0, inf). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…etch-bench#174) (ProjectASAP/sketch-bench#178) * rqe-optimizer: one study for sketch cost and accuracy (ProjectASAP/sketch-bench#174) The saturation study is now the only source of the optimizer's inputs. - study_saturation.py --phase optimizer-cost: one serial cost pass (5 runs, 3 warm-ups) and one accuracy pass per (sketch, config), plus the exact accumulators, at one shape: Zipf 1.1 over 1e4 keys (Pareto a = 2 for quantiles), 1e6 items -- the AutoSketch vs. ASAP evaluation's dataset. Reduced by atomic-costs to rqe_atomic_costs.json; fails on a skipped row. - AtomicCostEntry: required accuracy_metric (the query_accuracy key that is the row's accuracy) and measured_at; merge_accuracy removed. atomic-costs takes --accuracy-metric VARIANT=METRIC in place of --merge-accuracy. - rqe-optimizer: exact accumulators read the row's accuracy without merge brackets (accuracy_at removed); check_cost_table also refuses a row whose accuracy_metric isn't its family's or its curve's. small_problem reads the table from --saturation-dir (--cost-dir removed). - Delete scripts/export_rqe_optimizer_costs.sh. export_atomic_costs.sh (ASAPQuery's grid) passes --accuracy-metric. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: read a row's declared accuracy_metric, and require it be its family's row_accuracy read the family's key and ignored the row's accuracy_metric, so the two were only compared in check_cost_table. It now reads query_accuracy[accuracy_metric] and panics when that metric isn't the family's: one version, checked wherever a row's accuracy is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * study_saturation: run the cost shape on the curve grid; fix export_atomic_costs pass order - The accuracy grid also runs every config at the optimizer-cost shape (Zipf 1.1 over 1e4 keys, Pareto a = 2) unless --no-cost-shape, so each cost-table row is a point on a curve and check_cost_table checks every sketch row. The synthetic evaluation uses the same data and now reads an exact curve. - --phase optimizer-cost shows atomic-costs' error when it fails, and requires its summary to match the row count exactly. - export_atomic_costs.sh: cost pass first (flatten keeps the first pass's query timings, so accuracy-first skipped every row), accuracy pass once with no warm-ups (accuracy scores one run), and fail on skipped rows. - small_problem's missing-table message names --out DIR/optimizer_cost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * study_saturation: optimizer-cost skips frequency; test exact accumulators merge losslessly From review on ProjectASAP/sketch-bench#178: - --phase optimizer-cost measures only the families some capability plans (top-k, cardinality, quantile): no capability reads the frequency rows. - The optimizer takes exact accumulators to merge without loss, and the merge-accuracy sweep that covered them is gone, so a CLI test now checks that each one merged from 4 shards scores exactly like one (error 0). - read_curve's exit names --no-cost-shape for older accuracy runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ojectASAP/sketch-bench#158) (ProjectASAP/sketch-bench#179) * rqe-optimizer: merged KLL and top-k answers read the merge curves (ProjectASAP/sketch-bench#158) Since ProjectASAP/sketch-bench#171 every sketch read its plain curve at the lookback's item count, whatever the window, so a KLL or top-k answer merged from L/x windows got the unmerged sketch's accuracy. SaturationCurves now loads saturation_merge_curve.csv and reads it for those two families at m = L/x: the worse of the measured shard counts either side, the largest past it, and no accuracy without a merge curve. A later run's point keeps an earlier run's merge curves when it has none (out_1e9 measures none). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: merge curves extrapolate only from a saturation they cover From review on ProjectASAP/sketch-bench#179: - Past a curve's last checkpoint, its plateau only if n_sat is at or before it: a merge curve stops at 1e7, while its point's n_sat may come from the 1e9 run. - A later run keeps an earlier run's merge curves at shard counts it didn't measure, moved rather than cloned. - load refuses a candidate KLL or top-k point with no merge curve, rather than leaving every merged deployment silently without accuracy. - shards must be a whole count >= 1; univmon-topk merges lossily too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: no merged accuracy past the measured shard counts From the second review of ProjectASAP/sketch-bench#179: - Past the largest measured shard count, a merged KLL or top-k answer has no accuracy, rather than the largest count's: merge error grows with the count, and the study measures whatever counts a workload needs. - load refuses only a study with no merge curves at all (run without --merge-shards-list). A point without one, such as a 1e9-only point, just has no merged accuracy. - agrees_at takes the plateau under the same rule as curve_error_at (n_sat at or before the last checkpoint). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: merged accuracy only where the merge curves measured it From the third review of ProjectASAP/sketch-bench#179: - The merge count is Deployment::query_instance_count, as in the cost model: no answer for a zero window or a lookback that isn't whole windows. - Past a merge curve's last N the merged answer is unknown. The plain curve's saturation says nothing about the merged sketch. A curve's last checkpoint itself is read as measured. - load refuses each lossy candidate sketch that has no merge curve at all, not only a study with none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…easured nothing (ProjectASAP/sketch-bench#180) * rqe-optimizer: fall back to each sketch's guarantee where the study measured nothing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: test the guarantee fallback end to end Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: a top-k guarantee for every key at once; fallbacks no better than measured From review on ProjectASAP/sketch-bench#180: - CMS top-k: the per-key 1 − e^−rows wasn't a guarantee for the whole top k. Now Markov per row, independent rows and a union bound over the K − k other keys give each top key's failure probability; keys count from the most frequent while the summed failure stays within 5%. - A fallback is never better than the last measurement it extends, and it applies only at or above the first N of the curves it would read, merge curves included. - HLL uses the one-sided 95% normal quantile (1.645), as CONFIDENCE says. - The Zipf normalizer is exact for the first 1000 terms and an Euler–Maclaurin tail beyond, so large K is O(1). - accuracy_with_source reports the source of the worst value only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: fallback floors on both bracketing merge curves; two-sided rank and relative errors From the second review of ProjectASAP/sketch-bench#180: - A fallback's floor, and its first-N gate, read the measured shard counts either side of the merge count, the curves merged_error_at would have read, not only the smaller one or every merge curve. - KLL rank error and HLL relative error are absolute, so their 95% bounds are two-sided (KLL scaled by sqrt(ln 40 / ln 200), HLL z = 1.96). The top-k bound stays one-sided: CMS only overestimates. - Docs: the source is the returned value's; DDSketch's alpha isn't a hard bound on the measured metric, which the floor covers; the top-k bound assumes the heap ranks by final CMS estimates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…-table accuracy (ProjectASAP/sketch-bench#182) * top-k: heap = m·k per deployment; heap a runtime knob; keep summary rows on --resume sketch-bench: - The top-k heap's capacity is a runtime param (`heap=` on cms-heap and the new countsketch-heap vocabulary, default and minimum TOPK_K = 32). The answered and graded k stays TOPK_K: a larger heap answers its heaviest k. Memory counts heap x HHItem. UnivMon and the CLI's TopkGT read TOPK_K. rqe-optimizer: - A heap top-k deployment answering from m merged windows keeps m·k. Candidates carry one heap per distinct m among a window's RAQEs, and a RAQE is eligible only where the heap holds its m·k. - Costs come from the cost table's heap rows, linear in the heap between and past the measured sizes. One measured size serves any heap up to it. - With heap >= m·k a merged answer reads as one sketch (plain curve, then the guarantee fallback), so top-k needs no merge curves; KLL still does. Curves match configs ignoring `heap`. check_cost_table skips heaps other than k. study_saturation: - --phase optimizer-cost measures top-k at heaps 32/128/512/2048 (TOPK_HEAPS). - --resume over a narrower grid keeps the saturation.csv rows of the other points, as the curve files keep theirs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * top-k heap: AutoSketch searches heap k only; one accuracy per config; flag single-heap pricing From the first review of ProjectASAP/sketch-bench#182: - AutoSketch never merges, so its search space holds heap-k rows only; larger heaps aren't an axis it tunes. - Every heap row priced from a config carries the heap-k row's accuracy, measured hit or interpolated, since the curves are measured at k. - check_cost_table flags a heap top-k config priced at one heap only: no merged candidate could be built from it. - heap reads any JSON number (128 or 128.0). - --resume keeps other points' saturation.csv rows in the accuracy phase only, tolerating older column sets. - UnivMon's top-k cut reuses top_k. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * top-k heap: fail loudly on one-heap pricing and duplicate rows; heap_needed Round-2 review of ProjectASAP/sketch-bench#182: - at_heap panics, naming the optimizer-cost rerun, when a config is priced at one heap smaller than the heap a candidate needs; it no longer drops the candidate. - heap_rows_by_shape panics on two rows at one heap (no heap and heap=k); AutoSketch's grid runs the same check. - heap_needed(lookback, window) and Deployment::heap_needed give m · k for is_eligible, the per-window candidate heaps and SaturationCurves::accuracy. - top_k selects the heaviest k with select_nth_unstable and sorts only those; the query cost is benchmarked. - CountSketchParams no longer owns countsketch-heap-* variants (pinned in each_algorithm_owns_its_variants_and_no_neighbours). - load_needs_no_merge_curves_for_heap_topk no longer splits another test from its doc comment; config_params' doc is one comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study_saturation: cost-table sketch accuracy is the curves' seed mean; CmsParams leaves cms-heap - --phase optimizer-cost rewrites each sketch row's accuracy as the mean over seeds 1..--seeds of one accuracy run at COST_N: the run the curves make at that N. One seed-42 run against a three-seed mean failed check_cost_table on seed variance alone (top-k rows=3 cols=256, KLL k=200). Top-k heaps above k take the heap-k mean; exact rows keep 0. - CmsParams no longer claims cms-heap-* variants, as CountSketchParams no longer claims countsketch-heap-*. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e a grid with holes (ProjectASAP/sketch-bench#186) * rqe-optimizer: bracket the shape parameter within each K The cost table's shape adds one (θ = 1.1, K = 1e4) point to the curve grid (ProjectASAP/sketch-bench#178). bracketing_points bracketed θ over every point's θ and K over every point's K, so a θ = 1.1 shape at another K asked for a (1.1, K) point that was never measured and got no accuracy. small_problem's top_endpoints_1h (θ = 1.1, K = 1e3) was unservable. Now θ is bracketed within each K, and K among the K values that can bracket θ. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: one pass per K for bracketing; equal shapes all count From review on ProjectASAP/sketch-bench#186: - Gather each K's bracketing points in one pass; points with equal shapes (10000 vs 1e4 in two CSVs) all count instead of failing a length check. - Quantile when every point is Pareto, not by the first point. - Doc: the K bracket falls back to neighbouring Ks that have the parameter (conservative) rather than returning None. - The test's extra point carries no merge curves of the point it copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study_saturation: the cost shape as a full grid row and column; keep the rectangular bracket From the second review of ProjectASAP/sketch-bench#186: rather than bracketing a sparse grid, keep the grid a full cross. --cost-shape now adds θ = 1.1 and K = 1e4 to --thetas and --cardinalities (and a = 2 to --alphas), so θ = 1.1 is measured at every K and K = 1e4 at every θ. bracketing_points goes back to the rectangular bracket with its completeness check: a hole in the grid leaves the shape without accuracy instead of bridging to farther points (accuracy needn't be monotone in K). Quantile is decided from every point, not the first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: refuse a curve grid that isn't a full cross, naming the holes From the third review of ProjectASAP/sketch-bench#186: - SaturationCurves::load refuses curves whose Zipf grid isn't a full cross of its θ and K values, naming the missing (θ, K). A hole used to leave every shape around it silently without accuracy. - Quantile is decided by the first point again (the grids hold one distribution per config). - The hole test removes the point by shape, not position; a study test checks a grid that already holds the cost shape adds nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* top-k: k is a per-query variable, not the constant 32 (ProjectASAP/sketch-bench#184) - sketch-bench: heap top-k variants take topk_k= (default 32); the heap defaults to k and a heap below k is refused; the answer is the heaviest k and the comparator grades precision@k at it (answered_k). - study_saturation: top-k curves run at each k in --topk-ks (10, 32, 100); k != 32 is keyed in the config string (topk_k=k), so k = 32 keeps its old keys and older files read unchanged. - rqe-optimizer: Raqe.topk_k (default TOPK_K); heap needed m·k and eligibility heap >= (L/x)·k; accuracy reads the curve at k, else the next larger measured k, and past every measured k the theory bound at k; output cost is k entries; AutoSketch searches the smallest measured heap >= k. - Docs: rqe_sketch_deployment_v1.md, saturation_study.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * top-k k: floor heaps, carry topk_k, share heap pricing with AutoSketch (ProjectASAP/sketch-bench#185 review) From the first review of ProjectASAP/sketch-bench#185: - A top-k need below the smallest measured heap prices at that heap, never extrapolated below; candidates are keyed by (heap, k) and carry topk_k when k != 32, so the deployed sketch answers k. A RAQE needs a deployment answering at least its k. - univmon-topk answers TOPK_K only: ineligible for any other k. - AutoSketch resolves heaps with the MILP's heap_row (same floor and interpolation/extension), index for index, instead of picking a measured heap >= k. - Accuracy picks the next larger measured k whose curves bracket the shape; past every such k, the guarantee at k, floored by the largest bracketing k's measurement at this N. Params compare without cloning. - One topk_k decoding (params_topk_k / config_topk_k): heap_capacity's missing heap is the row's k, as in sketch-bench. - --topk-ks is deduplicated and must include 32. - Top-k query CPU scales by k/32 (measured at 32), a first approximation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: the full-cross check is per top-k k Rebased onto ProjectASAP/sketch-bench#186. Its load check groups grid points by their full params, topk_k included, so each k's grid must be a full cross of θ and K on its own and the k dimension doesn't trip it. The study crosses each per-k top-k config with the cost-shape row and column like any config, and the curve-at-k lookup keeps the rectangular bracket within the chosen k. A test pins both: two k's on different grids load; a hole within one k is named. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * Address second ProjectASAP/sketch-bench#185 review: answered-k query cost, fixed-k families - Query CPU and output bytes scale by the deployment's answered k. - answered_k/serves_topk_k in lib.rs: a fixed-k family (univmon-topk) answers TOPK_K only; used by candidates::is_eligible and AutoSketch. - heap_row's floor sets an explicit heap; one topk_k encoder (topk_k_param) shared with with_topk_k. - priced_parts tags sketch vs tracker instead of pointer identity. - AutoSketch resolves heaps once per (k, top-k) in plan, indexes heap rows once, and keeps heap/topk_k off its search axes. - study_saturation: old k=32 curve files need --topk-ks 32 for cost and crossover; say so in the docstring and exit message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…g, type and SLA sweeps) (ProjectASAP/sketch-bench#177) * rqe-optimizer: quantile_scenarios example sweeping lookback and interval Each scenario is 10 quantile_over_time RAQEs on distinct metrics sharing one (lookback, interval); the example solves each with the MILP and prints one row per scenario. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: workload_scenarios example with grouping, type and SLA sweeps Renames quantile_scenarios to workload_scenarios and adds sweeps over mixed groupings, mixed query types, a latency SLA and an accuracy SLA. Each step prints the deployments the MILP runs and the queries each one serves. --sweep NAME runs one sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: workload_scenarios follows per-family accuracy and the study's cost table Raqe no longer names an accuracy metric, so each query keeps only its SLA and the output prints the chosen family's metric. The cost table is read from the saturation directory, so --cost-dir is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…as an SLA) (ProjectASAP/sketch-bench#188) * rqe-optimizer: batch latency SLA with provisioned vCPUs minimize_batch_sla bounds when an aligned batch of every RAQE finishes: the plan provisions V vCPUs shared by ingest (rho_ing) and the batch's merge and query work W, a fluid queue, so V >= rho_ing + W / SLA, and V >= the mean CPU. CPU is priced at V. A pair whose per-group work exceeds the SLA is forbidden. With a loose SLA the plan is minimize's. analytical_cost_model gains provisioned_vcpus and batch_makespan_ms to price any plan at an SLA. The model is in docs/rqe_sketch_deployment_v1.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: cost models 1 and 2, batch latency and query placement (design for review) Replaces the provisioned-V batch SLA write-up with the design under review: precompute and query jobs per RAQE firing, each on at most one core; an event-driven placement at a (CPU, memory) resource point gives batch latency; cost model 1 bills AUC, cost model 2 the peak; the MILP uses a linear surrogate and is verified by the placement. Four open questions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: precompute is ingest; queries merge then estimate; fractional capacity Precompute is inserting samples into window instances (ingest), a steady background load; each firing's query job merges the query window's instances and estimates, at most one core. Capacity is fractional and shared as rates. States sketch-bench's measurement assumption (single threaded, CPU time = wall time). Two open questions remain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: parallel ingest split by sample, compaction at window close, memory in three parts Ingest runs on ceil(rho_D) workers split by sample, each with its own open windows; a compaction job merges the partials at window close, on the query's critical path. Memory is ingest, storage and query, each counted once. Cost model 1 bills CPU and memory by use (elastic CPU, latency is the longest chain); cost model 2 provisions (C, M), billed at the peak, with the placement giving latency. The resource-point question is resolved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: the MILP for cost models 1 and 2; the placement sizes (C, M) Cost model 1 is exact and linear (elastic CPU: latency is the longest chain). Cost model 2's batch row is a surrogate that chooses the mapping; the placement then sizes C (smallest meeting the SLA) and M (its peak) for every method alike, replacing the re-solve loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: cost models 1 and 2 with query placement placement: a plan's load over time (ingest on ceil(rho) workers split by sample, compaction at window close, one query job per firing, each at most one core), cost model 1 (by use, elastic CPU, latency = longest chain) and cost model 2 (a provisioned (C, M) billed at the peak, sized by an event-driven placement: the smallest C meeting the SLA, M its peak). milp::minimize_cost_model: cost model 1 is exact and linear; cost model 2 uses a linear surrogate (batch work over the capacity ingest leaves, every query's memory at once), then corrects it with the placement's error on the chosen mapping and solves again, keeping the cheapest placed mapping. The earlier provisioned-V minimize_batch_sla is removed; minimize is unchanged. Tests: placement (one core per job, fractional capacity, shared batches, compaction before queries, backlog, both cost models); cost model 1 matches brute force with and without an SLA; cost model 2 meets the SLA and matches the brute-force placed optimum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: restructure the section as a design doc Title 'MILP with latency SLA constraints: cost models, query placement and batch latency'; in order: definitions, the two cost models, CPU and memory parts and how each is computed, batch latency vs. the SLA, the job placement algorithm, the MILP formulation, and the solution method (cost model 1 exact; cost model 2 surrogate, placement sizing and correction passes). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: plain-English explanation for every formula and the placement algorithm Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs, placement: one job placement for both cost models Cost model 1 is the placement with unlimited capacity (elastic CPU): every job starts when ready on its own core, so the latency is the longest chain and the closed form the MILP uses is exact; cost model 2 runs it with the provisioned C. A test checks the closed form against the placement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs, placement: a query is issued, not released Queries are issued at their firing time; compaction is triggered at a window close. The doc section and placement.rs use these terms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: map cost models 1 and 2 to cloud billing; memory limit in the placement Cost model 2 matches containers billed by allocation and VMs (queueing for a core or for memory lowers the peaks); cost model 1 matches idealized fine-grained autoscaling, where queueing saves nothing and the cost-latency trade-off comes from the plan (precompute vs. query-time merging). The placement makes a job wait when its memory does not fit in M, and cost model 2's sizing searches (C, M) for the cheapest pair meeting L. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: memory does not limit the placement Billing charges the memory allocated; whether an application queues jobs to fit a smaller allocation is its own policy, not billing. So the placement has no memory limit and M is the peak it reaches at C*. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer, docs: review fixes for cost models 1 and 2 - minimize_cost_model takes `allowed` candidates per RAQE, so PerQuery can rule out sharing: identical deployments as separate candidate copies each pay their own ingest (test). - One chain function (analytical_cost_model::chain_ms) and one SLA slack (meets_sla, 1e-9 relative) for the MILP's chain filter and the placement, so a chain equal to L is never feasible in one and not the other. - cost_model_2's doubling search stops after 64 steps. - The placement keeps a pending list of issued, unfinished jobs instead of scanning every job at each event. - A compaction holds the closed window's k partial copies while it runs: in AUC(memory), the placement's memory over time, and both MILPs' memory terms (test). - Docs: instances vs. groups in the formulas; the compaction memory row; per-RAQE latency bounds are not used; the search bounds; PerQuery via `allowed`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer, docs: cost model 2 is evaluated with an SLA only Without an SLA, cost model 2's capacity is not pinned down (at the mean load, a placement that can't use every core keeps a growing backlog, so its latency depends on how long it is simulated). cost_model_2 takes the SLA, and minimize_cost_model asserts one for cost model 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer, docs: cost by use only, latency reported, not constrained The paper's message is lower cost and lower latency, so the MILP drops the peak-billed cost model and every latency constraint: - usage.rs (replacing placement.rs): a plan's resource use (ingest on ceil(rho) workers split by sample, compaction at window close, one query job per firing), its cost w_cpu * AUC(CPU) + w_mem * AUC(memory), and its latency, the longest chain (CPU is elastic). - milp::minimize_usage_cost (replacing minimize_cost_model): exact and linear, with `allowed` candidates per RAQE for PerQuery. Cost model 2, the placement, the SLA rows, the correction passes and meets_sla are removed; minimize is unchanged. - Docs: "Cost by use and batch latency": definitions, the cost model, CPU and memory parts, batch latency (reported), the MILP, and how it is solved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer, docs: an optional latency bound to trace the cost-latency frontier minimize_usage_cost takes latency_bound_ms: pairs whose chain exceeds it are ruled out, so the plan is the cheapest at least that fast (exact, since CPU is elastic). Sweeping it traces each method's Pareto frontier; it is not an SLA. Test: matches brute force under four bounds, and looser bounds never cost more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: two versions (no SLA; batch latency SLA) and the job placement algorithm Version 1 minimizes cost by use and reports latency, with an optional bound that traces the frontier; version 2 requires the batch latency, from the job placement, to be at most an SLA. The placement algorithm is back: with elastic CPU it has the closed form (longest chain) the code uses, which keeps both versions exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: answer ProjectASAP/sketch-bench#188's review on the CPU means Every mean vCPU is an average rate over the same period (whole hyperperiods), so the parts add; ingest is already a rate (lambda_D = series / scrape interval), and only compaction and queries are divided by their period. AUC(CPU) is written as three sums to make that plain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ontier and SLA versions) (ProjectASAP/sketch-bench#138) * rqe-optimizer: evaluate AutoSketch vs ASAP on the synthetic and trace workloads #777 (paper §6.3), consolidating ProjectASAP/sketch-bench#138, ProjectASAP/sketch-bench#139 and ProjectASAP/sketch-bench#141. - rqe-optimizer/examples/autosketch_vs_asap.rs (traces and synthetic): ASAP (milp::minimize), PerQuery-CostAware and AutoSketch-Adapted. Costs and accuracy come from --saturation-dir, as the planner reads them: SaturationCurves::accuracy for ASAP (merge curves for KLL and top-k, ProjectASAP/sketch-bench#179), autosketch_accuracy for AutoSketch, and the one-study cost table (ProjectASAP/sketch-bench#178). Each run is scored by analytical_cost_model::score at #777's two weight settings and the SLA grid. - scripts/run_autosketch_vs_asap.sh and plot_autosketch_vs_asap.py: the trace workloads alibaba_v2022 and google_2011. boom is left out for now. - scripts/export_autosketch_eval_table.py --synthetic: one workload-only table per grid point, plus plan.tsv. The data constants come from study_saturation's COST_*. The traces path is unchanged from main. - scripts/run_autosketch_vs_asap_synthetic.sh and plot_autosketch_vs_asap_synthetic.py: run the synthetic grid, then plot. - scripts/profiling_time.py and the results README: ASAP's one-time profiling time, and how to reproduce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * rqe-optimizer: trace key queries keep one accumulator per key; report the accuracy source From review on ProjectASAP/sketch-bench#138: - A traces key query (sum by key) keeps one exact accumulator per key: its groups are the window's keys, not the table's one sketch. A stream's card(g) is the largest over its RQEs. - AutoSketch's benchmark lower bound uses one query of one instance, not the stream's q_r of them. - --weights that names none of the weight settings is an error. - The chosen report lists each RQE's accuracy source (ProjectASAP/sketch-bench#180's accuracy_with_source). - Drop a duplicated BoomTest; capability_of's comment matches the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: one accuracy level, p95; plots join one RQE set and charge AutoSketch's benchmark - One accuracy level for every workload, synthetic and traces: 95% in each family's metric (rank, relative value and relative error at most 0.05; top-k precision at least 0.95). The strictness dimension is gone. - fig_objective_vs_latency joins only points over one RQE set (solid for the full set, dashed for the subset tight SLAs keep). - fig_planning_time adds AutoSketch's charged benchmark time (lower bound and 60 s per probe), as #777 section 7 counts it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: summaries show Σ window/slide; synthetic summary to its own file AutoSketch-Adapted's ingest cost is window / slide overlapping sketches per deployment (window = the query's lookback), so the column explains its gap to ASAP. The synthetic and trace scripts both wrote summary.md into one results directory; the synthetic one is now summary_synthetic.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: RQEs carry their top-k k (ProjectASAP/sketch-bench#185) Raqe now has topk_k. The tables name it only in the query id (topk32_...), so the runner reads it there and stops on a top-k query that names none. On the p95 synthetic and trace tables every top-k query is k = 32, and the plans are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: p95 results for the synthetic grid and the traces Runs on clnode155 over the asap_sketchlib 0.3.0 study (ProjectASAP/sketch-bench#174, ProjectASAP/sketch-bench#178–ProjectASAP/sketch-bench#186): four synthetic workloads and alibaba_v2022 / google_2011, raw JSON, summaries and figures. The README gives the no-SLA headline, including Σ window/slide behind AutoSketch-Adapted's gap. profiling-time.json is refilled from the study's raw records (0.77 h; the K = 1e4 column refill of ProjectASAP/sketch-bench#186 is its own run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: figures and tables give each method's absolute cost Cost figures label every bar and point with its value; the summaries and the results README list each method's objective instead of its ratio to ASAP. The by-dimension figure's legend moves above the panels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: metrics dimension, one dashboard per metric (21 · m RQEs) Shared replicas on one metric stop adding RQEs past r = 8 (52 at r = 8, 57 at r = 64), so the grid keeps shared r ∈ {1, 8} and adds metrics m ∈ {1, 8, 16}: copy i of the template set reads its own metric data_i, so nothing is shared across copies and RQEs grow as 21 · m (168, 336). The planning-time figure plots this dimension. The m = 8 and 16 runs are added; the r = 64 result is removed. Existing tables are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: AutoSketch benchmarks each config once per metric Synthetic tables name the metric each RQE reads (data, or data_i in the metrics dimension), and AutoSketch-Adapted's benchmark time counts a probed config once per metric, on that metric's data: 540, 4320 and 8640 s (60 s per probe) for m = 1, 8, 16. Trace tables name none and read one metric per dataset, as before. All five synthetic workloads are rerun with the current runner on four nodes; objectives are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: cost vs. total estimated latency for every workload One column per workload (five synthetic, two traces), one row per weight setting, one line per method across the SLA grid, each point labeled with its cost. Total latency is the sum of the RQEs' estimated latencies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: finer SLA grid, {0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10} ms and none Checks whether AutoSketch-Adapted's memory objective makes it miss an SLA: it never does. Its plans never merge, so each RQE's latency is within 1.4% of the lowest any deployment reaches (equal for quantile, exact and trace RQEs), and RQEs no method can meet are excluded for all methods. All workloads rerun on four nodes; figures and summaries regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: AutoSketch probe list and measured benchmark time; traces with 6h/24h windows - The runner lists AutoSketch-Adapted's distinct probed (metric, config) pairs with the data shape each is benchmarked on, and drops the benchmark-time lower bound. scripts/autosketch_benchmark_time.py runs each probe's approxbench accuracy benchmark (1e8 items, exact baseline and scoring) and times it, as AutoSketch benchmarks probed configs. - Trace streams with more keys than samples per second are left out and listed (rounding their rate up to their key count inflated every method's cost). - The trace table is regenerated from ASAPQuery #812's skew summary (6h and 24h windows, a day of Alibaba). - Plots: one line per method over the full RQE set; AutoSketch's planning time with its measured benchmark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: cost by use (sketch-bench ProjectASAP/sketch-bench#188); Pareto frontier sweep; mixed set by default - The runner prices every plan by use (usage::usage_cost): w_cpu * AUC(CPU) + w_mem * AUC(memory), CPU elastic, latency the longest chain, reported. - ASAP is milp::minimize_usage_cost over every RQE; PerQuery the same over each RQE's own candidates (separate copies via `allowed`, no sharing); AutoSketch's plan is priced the same way. - No SLA grid. Each method's cost-latency frontier: ASAP and PerQuery are solved unbounded and for 12 bounds log-spaced from the tightest feasible one to the unbounded latency, plus AutoSketch's latency. - The synthetic grid is the mixed template set: shared r in {1, 8} and metrics m in {1, 8, 16} (50, 92, 400, 800 RQEs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: version 1 (no SLA) results on the mixed set, with AutoSketch's measured benchmark - Four workloads of the mixed template set (50, 92 with r = 8, 400 with m = 8, 800 with m = 16 RQEs), cost by use (ProjectASAP/sketch-bench#188), each method's cheapest plan and ASAP's and PerQuery's cost-latency frontiers (12 bounds plus AutoSketch's latency), CPU-only and Fargate weights, --runs 3, one node per workload. No RQE dropped, no sanity violation. - AutoSketch's benchmark time is measured: approxbench's accuracy run (1e8 items, exact baseline, scoring) once per distinct probed config and shape, serially on idle clnode155 (21-37 s each), charged once per probed (metric, config): 212, 1697 and 3394 s for m = 1, 8, 16. - plot_autosketch_vs_asap_synthetic.py: fig_frontier.png (cost vs. query latency, every point labeled), fig_planning_time.png (AutoSketch as search + measured benchmark; 60 s per probe only as a labeled reference), summary_synthetic.md. The SLA-era figures, the dashboard results and plot_autosketch_vs_asap_workloads.py are removed. - README: the cost model, version 1, how to reproduce, and the headline numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: version 2, a batch latency SLA, on the synthetic mixed set The runner adds version 2 of the evaluation beside version 1 (no latency constraint and the frontiers, unchanged in `results`): at each SLA of a grid (default 100, 300, 1000, 3000, 10000 ms; --slas-ms overrides), ASAP and PerQuery plan the cheapest plan whose batch latency (the longest chain) is at most the SLA, via minimize_usage_cost's latency bound, which is exact with elastic CPU. An SLA below a method's tightest feasible bound gives an infeasible record. AutoSketch's one plan is recorded at each SLA with meets_sla. Records go to `sla_results`, with sanity checks that every plan meets its SLA and that ASAP costs no more than PerQuery. Results on clnode109 (--runs 3) for the mixed set, r = 8 and m = 8, 16: no drops, no sanity violations; the tightest feasible SLA is 92 ms, and AutoSketch (latency 1011 ms) meets only 3000 and 10000 ms. AutoSketch's planning time is search plus its measured benchmark (scripts/autosketch_benchmark_time.py: approxbench accuracy runs at 1e8 items, 20-33 s each, serial and alone on the node): 197 s at m = 1, 3159 s at m = 16. scripts/plot_autosketch_vs_asap_sla.py draws fig_cost_vs_sla.png and summary_sla.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: remove the old trace results and figures They came from the earlier model (per-RQE SLA over a µs grid, mean-CPU cost); the traces are to be rerun on cost by use with both latency versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: trim the PR to what it needs - Result JSONs without per-RQE choices (`chosen`): the figures and summaries don't read them, and they were nearly all of the 2.9M lines; rerun the runner for them. - Machine files folded into the READMEs (all Intel Xeon E5-2683 v3, 56 cores); the synthetic run script no longer writes them. - The SLA-era trace scripts (plot_autosketch_vs_asap.py, run_autosketch_vs_asap.sh) are removed; the traces are to be rerun on cost by use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: keep only what the three figures need The PR's results are the frontier (version 1), cost vs. SLA (version 2) and planning time on the synthetic mixed set. The trace table goes back to the base version (the traces are to be rerun on cost by use, with their new table), and the ASAP profiling-time script and data, which no figure uses, are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: commit the inputs of the three figures rqe-optimizer/results/autosketch-vs-asap-inputs/ holds what the figures were computed from: the sketch-bench study on asap_sketchlib 0.3.0 (the cost table, sketches and exact aggregations, and the accuracy curves the runner reads as --saturation-dir) and the synthetic mixed-set tables with plan.tsv. The READMEs point the reproduce steps at them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: the raw approxbench records behind the figures' study rqe-optimizer/results/autosketch-vs-asap-inputs/raw/ holds the gzipped approxbench records (920 KB) that ../saturation/ was reduced from, with a README mapping each file to its machine, arguments, record count, run time and output. Rerunning them is 0.77 h of machine time over 5 machines in parallel (about 20-30 minutes of wall time). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…QEs (ProjectASAP/sketch-bench#190) * rqe-optimizer: let mergeable finer-group deployments serve coarser RAQEs A deployment grouped by G_d now serves a RAQE grouped by any subset G_r when its family merges across groups (exact sum/min/max, HLL, UnivMon cardinality, KLL, DD). The read is priced at G_r: card(G_d)·L/x − card(G_r) merges, with card(G_r) accumulators, probes and output. Accuracy reads G_r's shape and items, and KLL's merge curve at the average fan-out times L/x. Fine candidates take windows from coarser RAQEs. validate_facts rejects a label set with more groups than a superset. Closes ProjectASAP/sketch-bench#187 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: split RQE optimizer design by concern * docs: clarify RQE optimizer cost model status * docs: one cost table for every phase, with roll-ups Merge the cost-by-use CPU and memory tables into one table per phase (ingest, compaction, query job, storage) in terms of I_d and I_r, so roll-ups (G_r ⊂ G_d) are priced as the code prices them: ℓ = card(G_r)·c_qry + (I_d·n − I_r)·c_mrg, accumulators at G_r. The snapshot-AUC section now lists only how it differs (k = 1, query memory always held, per-RAQE serial latency) instead of repeating the table; the shape/size-law side-by-side gives each family's I_d, I_r and tracker; the proposed Hydra cost model is §3's table with I_d = I_r = 1, adding compaction and using the doc's symbols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: KLL's memory is sketch-bench's nominal footprint, not measured at 1e6 The cost table's KLL m is kll_footprint = 4·k·sizeof(T), independent of the item count (1600 B at k = 50 for 1e3, 1e6 and 1e8 items), so it is not "the size measured at 1e6 values" and does not grow with n. Say so, and how it compares with asap_sketchlib's preallocated capacity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: prune on every cost cost-by-use bills; fix stale comments Dominance pruning compared ingest, query latency, merge memory and storage. Cost by use also bills compaction (per window in every chain, per slide in CPU and byte-seconds) and query output memory, which differs between top-k configs answering different k. A direct RAQE never merges at query time, so a candidate with cheaper ingest but costlier merges could prune one that is cheaper by use. Compare those too; the new test fails without it. Also: workload_scenarios no longer says groupings never share; the exact-increase exclusion states its real (conservative) reason; merged_instance_count notes roll-ups; a circular doc comment; stale §N citations now name the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: consolidate the optimizer docs; add the Hydra admission design - candidates: eligibility allows subset groupings for families that merge across groups; generation shows the roll-up RAQE sets; accuracy is read at G_r with KLL's fan-out merge count, and past the measured merge counts the guarantee applies, floored by the last measurement; pruning lists every compared cost; the MILP lives only in the cost doc; one roll-up section. - cost model: top-k output and query CPU scale with k; k_D sums parts; mergeable_across_groups listed; KLL footprint points at ProjectASAP/sketch-bench#191/#192; Hydra moved out. - v1 and TODO: phases include compaction and the query job; TODOs match cost by use. - rqe_optimizer_hydra.md (new): Hydra in sketch-bench, accuracy bounds per combination (CMS, CountSketch, HLL, UnivMon, KLL) checked against the paper's Theorem 2 and the library (correlated inner seeds, fixed column hash, few rows), cost model, candidates and MILP edges, sketch-bench requirements and open decisions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: univmon-cardinality merges lossily; increase's real reason UnivMon's counters add exactly on merge, but each level's heavy-hitter heap is rebuilt from the union of the two heaps, so a key heavy only in the union is lost: a merged (or rolled-up) answer now reads merge curves, as KLL's does. exact-increase's accumulator merge joins two pieces of one counter in time, so merging two groups' counters is wrong; say so instead of "unconfirmed". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: check group counts only between label sets a plan uses validate_facts compared every pair of label sets in a metric's cardinality map, so an inconsistent entry no RAQE groups by rejected the whole workload. The roll-up merge count only reads RAQE groupings and the full label set, so check X ⊂ Y among those. Docs: why rate/increase, top-k and Hydra don't roll up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs(hydra): hydra-kll is measured in the same Hydra rerun Every Hydra measurement predates the admission requirements, so they are all rerun; hydra-kll goes in that run with the other four variants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: move the Hydra admission design out of this PR Keep ProjectASAP/sketch-bench#190 to roll-ups. docs/rqe_optimizer_hydra.md and its links move to a separate PR stacked on this one; the docs here only say Hydra is future work and doesn't roll up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: tests for roll-up DD memory, facts and pruning; doc fixes Tests that fail without their rule: - a DDSketch roll-up sizes its accumulator at the coarse grouping; - validate_facts rejects a RAQE grouping with more groups than series; - pruning drops a heap whose larger answered k only adds output memory. Docs: the KLL footprint paragraph now that ProjectASAP/sketch-bench#192 is in; univmon-cardinality reads merge curves like KLL (and has no guarantee past them); the TODO's done list covers roll-ups and the pruning costs; "temporal pre-merging"; enumerate.rs names the MILP variables z and u as the docs do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: name every lossy non-heap sketch that needs merge curves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zz_y <zeyingz@umd.edu>
…rojectASAP/sketch-bench#181) * docs: end-to-end planning flow from traces to an rqe-optimizer plan Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: kind/weight/range and data_shape, fanning out, and ProjectASAP/sketch-bench#180/#182/#183 in the planning flow From review on ProjectASAP/sketch-bench#181: - skew_summary.csv's kind, weight and range columns, and how ranges map onto data_shape (keyed by grouping: the worst case over its RAQEs' ranges). - A 'Fanning out over machines' section and scripts/merge_study_parts.py, which concatenates per-machine run directories, refusing a point in two parts. - Brought up to date: top-k needs no merge curves (heap m·k, ProjectASAP/sketch-bench#182), the guarantee fallback is merged (ProjectASAP/sketch-bench#180), main is through ProjectASAP/sketch-bench#183. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…h needs) (ProjectASAP/sketch-bench#193) * docs: Hydra admission design (accuracy, cost, candidates, sketch-bench needs) docs/rqe_optimizer_hydra.md: Hydra as sketch-bench runs it; accuracy bounds per combination (CMS, CountSketch, HLL, UnivMon, KLL), checked against the paper's Theorem 2 and the library; cost model as the cost doc's §3 table with I_d = I_r = 1; candidates and MILP edges (MILP unchanged, still linear); what sketch-bench must add, hydra-kll included in the same rerun; open decisions. Links from the other optimizer docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs(hydra): consolidate the Hydra notes already on main - cost model: the Shared + Fixed bullet no longer presents the subpopulations gate as Hydra's rule or fan-out as a deviation; it is a necessary check only, and admission is the measured predicate of rqe_optimizer_hydra.md §2.4. The HydraKLL column of the side-by-side table moves to Hydra §3; the key tracker's grouping is open (§6.1). - candidates.rs: measured_at_group_count's comment says the same. - configs/datagen hydra yamls: since asap_sketchlib 0.3.0 the sketch keys subsets by column; only the polars baseline still aliases (ProjectASAP/sketch-bench#74). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ch-bench#138) (ProjectASAP/sketch-bench#194) The eval's inputs README lists the saturation study's accuracy grid (out_grid_1e7_cost) and targeted 1e9 points (out_1e9), but .gitignore's *.csv kept their six CSVs out of ProjectASAP/sketch-bench#138, so the eval couldn't be rerun from the repo. Add a narrow exception for that directory and commit them (about 1 MB): the curves the committed results were planned with. Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ASAP/sketch-bench#202) * polars: tag Hydra baseline subset keys by column (ProjectASAP/sketch-bench#74) The polars exact baselines keyed a label subset by its `;`-joined values, asap_sketchlib 0.2.2's subkey. 0.3.0's Hydra tags each value with its column (`label0:a;label1:b`), so on label columns that share a value domain the baseline merged a key1 group with the same key2 group and scored nonzero against itself. subset_key now builds the 0.3.0 key over hydra_shared's `label{i}` schema (escaping `\`, `:` and `;`), empty labels keep their column, and every lookup goes through it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * polars: write subset key tags without a temporary String; fix test comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: interleaved merge split; merge exactness and bound tests `--merge-split contiguous|interleaved` (default contiguous, unchanged) picks how merge runs split the stream into shards. Interleaved reorders the stream so the existing contiguous `partition` deals it round-robin by record, at the contiguous split's shard sizes, so every row's folds take it unchanged. Tests: hydra-cms, hydra-cs and hydra-hll grids folded from 4 and 7 shards, under both splits, answer every probed subpopulation exactly as the single stream; and every scored group on a fixed 20k-record stream lands inside its hydra doc §2.2 bound (CMS also never under). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: child_of columns for hierarchical labels A `child_of: <index>` column with `fan_out: f` nests under an earlier string column: its value is the parent row's value, a `.`, and a child index in [0, f) drawn from the column's own distribution. Bad indices, a fan_out that disagrees with the domain, and fields that would not mean what they say on a child are refused before drawing. Adds configs/datagen/hydra_hier.yaml (region 4 -> service 25 -> endpoint 25, status 4, user_id value) for the hydra rows, docs, and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: score any target grouping; per-group error output Every hydra-* row (lib and polars, all five variants and hydra-univmon's four statistics) takes `--group-columns J,K,...` (default `0`, today's behavior) in place of the hardcoded SCORED_LABEL_COLUMN. A subpopulation probe is now positioned (`Vec<Option<String>>`, Some at the grouped columns), so `hydra_shared::query` asks any non-empty subset of the key columns and the polars baselines look up `subset_key` with that subset's mask (`group_key`, generalizing `prefix_key`). Each grouped comparator reports its per-group error in its own metric (relative error; mean rank error for hydra-kll) and summarises it as err_mean, err_p50, err_p90, err_max, groups_scored, plus schema_width, records and fanned_mass. `--per-group-out PATH` writes the rows as `group_key,n_q,error`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: whole uniform bounds for child_of; 2- and 3-label hierarchy cuts A uniform draw is continuous, so `uniform{0.0, 2.9}` with fan_out 2 drew child indices 0, 1 and 2; child_of now requires whole uniform bounds. hydra_hier_d2.yaml and hydra_hier_d3.yaml cut hydra_hier.yaml to its first 2 and 3 labels for the Hydra study's schema-width sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: the hydra_hier usage example passes --dtype Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: list every group in --per-group-out; escape keys; refuse no accuracy - Group keys escape `\`, `:` and `;` as polars_shared::subset_key does. - Groups that cannot be scored (zero true statistic, e.g. zero entropy) are written with an empty `error`, so n_q sums to the stream's records. - --per-group-out with no accuracy measurement is refused instead of writing an empty file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: KLL CDF and UnivMon sum rows; KLL cell merge buffer - hydra-kll-cdf (lib, polars): asks the KLL grid HydraQuery::Cdf(x) at each group's own values at phi = 0.01..0.99, scored by absolute CDF error (SubpopCdfErrorGT, comparator subpop-cdf-error). The polars baseline is the exact share at or below x over the same sorted groups. - hydra-univmon-sum (lib, polars): each record inserted with count = value, so the UnivMon grid's L1 is the group's sum, scored by relative error against exact per-group sums (SubpopSumGT, comparator subpop-sum). Integer value dtypes only; f64/str and values outside [0, i32::MAX] are refused by name. - memory_hydra_kll counts each cell as kll_lib_bytes does, merge buffer of cell_k items included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra-kll: the footprint's merge buffer is the steady state; tie tests The grid clones its cells from one template, and a cloned Vec keeps no capacity, so a cell's merge buffer is empty at build and grows to about cell_k items at its first compaction. Say the footprint counts it at that bound. The CDF test now asks a group of ties (at or below) and a mixed-tie group on a grid wide enough that it shares no cell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: record merge_split; review fixes - `merge_split` (contiguous|interleaved) is a record field next to `merge_shards` (BenchSection and MergeMetrics; schema regenerated). Merge records stamp it; a record without it split contiguously, so existing results parse, and flatten refuses two different splits in one row as it does two shard counts. - `MergeSplit` moves to aqpbm-core (it is a record type) and is parsed by clap directly instead of matching strings. - The interleaved copy of the stream is built only when a merge runs. - README states the round-robin exactly (contiguous shard sizes, so a short last shard shifts the pattern); MergeMetrics docs name the split. - hydra-cs bound test: note its margin (passes from beta ~4.8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study: --phase hydra on the eval's datasets; Hydra cost rows configs/datagen/hydra_http.yaml, hydra_http_latency.yaml and hydra_flows.yaml follow the §6.3 eval's http and flows schemas (labels in schema order, cardinalities, child_of, skews; user_id Zipf 0.8, latency Pareto a = 2, src_ip uniform). The flows DDoS burst is not generated: datagen draws each column independently. --phase hydra runs HYDRA_SWEEPS: the eval's variants on those datasets (every grouping, W 1024/4096/16384, N 1e5..1e7, 1/4/16 interleaved shards, 3 seeds) and every statistic on hydra_hier{_d2,_d3,}.yaml, one run per grouping with --per-group-out, into hydra_saturation.csv with the coverage columns err_max_cov_0.01/0.05 (share >= tau plus the largest group). --jobs under a memory budget, --resume, and --hydra-shard i/n to split it across nodes. --phase optimizer-cost adds Hydra rows (eval variants x W x dataset); MeasuredAt gains dataset and schema_width, the reducer reading the width off the row's scores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra-univmon-sum: the ground truth refuses negative values too The lib row inserts each value as a UnivMon count and refuses negatives; the truth (and so the polars baseline) accepted them, so a spec with negatives ran on polars and failed on lib. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study: Hydra cost rows at 3 runs; their test; datagen path from the script Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: Hydra cost rows' run count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study_saturation: atomic hydra table rewrite, stricter resume, schema checks --phase hydra --resume now writes the kept rows to hydra_saturation.csv.tmp and os.replace()s it over the table, and keeps a row only when it ends in a line end and every column parses (integers, numbers or "" for the error columns, non-empty text). The CSV format is unchanged, so tables written by 003eb31 resume as before. The eval-spec test also checks each label's cardinality, skew and child_of against PR ProjectASAP/sketch-bench#196's SCHEMAS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study_saturation: --hydra-variants/--hydra-ws narrow the Hydra cost rows too A hydra-univmon-cardinality grid at W=16384 merged from 16 shards outgrows a 251 GB node, and the cost phase can't resume; these flags let the cost phase leave a variant or width out, as they do the hydra phase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * aqpbm-datagen: scale_by/scale_range; hydra_http_latency scales latency per service A f64 column with `scale_by: <label column>` and `scale_range: [lo, hi]` multiplies each row by a factor fixed per label value, log-uniform in [lo, hi] from a hash of the value and the column's seed. Validation: earlier string label column, f64 only, 0 < lo <= hi. Absent fields keep every existing spec's output byte-identical. hydra_http_latency.yaml scales latency by service in [1, 10], so groups differ in latency scale and a Hydra-KLL mixture is no longer the queried group's own CDF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…dra and roll-ups (ProjectASAP/sketch-bench#196) * eval: multi-grouping anomaly-detection templates (http, flows) The generator gets a metric-level ordered label schema (labels with cardinality, child_of fan-out and skew; per-value data shape) and per-RQE `grouping` and `covers_share`. The 10 templates are now the "classic" set, byte-for-byte the committed tables; "all" adds templates 11, 12, 14, 15, 16, 17 on `http` and `flows`, whose streams each carry several groupings so roll-ups (and later Hydra) can share. The plan runs both sets over the grid. The runner builds each schema stream's MetricFacts from the schema (labels = schema + x; cardinality per used grouping and the full set), takes Raqe.grouping_labels from the RQE's grouping, maps the `cardinality` capability to HLL, and reports covers_share and roll-up deployment groupings in each choice. Classic output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: skewed http labels, per-RQE group coverage, fresh-checkout test Review of ProjectASAP/sketch-bench#196. http's labels are now Zipf-skewed (region 0.5, service 1.1, endpoint 1.1 within its service, status 2.0) and the flows DDoS burst targets a tail subnet, so covers_share selects a few groups. The generator computes each group's share (product of its labels' shares) and, per RQE, the covered groups (share >= tau, plus always the largest) and the smallest covered group's share and items per window. The runner reads a covered RQE's accuracy (ASAP and AutoSketch) at that group's items instead of the mean group's; covers_share = null is unchanged. The schema test reads an inline HLL curve fixture instead of the gitignored study curves. The synthetic plot keys runs by template set too, and both plot scripts title classic "mixed" and all "mixed + multi-grouping". README: the coverage rule, the N the runner uses, and which schema fields are descriptive. label_set.keys_per_window is commented as the universe K. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: read null-coverage RQEs at their smallest group; template 17 at {service, endpoint} - covers_share = null covers every group, so the runner reads its accuracy at the smallest group (min_covered_share), like a tau RQE's smallest covered group. Classic RQEs (no covers_share) are unchanged. - Template 17 (negative control) groups by {service, endpoint}: its smallest group holds 4.5e4 items in 5m, on the curves. - The generator asserts every schema RQE's covered_min_N >= 1e3 (drops the vacuous `assert covered`); a test checks the old full-label-set template fails it. - Covered facts go through validate_facts. - README: null semantics, template 17, and the numbers' provenance (fresh `all` table, committed saturation inputs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: hold covered RQEs to the worse of their smallest and largest group Per sketch-bench#189, KLL/DD/HLL error does not fall as N grows, so the largest covered group can be harder than the smallest. The generator also emits max_covered_share / covered_max_N (the largest group, always covered); the runner reads each covered RQE's accuracy at both the smallest and the largest covered group and keeps the worse (direction-aware), unknown if either read is. Classic RQEs and Hydra's lookup are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: template 12 groups by (dst_subnet, proto), not (dst_subnet, dst_port) (dst_subnet, dst_port) has 1e6 groups; answering each every minute takes about 49 s on one core, which set every plan's batch latency above the largest SLA (10 s) and collapsed the multi-grouping frontier. (dst_subnet, proto) has 3e3 groups and keeps three groupings over one stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tes, with measured accuracy by coverage (ProjectASAP/sketch-bench#201) * eval: multi-grouping anomaly-detection templates (http, flows) The generator gets a metric-level ordered label schema (labels with cardinality, child_of fan-out and skew; per-value data shape) and per-RQE `grouping` and `covers_share`. The 10 templates are now the "classic" set, byte-for-byte the committed tables; "all" adds templates 11, 12, 14, 15, 16, 17 on `http` and `flows`, whose streams each carry several groupings so roll-ups (and later Hydra) can share. The plan runs both sets over the grid. The runner builds each schema stream's MetricFacts from the schema (labels = schema + x; cardinality per used grouping and the full set), takes Raqe.grouping_labels from the RQE's grouping, maps the `cardinality` capability to HLL, and reports covers_share and roll-up deployment groupings in each choice. Classic output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: skewed http labels, per-RQE group coverage, fresh-checkout test Review of ProjectASAP/sketch-bench#196. http's labels are now Zipf-skewed (region 0.5, service 1.1, endpoint 1.1 within its service, status 2.0) and the flows DDoS burst targets a tail subnet, so covers_share selects a few groups. The generator computes each group's share (product of its labels' shares) and, per RQE, the covered groups (share >= tau, plus always the largest) and the smallest covered group's share and items per window. The runner reads a covered RQE's accuracy (ASAP and AutoSketch) at that group's items instead of the mean group's; covers_share = null is unchanged. The schema test reads an inline HLL curve fixture instead of the gitignored study curves. The synthetic plot keys runs by template set too, and both plot scripts title classic "mixed" and all "mixed + multi-grouping". README: the coverage rule, the N the runner uses, and which schema fields are descriptive. label_set.keys_per_window is commented as the universe K. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: read null-coverage RQEs at their smallest group; template 17 at {service, endpoint} - covers_share = null covers every group, so the runner reads its accuracy at the smallest group (min_covered_share), like a tau RQE's smallest covered group. Classic RQEs (no covers_share) are unchanged. - Template 17 (negative control) groups by {service, endpoint}: its smallest group holds 4.5e4 items in 5m, on the curves. - The generator asserts every schema RQE's covered_min_N >= 1e3 (drops the vacuous `assert covered`); a test checks the old full-label-set template fails it. - Covered facts go through validate_facts. - README: null semantics, template 17, and the numbers' provenance (fresh `all` table, committed saturation inputs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: hold covered RQEs to the worse of their smallest and largest group Per sketch-bench#189, KLL/DD/HLL error does not fall as N grows, so the largest covered group can be harder than the smallest. The generator also emits max_covered_share / covered_max_N (the largest group, always covered); the runner reads each covered RQE's accuracy at both the smallest and the largest covered group and keeps the worse (direction-aware), unknown if either read is. Classic RQEs and Hydra's lookup are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: template 12 groups by (dst_subnet, proto), not (dst_subnet, dst_port) (dst_subnet, dst_port) has 1e6 groups; answering each every minute takes about 49 s on one core, which set every plan's batch latency above the largest SLA (10 s) and collapsed the multi-grouping frontier. (dst_subnet, proto) has 3e3 groups and keeps three groupings over one stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: admit Hydra (HLL, UnivMon cardinality, KLL) as candidates, with measured accuracy by coverage - FamilyProperties for hydra-hll, hydra-univmon-cardinality and hydra-kll: one fixed-size grid, DeltaSet key tracker, no roll-up, and the new answers_any_subgrouping. Added to Capability::families, not to DEPLOYABLE_FAMILIES. - Candidates: on a metric with a hydra_dataset, one grid per schema Λ (each RAQE grouping and the union, where card(Λ) is known); eligible for any RAQE with ∅ ≠ G_r ⊆ Λ. The subpopulations ceiling is gone. - Accuracy: hydra_saturation.csv under the saturation dir (optional), keyed by variant, config, dataset and grouping; the column the RAQE's accuracy_covers_share selects; worst over records and seeds; lossy merges bracketed by merge_shards, unknown past the largest. - Runner: ASAP and PerQuery plan with undeployable families; RQEs carry covers_share; http/flows streams name their Hydra dataset; results report hydra_rqes for workloads Hydra may serve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: Hydra grids at the full schema from their dataset's rows Round-1 review of ProjectASAP/sketch-bench#201: - One Hydra grid per (stream, variant, config, window, slide), over the metric's full schema only (MetricFacts::hydra_dataset now carries the dataset and the schema), the width the study measures. - MeasuredAt gains optional `dataset` and `schema_width`; a Hydra row is a candidate on a metric only if measured on its dataset. - Runner: card(full schema) = product of label fan-outs, capped at the stream's series; AutoSketch also allows undeployable families (it skips Hydra itself, ProjectASAP/sketch-bench#159). - read_csv checks required columns and short rows; config params must be numbers. - Docs: Hydra held to the worst case, built only at the full schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: validate the Hydra schema, skip empty Hydra cells, reject long CSV rows validate_facts reports a Hydra schema outside the metric's labels or without a nonzero cardinality, instead of a later panic. The Hydra accuracy fold skips empty cells (no covered group scored in that run) and is unknown only when every row left the column empty. read_csv rejects rows longer than the header too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: test fixtures carry max_covered_share and label cardinalities After rebasing onto ProjectASAP/sketch-bench#196's two-ended coverage read: ProjectASAP/sketch-bench#201's runner test RQEs need max_covered_share, and ProjectASAP/sketch-bench#196's coverage test schema needs the label cardinality the full-schema Hydra grid reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: roll-up ablation, ASAP without roll-ups The runner adds the method `asap-norollup` ("ASAP (no roll-ups)"): ASAP's MILP, candidates, frontier sweep and SLA grid, with each RQE allowed only deployments at its own grouping or Hydra grids. Every result counts `rolled_up_rqes` (RQEs served by a finer non-Hydra deployment), and sanity checks asap <= asap-norollup <= perquery unbounded and at each SLA. The plot scripts draw it as a dashed violet line, add fig_rollup_ablation.png (unbounded cost with and without roll-ups, absolute, CPU and Fargate panels) and a roll-up section in both summaries; results without the method still plot. The run script passes --no-chosen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…n (§6.3) (ProjectASAP/sketch-bench#203) * polars: tag Hydra baseline subset keys by column (ProjectASAP/sketch-bench#74) The polars exact baselines keyed a label subset by its `;`-joined values, asap_sketchlib 0.2.2's subkey. 0.3.0's Hydra tags each value with its column (`label0:a;label1:b`), so on label columns that share a value domain the baseline merged a key1 group with the same key2 group and scored nonzero against itself. subset_key now builds the 0.3.0 key over hydra_shared's `label{i}` schema (escaping `\`, `:` and `;`), empty labels keep their column, and every lookup goes through it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: multi-grouping anomaly-detection templates (http, flows) The generator gets a metric-level ordered label schema (labels with cardinality, child_of fan-out and skew; per-value data shape) and per-RQE `grouping` and `covers_share`. The 10 templates are now the "classic" set, byte-for-byte the committed tables; "all" adds templates 11, 12, 14, 15, 16, 17 on `http` and `flows`, whose streams each carry several groupings so roll-ups (and later Hydra) can share. The plan runs both sets over the grid. The runner builds each schema stream's MetricFacts from the schema (labels = schema + x; cardinality per used grouping and the full set), takes Raqe.grouping_labels from the RQE's grouping, maps the `cardinality` capability to HLL, and reports covers_share and roll-up deployment groupings in each choice. Classic output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * polars: write subset key tags without a temporary String; fix test comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: interleaved merge split; merge exactness and bound tests `--merge-split contiguous|interleaved` (default contiguous, unchanged) picks how merge runs split the stream into shards. Interleaved reorders the stream so the existing contiguous `partition` deals it round-robin by record, at the contiguous split's shard sizes, so every row's folds take it unchanged. Tests: hydra-cms, hydra-cs and hydra-hll grids folded from 4 and 7 shards, under both splits, answer every probed subpopulation exactly as the single stream; and every scored group on a fixed 20k-record stream lands inside its hydra doc §2.2 bound (CMS also never under). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: child_of columns for hierarchical labels A `child_of: <index>` column with `fan_out: f` nests under an earlier string column: its value is the parent row's value, a `.`, and a child index in [0, f) drawn from the column's own distribution. Bad indices, a fan_out that disagrees with the domain, and fields that would not mean what they say on a child are refused before drawing. Adds configs/datagen/hydra_hier.yaml (region 4 -> service 25 -> endpoint 25, status 4, user_id value) for the hydra rows, docs, and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: score any target grouping; per-group error output Every hydra-* row (lib and polars, all five variants and hydra-univmon's four statistics) takes `--group-columns J,K,...` (default `0`, today's behavior) in place of the hardcoded SCORED_LABEL_COLUMN. A subpopulation probe is now positioned (`Vec<Option<String>>`, Some at the grouped columns), so `hydra_shared::query` asks any non-empty subset of the key columns and the polars baselines look up `subset_key` with that subset's mask (`group_key`, generalizing `prefix_key`). Each grouped comparator reports its per-group error in its own metric (relative error; mean rank error for hydra-kll) and summarises it as err_mean, err_p50, err_p90, err_max, groups_scored, plus schema_width, records and fanned_mass. `--per-group-out PATH` writes the rows as `group_key,n_q,error`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: whole uniform bounds for child_of; 2- and 3-label hierarchy cuts A uniform draw is continuous, so `uniform{0.0, 2.9}` with fan_out 2 drew child indices 0, 1 and 2; child_of now requires whole uniform bounds. hydra_hier_d2.yaml and hydra_hier_d3.yaml cut hydra_hier.yaml to its first 2 and 3 labels for the Hydra study's schema-width sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: skewed http labels, per-RQE group coverage, fresh-checkout test Review of ProjectASAP/sketch-bench#196. http's labels are now Zipf-skewed (region 0.5, service 1.1, endpoint 1.1 within its service, status 2.0) and the flows DDoS burst targets a tail subnet, so covers_share selects a few groups. The generator computes each group's share (product of its labels' shares) and, per RQE, the covered groups (share >= tau, plus always the largest) and the smallest covered group's share and items per window. The runner reads a covered RQE's accuracy (ASAP and AutoSketch) at that group's items instead of the mean group's; covers_share = null is unchanged. The schema test reads an inline HLL curve fixture instead of the gitignored study curves. The synthetic plot keys runs by template set too, and both plot scripts title classic "mixed" and all "mixed + multi-grouping". README: the coverage rule, the N the runner uses, and which schema fields are descriptive. label_set.keys_per_window is commented as the universe K. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: read null-coverage RQEs at their smallest group; template 17 at {service, endpoint} - covers_share = null covers every group, so the runner reads its accuracy at the smallest group (min_covered_share), like a tau RQE's smallest covered group. Classic RQEs (no covers_share) are unchanged. - Template 17 (negative control) groups by {service, endpoint}: its smallest group holds 4.5e4 items in 5m, on the curves. - The generator asserts every schema RQE's covered_min_N >= 1e3 (drops the vacuous `assert covered`); a test checks the old full-label-set template fails it. - Covered facts go through validate_facts. - README: null semantics, template 17, and the numbers' provenance (fresh `all` table, committed saturation inputs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * datagen: the hydra_hier usage example passes --dtype Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: list every group in --per-group-out; escape keys; refuse no accuracy - Group keys escape `\`, `:` and `;` as polars_shared::subset_key does. - Groups that cannot be scored (zero true statistic, e.g. zero entropy) are written with an empty `error`, so n_q sums to the stream's records. - --per-group-out with no accuracy measurement is refused instead of writing an empty file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: KLL CDF and UnivMon sum rows; KLL cell merge buffer - hydra-kll-cdf (lib, polars): asks the KLL grid HydraQuery::Cdf(x) at each group's own values at phi = 0.01..0.99, scored by absolute CDF error (SubpopCdfErrorGT, comparator subpop-cdf-error). The polars baseline is the exact share at or below x over the same sorted groups. - hydra-univmon-sum (lib, polars): each record inserted with count = value, so the UnivMon grid's L1 is the group's sum, scored by relative error against exact per-group sums (SubpopSumGT, comparator subpop-sum). Integer value dtypes only; f64/str and values outside [0, i32::MAX] are refused by name. - memory_hydra_kll counts each cell as kll_lib_bytes does, merge buffer of cell_k items included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra-kll: the footprint's merge buffer is the steady state; tie tests The grid clones its cells from one template, and a cloned Vec keeps no capacity, so a cell's merge buffer is empty at build and grows to about cell_k items at its first compaction. Say the footprint counts it at that bound. The CDF test now asks a group of ties (at or below) and a mixed-tie group on a grid wide enough that it shares no cell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra: record merge_split; review fixes - `merge_split` (contiguous|interleaved) is a record field next to `merge_shards` (BenchSection and MergeMetrics; schema regenerated). Merge records stamp it; a record without it split contiguously, so existing results parse, and flatten refuses two different splits in one row as it does two shard counts. - `MergeSplit` moves to aqpbm-core (it is a record type) and is parsed by clap directly instead of matching strings. - The interleaved copy of the stream is built only when a merge runs. - README states the round-robin exactly (contiguous shard sizes, so a short last shard shifts the pattern); MergeMetrics docs name the split. - hydra-cs bound test: note its margin (passes from beta ~4.8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study: --phase hydra on the eval's datasets; Hydra cost rows configs/datagen/hydra_http.yaml, hydra_http_latency.yaml and hydra_flows.yaml follow the §6.3 eval's http and flows schemas (labels in schema order, cardinalities, child_of, skews; user_id Zipf 0.8, latency Pareto a = 2, src_ip uniform). The flows DDoS burst is not generated: datagen draws each column independently. --phase hydra runs HYDRA_SWEEPS: the eval's variants on those datasets (every grouping, W 1024/4096/16384, N 1e5..1e7, 1/4/16 interleaved shards, 3 seeds) and every statistic on hydra_hier{_d2,_d3,}.yaml, one run per grouping with --per-group-out, into hydra_saturation.csv with the coverage columns err_max_cov_0.01/0.05 (share >= tau plus the largest group). --jobs under a memory budget, --resume, and --hydra-shard i/n to split it across nodes. --phase optimizer-cost adds Hydra rows (eval variants x W x dataset); MeasuredAt gains dataset and schema_width, the reducer reading the width off the row's scores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * hydra-univmon-sum: the ground truth refuses negative values too The lib row inserts each value as a UnivMon count and refuses negatives; the truth (and so the polars baseline) accepted them, so a spec with negatives ran on polars and failed on lib. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study: Hydra cost rows at 3 runs; their test; datagen path from the script Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * docs: Hydra cost rows' run count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study_saturation: atomic hydra table rewrite, stricter resume, schema checks --phase hydra --resume now writes the kept rows to hydra_saturation.csv.tmp and os.replace()s it over the table, and keeps a row only when it ends in a line end and every column parses (integers, numbers or "" for the error columns, non-empty text). The CSV format is unchanged, so tables written by 003eb31 resume as before. The eval-spec test also checks each label's cardinality, skew and child_of against PR ProjectASAP/sketch-bench#196's SCHEMAS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study_saturation: --hydra-variants/--hydra-ws narrow the Hydra cost rows too A hydra-univmon-cardinality grid at W=16384 merged from 16 shards outgrows a 251 GB node, and the cost phase can't resume; these flags let the cost phase leave a variant or width out, as they do the hydra phase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: hold covered RQEs to the worse of their smallest and largest group Per sketch-bench#189, KLL/DD/HLL error does not fall as N grows, so the largest covered group can be harder than the smallest. The generator also emits max_covered_share / covered_max_N (the largest group, always covered); the runner reads each covered RQE's accuracy at both the smallest and the largest covered group and keeps the worse (direction-aware), unknown if either read is. Classic RQEs and Hydra's lookup are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * aqpbm-datagen: scale_by/scale_range; hydra_http_latency scales latency per service A f64 column with `scale_by: <label column>` and `scale_range: [lo, hi]` multiplies each row by a factor fixed per label value, log-uniform in [lo, hi] from a hash of the value and the column's seed. Validation: earlier string label column, f64 only, 0 < lo <= hi. Absent fields keep every existing spec's output byte-identical. hydra_http_latency.yaml scales latency by service in [1, 10], so groups differ in latency scale and a Hydra-KLL mixture is no longer the queried group's own CDF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: template 12 groups by (dst_subnet, proto), not (dst_subnet, dst_port) (dst_subnet, dst_port) has 1e6 groups; answering each every minute takes about 49 s on one core, which set every plan's batch latency above the largest SLA (10 s) and collapsed the multi-grouping frontier. (dst_subnet, proto) has 3e3 groups and keeps three groupings over one stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: admit Hydra (HLL, UnivMon cardinality, KLL) as candidates, with measured accuracy by coverage - FamilyProperties for hydra-hll, hydra-univmon-cardinality and hydra-kll: one fixed-size grid, DeltaSet key tracker, no roll-up, and the new answers_any_subgrouping. Added to Capability::families, not to DEPLOYABLE_FAMILIES. - Candidates: on a metric with a hydra_dataset, one grid per schema Λ (each RAQE grouping and the union, where card(Λ) is known); eligible for any RAQE with ∅ ≠ G_r ⊆ Λ. The subpopulations ceiling is gone. - Accuracy: hydra_saturation.csv under the saturation dir (optional), keyed by variant, config, dataset and grouping; the column the RAQE's accuracy_covers_share selects; worst over records and seeds; lossy merges bracketed by merge_shards, unknown past the largest. - Runner: ASAP and PerQuery plan with undeployable families; RQEs carry covers_share; http/flows streams name their Hydra dataset; results report hydra_rqes for workloads Hydra may serve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: Hydra grids at the full schema from their dataset's rows Round-1 review of ProjectASAP/sketch-bench#201: - One Hydra grid per (stream, variant, config, window, slide), over the metric's full schema only (MetricFacts::hydra_dataset now carries the dataset and the schema), the width the study measures. - MeasuredAt gains optional `dataset` and `schema_width`; a Hydra row is a candidate on a metric only if measured on its dataset. - Runner: card(full schema) = product of label fan-outs, capped at the stream's series; AutoSketch also allows undeployable families (it skips Hydra itself, ProjectASAP/sketch-bench#159). - read_csv checks required columns and short rows; config params must be numbers. - Docs: Hydra held to the worst case, built only at the full schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * rqe-optimizer: validate the Hydra schema, skip empty Hydra cells, reject long CSV rows validate_facts reports a Hydra schema outside the metric's labels or without a nonzero cardinality, instead of a later panic. The Hydra accuracy fold skips empty cells (no covered group scored in that run) and is unknown only when every row left the column empty. read_csv rejects rows longer than the header too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: test fixtures carry max_covered_share and label cardinalities After rebasing onto ProjectASAP/sketch-bench#196's two-ended coverage read: ProjectASAP/sketch-bench#201's runner test RQEs need max_covered_share, and ProjectASAP/sketch-bench#196's coverage test schema needs the label cardinality the full-schema Hydra grid reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: roll-up ablation, ASAP without roll-ups The runner adds the method `asap-norollup` ("ASAP (no roll-ups)"): ASAP's MILP, candidates, frontier sweep and SLA grid, with each RQE allowed only deployments at its own grouping or Hydra grids. Every result counts `rolled_up_rqes` (RQEs served by a finer non-Hydra deployment), and sanity checks asap <= asap-norollup <= perquery unbounded and at each SLA. The plot scripts draw it as a dashed violet line, add fig_rollup_ablation.png (unbounded cost with and without roll-ups, absolute, CPU and Fargate panels) and a roll-up section in both summaries; results without the method still plot. The run script passes --no-chosen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: AutoSketch vs. ASAP with Hydra candidates and a roll-up ablation Results of the §6.3 synthetic eval (#777) rerun with roll-ups (ProjectASAP/sketch-bench#190) and their ablation, Hydra candidates measured on the eval's own data (ProjectASAP/sketch-bench#195–ProjectASAP/sketch-bench#202), the multi-grouping template set (ProjectASAP/sketch-bench#196), ProjectASAP/sketch-bench#192's KLL footprint and a re-measured cost table. 8 workloads, 0 RQEs dropped, 0 sanity violations. Roll-ups serve 40% of the multi-grouping RQEs from a finer deployment and halve ASAP's deployments, but save only 3.3–3.6% (CPU) / 4.4–4.8% (Fargate). Hydra is eligible for many RQEs but never chosen: its insert fans out to every label subset (1.9–5.1 µs per record vs. 3.85 ns for a per-group HLL). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: cube template and a multi-grouping-only workload Template 16 is now p99 latency GROUP BY CUBE(region, service, endpoint), all 7 groupings, replacing {service}, {region, service}, {service, status}. A new `multigroup` template set runs the multi-grouping templates alone, so the roll-up ablation is not diluted by the classic set. Roll-ups now save 60% on multi-grouping (0.551 -> 0.221 vCPU, 26 -> 5 deployments) and 8% on mixed + multi-grouping. Classic is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * eval: each workload's chosen plans, gzipped Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rojectASAP/sketch-bench#205) Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ctASAP/sketch-bench#204 P2) (ProjectASAP/sketch-bench#208) export_autosketch_eval_table.py --synthetic adds workloads[0].e2e to classic tables: each PromQL query with its repeat interval and the RQEs it reads (template 10's ratio reads two), and the metrics to generate (gauge data, counter data_total; job and label_0 child_of job). The runner does not read it, so the RQEs and the committed plans are unchanged. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ProjectASAP/sketch-bench#207) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…-bench#209) * eval: e2e block for multi-grouping and all tables (ProjectASAP/sketch-bench#204) export_autosketch_eval_table.py --synthetic now writes workloads[0].e2e to every table. Schema templates have no PromQL (SQL is rendered downstream), so each schema RQE gets one structured query entry: table, value, capability, grouping, quantile (p99: 0.99), window_secs, interval_secs and its rqes. Each schema metric is a `records` table (`http`, `flows`; `data_i_http` under metric `data_i`) with SCHEMAS' labels, rate as records_per_sec, and values (scaled_by, anomaly, burst as in SCHEMAS). Classic entries are unchanged. The multigroup and all tables in autosketch-vs-asap-hydra/inputs/tables are regenerated; minus e2e they equal the committed ones byte for byte, and the classic tables are unchanged. Their roll-up plans were already exported by ProjectASAP/sketch-bench#203 (plans/*.plans.json.gz); a rerun of multigroup-shared1 and all-shared1 reproduces all 36 plans of each exactly, so the plans stand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: one e2e entry per distinct schema query (ProjectASAP/sketch-bench#204) Schema query entries identical apart from `rqes` (templates 14, 15 and 17 share some groupings) are merged into one entry whose `rqes` lists every RQE it reads, sorted, so each distinct SQL query appears once. Classic entries are unchanged. The multigroup and all tables are regenerated; minus e2e they still equal main's byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: wrap the e2e_block docstring Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: save ASAPQuery SQL in structured e2e queries --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Milind Srivastava <milindsrivastava1997@gmail.com>
…rojectASAP/sketch-bench#210) * eval: e2e block for multi-grouping and all tables (ProjectASAP/sketch-bench#204) export_autosketch_eval_table.py --synthetic now writes workloads[0].e2e to every table. Schema templates have no PromQL (SQL is rendered downstream), so each schema RQE gets one structured query entry: table, value, capability, grouping, quantile (p99: 0.99), window_secs, interval_secs and its rqes. Each schema metric is a `records` table (`http`, `flows`; `data_i_http` under metric `data_i`) with SCHEMAS' labels, rate as records_per_sec, and values (scaled_by, anomaly, burst as in SCHEMAS). Classic entries are unchanged. The multigroup and all tables in autosketch-vs-asap-hydra/inputs/tables are regenerated; minus e2e they equal the committed ones byte for byte, and the classic tables are unchanged. Their roll-up plans were already exported by ProjectASAP/sketch-bench#203 (plans/*.plans.json.gz); a rerun of multigroup-shared1 and all-shared1 reproduces all 36 plans of each exactly, so the plans stand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: one e2e entry per distinct schema query (ProjectASAP/sketch-bench#204) Schema query entries identical apart from `rqes` (templates 14, 15 and 17 share some groupings) are merged into one entry whose `rqes` lists every RQE it reads, sorted, so each distinct SQL query appears once. Classic entries are unchanged. The multigroup and all tables are regenerated; minus e2e they still equal main's byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: wrap the e2e_block docstring Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: drop classic templates 7 and 9 (rates by job, top-k by rate) `sum by (job) (rate(data_total[w]))` (template 7) and `topk(32, sum by (label_0) (rate(data_total[w])))` (template 9) leave the §6.3 evaluation (#777): ASAPQuery's engine would need per-series counter deltas to serve them. Per-series `rate` (template 6) stays, so `data_total` is still generated. The dashboard set drops the same two (D3, D4). Templates keep their numbers. Classic is now 8 templates, 42 RQEs (46 PromQL queries); all is 104 RQEs. Multigroup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: regenerate the classic and all tables without templates 7 and 9 export_autosketch_eval_table.py --synthetic, with each table's committed --revision (classic 8daa0bc, all and multigroup b27934e). The multigroup tables regenerate byte for byte. Classic: 42 RQEs (r = 8: 76; m = 8, 16: 336, 672); all: 104 (200; 832, 1664). The e2e blocks no longer hold either query; rate(data_total[w]) remains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: render template 2's e2e PromQL as topk(32, sum_over_time(data[w])) label_0 names one series (child_of job, unique values), so `sum by (label_0)` only drops `job`: the ranking and values are the same. ASAPQuery's planner supports the two-level shape today, not the three-level one (#744, #772). The RQE (top-k keyed by label_0, topk32_sum_by_label0_sum_over_time) is unchanged; only the classic and all tables' e2e queries change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: rerun §6.3 without classic templates 7 and 9 The classic and all workloads, rerun exactly as ProjectASAP/sketch-bench#203 (`--runs 3 --no-chosen`, the same saturation inputs; plans from a `--runs 1` rerun with chosen plans kept). Multigroup tables are unchanged and keep their results; a rerun of multigroup-shared1 reproduces its costs and plans exactly. AutoSketch's benchmark times are reused: every probe was already measured, and each workload's benchmark time is unchanged. Classic (42 RQEs): ASAP 2.38 vCPU (was 3.54), PerQuery 7.77 (10.5), AutoSketch 2,739 (3,716); at AutoSketch's latency ASAP 4.22 (5.67). All (104 RQEs): ASAP 2.60 (3.76), no roll-ups 2.93 (4.09), roll-ups save 11% (8%). Zero dropped RQEs, sanity violations or Hydra choices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: save ASAPQuery SQL in structured e2e queries --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Milind Srivastava <milindsrivastava1997@gmail.com>
ProjectASAP/sketch-bench#212) * eval: e2e block for multi-grouping and all tables (ProjectASAP/sketch-bench#204) export_autosketch_eval_table.py --synthetic now writes workloads[0].e2e to every table. Schema templates have no PromQL (SQL is rendered downstream), so each schema RQE gets one structured query entry: table, value, capability, grouping, quantile (p99: 0.99), window_secs, interval_secs and its rqes. Each schema metric is a `records` table (`http`, `flows`; `data_i_http` under metric `data_i`) with SCHEMAS' labels, rate as records_per_sec, and values (scaled_by, anomaly, burst as in SCHEMAS). Classic entries are unchanged. The multigroup and all tables in autosketch-vs-asap-hydra/inputs/tables are regenerated; minus e2e they equal the committed ones byte for byte, and the classic tables are unchanged. Their roll-up plans were already exported by ProjectASAP/sketch-bench#203 (plans/*.plans.json.gz); a rerun of multigroup-shared1 and all-shared1 reproduces all 36 plans of each exactly, so the plans stand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: one e2e entry per distinct schema query (ProjectASAP/sketch-bench#204) Schema query entries identical apart from `rqes` (templates 14, 15 and 17 share some groupings) are merged into one entry whose `rqes` lists every RQE it reads, sorted, so each distinct SQL query appears once. Classic entries are unchanged. The multigroup and all tables are regenerated; minus e2e they still equal main's byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: wrap the e2e_block docstring Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: drop classic templates 7 and 9 (rates by job, top-k by rate) `sum by (job) (rate(data_total[w]))` (template 7) and `topk(32, sum by (label_0) (rate(data_total[w])))` (template 9) leave the §6.3 evaluation (#777): ASAPQuery's engine would need per-series counter deltas to serve them. Per-series `rate` (template 6) stays, so `data_total` is still generated. The dashboard set drops the same two (D3, D4). Templates keep their numbers. Classic is now 8 templates, 42 RQEs (46 PromQL queries); all is 104 RQEs. Multigroup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: regenerate the classic and all tables without templates 7 and 9 export_autosketch_eval_table.py --synthetic, with each table's committed --revision (classic 8daa0bc, all and multigroup b27934e). The multigroup tables regenerate byte for byte. Classic: 42 RQEs (r = 8: 76; m = 8, 16: 336, 672); all: 104 (200; 832, 1664). The e2e blocks no longer hold either query; rate(data_total[w]) remains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: render template 2's e2e PromQL as topk(32, sum_over_time(data[w])) label_0 names one series (child_of job, unique values), so `sum by (label_0)` only drops `job`: the ranking and values are the same. ASAPQuery's planner supports the two-level shape today, not the three-level one (#744, #772). The RQE (top-k keyed by label_0, topk32_sum_by_label0_sum_over_time) is unchanged; only the classic and all tables' e2e queries change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: rerun §6.3 without classic templates 7 and 9 The classic and all workloads, rerun exactly as ProjectASAP/sketch-bench#203 (`--runs 3 --no-chosen`, the same saturation inputs; plans from a `--runs 1` rerun with chosen plans kept). Multigroup tables are unchanged and keep their results; a rerun of multigroup-shared1 reproduces its costs and plans exactly. AutoSketch's benchmark times are reused: every probe was already measured, and each workload's benchmark time is unchanged. Classic (42 RQEs): ASAP 2.38 vCPU (was 3.54), PerQuery 7.77 (10.5), AutoSketch 2,739 (3,716); at AutoSketch's latency ASAP 4.22 (5.67). All (104 RQEs): ASAP 2.60 (3.76), no roll-ups 2.93 (4.09), roll-ups save 11% (8%). Zero dropped RQEs, sanity violations or Hydra choices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae * eval: save ASAPQuery SQL in structured e2e queries * eval: §6.3 workloads re-planned at 10 samples/s for the e2e run export_autosketch_eval_table.py --synthetic gains --samples-per-sec (default 200): it sets SYNTHETIC_SAMPLES_PER_SEC and scales the schema metrics' rates, and leaves out (logged) a schema RQE whose smallest covered group falls below the curves' first N, with its e2e query, where the default asserts. Without the flag the export is unchanged (the committed 200/s tables, byte for byte but the final newline). rqe-optimizer/results/e2e-rate10/: the classic, all and multigroup tables (shared = 1) at 10 samples/s, the runner's results (p95, --runs 3 --no-chosen) and each method's plans (--runs 1, extracted by the new scripts/extract_autosketch_plans.py, which reproduces the committed -hydra plans). Two template-16 RQEs (p99 by (region, endpoint) and (region, service, endpoint) over 5m, 403 items in the smallest group) are left out; the runner would drop them as unservable and picks the same plans either way. Every other RQE is served. Cheapest plan, CPU weights, 200/s -> 10/s: classic ASAP 2.38 -> 0.309 vCPU, PerQuery 7.77 -> 0.725, AutoSketch 2,739 -> 137; all ASAP 2.60 -> 0.404; multigroup ASAP 0.221 -> 0.095. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Milind Srivastava <milindsrivastava1997@gmail.com>
…#206) The runner's --target takes acc90, acc95 (p95, as in the committed runs) or acc99: NN% accuracy in each family's own metric. The 12 synthetic workloads rerun at 90% and 99%; scripts/plot_autosketch_vs_asap_targets.py plots cost vs. target. 90% costs the same as 95%: the 95% plans already use the cheapest measured configs. 99% raises ASAP's cost 9% on mixed and 2.8x on multi-grouping; ASAP stays the cheapest at every target. Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tASAP/sketch-bench#213) * study: KLL k 25-800 (2x steps), DDSketch alpha 0.1, Hydra W 65536; --sketches/--configs to add configs to an existing study Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study: Hydra cost rows past W 16384 merge 4 shards (16 grids of 3.2 GB do not fit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: §6.3 rerun on the denser grid (KLL k 25-800, DDSketch alpha 0.1, Hydra W 65536) Inputs: the new configs' curves appended to autosketch-vs-asap-inputs (raw records in raw/dense_*), 7 cost rows (63 in all), and 999 Hydra runs at W = 65536 (7,778 in all). Results: the 12 workloads at 95% (with plans) and at 90% and 99%, on ProjectASAP/sketch-bench#210's tables. CPU cost falls 0-1.4%, Fargate cost up to 13% (smaller KLL/DDSketch nearer the target). 90% still equals 95%. Hydra is never chosen, even at W = 65536. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * review: one filter helper with name checks and tests; README provenance and claims - filtered_configs()/check_filters(): --sketches or --configs skips the exact rows; a misspelled name exits; tests for both. - The wide-merge comment says per fold, with operands 4x larger. - rqe-optimizer README: the -hydra row's 2026-10-14 rerun, the one solver-tolerance violation, e2e-rate10 planned on the pre-ProjectASAP/sketch-bench#213 grid. - -hydra README: rerun provenance moved to Reproduce; hydra-kll's figure marked unmerged, with the all-merge-counts value. - raw README: measurement dates, clnode130. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * review round 2: --sketches help names the Hydra flags; planning-time comparability; rewrap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#215) * eval: instant RQEs read one scrape, not 1 s A PromQL instant query reads one sample per series, but the synthetic spatial templates (sum by (job), quantile by (job)) looked back 1 s, so their N grew with the data rate (2e5 per job at 200/s, 1e4 at 10/s). They now look back one scrape interval (1 / samples_per_sec_per_series s), so N = series per job = 1e3 at any rate. autosketch_vs_asap scrapes a metric with samples_per_sec_per_series at that rate (record metrics keep 1 s), so a sub-second lookback holds whole scrapes, and writes sub-second windows and slides as fractional seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: reject samples/s rates that don't divide 1000 An instant RQE's lookback is one scrape, 1000 / rate ms, and the runner keeps lookbacks and scrape intervals in whole ms. A rate that doesn't divide 1000 (3/s gives 333 ms) truncates, and every synthetic RQE fails validation; above 1000/s the scrape is 0 ms, and 0 divides by zero. The exporter rejects such a --samples-per-sec, and the runner such a samples_per_sec_per_series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: fail loudly when an RQE's lookback isn't whole scrapes A metric missing from the e2e block falls back to a 1 s scrape, so a 5 ms instant lookback had no candidate window and was dropped as unservable, with only the dropped count logged. The runner now asserts that each RQE's lookback and interval are multiples of its stream's scrape, and rounds them to ms instead of truncating. The e2e_block docstring and the byte-for-byte test no longer claim the runner ignores the e2e block; the test checks every RQE metric's samples_per_sec_per_series before dropping the block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…/sketch-bench#216) * eval: one RQE per schema query, the strictest template's (ProjectASAP/sketch-bench#214) Templates 14, 15 and 17 share some groupings, so one query string carried several RQEs with different accuracy scopes. The MILP could serve them from different deployments, which the engine can't do for one query string. By default only the strictest template's RQE is kept (every group, then 1%, then 5%), and an infeasible strictest copy is an error. --keep-duplicate-queries keeps every copy; the §6.3 tables reproduce with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * review: stale RQE counts, e2e-rate10 command and help text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ry (ProjectASAP/sketch-bench#217) Regenerates rqe-optimizer/results/e2e-rate10 (classic, all, multigroup at shared = 1, 10 samples/s) on main: the denser grid's cost table and curves (ProjectASAP/sketch-bench#213), instant RQEs reading one scrape (ProjectASAP/sketch-bench#215), and one RQE per schema query (ProjectASAP/sketch-bench#216). Tables, --runs 3 results and --runs 1 plans; the README gives the RQE counts, costs, classic deployments and ASAPQuery's --data-ingestion-interval-ms 100. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* eval: RQE expression trees in e2e tables Every e2e query gets `expr`, its RQE expression tree: a leaf per RQE it reads (id, metric or table, capability, lookback_ms, grouping, phi/k/value), and a `/` node for template 10. Rate queries, and schema queries merged across several RQEs (--keep-duplicate-queries), get none. Regenerate the e2e-rate10 tables: only `expr` is added, so the RQE ids and plans are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * eval: leaf capabilities as a set, renaming only topk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ch#220) Re-exports the 8 classic and all tables of autosketch-vs-asap-hydra (--keep-duplicate-queries; only the instant RQEs change, now 5 ms) and reruns them: --runs 3 results, --runs 1 plans, and the 90%/99% runs of autosketch-vs-asap-targets. The multi-grouping tables are unchanged, so their results stand. AutoSketch's probes are unchanged, so benchmark times are reused. Figures, summaries and READMEs regenerated. ASAP's cheapest CPU rises ~15% on mixed and mixed + multi-grouping (one more exact-sum deployment per metric); PerQuery, AutoSketch and latency bounds are unchanged. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…h-bench#229) * eval: §6.3.1 case studies swept one knob at a time Rebuilds the performance-monitoring and anomaly-detection case studies as runner tables and sweeps latency SLA, refresh interval, sample rate, series, cardinality and a drill-down query to find where the plan changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: report only ASAP's plans in the case studies Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: case-study summary reads each MV's grouping from the runner Review of ProjectASAP/sketch-bench#229: - summarize_case_studies.py guessed an MV's grouping as the finest of the queries it serves, so an MV grouped (region, service) that serves only `by (service)` (anomaly statuses=2, SLAs 0.885-1.41 ms) printed as `by (service)`. It now reads `deployment_grouping`, which the runner writes wherever the grouping differs from the RQE's. Regenerated summary.md: that label is fixed; every other change is label order (the runner's groupings are sorted). - case_study_tables.anomaly() zipped queries with `intervals or [None] * 7`, so a short `intervals` silently dropped the drill-down. It now requires one interval per query. The regenerated tables are byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e type) (ProjectASAP/sketch-bench#221) * eval: §6.3 95% results of the 8 changed tables rerun on CloudLab The 8 classic and all tables' 95% runs, rerun on clnode109/155/167/178 (the multi-grouping runs' nodes, each running its share serially), so every planning time comes from one node type. Every cost, latency, frontier point and SLA result matches ProjectASAP/sketch-bench#220's desktop run exactly; only planning times change (CloudLab is 2-2.7x slower). Planning-time figure and table updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: §6.3 paper figures and the §6.3.1 case studies - scripts/plot_sec63_paper.py: the paper's cost bars (replacing Table 5), frontier (without the no-roll-up line) and roll-up ablation, from the committed results, into autosketch-vs-asap-hydra/paper/. - scripts/export_case_studies.py and results/case-studies-6.3.1/: the performance-monitoring and anomaly-detection workloads, every method's plan (chosen kept), and a README of what each method deploys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plot_sec63_paper: cost bars drawn at print size (figure* 7 in, and a stacked 3.4 in column version), larger fonts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plot_sec63_paper: larger fonts in the figure* cost bars; one label per bar (ASAP's cost, each baseline's multiple), no overlaps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plot_sec63_paper: figure* cost bars trimmed to the drawing; larger axis titles; ASAP's cost on its bar, baseline multiples inside theirs, darker bar colors Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plot_sec63_paper: cost bars on one set of axes at one column's width; LiteOlly, LiteOllyPerQuery, AutoSketch baseline; W-RangeHeavy/W-RollupHeavy/W-Mixed (the three-panel version kept as sec63_cost_bars_wide) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * plot_sec63_paper: the single-axes cost bars under Fargate prices too (sec63_cost_bars_fargate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…te) (ProjectASAP/sketch-bench#231) The 38 sweep tables rerun without --weights cpu. CPU-only costs match the committed results exactly; the Fargate results are added, and the README notes the three plans that differ under Fargate prices. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…its history The RQE deployment optimizer (MILP, candidates, cost model, saturation accuracy), its AutoSketch-vs-ASAP evaluation runner, the scripts that run, plot and summarize it, its committed results, and its design docs, from ProjectASAP/sketch-bench@c750fa6. History is filtered to these paths; sketch-bench issue references in commit messages are rewritten to ProjectASAP/sketch-bench#N. The eval-table exporters stay in sketch-bench, next to the measurements they read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r uses it by path rqe-optimizer joins the workspace, and asap-planner-rs depends on it by path instead of on sketch-bench's main by git (whose lock pinned an old rev, ASAPQuery#836). Its one sketch-bench dependency left is aqpbm-core, the atomic-cost table's row types, pinned to the rev the crate was imported from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The imported scripts never ran ASAPQuery's hooks: black-format them, end the files with a newline, and fix the two lint errors left (a lambda assigned to a name; shellcheck on the intentional single quotes of an `sh -c` script). check-added-large-files skips rqe-optimizer/results/, whose result JSON reaches ~2 MB. No behavior change: summary.md regenerates byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paths in the imported files were relative to sketch-bench's root: - the moved scripts and docs are now under rqe-optimizer/scripts and rqe-optimizer/docs; - what stayed in sketch-bench (the saturation study, the eval-table exporters, approxbench) is written sketch-bench/..., and commands run it from $SKETCH_BENCH, a sketch-bench checkout; relative links to it are GitHub links; - bare sketch-bench issue numbers are now sketch-bench#N (numbers past sketch-bench's range were already ASAPQuery's and stay bare). The README says what moved and what stayed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Dockerfiles copy rqe-optimizer, which the planner now depends on by path; .dockerignore drops its results and data. Rust, Docker and accuracy workflows trigger on rqe-optimizer changes. Planner comments credit rqe-optimizer for DEPLOYABLE_FAMILIES, Raqe and is_eligible. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Moves the RQE deployment optimizer from ProjectASAP/sketch-bench into this repo as the workspace crate
rqe-optimizer/, together with its full history (fromsketch-bench@c750fa6):autosketch_vs_asap;rqe-optimizer/scripts/;rqe-optimizer/results/, 41 MB on disk, about 20 MB in git history);rqe-optimizer/docs/.asap-planner-rsnow depends on it by path. Before, it used a git dependency on sketch-benchmainwhose lockfile pinned an old rev (ASAPQuery#836).What stays in sketch-bench: the benchmarks and the analysis on top of them, including what turns measurements into solver input: the saturation study, the cost table, and the eval-table exporters. The crate's only sketch-bench dependency left is
aqpbm-core, for the atomic-cost table's row types, pinned toc750fa6.Commits
--allow-unrelated-histories. Commit messages' bare#N(N ≤ 231) becameProjectASAP/sketch-bench#N; larger numbers were already ASAPQuery's and stay bare.good_lp/csvas direct dependencies.check-added-large-filesskipsrqe-optimizer/results/.rqe-optimizer/....sketch-bench/..., and commands use$SKETCH_BENCH.sketch-bench#N.asap-planner-rsno longer call it "sketch-bench's rqe-optimizer".Verification
cargo clippy --workspace --all-targets --all-features --locked -- -D warningsandcargo fmt --checkpass.cargo test -p rqe-optimizer --all-targetsand-p asap_plannerpass. That includes the runner's tests, and the planner now builds against the current solver rather than the old pinned rev.anomaly-base-0,perf-base-0andanomaly-drilldown_endpoints-50.summarize_case_studies.pyregeneratessummary.mdbyte for byte after black.The companion sketch-bench PR removes the moved files. Merge this one first.
🤖 Generated with Claude Code