Repository navigation
memo-milp misses rh8-q1h60 at 280 s (1.34e-05 vs 3.83e-06): suspected decode cut of a valid stricter site threshold #928
Description
Problem, by example
rh8-q1h60: RangeHeavy 8 users, the quantile queries over 1h at 60 s steps. Five queries share one SummaryMerge; query 4 is a ratio of two quantiles (q0.9 / q0.5) over that shared state. Fargate prices, raw data retained, guarantees enforced:
stage_pipeline --workload rh8-q1h60.json \
--deployment-capabilities '{"raw_data_retained":true,"query_time_results_retained":false}' \
--cost-model aws-fargate-us-east-1 --enforce-accuracy-guarantees \
--search memo-milp --milp-time-limit-s 280| cost per s | rounds | gap | |
|---|---|---|---|
| legacy reference | 3.832852e-06 | – | – |
| memo-milp on main (1800 s limit) | 3.832840e-06 | 97 | 0 (772 s) |
| memo-milp on the integration branch 04151ce (280 s limit) | 8.53e-06 (other runs 1.34e-05, the warm start) | 110–153 | 0.71, time limit |
| memo-milp after the PR-B merge, 0f8bf09 (280 s limit) | 1.342e-05 | 162 | 0.814, time limit |
Expected: the 3.83e-06 plan, certified, well within 280 s.
Suspected cause, and what the diagnosis found
Suspicion: solution decoding evaluates one site threshold per choice set, and when Stage 3 rejects it, the no-good cut removes the whole choice set, including the same choices at a stricter (larger) threshold that would be valid. That would also allow a wrong certificate: a valid plan cut away, then a bound that no longer covers it.
Diagnosis so far (restricted-model runs on 04151ce):
- The optimum's choice set is feasible in the model at 3.832831e-06 <= Stage 3 3.832840e-06.
- On a neighbouring set, round 0 picks site threshold 0 at 2.3959e-06 (Stage 3 rejects: AccuracyTargetMissed on query 4, relative bound 0.105); round 1 picks threshold 1 of the same set at 3.832831e-06, which Stage 3 accepts. So thresholds are separate versions and the no-good cut is per version there: that case does not lose the valid plan.
- The time is spent elsewhere: many choice sets at objective 2.3959e-06 to 2.4904e-06 (different pane candidates under the ratio) each fail the ratio's accuracy target at the smaller threshold and are cut one per round, ~1.8 s each. The pair accuracy rule (
pair_guarantee_rules, model.rs) only fires when every threshold version fails, so it rules none of them out.
The decode-threshold mechanism is not confirmed. This issue stays open until a test shows either that it cannot happen or that it does and is fixed.
Proposal
- A failing-first test: a choice set rejected at one site threshold and valid at a stricter one; assert the loop still returns (and certifies) the stricter plan.
- The pair accuracy rule checks each child choice pair at its own site thresholds plus the largest size, ruling out a version only if it fails at its own and the largest size, tied to the site threshold variable
u, not to the choice set.
Scope and acceptance
- rh8-q1h60 certifies at <= 3.832852e-06 within 280 s.
- The fixed regression set keeps its costs (q24h60, w840*, i840, j839 raw/no raw);
lower_boundtests and the 20-seed three-way pass. - The new test from step 1.
Status and related work
- A WIP fix for step 2 is on
fix/memo-rh8(5c28842, on top of the q24h60 pair rule ee17281 fromfix/memo-q24h-ingestion): rh8-q1h60 certifies at 3.832840e-06 in 17.5 s, 19 rounds; a failing-first testthe_first_solution_for_a_ratio_beside_three_quantiles_meets_its_targetis added. Not yet gated or reviewed, and it must be checked for exactness (per-version exclusion tied tou). - Part of Tracking: plan every whole evaluation workload within 3 minutes (exact pruning, memo-group search) #879; follows the memo-milp PR-B merge.
Activity
Memo MILP search: clusters, solution decoding, warm start, accuracy r…