Skip to content

feat: jointly price supplier and worker configurations - #939

Merged
zzylol merged 37 commits into
mainfrom
feat/supplier-dependent-direct-cost
Oct 26, 2026
Merged

zzylol merged 37 commits into
mainfrom
feat/supplier-dependent-direct-cost

Conversation

@zzylol

@zzylol zzylol commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem and resulting behavior

A consumer can have different costs and output statistics depending on its selected supplier or state configuration. Ingestion suppliers can also cross worker-count boundaries. The direct-cost extractor previously kept one priced interface per physical candidate and discarded differing interfaces from that direct proposal.

This PR retains finite local pricing options and selects them jointly with supplier connections in the same MILP. All readers of a shared physical candidate must agree on one pricing option. The objective still charges its owned work/state once, including the executions required by its consumers.

Based on #935; this PR contains the supplier/configuration/worker extension separately. Complex schedule extensions are not included.

Scenarios handled

Scenario Before this PR After this PR
Supplier output statistics change a downstream operation's cost Only the first certified interface entered a direct proposal. Both interfaces and their downstream bindings participate jointly; forced choices agree with the independent DAG evaluator.
One supplier candidate itself has multiple priced configurations A consumer inherited only one interface. Compatibility records the supplier candidate and its pricing option; differences propagate downstream.
Equivalent ingestion input computations require different worker counts One worker bucket was retained. Separate options preserve their upstream-work certificates and compaction delay. A pricing-kernel regression compares a raw scan with an identity projection of the same input, with 1/2 worker buckets, against the evaluator. This fixture does not claim runtime installation support for a Project-pane chain.
Two readers share one producer but prefer conflicting interfaces A fixed profile could not express the choice. At most one pricing option activates for that physical producer; readers cannot mix incompatible views or double-charge it.
Potential cycles among candidate connections A topological-only extension rejected the whole pricing domain. Bounded local-interface propagation covers legal unary paths; final MILP ordering excludes cyclic executions.
Rejected plans, reader requirements, and resource capacities Rules use physical candidate IDs. Internal pricing-option expansion preserves those identities and projects selections/cuts back correctly.
SQL distinct rollup, three SQL quantile groupings, and PromQL quantile rollup Existing production direct-cost acceptance. Tests require an actual joint-price solve and a validated direct-cost plan, while allowing a cheaper exact result.

Pricing-kernel fixtures validate cost propagation and identity mapping; they do not introduce new logical-equivalence rules.

Key code interfaces added / modified

  • extract_price_options: derive and deduplicate certified local pricing interfaces and compatible supplier bindings using existing operator pricing rules.
  • JointPriceProfile / CandidatePriceOption: local resource/work profiles and producer-configuration compatibility.
  • solve_joint_price_proposal_with_metrics: lift options into the MILP, preserve one physical identity, and reconstruct original candidate IDs.
  • CandidateConnectionReport.joint_price_solve_calls: distinguish joint-price selection from fixed-profile direct fallback.

Major algorithm changes

  • Reuse certified local pricing for each supplier interface, without enumerating complete workload plans.
  • Select at most one price option per physical candidate; map compatible edges, reader/resource constraints, and rejected-assignment cuts into the internal option graph.
  • Use dependency order for acyclic catalogs and at most N propagation rounds for potential cycles. Query-time interfaces omit cumulative work that is irrelevant to ingestion worker sizing.
  • Reserve an interface-generation budget and report explicit fallback on domain/budget limits. The bounded production seed uses the first interchangeable execution copy, retaining all its pricing options; full search retains every copy. Restricted bounds are never reported as full-catalog bounds.

Scope and limits

This extends the currently certified unary exact/sketch/pane cost domain. Multi-input pricing, wider-scan sharing that needs a selected maximum-lookback witness, unsupported accuracy/retention contexts, and schedules outside existing bounded encodings still require evaluator refinement. Worker count is derived from selected upstream work under the current evaluator, not a new runtime scheduling decision. Local interfaces are capped at 50,000 and generation is deadline-bound. The proposed general DP/Pareto frontier algorithm remains unimplemented.

Validation

  • 357 plan-selection library tests passed.
  • 22 integration tests passed: tiny-catalog equivalence (14), production rollups (3), controlled/production rollups (2), sketch rollup planning (3).
  • Strict clippy passed for plan-selection library/tests and production rollup integration; formatting and diff checks passed.
  • Independent review of production modeling/projection changes found no blocker. Test author reviewed production code; implementation author ran acceptance.

Latest local production acceptance (modeled cost per second):

Workload Selected cost Reported planning time Status
PromQL quantile grouping rollup 1.0799730566743272e-11 30.24s Validated incumbent at time limit; exact wins
SQL distinct region/service rollup 1.7931682458222749e-7 30.02s Validated incumbent at time limit
SQL quantile three groupings 1.7377853804201427e-7 0.43s Optimum reported for this generated finite model

All three recorded one joint-price solve and a validated direct-cost plan, with no direct fallback reason. These are model/evaluator estimates, not runtime measurements or guarantees of future planning time. Full CI is pending.

CI regression scenarios

  • Interrupted evaluation archives: only completed selector offers enter the evaluated archive. Compiler deadlines during frozen or complete-catalog evaluation no longer create a physical plan without a cost or rejection. The deterministic timeout regression failed before this correction; all 10 objective-search tests and the unchanged example-1 archive-accounting test pass.
  • Frequency executor fixture: execute the explicit exact and UnivMon local rule candidates instead of arbitrary sized states visited by a bounded optimization search. The failed distinct-count run used heap 338,000 with failure probability 0.0001, although the test requested 0.1. The explicit candidate uses heap 338 and the requested 0.1. Default runtime memory limits and numerical-result assertions are unchanged; a new assertion checks the requested failure probability. All three exact/UnivMon execution cases pass (0.03s).

CI repair and current-main integration

Scenario: three repeated/different-window SQL COUNT DISTINCT outputs with an explicit confidence target and a 10-second planning budget. A feasibility seed previously kept choosing estimators with an unresolved confidence contract. The seed now excludes those proposals; the complete catalog and final accuracy validation remain unchanged, and there is no approximate-candidate price surcharge. Includes latest main (#938 family capability consistency). This preserves the original test budget and accuracy requirements.

Candidate-cost overflow recovery

Scenario: Example 2 can select a physical configuration whose analytical cost arithmetic overflows. Previously that stopped search before a usable incumbent. Scalar-price search now skips the unpriceable assignment, records cost_not_representable, and continues looking for a validated plan. Unknown cost is not infeasibility: exclusions of unpriceable assignments invalidate full-domain optimality/infeasibility claims and reset the reported bound conservatively. Other evaluator errors remain fatal. The new regression covers both a later feasible candidate and an entirely unpriceable domain; Example 2 retains its original requirements and budget.

TopK contract regression

Scenario: a temporal TopK search visits a tighter error allocation first, with heap capacity 200 instead of the default 100. The integration test now verifies every visited heap-backed candidate preserves the requested k, minimum capacity, entity identity and update expression, rather than requiring a fixed first-candidate capacity. Default sizing remains covered by its unit test. The original assertion failed locally; all 13 tests in promql_to_post_asap pass after this correction.

Stable sketch execution and coverage fixtures

Scenarios: a search archive may contain tightened, large sketch configurations or multiple KLL capacities. Rollup execution and coverage tests now use the existing explicit local-rule fixtures, so they test all intended sketch rules and coverage semantics without depending on solver visitation. The rollup memory limit and result assertions remain unchanged. The former fixtures failed locally; corrected rollup (3) and coverage (8) tests pass. The integration package was exercised in batches: 330 passed, 23 pre-existing ignored; affected tests also pass strict clippy. Full remote CI passed before merge.

Combined delivery with #940

#940 was merged into this branch after its full CI passed. This PR now also delivers multi-input supplier-dependent pricing and mixed evaluation schedules, consolidates schedule encoding into one implementation, and preserves feasibility fallback when an optional direct profile is unsupported. The combined code tree is identical to the validated #940 head; the final combined branch passed full CI before merging to main.

zzylol added 26 commits October 24, 2026 13:43
…p-decomposition

# Conflicts:
#	docs/design_docs/proposals/joint-planning-search.md
…tion' into feat/supplier-dependent-direct-cost
@zzylol
zzylol changed the base branch from docs/joint-planning-dp-decomposition to main October 26, 2026 19:50
* feat: jointly price multi-input plans and mixed schedules

* fix(planner): preserve search after unsupported direct pricing

* refactor(milp): keep one mixed-schedule representation and encoder
@zzylol
zzylol merged commit 3ac566a into main Oct 26, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant