Repository navigation
feat: jointly price supplier and worker configurations - #939
Merged
Merged
Conversation
…p-decomposition # Conflicts: # docs/design_docs/proposals/joint-planning-search.md
…tion' into feat/supplier-dependent-direct-cost
…r-dependent-direct-cost
…r-dependent-direct-cost
…r-dependent-direct-cost
zzylol
changed the base branch from
docs/joint-planning-dp-decomposition
to
main
October 26, 2026 19:50
* feat: jointly price multi-input plans and mixed schedules * fix(planner): preserve search after unsupported direct pricing * refactor(milp): keep one mixed-schedule representation and encoder
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and resulting behavior
A consumer can have different costs and output statistics depending on its selected supplier or state configuration. Ingestion suppliers can also cross worker-count boundaries. The direct-cost extractor previously kept one priced interface per physical candidate and discarded differing interfaces from that direct proposal.
This PR retains finite local pricing options and selects them jointly with supplier connections in the same MILP. All readers of a shared physical candidate must agree on one pricing option. The objective still charges its owned work/state once, including the executions required by its consumers.
Based on #935; this PR contains the supplier/configuration/worker extension separately. Complex schedule extensions are not included.
Scenarios handled
Pricing-kernel fixtures validate cost propagation and identity mapping; they do not introduce new logical-equivalence rules.
Key code interfaces added / modified
extract_price_options: derive and deduplicate certified local pricing interfaces and compatible supplier bindings using existing operator pricing rules.JointPriceProfile/CandidatePriceOption: local resource/work profiles and producer-configuration compatibility.solve_joint_price_proposal_with_metrics: lift options into the MILP, preserve one physical identity, and reconstruct original candidate IDs.CandidateConnectionReport.joint_price_solve_calls: distinguish joint-price selection from fixed-profile direct fallback.Major algorithm changes
Scope and limits
This extends the currently certified unary exact/sketch/pane cost domain. Multi-input pricing, wider-scan sharing that needs a selected maximum-lookback witness, unsupported accuracy/retention contexts, and schedules outside existing bounded encodings still require evaluator refinement. Worker count is derived from selected upstream work under the current evaluator, not a new runtime scheduling decision. Local interfaces are capped at 50,000 and generation is deadline-bound. The proposed general DP/Pareto frontier algorithm remains unimplemented.
Validation
Latest local production acceptance (modeled cost per second):
All three recorded one joint-price solve and a validated direct-cost plan, with no direct fallback reason. These are model/evaluator estimates, not runtime measurements or guarantees of future planning time. Full CI is pending.
CI regression scenarios
CI repair and current-main integration
Scenario: three repeated/different-window SQL COUNT DISTINCT outputs with an explicit confidence target and a 10-second planning budget. A feasibility seed previously kept choosing estimators with an unresolved confidence contract. The seed now excludes those proposals; the complete catalog and final accuracy validation remain unchanged, and there is no approximate-candidate price surcharge. Includes latest main (#938 family capability consistency). This preserves the original test budget and accuracy requirements.
Candidate-cost overflow recovery
Scenario: Example 2 can select a physical configuration whose analytical cost arithmetic overflows. Previously that stopped search before a usable incumbent. Scalar-price search now skips the unpriceable assignment, records
cost_not_representable, and continues looking for a validated plan. Unknown cost is not infeasibility: exclusions of unpriceable assignments invalidate full-domain optimality/infeasibility claims and reset the reported bound conservatively. Other evaluator errors remain fatal. The new regression covers both a later feasible candidate and an entirely unpriceable domain; Example 2 retains its original requirements and budget.TopK contract regression
Scenario: a temporal TopK search visits a tighter error allocation first, with heap capacity 200 instead of the default 100. The integration test now verifies every visited heap-backed candidate preserves the requested k, minimum capacity, entity identity and update expression, rather than requiring a fixed first-candidate capacity. Default sizing remains covered by its unit test. The original assertion failed locally; all 13 tests in
promql_to_post_asappass after this correction.Stable sketch execution and coverage fixtures
Scenarios: a search archive may contain tightened, large sketch configurations or multiple KLL capacities. Rollup execution and coverage tests now use the existing explicit local-rule fixtures, so they test all intended sketch rules and coverage semantics without depending on solver visitation. The rollup memory limit and result assertions remain unchanged. The former fixtures failed locally; corrected rollup (3) and coverage (8) tests pass. The integration package was exercised in batches: 330 passed, 23 pre-existing ignored; affected tests also pass strict clippy. Full remote CI passed before merge.
Combined delivery with #940
#940 was merged into this branch after its full CI passed. This PR now also delivers multi-input supplier-dependent pricing and mixed evaluation schedules, consolidates schedule encoding into one implementation, and preserves feasibility fallback when an optional direct profile is unsupported. The combined code tree is identical to the validated #940 head; the final combined branch passed full CI before merging to main.