Skip to content

fix(speculate): write the merge state before dispatching the batch - #585

Draft
behinddwalls wants to merge 1 commit into
mainfrom
preetam/speculate-write-before-dispatch
Draft

fix(speculate): write the merge state before dispatching the batch#585
behinddwalls wants to merge 1 commit into
mainfrom
preetam/speculate-write-before-dispatch

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

applyOutcome published a batch to the merge topic before writing BatchStateMerging, so a lost compare-and-swap could leave Runway acting on an outcome that was never recorded. That ordering existed to avoid a stall, and the stall is real: nothing re-drives a batch stuck in Merging. Process self-heals only terminal and Created batches, finalize walks only heads that are still speculating, and the sole production reader of BatchStateMerging is the cancel controller — so a batch written Merging whose dispatch never went out would sit there forever.

Giving that stall a repair path lets the write come first, which is the ordering the rest of the state machine already wants.

What?

applyOutcome is restructured into decide-state → recover → write → dispatch. The terminal bool falls out: a second switch mirrors the first and dispatches merge or conclude once the state write has landed.

recoverable is hoisted above the switch so a cascade-decided merge gets a recovery message too, not just a cascade-decided failure. It needs one for the same reason: the write drops the batch out of the speculating set, and Process's self-heal only ever names the trigger batch.

Process gains a BatchStateMerging branch that re-sends the dispatch through the new dispatchMerge helper. That keeps the stable IntentID, the inverse of fanout's UniqueID — for conclude a stable ID would suppress the repair, for merge it is what stops Runway merging the batch twice.

One side benefit: a lost state CAS now means the dispatch is never sent at all, narrowing the window where a cancelled batch has a live merge request against it.

Test Plan

bazel test //submitqueue/... //platform/... — 68 tests pass

New coverage: the dispatch follows the state write; a lost CAS publishes nothing; a cascade-merged batch gets its recovery signal before the write; Process on a Merging batch re-dispatches; dispatchMerge reuses one message ID per batch.

TestProcess_MergingRunsButDoesNotAct asserted the old behaviour — that a Merging batch publishes nothing — and is replaced by TestProcess_MergingSelfHeals.

Stack

  1. @ fix(speculate): write the merge state before dispatching the batch #585
  2. feat(speculate): hold speculating until the batch can be sent to merge #586

## Summary

### Why?

`applyOutcome` published a batch to the merge topic before writing `BatchStateMerging`, so a lost compare-and-swap could leave Runway acting on an outcome that was never recorded. That ordering existed to avoid a stall, and the stall is real: nothing re-drives a batch stuck in `Merging`. `Process` self-heals only terminal and `Created` batches, `finalize` walks only heads that are still speculating, and the sole production reader of `BatchStateMerging` is the cancel controller — so a batch written `Merging` whose dispatch never went out would sit there forever.

Giving that stall a repair path lets the write come first, which is the ordering the rest of the state machine already wants.

### What?

`applyOutcome` is restructured into decide-state → recover → write → dispatch. The `terminal` bool falls out: a second switch mirrors the first and dispatches merge or conclude once the state write has landed.

`recoverable` is hoisted above the switch so a cascade-decided *merge* gets a recovery message too, not just a cascade-decided failure. It needs one for the same reason: the write drops the batch out of the speculating set, and `Process`'s self-heal only ever names the trigger batch.

`Process` gains a `BatchStateMerging` branch that re-sends the dispatch through the new `dispatchMerge` helper. That keeps the stable `IntentID`, the inverse of `fanout`'s `UniqueID` — for conclude a stable ID would suppress the repair, for merge it is what stops Runway merging the batch twice.

One side benefit: a lost state CAS now means the dispatch is never sent at all, narrowing the window where a cancelled batch has a live merge request against it.

## Test Plan

✅ `bazel test //submitqueue/... //platform/...` — 68 tests pass

New coverage: the dispatch follows the state write; a lost CAS publishes nothing; a cascade-merged batch gets its recovery signal before the write; `Process` on a `Merging` batch re-dispatches; `dispatchMerge` reuses one message ID per batch.

`TestProcess_MergingRunsButDoesNotAct` asserted the old behaviour — that a `Merging` batch publishes nothing — and is replaced by `TestProcess_MergingSelfHeals`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant