Skip to content

fix(core): a stuck seek in check or snapshot fails named before the protocol timeout - #6596

Open
miguel-heygen wants to merge 3 commits into
mainfrom
fix/seek-barrier-bounded
Open

miguel-heygen wants to merge 3 commits into
mainfrom
fix/seek-barrier-bounded

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

A seek waits for everything it started (video frames, frame sources, scene windows, motion blur, effects, image decodes) before a frame is captured. If one of those holds never settled, check and snapshot sat until Puppeteer's protocol timeout (180 s) fired, with a message that said nothing about why.

Now the caller can bound the wait, and the error names what is still pending:

seek did not settle within 170000 ms: still waiting on frame-source
  • __hfWaitForSeekCompletion({ timeoutMs }) ends at the caller's limit. Without timeoutMs it waits as before, so export renders keep the limits they already had: protocolTimeout, scaled by canvas size and settable by the user.
  • check (the overlap grid) and snapshot / settleCompositionSeek pass SEEK_SETTLES_BEFORE_PROTOCOL_TIMEOUT_MS, which is 170 s, 10 s under the protocol timeout their browser runs with. A seek that used to finish still finishes; one that used to die at 180 s with an opaque timeout now fails at 170 s, named. The grid runs all its seeks in one page call, so it spends one 170 s budget across all of them.
  • Every hold registers with its kind (registerSeekCompletion(promise, hold)), so the error can say which one is stuck.
  • Holds registered while the barrier waits share the same deadline; a stream of new holds can't extend it.

What it does not fix

  • It does not make a slow seek faster.
  • It does not fix the many-clips snapshot hang. That is a renderer crash, handled separately.

Tests

seek-dispatch.test.ts:

  • A hold that never settles ends at the caller's deadline and is named.
  • Holds registered while waiting end at the same deadline, in under 1 s.
  • With no timeoutMs, a seek waits as long as its holds take (fake timers, 30 min).
  • Holds that settle in time resolve as before.

The first two fail on main, where the wait had no deadline. The CLI capture tests now expect the limit to be passed.

Results on Linux:

  • Core seek tests: 3 runs in a row, 14/14 each time.
  • Full CLI unit project: 418 files, 5810 tests.
  • Fallow audit: clean.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits · head c74bb142

The 30 s default also applies to export renders, not just check/snapshot. That turns renders that are slow but valid into failures. Details inline.

Scope and checks
  • Read all 7 files. Also checked every caller of __hfWaitForSeekCompletion: engine frameCapture/drawElementService/riskFrames/videoFrameInjector, cli captureCompositionFrame/checkBrowser/motionShot, and studio useShadowPreviewReload. None of them pass timeoutMs.
  • The producer's virtual-time shim freezes Date.now while the barrier waits, so deadline - Date.now() still works out to 30 s of real time. That part is fine.
  • Ran locally on the head: seek-dispatch, init.seekOrder and frameSources tests (35 pass); core tsc is clean. CI was still running when I posted.

(completion) => options?.videoFrames !== false || !_videoFrameCompletions.has(completion),
([, hold]) => options?.videoFrames !== false || hold !== "video-frame",
);
const timeoutMs = options?.timeoutMs ?? SEEK_SETTLE_CAP_MS;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: every engine/producer render goes through this default. waitForPendingSeekCompletion (engine frameCapture.ts:3190) calls __hfWaitForSeekCompletion() with no options. The 180 s in the PR body is the CLI's Puppeteer default (captureTimeout.ts). Renders run under protocolTimeout, which is 300 s by default, scales with canvas area up to 30 min (scaleProtocolTimeoutForComposition exists because slow-but-valid seek/paint calls on large canvases were being killed), and can be raised by the user.

So with this change, a render that used to finish now fails after 30 s, with no way to raise the limit. Example: a frame source whose ready loads a heavy model or scene for 45 s on a cold worker. docs/guides/frame-sources.md says "ready holds initialization and export capture" and documents no limit. The same goes for a slow software-GL vfx/motion-blur paint at 4K.

Fix: have the engine pass timeoutMs derived from its effective protocol timeout, minus a margin so the named error still wins the race. Keep 30 s, or the CLI's budget, for check/snapshot. If 30 s is meant to be the new contract for renders too, document it in the frame-sources guide and make it configurable.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in ce1d96b. The core wait no longer has a default limit: without timeoutMs it waits as before, so renders keep their scaled protocolTimeout. Only the CLI's check grid and settleCompositionSeek (snapshot) pass a limit, 170 s, 10 s under the 180 s protocol timeout their browser runs with. Anything that finished before still finishes; a seek that hit the opaque 180 s timeout now fails at 170 s and names the stuck hold.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed in ce1d96b. The core wait has no default limit anymore, so renders keep their scaled protocolTimeout. Only check and snapshot pass 170 s.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 143 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (2)

  • seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true
  • seqrotatenudge-none-pct-r0-nested-z100: tracking 0.02, pressJump 0, drop 0, reload 3.03, render 0.06, renderKey -, undo false, teleport true / tracking 0.02, pressJump 0, drop 0, reload 0, render 0.06, renderKey -, undo true, teleport true / tracking 0.02, pressJump 0, drop 0, reload 0, render 0.06, renderKey -, undo true, teleport true

@miguel-heygen miguel-heygen changed the title fix(core): a seek that never settles ends after 30 s and names what it waits on fix(core): a stuck seek in check or snapshot fails named before the protocol timeout Oct 27, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 1 resolved, 0 open, 0 new · c74bb14 → ce1d96b. CI is green; the first CI run lost edit-accuracy shard 4 to a stalled Chrome download, and the second run on this head passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant