Skip to content

feat(preview): an unplayable video shows its own frame while its copy is made - #6589

Open
miguel-heygen wants to merge 7 commits into
mainfrom
feat/preview-still-frame
Open

miguel-heygen wants to merge 7 commits into
mainfrom
feat/preview-still-frame

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #6586.

What

While the copy of a video the browser cannot play is being made (HEVC without a decoder, ProRes), the video shows its own frame at the playhead instead of black, and the existing copy-progress report now runs at Full quality for those videos, as it already did for lighter copies below Full.

  • The server cuts one JPEG of the source at the asked time (?hf-proxy=…&hf-proxy-still=<seconds>), to a tenth of a second, sized like the copy's box, with one ffmpeg -frames:v 1. It sits behind the same eligibility gate as the copy and is cached next to it, so a second ask costs nothing.
  • The runtime sets that URL as the video's poster on each ask while the video is at the playhead and has no poster of its own, and removes it when the copy has loaded. A failed copy leaves the frame up.
  • previewCopies counts the copy a file cannot play without separately from the lighter ones: it is reported at any level, and copiesReady (which the host reads for lighter copies) is unchanged.

Why

On a machine with no HEVC decoder a project of many HEVC clips showed only its captions over black until the copies were made, with nothing to say why. The copy-progress report only ran below Full quality, so at Full there was no sign of progress either.

Measured

24 separate 1080x1920 HEVC clips of 8 s, back to back, captions over them, in Studio served by the built CLI on a Linux box whose Chrome has no HEVC decoder (the same as a Windows machine without the HEVC extension). Seconds from the first clip appearing to its first picture, each run with a cold copy cache:

Run Before (the copy plays) After (the still arrives)
night 5.9 s 0.13 s
paper 8.8 s 1.8 s
paper, repeated n/a 5.3 s

The wait for the copy is the same in both, and the box was shared, so the spread follows its load: the still is a separate ffmpeg that competes for the same CPU as the copies. In every run the still came before the copy.

Before

Two seconds after the first clip appears (night, then paper): the clip is black, only its caption shows.

before, night
before, paper

After

The same moment: the clip's own frame stands in until the copy lands.

after, night
after, paper

Tests

  • mediaProxy.test.ts: the poster is set for a video at the playhead and not for an idle one, carries the source time, and is removed when the copy loads.
  • previewCopies.test.ts: the unplayable file's copy is counted at Full, a lighter one is not, and copiesReady ignores the former.
  • preview.test.ts: the route serves the JPEG behind the same gate and refuses a bad time or a request without the proxy param.
  • proxyTranscoder.test.ts: a real ffmpeg cuts a JPEG, and the same tenth of a second finds the cached one.
  • The poster test fails with the poster disabled.

heygengenesis[bot]
heygengenesis Bot previously approved these changes Oct 27, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 0 important · 2 nits · head 87493be9

The still route sits behind the same eligibility gate as the copy, and the poster is only set for non-light copies and cleared on load or when the copy stops being live. Two nits are inline.

Scope and checks
  • Read all 12 files. Also checked other poster readers in the runtime (sceneUrls and the asset-failure diagnostic in init.ts): both are unaffected. The retry path now also picks up failed unplayable copies at Full, through swapToProxy.
  • Ran locally on the head: core mediaProxy/previewCopies tests (66 pass); full core suite (5465 pass; the only failures are 3 studio-api files that fail on import because studio-server isn't built here, unrelated); full studio-server suite (1555 pass); resolveStill with real ffmpeg (ran, not skipped). oxlint, oxfmt and tsc are clean for core and studio-server.
  • Because the PR is stacked on #6586, the main CI workflow hasn't run. The Comments, regression, preview-regression and Player perf runs passed on this head. Required CI (including the core size budget in Build) runs after the retarget to main.
  • I couldn't check how the desktop host shows a preview-copies report at Full, because hyperframes-internal is outside my access.

const source = canonicalizeProxySource(projectDir, absoluteSourcePath);
const at = Math.max(0, Math.round(atSeconds * 10) / 10);
const key = fastCopyKey(source.sourcePath, PROXY_PARAMS_VERSION, `still@${at}`, box);
const cachePath = join(source.projectDir, CACHE_DIR_NAME, `${key}.jpg`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: proxyCache.ts only budgets and idles out names whose extension is in PROXY_EXTENSIONS (the copy variants), so finished <key>.jpg stills are never evicted. Every cold copy of an unplayable clip leaves one behind in .transcode-cache until hyperframes clean runs. Each one is small, but a still for a source that later changes is never reclaimed. To fix, add .jpg to the swept set, or delete the still once its copy lands.

box: PreviewProxyBox | null,
): void {
const at = sourceTimeAtPlayhead(el);
if (at === undefined || stills.has(el) || el.hasAttribute("poster")) return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: stills.has(el) (and the poster you just set) pins the frame from the first ask. If someone scrubs inside the same clip while the copy is still being made, the poster keeps showing the old time instead of the frame at the playhead. That's still better than black, but it doesn't match the body's "on each ask". If you want it to follow the playhead, re-set the poster when the rounded at changes and stills.has(el) is true.

Base automatically changed from fix/proxy-playhead-first to main October 27, 2026 20:32
@miguel-heygen
miguel-heygen dismissed heygengenesis[bot]’s stale review October 27, 2026 20:32

The base branch was changed.

@github-actions

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 138 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

  • seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant