perf(recording): Windows fragmented M4S readback overlap and GPU scale by AatmanAJ · Pull Request #3586 · CapSoftware/Cap

AatmanAJ · 2026-10-29T23:34:24Z

Summary

Fragmented M4S (instant + studio default on Windows): Split D3D→FFmpeg readback from H.264 encode on a bounded staging channel so capture can keep producing while the prior frame encodes. When the encoder would have used internal swscale (scale and/or BGRA→YUV), enable prefer_external_conversion and feed frames through AsyncConverterPool (D3D11 video processor when available) with queue_preconverted_frame_to, overlapping GPU/CPU conversion with readback and libx264/NVENC work.
Instant mode capture sizing: Apply the same max-width capture constraint on Windows as macOS so VideoInfo matches the encode resolution instead of capturing native 4K and scaling twice (capture scaler + encoder swscale).
Display resize scaling: Prefer D3D11 VideoProcessorBlt for dimension mismatches and read back only at the target resolution; fall back to the existing CPU swscale path when the GPU scaler cannot be created.

Not in this PR (follow-up)

Wiring Media Foundation H.264 from WindowsMuxer into fragmented M4S (zero-copy texture encode for studio/instant segments).
Eliminating the final BGRA CPU readback entirely (would require MF or hardware FFmpeg with D3D surfaces).

Test plan

Windows instant recording at 1920p cap on a >1080p display: verify lower CPU vs main, stable frame pacing, acceptable quality.
Windows studio fragmented recording (display): same smoke test.
Resize display during recording: confirm GPU scale path logs and no corruption.
cargo check -p cap-recording (CI).

Confidence Score: 5/5

The PR appears safe to merge, with all previous findings addressed and no new actionable issues found.

Summary

This PR overlaps Windows frame readback, conversion, and encoding, and adds GPU scaling with a CPU fallback.

Windows fragmented recordings stage frames before converting and encoding them.
Windows instant recording limits capture size to the output dimensions.
Windows display capture tries D3D11 scaling before CPU scaling.
RGB-to-YUV conversion now uses BT.709 color settings.
Linux recording checks prepare the kernel and Hyprland display setup.
The export check counts GIF frames by walking the image blocks.
The codec check uses a higher bitrate for its smallest output size.

_{Reviews (5) · Last reviewed commit: "fix(ci): walk GIF blocks when measuring ..." · Reviewed by Greptile}

…cale Instant and studio fragmented M4S paths stage D3D readback ahead of encode, run pixel conversion through AsyncConverterPool when needed, scale mismatched capture sizes on the D3D11 video processor before readback, and align Windows instant max capture size with encode resolution to avoid redundant swscale.

vercel · 2026-10-29T23:34:29Z

Someone is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

greptile-apps · 2026-10-29T23:39:00Z

 use tokio_util::{future::FutureExt as _, sync::CancellationToken};
 use tracing::*;

+mod gpu_scale;


Windows cannot find the module

mod gpu_scale; looks for screen_capture/windows/gpu_scale.rs or screen_capture/windows/gpu_scale/mod.rs, but the new file is at screen_capture/gpu_scale.rs. Every Windows build fails with a missing-module error. Move the file under windows/, declare it in the parent module, or add an explicit path.

Suggested change

mod gpu_scale;

#[path = "gpu_scale.rs"]

mod gpu_scale;

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/recording/src/sources/screen_capture/windows.rs Line: 32 Comment: **Windows cannot find the module** `mod gpu_scale;` looks for `screen_capture/windows/gpu_scale.rs` or `screen_capture/windows/gpu_scale/mod.rs`, but the new file is at `screen_capture/gpu_scale.rs`. Every Windows build fails with a missing-module error. Move the file under `windows/`, declare it in the parent module, or add an explicit path. ```suggestion #[path = "gpu_scale.rs"] mod gpu_scale; ``` --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-29T23:39:00Z

+            PastFrames: 0,
+            FutureFrames: 0,
+            ppPastSurfaces: ptr::null_mut(),
+            pInputSurface: std::mem::transmute_copy(&input_view),


Unsafe call blocks compilation

scale_texture calls std::mem::transmute_copy outside an unsafe block. Once the module path is fixed, this causes another Windows compile error. Wrap the call in an explicit unsafe block or construct pInputSurface without the transmute.

Suggested change

pInputSurface: std::mem::transmute_copy(&input_view),

pInputSurface: unsafe { std::mem::transmute_copy(&input_view) },

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/recording/src/sources/screen_capture/gpu_scale.rs Line: 158 Comment: **Unsafe call blocks compilation** `scale_texture` calls `std::mem::transmute_copy` outside an `unsafe` block. Once the module path is fixed, this causes another Windows compile error. Wrap the call in an explicit `unsafe` block or construct `pInputSurface` without the transmute. ```suggestion pInputSurface: unsafe { std::mem::transmute_copy(&input_view) }, ``` --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-29T23:39:00Z

+        while let Some(converted) = pool.try_recv() {
+            let timestamp = Duration::from_micros(converted.frame.pts().unwrap_or(0).max(0) as u64);
+            let encode_start = std::time::Instant::now();
+            match encoder.lock() {
+                Ok(mut enc) => {
+                    if let Err(error) = enc.queue_preconverted_frame(converted.frame, timestamp) {


Recorded frames can go backward

The pool uses two to four independent workers, so a delayed worker can return an older frame after a newer one. drain_converted encodes results immediately and ignores converted.sequence. update_pts moves backward timestamps forward rather than restoring frame order, so recordings can show older screen content after newer content with incorrect timing. Preserve capture order, or discard stale results, in both this drain and the final drain.

Knowledge Base Used: Media encoding and muxing

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/recording/src/output_pipeline/win_fragmented_m4s.rs Line: 1138-1143 Comment: **Recorded frames can go backward** The pool uses two to four independent workers, so a delayed worker can return an older frame after a newer one. `drain_converted` encodes results immediately and ignores `converted.sequence`. `update_pts` moves backward timestamps forward rather than restoring frame order, so recordings can show older screen content after newer content with incorrect timing. Preserve capture order, or discard stale results, in both this drain and the final drain. **Knowledge Base Used:** [Media encoding and muxing](https://app.greptile.com/cap/-/custom-context/knowledge-base/capsoftware/cap/-/docs/media-encoding-and-muxing.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-29T23:39:00Z

+            AsyncConverterPool::from_config(conversion_config, pool_config)
+                .map_err(|error| anyhow!("Failed to create Windows M4S converter pool: {error}"))?,


Software recordings change colors

The external conversion path drops the encoder's BT.709 color setup. With libx264, the required output is YUV420P, which the D3D11 converter does not support. The pool therefore falls back to SwscaleConverter, which uses swscale's default BT.601 coefficients, while H.264 still labels the video BT.709. This shifts recorded colors compared with the previous path. Apply the same BT.709 limited-range setup to external RGB-to-YUV conversion.

Knowledge Base Used: Media encoding and muxing

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/recording/src/output_pipeline/win_fragmented_m4s.rs Line: 1107-1108 Comment: **Software recordings change colors** The external conversion path drops the encoder's BT.709 color setup. With `libx264`, the required output is `YUV420P`, which the D3D11 converter does not support. The pool therefore falls back to `SwscaleConverter`, which uses swscale's default BT.601 coefficients, while H.264 still labels the video BT.709. This shifts recorded colors compared with the previous path. Apply the same BT.709 limited-range setup to external RGB-to-YUV conversion. **Knowledge Base Used:** [Media encoding and muxing](https://app.greptile.com/cap/-/custom-context/knowledge-base/capsoftware/cap/-/docs/media-encoding-and-muxing.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-29T23:39:00Z

+                self.target_width,
+                self.target_height,
+            );
+            self.gpu_scaler_source = self.gpu_scaler.as_ref().map(|_| (src_width, src_height));


Failed GPU setup repeats

When try_new fails, gpu_scaler_source stays None. Every scaled frame then repeats device checks and processor creation before taking the CPU fallback. This adds repeated work on unsupported devices, especially now that capped Instant captures can need scaling throughout the recording. Cache the attempted source size even when creation fails, and retry only after the size or device changes.

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/recording/src/sources/screen_capture/windows.rs Line: 231 Comment: **Failed GPU setup repeats** When `try_new` fails, `gpu_scaler_source` stays `None`. Every scaled frame then repeats device checks and processor creation before taking the CPU fallback. This adds repeated work on unsupported devices, especially now that capped Instant captures can need scaling throughout the recording. Cache the attempted source size even when creation fails, and retry only after the size or device changes. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Point the nested gpu_scale module at the sibling source file and wrap transmute_copy in an unsafe block for Rust 2024.

Preserve capture sequence through the async converter pool, apply BT.709 limited-range coefficients for RGB swscale fallbacks, and cache failed GPU scaler setup per source size.

AatmanAJ · 2026-10-30T00:01:58Z

@greptile-apps please re-review. Addressed the open P1/P2 items in the latest push on this branch.

greptile-apps · 2026-10-30T00:05:27Z

+        pending_converted.insert(converted.sequence, converted);
+        while let Some(converted) = pending_converted.remove(&next_encode_sequence) {
+            encode_converted(converted)?;
+        }


Recording stops after one frame

next_encode_sequence never advances, so the default external-conversion path encodes only sequence 0. Every later converted frame stays in pending_converted until shutdown, which also checks only sequence 0. Recordings lose nearly all video while memory keeps growing.

Advance the sequence after encoding and handle missing results without waiting forever: AsyncConverterPool can drop submitted frames when its queues fill.

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/recording/src/output_pipeline/win_fragmented_m4s.rs Line: 1168-1171 Comment: **Recording stops after one frame** `next_encode_sequence` never advances, so the default external-conversion path encodes only sequence 0. Every later converted frame stays in `pending_converted` until shutdown, which also checks only sequence 0. Recordings lose nearly all video while memory keeps growing. Advance the sequence after encoding and handle missing results without waiting forever: `AsyncConverterPool` can drop submitted frames when its queues fill. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

…fter runtime failure

…skip lost ones

AatmanAJ · 2026-10-30T09:32:52Z

@greptile-apps review

…rness Hyprland 0.57 dropped hyprland.conf and only reads hyprland.lua, so the harness's monitor mode, session export and waybar were ignored and the output came up at vkms's preferred 1024x768. The screenshot scenario then reported screenshot_size=1024x768 instead of yes. Write the same settings as Lua and use the Lua form of the focus dispatcher.

…mera job The ubuntu-24.04 runner's 7.0.0-1017-azure kernel has no linux-modules-extra package, so apt-get failed before the camera test ran. That kernel ships videodev in its base modules (the desktop workflow's X11 camera step loads v4l2loopback there with only the headers), so install the extra modules only when videodev is missing and the package exists, install the headers v4l2loopback-dkms builds against, and fail clearly when videodev still cannot load.

…set bitrate At 80 kbps the paravirtualised VideoToolbox HEVC encoder on the CI macOS VMs occasionally left a blurred camera code cell at margin 0.17, below the reader's 0.18 threshold. 138,240 bps is the Web export preset (0.08 bpp), still below the Social preset the verify exports use.

AatmanAJ · 2026-10-30T10:39:22Z

@greptile-apps review

AatmanAJ · 2026-10-30T11:03:16Z

@greptile-apps review

greptile-apps Bot reviewed Oct 29, 2026

View reviewed changes

Aatman Jain added 2 commits October 29, 2026 23:54

fix: resolve Windows gpu_scale module path and unsafe transmute

767b081

Point the nested gpu_scale module at the sibling source file and wrap transmute_copy in an unsafe block for Rust 2024.

fix(recording): ordered M4S conversion and BT.709 swscale

e077503

Preserve capture sequence through the async converter pool, apply BT.709 limited-range coefficients for RGB swscale fallbacks, and cache failed GPU scaler setup per source size.

greptile-apps Bot reviewed Oct 30, 2026

View reviewed changes

AatmanAJ force-pushed the perf/win-fragmented-gpu-encode branch from 4e93812 to e077503 Compare October 30, 2026 00:26

Aatman Jain added 4 commits October 30, 2026 09:32

fix(recording): resolve Windows gpu_scale module and disable scaler a…

e9dd083

…fter runtime failure

fix(recording): reorder converted Windows M4S frames by sequence and …

06e675c

…skip lost ones

test(frame-converter): cover BT.709 swscale RGB conversion

9931c67

fix(frame-converter): use BT.709 for D3D11 RGB to YCbCr conversion

b6870e3

Aatman Jain added 3 commits October 30, 2026 10:28

fix(ci): walk GIF blocks when measuring the exported GIF's length

ea8bf7c

Navigation Menu

Repository navigation

perf(recording): Windows fragmented M4S readback overlap and GPU scale - #3586

perf(recording): Windows fragmented M4S readback overlap and GPU scale#3586
AatmanAJ wants to merge 11 commits into
CapSoftware:mainfrom
AatmanAJ:perf/win-fragmented-gpu-encode

AatmanAJ commented Oct 29, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

vercel Bot commented Oct 29, 2026

Uh oh!

greptile-apps Bot Oct 29, 2026

Uh oh!

greptile-apps Bot Oct 29, 2026

Uh oh!

greptile-apps Bot Oct 29, 2026

Uh oh!

greptile-apps Bot Oct 29, 2026

Uh oh!

greptile-apps Bot Oct 29, 2026

Uh oh!

AatmanAJ commented Oct 30, 2026

Uh oh!

greptile-apps Bot Oct 30, 2026

Uh oh!

AatmanAJ commented Oct 30, 2026

Uh oh!

AatmanAJ commented Oct 30, 2026

Uh oh!

AatmanAJ commented Oct 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	pInputSurface: std::mem::transmute_copy(&input_view),
	pInputSurface: unsafe { std::mem::transmute_copy(&input_view) },

		AsyncConverterPool::from_config(conversion_config, pool_config)
		.map_err(\|error\| anyhow!("Failed to create Windows M4S converter pool: {error}"))?,

Conversation

AatmanAJ commented Oct 29, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Not in this PR (follow-up)

Test plan

Confidence Score: 5/5

Summary

Uh oh!

vercel Bot commented Oct 29, 2026

Uh oh!

greptile-apps Bot Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

AatmanAJ commented Oct 30, 2026

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

AatmanAJ commented Oct 30, 2026

Uh oh!

AatmanAJ commented Oct 30, 2026

Uh oh!

AatmanAJ commented Oct 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

AatmanAJ commented Oct 29, 2026 •

edited by greptile-apps Bot

Loading