Video Capture Pipeline: WGC, D3D11, and Media Foundation

Flashback’s video capture pipeline is built entirely on Windows-native APIs: Windows Graphics Capture (WGC) acquires frames as GPU textures, Direct3D 11 manages the texture lifetime, a Video Processor MFT converts and scales the frames, and a hardware or software H.264 encoder MFT produces the final H.264 bitstream that IMFSinkWriter muxes into an MP4 container. All GPU-to-GPU copies mean the frame data rarely touches system memory.

The capture pipeline is Windows-only. The Rust modules compile stubs for other targets, but all real capture logic lives inside #[cfg(target_os = "windows")] blocks in capture.rs.

Frame Acquisition with Windows Graphics Capture

WGC is Microsoft’s sanctioned screen-capture API, available from Windows 10 1903 onwards. Flashback creates a Direct3D11CaptureFramePool in free-threaded mode (so the FrameArrived callback fires on a system thread pool thread rather than a UI thread) and starts a GraphicsCaptureSession against either a monitor handle or an application window handle.

let frame_pool = Direct3D11CaptureFramePool::CreateFreeThreaded(
    &d3d_device,
    DirectXPixelFormat::B8G8R8A8UIntNormalized,
    2,          // two buffered frames
    size,       // capture dimensions (rounded to even)
)?;
let session = frame_pool.CreateCaptureSession(&item)?;
let _ = session.SetIsBorderRequired(false); // no yellow highlight border

WGC delivers frames as IDirect3D11Texture2D surfaces in BGRA format. Each FrameArrived callback retrieves the surface, extracts the underlying ID3D11Texture2D via IDirect3DDxgiInterfaceAccess, copies it to a slot in a private ring of textures owned by Flashback, and then releases the WGC frame immediately — WGC reuses its own textures as soon as the frame is closed.

Frame-Rate Limiting

WGC delivers frames whenever the display surface changes — it is inherently variable-rate. Flashback applies a per-frame timestamp check (keep_frame) before copying the texture, discarding frames that arrive faster than the configured FPS target. This avoids over-encoding and keeps the encoder load bounded at high refresh rates.

Constant-Frame-Rate Pacing

Clips are recorded at a strict CFR. The FrameArrived handler only ever updates the “latest texture” slot; a separate CFR pacing loop (running on the capture thread) wakes up once per frame-slot and emits whichever texture was most recently received, duplicating it if no new frame arrived since the last slot. Duplicated frames encode as near-zero-cost P-frames (“skip frames”) in Baseline H.264.

while emitted < cur {
    emitted += 1;
    let pts = cfr_pts(emitted, fps);
    let dur = cfr_pts(emitted + 1, fps) - pts;
    let _ = e.emit_paced(pts, dur);
}

Even-Dimension Enforcement

NV12 and H.264 require that both width and height be even. Game windows can have odd dimensions, which would cause MF_E_INVALIDMEDIATYPE. Flashback rounds both dimensions down to the nearest even value before creating the frame pool:

size.Width  = size.Width.max(2)  & !1;
size.Height = size.Height.max(2) & !1;

D3D11 Texture Pipeline

Flashback creates a single ID3D11Device with both D3D11_CREATE_DEVICE_BGRA_SUPPORT (required for WGC interop) and D3D11_CREATE_DEVICE_VIDEO_SUPPORT (required for sharing the device with Media Foundation hardware encoders). The same device object is wrapped as an IDirect3DDevice (WinRT) for the frame pool and as an IMFDXGIDeviceManager for Media Foundation. Multi-thread protection is enabled on the device context so that the FrameArrived callback thread and the CFR pacing thread can access it safely:

if let Ok(mt) = ctx.cast::<ID3D11Multithread>() {
    let _ = unsafe { mt.SetMultithreadProtected(true) };
}

Manual recording maintains a ring of six D3D11_USAGE_DEFAULT BGRA textures. The FrameArrived handler copies the WGC surface into the next ring slot (GPU→GPU, no CPU involvement) and marks that slot as “latest”. The CFR loop wraps the latest texture into a MFCreateDXGISurfaceBuffer and hands it to the IMFSinkWriter. The SinkWriter’s built-in Video Processor MFT performs BGRA→NV12 conversion and optional scaling entirely on the GPU. Instant Replay uses a separate BGRA ring and a channel to decouple the FrameArrived callback from the encoding pump thread. The pump runs a Video Processor MFT (BGRA→NV12 + optional scale) and then feeds the H.264 encoder MFT directly, bypassing IMFSinkWriter so encoded packets can be intercepted and pushed into the ring buffer.

Hardware Encoder Selection

At startup Flashback enumerates available H.264 encoders via MFTEnumEx and selects one according to a configurable preference (encoder_pref). The selection order is:

NVENC — NVIDIA’s hardware H.264 encoder (keyword match: “nvenc” / “nvidia”)
AMF — AMD’s Advanced Media Framework encoder (keyword match: “amf” / “amd”)
Quick Sync — Intel’s Quick Sync Video encoder (keyword match: “quick sync” / “intel”)
First available hardware encoder — fallback if the preferred vendor is not present
Software (MS H.264) — MFT_ENUM_FLAG_SYNCMFT | MFT_ENUM_FLAG_TRANSCODE_ONLY

fn pick_hw_encoder(pref: &str) -> Result<Option<IMFActivate>> {
    // Enumerate hardware H.264 encoders sorted by quality
    MFTEnumEx(
        MFT_CATEGORY_VIDEO_ENCODER,
        MFT_ENUM_FLAG_HARDWARE | MFT_ENUM_FLAG_SORTANDFILTER,
        None,
        Some(&info),  // MFVideoFormat_H264 output
        &mut activates,
        &mut count,
    )?;
    // Match against vendor keywords; fall back to first if no match
}

The hardware encoder is always configured with H.264 Baseline profile (66) — no B-frames, no reordering. This is essential because Flashback uses a FIFO queue (pts_fifo) to assign timestamps to encoder output packets: the N-th output packet gets the N-th input timestamp. B-frames would break this ordering guarantee. Periodic IDR frames are requested via ICodecAPI:

let gop = (fps / 2).max(8).min(60); // IDR roughly every ~0.5 s
codec.SetValue(&CODECAPI_AVEncMPVGOPSize, &variant_u32(gop))?;
codec.SetValue(&CODECAPI_AVEncMPVDefaultBPictureCount, &variant_u32(0))?;

Setting FLASHBACK_FORCE_SW_ENCODER=1 in the environment forces the software encoder regardless of hardware availability. This is useful for validating the software fallback path on machines that have hardware encoders.

Media Foundation SinkWriter (Manual Recording)

For manual recording, IMFSinkWriter manages the full encode-and-mux pipeline. Flashback declares a video output type (H.264) and a BGRA/ARGB32 input type; the SinkWriter automatically inserts a Video Processor MFT to convert and scale between the two. Audio tracks (AAC, one for system audio and optionally one for microphone) are added as additional streams with PCM input and AAC output types.

// Output type: H.264 at target bitrate and resolution
out_type.SetGUID(&MF_MT_MAJOR_TYPE, &MFMediaType_Video)?;
out_type.SetGUID(&MF_MT_SUBTYPE, &MFVideoFormat_H264)?;
out_type.SetUINT32(&MF_MT_AVG_BITRATE, bitrate)?;
let stream = writer.AddStream(&out_type)?;

// Input type: BGRA at capture resolution (SinkWriter scales internally)
in_type.SetGUID(&MF_MT_SUBTYPE, &MFVideoFormat_ARGB32)?;
writer.SetInputMediaType(stream, &in_type, None)?;

writer.BeginWriting()?;

The SinkWriter is configured with MF_SINK_WRITER_DISABLE_THROTTLING so it never blocks the capture thread waiting for the encoder to drain, and with MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS to allow the hardware encoder MFT to be inserted automatically.

Instant Replay Ring Buffer

The Instant Replay mode buffers a configurable duration of pre-encoded H.264 and AAC packets in memory so they can be flushed to disk on demand without re-encoding. This is fundamentally different from manual recording, where the SinkWriter owns the encoder.

struct ReplayBuffer {
    packets: VecDeque<Packet>,   // H.264 packets
    seq_header: Vec<u8>,         // SPS/PPS header
    width: u32,
    height: u32,
    fps: u32,
    bitrate: u32,
    window_ns: i64,              // buffer duration in 100-ns units
    sys_audio: Option<AudioTrackBuf>,
    mic_audio: Option<AudioTrackBuf>,
}

struct Packet {
    data: Vec<u8>,
    time: i64,
    dur: i64,
    key: bool,   // true for IDR frames
}

Buffer Trimming

The ring buffer trims old packets on every push. To ensure the saved clip always starts at an IDR frame, trimming finds the latest keyframe older than the window cutoff and discards everything before it:

fn trim(&mut self) {
    let cutoff = latest - self.window_ns;
    let mut anchor = 0usize;
    for (i, p) in self.packets.iter().enumerate() {
        if p.time > cutoff { break; }
        if p.key { anchor = i; }
    }
    for _ in 0..anchor { self.packets.pop_front(); }
}

Audio track buffers trim by time only (no keyframe concept for AAC); packets older than the window are dropped.

Saving a Replay

save_replay locks the buffer, clones the packets starting from the first IDR frame, releases the lock, and then muxes the clone into a new MP4 file on a dedicated MTA thread — all without pausing the live encoding pump:

pub fn save_replay(source: &str) -> Option<String> {
    // Clone under lock, then release before touching disk
    let (packets, .., sys_audio, mic_audio) = {
        let buf = buffer.lock().unwrap();
        // collect packets starting from the first keyframe
        (pkts, buf.width, buf.height, buf.fps, ...)
    };
    // mux_replay writes a fresh MP4 via IMFSinkWriter in passthrough mode
    std::thread::spawn(move || mux_replay(...)).join()
}

The mux thread creates a new IMFSinkWriter, declares the video stream in passthrough mode (H.264 in → H.264 out, no re-encode), writes every buffered packet with its original timestamp, and calls Finalize().

If no keyframe has arrived yet when save_replay is called, the save fails gracefully. In practice this occurs only within the first ~0.5 s of starting the replay, before the first periodic IDR.

Encoder Pump (Async vs. Sync)

Hardware encoders expose IMFMediaEventGenerator and operate asynchronously: the pump must wait for ME_TRANSFORM_NEED_INPUT events before feeding frames and drain ME_TRANSFORM_HAVE_OUTPUT events to retrieve encoded packets. Software encoders are synchronous MFTs: the pump calls ProcessInput and immediately drains ProcessOutput in the same call. Both paths maintain the same CFR pacing logic; the only difference is how they interact with the encoder MFT:

fn run_pump(pipe: &ReplayPipeline, ...) {
    if pipe.enc_events.is_some() {
        run_pump_async(pipe, ...);  // hardware: event-driven
    } else {
        run_pump_sync(pipe, ...);   // software: synchronous
    }
}

MMCSS Thread Priority

Both the capture thread (manual recording) and the replay pump thread register with the Multimedia Class Scheduler Service under the "Capture" task. This elevates their scheduling priority above normal user threads, ensuring the capture pipeline is not starved by a CPU-intensive game running in the foreground.

let _mmcss = MmcssTask::new("Capture");
// MmcssTask is a RAII guard: AvRevertMmThreadCharacteristics on drop

Windows timer resolution is also tightened to 1 ms via timeBeginPeriod(1) for the duration of capture, which makes the CFR pacing sleep (std::thread::sleep) precise enough for accurate frame timing.

Out-of-Focus Card

When a game window is minimized during an Instant Replay session, WGC stops delivering new frames. Rather than filling the gap with frozen duplicates, Flashback composites a single out-of-focus card — the last real frame, Gaussian-blurred and darkened, with the Flashback logo and the text “The game is out of focus” — and encodes it at a low cadence (~2 fps) for the duration of the minimized period. The card is rendered by overlay::OutOfFocusCard using Direct2D and DirectWrite on the same ID3D11Device as the capture pipeline (zero additional device overhead):

// overlay.rs
let blur = ctx.CreateEffect(&CLSID_D2D1GaussianBlur)?;
// standard deviation ∝ capture height, optimization = SPEED
blur.SetInput(0, &src_bmp, true);
ctx.BeginDraw();
ctx.DrawImage(&blur.GetOutput()?, ...);   // blurred frame
ctx.FillRectangle(&full, &self.dim);      // 45% black overlay
ctx.DrawBitmap(&self.logo, ...);          // app logo PNG
ctx.DrawText(&text, ...);                 // "The game is out of focus"
ctx.EndDraw(None, None)?;

A forced IDR is inserted both at the transition into minimized state and at the transition back to the game, so the clip can be trimmed cleanly at either boundary without visual ghosting from cross-GOP references.

Architecture

How the Rust backend and SvelteKit frontend fit together, including the command model and window setup.

Audio Pipeline

WASAPI loopback, per-track threads, PCM conversion, downmix, and AAC encoding with IMFTransform.

Instant Replay

User-facing guide to configuring and triggering Instant Replay.

Encoder Configuration

How to select NVENC, AMF, Quick Sync, or software encoding in the settings UI.

Architecture

Tauri Commands

Video Capture Pipeline: WGC, D3D11, and Media Foundation

Frame Acquisition with Windows Graphics Capture

Frame-Rate Limiting

Constant-Frame-Rate Pacing

Even-Dimension Enforcement

D3D11 Texture Pipeline

Hardware Encoder Selection

Media Foundation SinkWriter (Manual Recording)

Instant Replay Ring Buffer

Buffer Trimming

Saving a Replay

Encoder Pump (Async vs. Sync)

MMCSS Thread Priority

Out-of-Focus Card

Architecture

Audio Pipeline

Instant Replay

Encoder Configuration

Build docs developers (and LLMs) love

Architecture

Tauri Commands

Documentation Index

​Frame Acquisition with Windows Graphics Capture

​Frame-Rate Limiting

​Constant-Frame-Rate Pacing

​Even-Dimension Enforcement

​D3D11 Texture Pipeline

​Hardware Encoder Selection

​Media Foundation SinkWriter (Manual Recording)

​Instant Replay Ring Buffer

​Buffer Trimming

​Saving a Replay

​Encoder Pump (Async vs. Sync)

​MMCSS Thread Priority

​Out-of-Focus Card

​Related Pages

Architecture

Audio Pipeline

Instant Replay

Encoder Configuration

Build docs developers (and LLMs) love

Frame Acquisition with Windows Graphics Capture

Frame-Rate Limiting

Constant-Frame-Rate Pacing

Even-Dimension Enforcement

D3D11 Texture Pipeline

Hardware Encoder Selection

Media Foundation SinkWriter (Manual Recording)

Instant Replay Ring Buffer

Buffer Trimming

Saving a Replay

Encoder Pump (Async vs. Sync)

MMCSS Thread Priority

Out-of-Focus Card

Related Pages