Skip to main content

Overview

The GPU engine (src/engine/WebGPUEngine.ts) is a thin facade that coordinates all rendering subsystems. Preview, scrubbing, and export run through the same WebGPU ping-pong compositor. Video textures are imported as GPUExternalTexture — zero-copy, no CPU roundtrip.

WebGPUContext

GPU adapter, device, canvas configuration, and device loss recovery.

RenderTargetManager

Ping-pong buffers, independent preview buffers, and effect temp textures.

CompositorPipeline

37 blend modes, 3D transforms, and inline color effects in a single WGSL shader per layer.

ScrubbingCache

3-tier frame cache: 300 GPU textures, last-frame fallback, and 900-frame RAM Preview.

Architecture

Engine directory structure

src/engine/
  WebGPUEngine.ts           # Thin facade orchestrating all subsystems
  WebCodecsPlayer.ts        # Full WebCodecs decoding (MP4Box demux + VideoDecoder)
  WebCodecsExportMode.ts    # Sequential export frame decoding
  ParallelDecodeManager.ts  # Parallel multi-clip decoding for export
  featureFlags.ts           # Runtime feature toggles
  webCodecsTypes.ts         # Shared MP4Box/sample types
  index.ts                  # Public exports
  core/
    WebGPUContext.ts         # GPU adapter/device/canvas management + device loss recovery
    RenderTargetManager.ts   # Ping-pong buffers, independent preview buffers, effect temp textures
    types.ts                 # Layer, BlendMode, EngineStats, LayerRenderData, blend mode map
  render/
    RenderLoop.ts            # RAF loop with idle detection, frame rate limiting, watchdog
    RenderDispatcher.ts      # Orchestrates per-frame render: collect -> composite -> output
    Compositor.ts            # Ping-pong compositing with inline + complex effects
    LayerCollector.ts        # Imports textures from video/image/text/nested sources
    NestedCompRenderer.ts    # Pre-renders nested compositions to offscreen textures
    layerEffectStack.ts      # Splits effects into inline vs. complex via splitLayerEffects()
  pipeline/
    CompositorPipeline.ts    # GPU pipelines for standard + external composite + copy shaders
    OutputPipeline.ts        # Final output to canvas with optional transparency grid
    SlicePipeline.ts         # Corner-pin warped slices with 16x16 subdivision
  texture/
    TextureManager.ts        # Image/canvas/video/ImageBitmap texture creation and caching
    MaskTextureManager.ts    # Per-layer mask textures with white fallback
    ScrubbingCache.ts        # LRU frame cache for scrubbing, last-frame cache, RAM Preview
  managers/
    CacheManager.ts          # Owns ScrubbingCache lifecycle + video time tracking
    ExportCanvasManager.ts   # OffscreenCanvas for zero-copy VideoFrame export
    OutputWindowManager.ts   # External popup windows for multi-display output
  video/
    VideoFrameManager.ts     # requestVideoFrameCallback tracking for frame readiness
  stats/
    PerformanceStats.ts      # FPS, timing, frame drop tracking with ring buffer
  analysis/
    OpticalFlowAnalyzer.ts   # GPU Lucas-Kanade optical flow (compute shaders)
    ScopeRenderer.ts         # Delegates to waveform/histogram/vectorscope
    HistogramScope.ts        # GPU-accelerated histogram (compute + render)
    WaveformScope.ts         # GPU-accelerated waveform with phosphor glow
    VectorscopeScope.ts      # GPU-accelerated vectorscope (BT.709 CbCr)
  audio/
    AudioMixer.ts            # Multi-track mixing with mute/solo
    AudioEncoder.ts          # WebCodecs AAC/Opus encoding
    AudioExportPipeline.ts   # Orchestrates complete audio export
  export/
    FrameExporter.ts         # Frame-by-frame export orchestrator
    VideoEncoderWrapper.ts   # WebCodecs VideoEncoder + mp4/webm muxers

Engine facade

WebGPUEngine (Facade)
  Core:
  ├── WebGPUContext             # GPU adapter, device, canvas configuration
  ├── RenderTargetManager       # Ping-pong + independent + effect temp textures
  ├── PerformanceStats          # FPS, timing, drops

  Render:
  ├── RenderLoop                # RAF with idle detection + watchdog
  ├── RenderDispatcher          # Orchestrates per-frame render
  ├── LayerCollector            # Texture import from all source types
  ├── Compositor                # Ping-pong compositing + effects
  └── NestedCompRenderer        # Offscreen nested composition rendering

  Pipelines:
  ├── CompositorPipeline        # Standard + external composite GPU pipelines
  ├── EffectsPipeline           # Per-effect GPU pipelines (30+ effects)
  ├── OutputPipeline            # Final output with transparency grid
  └── SlicePipeline             # Corner-pin warped slice output

  Textures:
  ├── TextureManager            # Image/canvas/video/bitmap textures
  ├── MaskTextureManager        # Per-layer mask textures
  └── VideoFrameManager         # RVFC frame readiness tracking

  Managers:
  ├── CacheManager              # ScrubbingCache + video time tracking
  ├── ExportCanvasManager       # Zero-copy export via OffscreenCanvas
  └── OutputWindowManager       # External popup windows

Core components

WebGPUContext (src/engine/core/WebGPUContext.ts)

Manages the GPU adapter, logical device, and canvas surface.
// Initialization parameters
- GPU adapter: powerPreference configurable ('high-performance' | 'low-power')
- Device: requiredLimits: { maxTextureDimension2D: 4096 }
- Canvas: preferred format via navigator.gpu.getPreferredCanvasFormat()
- Alpha mode: 'opaque' for preview canvases, 'premultiplied' for export
- Sampler: linear filtering, clamp-to-edge
- Device loss recovery: 100ms delay before re-initialization (up to 3 attempts)
Device loss recovery flow:
Device Lost → notify callbacks → clean GPU resources → wait 100ms →
  → re-initialize (up to 3 attempts) → recreate all resources →
  → reconfigure canvases → restart render loop → notify restored

RenderTargetManager (src/engine/core/RenderTargetManager.ts)

Allocates and owns all GPU render textures. All textures are rgba8unorm.
TexturePurpose
Ping / PongMain compositing double-buffer
Independent Ping / PongSeparate buffers for multi-composition preview
Effect Temp 1 / 2Pre-processing source textures for complex effects
Black texture1×1 fallback for empty frame output
createPingPongTextures() nulls texture references for GC instead of calling .destroy() to avoid Destroyed texture used in a submit warnings (VRAM leak fix).

Texture types

All texture handling lives in src/engine/texture/TextureManager.ts and src/engine/texture/MaskTextureManager.ts.
SourceGPU typeCopy behaviorCaching
HTMLVideoElementGPUExternalTexture (texture_external)Zero-copyPer-video last-frame + scrubbing cache
HTMLVideoElement on Firefoxtexture_2d<f32>Copied to persistent texture per frameReused per layer to avoid 30+ tex/sec
VideoFrame (WebCodecs)GPUExternalTexture (texture_external)Zero-copyNone — frame buffer managed by decoder
HTMLImageElementtexture_2d<f32> (rgba8unorm)Copied onceBy HTMLImageElement reference
HTMLCanvasElement (text clips)texture_2d<f32> (rgba8unorm)Copied onceBy canvas reference
ImageBitmap (Native Helper)texture_2d<f32> (rgba8unorm)Re-uploaded per frameReused by layer ID
Firefox does not support importExternalTexture reliably for HTMLVideoElement. The htmlVideoPreviewFallback.ts path copies frames to a persistent texture_2d<f32> to avoid intermittent black frames.

Compositing pipeline

Four GPU render pipelines (src/engine/pipeline/CompositorPipeline.ts)

  1. Standard composite — image/canvas textures (texture_2d<f32>)
  2. External composite — video textures (texture_external)
  3. Standard copy — texture-to-texture copy for effect pre-processing
  4. External copy — external texture to rgba8unorm for effect pre-processing

Inline vs. complex effects

Effects are classified at composite time by src/engine/render/layerEffectStack.ts:
  • Inline effects (brightness, contrast, saturation, invert) — applied as uniforms in the composite shader; no extra render passes
  • Complex effects (blur, pixelate, glow, etc.) — require separate pre-processing render passes on the source texture before compositing

Ping-pong compositing

Clear Ping → transparent

Layer 1 → Read Ping, Write Pong (composite)
Layer 2 → Read Pong, Write Ping (composite)
Layer 3 → Read Ping, Write Pong (composite)
...
Final → Output Pipeline → Canvas(es)

Layer uniform structure (96 bytes / 24 floats)

// CompositorPipeline uniform layout
[0]  opacity: f32
[1]  blendMode: u32              // 0-36 (37 blend modes)
[2]  positionX: f32
[3]  positionY: f32
[4]  scaleX: f32
[5]  scaleY: f32
[6]  rotationZ: f32              // radians
[7]  sourceAspect: f32
[8]  outputAspect: f32
[9]  time: f32                   // for dissolve effects
[10] hasMask: u32                // 0 or 1
[11] maskInvert: u32             // 0 or 1
[12] rotationX: f32              // radians
[13] rotationY: f32              // radians
[14] perspectiveDistance: f32    // default 2.0
[15] maskFeather: f32            // blur radius in pixels
[16] maskFeatherQuality: u32     // 0=low, 1=med, 2=high
[17] positionZ: f32              // depth position
[18] inlineBrightness: f32       // 0 = no change
[19] inlineContrast: f32         // 1 = no change
[20] inlineSaturation: f32       // 1 = no change
[21] inlineInvert: u32           // 0 or 1

WGSL shaders

Total WGSL: ~2,565 lines in files, ~3,000 lines including inline.
FileLinesPurpose
src/shaders/composite.wgsl618Blending + 37 modes + inline effects + mask feathering
src/shaders/opticalflow.wgsl326Motion analysis (compute shaders)
src/shaders/effects.wgsl243Legacy inline GPU effects
src/effects/_shared/common.wgsl154Shared effect utilities
src/shaders/output.wgsl83Passthrough + transparency grid + stacked alpha
src/shaders/slice.wgsl33Corner-pin warped slice rendering
30 effect shaders (src/effects/)~1,108Individual effect shaders
~435 lines of WGSL are inlined in src/engine/pipeline/CompositorPipeline.ts (copy shader ~30 lines, external copy ~30 lines, external composite shader ~375 lines with all 37 blend modes). These are not in separate .wgsl files.

Blend modes (37 total)

CategoryCountExamples
Normal3Normal, Dissolve, Behind
Darken6Multiply, Darken, Color Burn
Lighten6Screen, Add, Lighten
Contrast7Overlay, Soft Light, Hard Light
Inversion5Difference, Exclusion
Component4Hue, Saturation, Color, Luminosity
Stencil5Stencil Alpha, Silhouette Luma
Alpha Add1Alpha Add

Effect shaders (30 total)

CategoryShaders
blur/box, gaussian, motion, radial, zoom
color/brightness, contrast, exposure, hue-shift, invert, levels, saturation, temperature, vibrance
distort/bulge, kaleidoscope, mirror, pixelate, rgb-split, twirl, wave
keying/chroma-key
stylize/edge-detect, glow, grain, posterize, scanlines, sharpen, threshold, vignette

Scrubbing cache

Three tiers of frame caching in src/engine/texture/ScrubbingCache.ts:
TierPurposeKeyCapacityEviction
Tier 1 — GPU texturesInstant access during timeline scrubvideoSrc:quantizedTime (30fps)300 frames (~10s at 30fps)LRU via Map insertion order
Tier 2 — Last frameVisible during seeks/pausesHTMLVideoElement reference1 per videoOverwrite
Tier 3 — RAM PreviewFully composited frames for instant playbackQuantized time (30fps)900 frames / 512 MBLRU, frame count + memory limit
A separate GPU frame cache (max 60 textures) avoids CPU-to-GPU re-upload during RAM Preview playback. When the cache is warm, scrubbing does not decode at all.

Optical flow

src/engine/analysis/OpticalFlowAnalyzer.ts implements GPU Lucas-Kanade optical flow via compute shaders (src/shaders/opticalflow.wgsl, 326 lines). Analysis runs at 160×90 pixels. Compute pipeline stages:
  1. Grayscale conversion (BT.601)
  2. Gaussian blur 5×5 (sigma=1.0)
  3. Pyramid downsampling (3 levels)
  4. Spatial gradients (Ix, Iy)
  5. Temporal gradient (It)
  6. Lucas-Kanade solver
  7. Statistics aggregation
Output:
interface MotionResult {
  total: number;       // Overall motion 0-1
  global: number;      // Camera/scene motion 0-1
  local: number;       // Object motion 0-1
  isSceneCut: boolean; // True if likely a scene cut
}

Video scopes

Three GPU-accelerated scopes in src/engine/analysis/, all using @workgroup_size(16, 16) compute shaders with atomic<u32> storage buffers:
ScopeComputeRenderColor space
HistogramBins pixels into 256-bin R/G/B/Luma histograms via atomicAddBar chart visualizationsRGB + BT.709 luma
WaveformAccumulates per-column intensity with sub-pixel weightingPhosphor glow visualizationsRGB + luma
VectorscopeMaps pixels to CbCr coordinates via BT.709 coefficientsDot plot with graticuleBT.709 CbCr

Export pipeline

Export runs through src/engine/export/FrameExporter.ts:
  1. Initialize VideoEncoderWrapper + AudioExportPipeline
  2. Set engine resolution to export resolution
  3. Initialize export OffscreenCanvas for zero-copy path
  4. For each frame: seek → build layers → render → new VideoFrame(offscreenCanvas) → encode via VideoEncoder
  5. Mux to container (mp4-muxer or webm-muxer)
  6. Encode audio
  7. Return Blob
No readPixels(), no getImageData(). The GPU renders, WebCodecs encodes.

Codec support

CodecContainerCodec string
H.264 (AVC)MP4avc1.4d0028
H.265 (HEVC)MP4hvc1.1.6.L93.B0
VP9MP4, WebMvp09.00.10.08
AV1MP4, WebMav01.0.04M.08

Feature flags

Runtime toggles via window.__ENGINE_FLAGS__ in src/engine/featureFlags.ts:
flags = {
  useRenderGraph: false,           // Render Graph executor (stubs — not ready)
  useDecoderPool: false,           // Shared decoder pool (not wired yet)
  useFullWebCodecsPlayback: false, // Preview uses HTML video by default;
                                   // WebCodecs is used for export and full-mode only
}
// Toggle in browser console
window.__ENGINE_FLAGS__.useFullWebCodecsPlayback = true

Performance

Frame rate targets

ModeTargetDrop threshold
Playback60 fps (16.67ms)>33ms
Scrubbing30 fps (33ms)>66ms

Bottleneck categories

CodeMeaning
slow_importTexture upload took >50% of frame budget
slow_renderCompositing took >16.67ms
slow_rafRAF gap exceeded 2× target (missed frames)

Idle mode

After 1 second of inactivity the render loop pauses (RAF stays alive for wake-up). Idle detection is suppressed until the first play event to keep video GPU surfaces warm after page reload.

Troubleshooting

ProblemSolution
15 fps on LinuxEnable Vulkan: chrome://flags/#enable-vulkan
”Device mismatch”HMR broke singleton — refresh the page
Black canvas after reloadVideo GPU surface not warm — press play once, or wait for preCacheVideoFrame
WebCodecs failsAutomatically falls back to HTMLVideoElement
Device lostAuto-recovery up to 3 attempts, then manual page reload
Integrated GPU selectedWindows: Graphics Settings → Add Chrome/Edge → Options → High Performance
// Check GPU status
chrome://gpu

// Debug commands
window.__ENGINE_FLAGS__                                          // view/toggle feature flags
Logger.enable('WebGPU,Compositor,RenderLoop')                    // enable engine logging
Logger.enable('LayerCollector,TextureManager')                   // debug texture import

Build docs developers (and LLMs) love