Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/Shyamalp16/CloudGaming/llms.txt

Use this file to discover all available pages before exploring further.

Host Runtime

The CloudGaming host is a high-performance C++ application that captures game video/audio and streams it to clients via WebRTC.

Architecture Overview

The host runtime consists of several specialized subsystems:
  • Video Capture: Windows Graphics Capture (WGC) + D3D11
  • Audio Capture: WASAPI with process loopback
  • Video Encoding: FFmpeg with hardware acceleration (NVENC/QSV/AMF)
  • Audio Encoding: Opus codec
  • WebRTC Transport: Go/Pion for peer connection management
  • Input Handling: Direct input injection

Video Capture & Encoding Pipeline

Windows Graphics Capture (WGC)

The host uses Windows Graphics Capture API to capture the game window at the compositor level:
// From Host/main.cpp:113-129
auto item = WindowUtils::CreateItem(hwnd);
GraphicsAndCapture::CaptureContext cap;
GraphicsAndCapture::InitializeCapture(cap, d3d, item);
GraphicsAndCapture::Start(cap);
Key Features:
  • Compositor-level capture (no game modification needed)
  • Hardware-accelerated texture sharing
  • Support for windowed and fullscreen modes
  • Configurable frame pacing via MinUpdateInterval

D3D11 Video Processing

BGRA textures from WGC are converted to NV12 using D3D11 VideoProcessor for optimal encoder compatibility:
// From Host/Encoder.cpp:481-553
static bool InitializeVideoProcessor(ID3D11Device* device, int width, int height) {
    // Create VideoProcessor for BGRA -> NV12 conversion
    g_videoDevice->CreateVideoProcessorEnumerator(&desc, g_vpEnumerator.GetAddressOf());
    g_videoDevice->CreateVideoProcessor(g_vpEnumerator.Get(), 0, g_videoProcessor.GetAddressOf());
    
    // Set BT.709 color space for accurate color reproduction
    D3D11_VIDEO_PROCESSOR_COLOR_SPACE inputCS{};
    inputCS.RGB_Range = 0;      // full-range RGB (0-255)
    inputCS.YCbCr_Matrix = 1;   // BT.709 (HD)
    inputCS.Nominal_Range = 2;  // 0-255 full range
    g_videoContext->VideoProcessorSetStreamColorSpace(g_videoProcessor.Get(), 0, &inputCS);
}
Optimizations:
  • LRU cache for D3D11 views (avoids per-frame allocations)
  • Pre-validated format support
  • Primed texture views for first-frame performance

Hardware Encoding

The encoder automatically selects the best hardware encoder based on GPU vendor:
// From Host/Encoder.cpp:1193-1227
case 0x10DE: // NVIDIA
    encoderName = "h264_nvenc";
    av_dict_set(&opts, "preset", "p5", 0);      // Fast low-latency
    av_dict_set(&opts, "tune", "ull", 0);       // Ultra-low-latency
    av_dict_set(&opts, "rc", "cbr", 0);         // Constant bitrate
    av_dict_set(&opts, "async_depth", "2", 0);  // Minimal buffering
    av_dict_set(&opts, "surfaces", "3", 0);     // async_depth + 1
    av_dict_set(&opts, "spatial_aq", "1", 0);   // Adaptive quantization
    av_dict_set(&opts, "aq-strength", "4", 0);  // Balanced quality/speed
Low-Latency Settings:
  • VBV buffer = 1x bitrate (minimal buffering)
  • B-frames disabled
  • Repeat headers enabled for keyframe recovery
  • BT.709 color metadata in SPS VUI
// From Host/Encoder.cpp:1252-1257
case 0x8086: // Intel
    encoderName = "h264_qsv";
    av_dict_set(&opts, "preset", "veryfast", 0);
    av_dict_set(&opts, "zerolatency", "1", 0);
    av_dict_set(&opts, "repeat-headers", "1", 0);
// From Host/Encoder.cpp:1258-1262
case 0x1002: // AMD
    encoderName = "h264_amf";
    av_dict_set(&opts, "usage", "lowlatency_high_quality", 0);
    av_dict_set(&opts, "repeat-headers", "1", 0);

Frame Ring Buffer

A ring buffer of hardware frames minimizes allocation overhead:
// From Host/Encoder.cpp:140-143
static std::vector<AVFrame*> g_hwFrames;
static int g_hwFrameIndex = 0;
static int g_hwFramePoolSize = 4; // Configurable pool size

// From Host/Encoder.cpp:695-702
bool AcquireHwInputSurface(int &slotIndexOut, ID3D11Texture2D** nv12TextureOut) {
    slotIndexOut = g_hwFrameIndex;
    g_hwFrameIndex = (g_hwFrameIndex + 1) % static_cast<int>(g_hwFrames.size());
    AVFrame* hw = g_hwFrames[slotIndexOut];
    *nv12TextureOut = (ID3D11Texture2D*)hw->data[0];
    return true;
}

Bitrate Adaptation

Adaptive bitrate control responds to network congestion:
// From Host/Encoder.cpp:923-975
void OnRtcpFeedback(double packetLoss, double rtt, double jitter) {
    auto now = std::chrono::steady_clock::now();
    auto since = std::chrono::duration_cast<std::chrono::milliseconds>(now - g_lastChange).count();
    
    // Reduce bitrate on packet loss
    if (packetLoss >= g_minPliLossThreshold.load()) {
        if (since >= g_decreaseCooldownMs) {
            g_congestionCeiling = static_cast<int>(g_currentBitrate * 0.9);
            double factor = (packetLoss >= 0.10) ? 0.6 : 0.8;
            int target = static_cast<int>(g_currentBitrate * factor);
            g_currentBitrate = std::max(g_minBitrateController, target);
            AdjustBitrate(g_currentBitrate);
        }
    }
    
    // Increase bitrate when stable
    g_cleanSamples++;
    if (since >= g_increaseIntervalMs && g_cleanSamples >= g_cleanSamplesRequired) {
        int target = g_currentBitrate + g_increaseStep;
        if (target <= effectiveMax) {
            g_currentBitrate = target;
            AdjustBitrate(g_currentBitrate);
        }
    }
}
Default Values (WAN-optimized):
  • Start: 8 Mbps
  • Min: 4 Mbps
  • Max: 12 Mbps
  • Increase step: +1 Mbps
  • Decrease cooldown: 1000ms

Key Source Files

Encoder.cpp

Video encoding pipeline with hardware acceleration and adaptive bitrate control.

AudioCapturer.cpp

WASAPI audio capture with Opus encoding and dedicated processing threads.

main.cpp

Main entry point with configuration loading and component initialization.

main.go

Go/Pion WebRTC integration with buffer pool and RTP packet handling.
Performance Tips:
  • Use NVENC preset p5 for best latency/quality balance
  • Set audio frame size to 5ms for ultra-low-latency
  • Enable adaptive bitrate control for WAN deployments
  • Monitor EAGAIN events to detect encoder congestion

Build docs developers (and LLMs) love