Server Monitoring: Prometheus and Grafana

The GameServer exposes a Prometheus text-format metrics endpoint at http://localhost:9108/metrics and a liveness check at http://localhost:9108/health. A pre-configured Docker Compose stack in ServerSolution/Infra/ pairs Prometheus (scraping every 5 seconds) with Grafana, which auto-provisions the MMO / MMO GameServer Overview dashboard on startup.

MetricsHttpServer

The HTTP server is controlled by two constants in Config.h:

constexpr bool EnablePrometheusEndpoint = true;
constexpr uint16 PrometheusHttpPort     = 9108;

When EnablePrometheusEndpoint is true the server starts a lightweight single-threaded TCP listener on PrometheusHttpPort before the IOCP loop begins. The two endpoints it serves are:

Path	Method	Description
`/metrics`	`GET`	Prometheus text format — all `NetworkMetrics::Snapshot` fields
`/health`	`GET`	Returns `200 OK` with body `ok`

Verify the endpoint

Invoke-WebRequest -UseBasicParsing http://localhost:9108/health
Invoke-WebRequest -UseBasicParsing http://localhost:9108/metrics

Starting the monitoring stack

Build and start the GameServer

Make sure the GameServer process is running before starting the containers. The Prometheus scrape job targets host.docker.internal:9108, so the endpoint must be reachable from Docker.

Set the Grafana admin password

The GRAFANA_ADMIN_PASSWORD variable is required by the Compose file — it has no built-in default in production mode.

$env:GRAFANA_ADMIN_PASSWORD = "your-password"

Start Prometheus and Grafana

Run the following from the ServerSolution/Infra/ directory:

cd ServerSolution\Infra
docker compose up -d prometheus grafana

The Compose file pins exact image versions (prom/prometheus:v3.13.0, grafana/grafana:13.1.1) and mounts the provisioning files as read-only volumes so the dashboard is always present on a clean start.

Open the dashboards

Service	URL
Prometheus targets	`http://localhost:9090/targets`
Grafana	`http://localhost:3000`

Log in to Grafana with user admin and the password set above. The MMO / MMO GameServer Overview dashboard is pre-loaded in the MMO folder.

NetworkMetrics.Snapshot fields

The /metrics response is derived from NetworkMetrics::Snapshot. The fields are grouped below by subsystem.

Connection counts

Field	Description
`connectedSessions`	Total TCP sessions currently connected
`authenticatedSessions`	Sessions that have completed login
`udpBoundSessions`	Sessions that have completed UDP hello handshake

TCP receive / send

Field	Description
`recvBytes`	Cumulative bytes received over TCP
`recvPackets`	Cumulative TCP packets received
`recvPayloadCopyBytes`	Cumulative bytes copied when reading TCP payloads
`sendPackets`	Cumulative TCP packets sent
`sendPayloadCopyBytes`	Cumulative bytes copied when building TCP send payloads
`sendCalls`	Number of `WSASend` calls made
`scatterGatherSendCalls`	`WSASend` calls that used scatter-gather buffers

UDP

Field	Description
`udpRecvPackets`	Cumulative UDP datagrams received
`udpRecvBytes`	Cumulative UDP bytes received
`udpSendPackets`	Cumulative UDP datagrams sent
`udpSendBytes`	Cumulative UDP bytes sent
`udpSendFailures`	UDP send calls that returned an error

Movement commands

Field	Description
`moveReqReceived`	`C_MoveReq` UDP packets received
`moveCommandsQueued`	Commands enqueued to the simulation queue
`moveCommandsDropped`	Commands dropped because the queue was full
`moveCommandsApplied`	Commands processed by the simulation tick
`moveCommandsStale`	Commands discarded because a newer input arrived first
`moveNotifyPackets`	`S_MoveNotify` packets sent
`moveNotifyRecipients`	Total recipient count across all notify packets
`movementBatchPackets`	`S_MoveBatchNotify` packets sent
`movementBatchEntries`	Total entries across all batch notify packets

Movement replication

Field	Description
`movementReplicationPublished`	Position snapshots published to the replication buffer
`movementReplicationReplaced`	Snapshots overwritten in the buffer by a newer position before dispatch
`movementReplicationDropped`	Snapshots dropped from the replication buffer
`movementReplicationDispatched`	Snapshots dispatched to clients in a replication cycle
`movementReplicationCycles`	Total replication cycles completed
`movementReplicationCycleTotalMicros`	Cumulative microseconds spent in replication cycles
`movementReplicationCycleMaxMicros`	Slowest single replication cycle in microseconds
`movementReplicationPending`	Snapshots currently waiting to be dispatched
`movementReplicationPendingMax`	Peak pending snapshot count observed
`movementReplicationWorkSamples`	Number of replication work-phase samples recorded
`movementReplicationGatherTotalMicros`	Cumulative microseconds spent gathering snapshots
`movementReplicationGatherMaxMicros`	Slowest single gather phase in microseconds
`movementReplicationSendTotalMicros`	Cumulative microseconds spent sending replication datagrams
`movementReplicationSendMaxMicros`	Slowest single send phase in microseconds
`movementReplicationAoiTotalMicros`	Cumulative microseconds spent on AoI queries during replication
`movementReplicationAoiMaxMicros`	Slowest single AoI phase in microseconds
`movementReplicationVisibilityTotalMicros`	Cumulative microseconds spent on visibility checks
`movementReplicationVisibilityMaxMicros`	Slowest single visibility phase in microseconds
`movementReplicationBatchBuildTotalMicros`	Cumulative microseconds building batch notify datagrams
`movementReplicationBatchBuildMaxMicros`	Slowest single batch-build phase in microseconds

Game simulation

Field	Description
`gameSimulationTicks`	Number of simulation ticks completed
`gameSimulationOverBudgetTicks`	Ticks that exceeded the 33 ms budget
`gameSimulationTickTotalMicros`	Cumulative tick wall-clock time in microseconds
`gameSimulationTickMaxMicros`	Slowest single tick in microseconds
`gameSimulationRuntime.missedTicks`	Ticks skipped because the previous tick ran too long
`gameSimulationRuntime.commandQueueDepth`	Command queue depth at the last runtime sample
`gameSimulationRuntime.commandQueueDepthMax`	Peak command queue depth observed
`gameSimulationRuntime.commandsDrained`	Commands drained in the last simulation drain pass
`gameSimulationRuntime.commandWaitSamples`	Number of command-wait latency samples recorded
`gameSimulationRuntime.commandWaitTotalMicros`	Cumulative command wait time in microseconds
`gameSimulationRuntime.commandWaitP95Micros`	95th-percentile command wait latency in microseconds
`gameSimulationRuntime.commandWaitP99Micros`	99th-percentile command wait latency in microseconds
`gameSimulationRuntime.commandWaitMaxMicros`	Maximum command wait latency in microseconds
`gameSimulationRuntime.drainBatches`	Number of drain batches executed
`gameSimulationRuntime.drainTotalMicros`	Cumulative drain execution time in microseconds
`gameSimulationRuntime.drainMaxMicros`	Slowest single drain batch in microseconds
`gameSimulationRuntime.tickTimingSamples`	Number of tick-lateness timing samples
`gameSimulationRuntime.tickLatenessTotalMicros`	Cumulative tick lateness in microseconds
`gameSimulationRuntime.tickLatenessMaxMicros`	Maximum tick lateness in microseconds
`gameSimulationPhaseSamples[*]`	Per-phase sample counts (CharacterStateSave, PlayerMovement, PlayerCombat, MonsterCombat, MonsterSimulation, MonsterNotifications)
`gameSimulationPhaseTotalMicros[*]`	Per-phase cumulative time in microseconds
`gameSimulationPhaseMaxMicros[*]`	Per-phase peak time in microseconds

Active movers

Field	Description
`activeMoverSamples`	Number of active-mover count samples taken
`activeMoverTotal`	Sum of active-mover counts across all samples
`activeMoverMax`	Peak active-mover count observed

Job queue

Field	Description
`jobQueued`	Session jobs enqueued
`jobDropped`	Session jobs dropped because `MaxSessionJobQueueSize` (1024) was exceeded

Process

Field	Description
`processCpuPercent`	Process-level CPU utilisation (0–100 × core count)
`processWorkingSetBytes`	Working set (physical memory) of the process
`processPrivateBytes`	Private committed memory
`processHandleCount`	Open OS handle count
`processThreadCount`	Active thread count

Logging

Field	Description
`logDropped`	Log events dropped because the async queue was full

CSV metrics log

In addition to the Prometheus endpoint, the server writes a rolling CSV log controlled by:

constexpr bool EnablePerfCsvLog        = true;
constexpr int  PerfLogIntervalSeconds  = 5;
constexpr const char* PerfCsvLogPath   = "Logs/perf_metrics.csv";

When EnablePerfCsvLog is true a new row is appended every 5 seconds. This file is the primary input for the PerformanceReport tool.

Watch gameSimulationOverBudgetTicks and gameSimulationRuntime.missedTicks as your primary at-a-glance health signals. A single non-zero missedTicks reading means the simulation loop fell a full 33 ms behind — investigate gameSimulationTickMaxMicros and per-phase timings first before tuning thread counts.

Docker Desktop’s Prometheus container reaches the Windows host via host.docker.internal:9108. If the Target shows DOWN on http://localhost:9090/targets, confirm that the GameServer is running and that Windows Firewall allows inbound TCP on port 9108.

Get Started

Server Setup

Core Systems

Game Features

Operations

Server Monitoring: Prometheus and Grafana

MetricsHttpServer

Verify the endpoint

Starting the monitoring stack

NetworkMetrics.Snapshot fields

CSV metrics log

Build docs developers (and LLMs) love

Get Started

Server Setup

Core Systems

Game Features

Operations

Documentation Index

​MetricsHttpServer

​Verify the endpoint

​Starting the monitoring stack

​NetworkMetrics.Snapshot fields

​CSV metrics log

Build docs developers (and LLMs) love

MetricsHttpServer

Verify the endpoint

Starting the monitoring stack

NetworkMetrics.Snapshot fields

CSV metrics log