Monitors

Overview

Monitors track a single numeric server metric over time. Each monitor points to one value in the live server data, casts it to a specific data type (integer, float, bytes, seconds, or milliseconds), and xyOps stores the samples in a time-series database.

A monitor evaluates its expression once per minute on each matching server. Results are stored and graphed at multiple resolutions (hourly, daily, monthly, yearly).

How It Works

Every minute, each satellite sends a fresh ServerMonitorData snapshot to the primary conductor:

Expression Evaluation

xyOps evaluates the monitor’s source expression against the current server monitor data sample

Type Casting

The value is type-cast using the monitor’s data type and optional match regex

Delta Computation

If the monitor is configured as a delta monitor, its rate is computed from the previous absolute value (and optionally divided by elapsed seconds)

Storage

The value is inserted into the server’s time-series for all resolutions (hourly, daily, monthly, yearly)

Monitor expressions run against the live ServerMonitorData context only. They do not depend on other monitors. Alerts evaluate immediately after monitors are computed.

Storage Systems

Monitors are stored in four time-series systems with different granularities:

hourly

system

Date format: [yyyy]/[mm]/[dd]/[hh]
Epoch division: 60 (1 sample per minute)
Single only: true (prevents duplicate submissions)

daily

system

Date format: [yyyy]/[mm]/[dd]
Epoch division: 120 (~720 samples per day, every 2 minutes)

monthly

system

Date format: [yyyy]/[mm]
Epoch division: 3600 (~720 samples per month, every hour)

yearly

system

Date format: [yyyy]
Epoch division: 43200 (~730 samples per year, every 12 hours)

Creating and Editing Monitors

Go to Admin → Monitors to configure:

Title

string

required

Display name for the graph

Display

boolean

Toggle to show/hide graph in the UI without deleting it

Icon

string

Optional Material Design Icon displayed next to the title

Server Groups

array

Restrict evaluation to specific groups (leave blank for all servers)

Data Expression

string

required

An expression that extracts or computes a single numeric value from ServerMonitorData

Data Match

regex

Optional regular expression to extract a number from a string value

Data Type

enum

required

Controls parsing and display: integer, float, bytes, seconds, milliseconds

Delta Features

object

For counter-style sources, compute a delta and optionally divide by elapsed time to get a rate per second. Supports zero-minimum clamp.

Min Vert Range

number

Set a minimum Y-axis range (e.g., 100 for percentages)

Data Suffix

string

Optional unit shown in labels (e.g., %, /sec, ms)

Use the “Test…” button to evaluate your expression on a live server before saving. Click the search icon to open the Server Data Explorer and browse live ServerMonitorData paths.

Expressions

Monitor expressions are evaluated in xyOps Expression Syntax (JavaScript-style) using the current ServerMonitorData object as context.

Examples

// Basic metric: CPU usage as a float percentage
cpu.currentLoad

// Array index: 1-minute load average
load[0]

// Object path: current active connections
stats.network.conns

// Math/composition: memory used %
100 - memory.available / memory.total * 100

// Guarded math: coalesce missing to 0
stats && stats.network ? stats.network.rx_bytes : 0

Expressions must resolve to a single numeric value before final casting. Do not reference monitors.* or deltas.* in monitor expressions - those are only available in alert expressions.

Data Match

If your expression yields a string and the number is embedded within it, set Data Match to a regular expression to extract exactly one numeric value.

Example: Open Files Monitor

Expression: commands.open_files
Data Match: (\d+)
Result: Extracts the first integer from a string like "1056\t0\t9223372036854775807"

If the regex includes a capture group, the first group is used; otherwise the entire match is used.

Delta Monitors

Some data sources are absolute counters that only ever increase (OS network byte totals, disk I/O byte counters). For these:

Calc as Delta

boolean

Stores the change since the previous minute instead of the absolute counter

Divide by Time

boolean

Divides the delta by elapsed seconds between samples to produce a per-second rate

Zero Minimum

boolean

Clamp negative spikes to a specific minimum (commonly 0) to avoid dips after reboots or counter resets

Code Implementation

Delta calculation from /workspace/source/lib/monitor.js:255-289:

monitor.js:255-289

delta_defs.forEach( function(mon_def) {
    // host_data.data.monitors always contains the absolute counter
    // only the sparse `data` object (for the timeline) will have the delta
    if (!(mon_def.id in host_data.data.monitors)) {
        // first time for server, set delta to 0
        params.data.deltas[ mon_def.id ] = 0;
        return;
    }
    
    var old_value = host_data.data.monitors[ mon_def.id ] || 0;
    var delta = data[ mon_def.id ] - old_value;
    if (!old_value) delta = 0;
    
    if ((mon_def.delta_min_value !== false) && (delta < mon_def.delta_min_value)) {
        delta = mon_def.delta_min_value;
    }
    if (mon_def.divide_by_delta && host_data.date) {
        var elapsed = (params.date - host_data.date) || 1;
        delta /= elapsed;
    }
    if (mon_def.data_type.match(/^(integer|bytes|seconds|milliseconds)$/)) {
        delta = Math.floor(delta);
    }
    
    params.data.deltas[ mon_def.id ] = delta;
    // Note: params.data.monitors is deliberately NOT overwritten here,
    // as it needs the absolute value for next time around.
} );

QuickMon (Quick Monitors)

QuickMon are lightweight, predefined real-time monitors sampled every second on each server:

Retention: Last 60 seconds per server stored in memory
Display: Real-time graphs and gauges on Server and Group pages
Snapshots: Most recent 60-second series embedded into all snapshots
Config: Defined in config.json under quick_monitors

Configuration from config.json

config.json:136-148

"quick_monitors": [
    { "id": "_qm_cpu_load", "title": "CPU Load Average", "source": "cpu.avgLoad", "type": "float", "suffix": "", "min_vert_scale": 1.0 },
    { "id": "_qm_cpu_usage", "title": "CPU Usage %", "source": "cpu.currentLoad", "type": "float", "suffix": "%", "min_vert_scale": 100 },
    
    { "id": "_qm_mem_used", "title": "Memory in Use", "source": "mem.used", "type": "bytes", "suffix": "" },
    { "id": "_qm_mem_avail", "title": "Memory Available", "source": "mem.available", "type": "bytes", "suffix": "" },
    
    { "id": "_qm_disk_read_sec", "title": "Disk Read (Bytes)", "source": "fs.rx", "type": "bytes", "suffix": "/sec", "min_vert_scale": 1024, "delta": true, "delta_min_value": 0, "divide_by_delta": true },
    { "id": "_qm_disk_write_sec", "title": "Disk Write (Bytes)", "source": "fs.wx", "type": "bytes", "suffix": "/sec", "min_vert_scale": 1024, "delta": true, "delta_min_value": 0, "divide_by_delta": true },
    
    { "id": "_qm_net_in_sec", "title": "Network In (Bytes)", "source": "net.rx", "type": "bytes", "suffix": "/sec", "min_vert_scale": 1024, "delta": true, "delta_min_value": 0, "divide_by_delta": true },
    { "id": "_qm_net_out_sec", "title": "Network Out (Bytes)", "source": "net.tx", "type": "bytes", "suffix": "/sec", "min_vert_scale": 1024, "delta": true, "delta_min_value": 0, "divide_by_delta": true }
]

QuickMon complements minute-level monitors: use QuickMon for immediate visibility, and standard monitors for historical analysis and alerting.

Default Monitors

xyOps ships with a set of standard monitors:

Monitor	Expression	Type	Delta	Notes
Load Average	`load[0]`	float	No	1-minute load average
CPU Usage	`cpu.currentLoad`	float	No	Suffix: `%`, Min range: `100`
Memory in Use	`memory.used`	bytes	No	Total memory in use
Memory Available	`memory.available`	bytes	No	Available memory
Network Connections	`stats.network.conns`	integer	No	Active socket connections
Disk Usage	`mounts.root.use`	float	No	Root filesystem usage %, Min range: `100`
Disk Read	`stats.fs.rx`	bytes	Yes	Bytes/sec, Delta + Divide by Time
Disk Write	`stats.fs.wx`	bytes	Yes	Bytes/sec, Delta + Divide by Time
Disk I/O	`stats.io.tIO`	integer	Yes	Total I/O ops/sec
I/O Wait	`cpu.totals.iowait`	float	No	CPU I/O wait % (Linux only)
Open Files	`commands.open_files`	integer	No	Uses Data Match: `(\d+)` (Linux only)
Network In	`stats.network.rx_bytes`	bytes	Yes	Bytes/sec in
Network Out	`stats.network.tx_bytes`	bytes	Yes	Bytes/sec out
Processes	`processes.all`	integer	No	Total number of processes
Active Jobs	`jobs`	integer	No	Number of active xyOps jobs

Examples and Recipes

Track Specific Process Memory

// Exact name match
processes.list[.command == 'ffmpeg'].memRss

// Substring match
find( processes.list, 'command', 'ffmpeg' ).memRss

Type: bytes

Memory Used Percentage

100 - memory.available / memory.total * 100

Type: float, Suffix: %, Min Vert Range: 100

Root Free Space (in GB)

(mounts.root.available) / (1024 * 1024 * 1024)

Type: float, Suffix: GB

TCP LISTEN Sockets

// Using filter
count( conns[.state == 'LISTEN'] )

// Or using stats
stats.network.states.listen

Type: integer

For advanced metrics, write a Monitor Plugin that emits structured data, then point a monitor expression at it and optionally use data_match to extract the value.

Get Started

Core Concepts

Monitoring & Alerts

Organization

Operations

Plugins & Extensions

Advanced

Overview

How It Works

Storage Systems

Creating and Editing Monitors

Expressions

Examples

Data Match

Delta Monitors

Code Implementation

QuickMon (Quick Monitors)

Configuration from config.json

Default Monitors

Examples and Recipes

Build docs developers (and LLMs) love

Get Started

Core Concepts

Monitoring & Alerts

Organization

Operations

Plugins & Extensions

Advanced

Documentation Index

​Overview

​How It Works

​Storage Systems

​Creating and Editing Monitors

​Expressions

​Examples

​Data Match

​Delta Monitors

​Code Implementation

​QuickMon (Quick Monitors)

​Configuration from config.json

​Default Monitors

​Examples and Recipes

Build docs developers (and LLMs) love

Overview

How It Works

Storage Systems

Creating and Editing Monitors

Expressions

Examples

Data Match

Delta Monitors

Code Implementation

QuickMon (Quick Monitors)

Configuration from config.json

Default Monitors

Examples and Recipes