Jobs

Overview

Jobs are the runtime instances of events in xyOps. When an event trigger fires or a user manually launches an event, the system creates a job object that tracks the execution from start to completion. Each job has a unique ID, full lifecycle state, streaming logs, performance metrics, and configurable limits and actions.

A job is essentially a running instance of an event configuration. Think of events as templates and jobs as their executions.

Job Lifecycle

Jobs progress through several states during their lifetime:

Job States

State	Description
`ready`	Job is ready to start, checking limits and selecting server
`starting`	Job is executing start actions before remote launch
`active`	Job is running on remote server
`queued`	Job is waiting in queue due to concurrency or server availability limits
`start_delay`	Job is delayed before starting (from trigger or retry)
`retry_delay`	Job is waiting before retry attempt
`complete`	Job has finished (success, error, or abort)

Job Creation

When a job is created from an event:

Copy Event Configuration

The job starts as a copy of the event plus trigger context and any user/API overrides

Assign Unique ID

Job receives a unique job.id (prefixed with ‘j’), and job.event is set to the event’s ID

Merge Defaults

Category-defined actions/limits and universal defaults are merged into the job

Prepare Environment

Custom environment variables are set up, plugin parameters are copied, and placeholders are resolved

Job Execution

Parameter Resolution

Event parameters support placeholder substitution using {{ macros }} that resolve against the job context:

{
  "params": {
    "output_file": "/data/{{event.id}}/{{job.id}}.csv",
    "username": "{{input.data.username}}",
    "timestamp": "{{job.now}}"
  }
}

Remote Execution

Once a server is chosen:

Job state moves to active and job.started is set to current time
Launch command is sent to the server’s xySat agent via WebSocket
Plugin executes with final parameters and environment variables
Output streams into job log in real-time
CPU/memory/IO metrics are sampled every second

Job Updates

The server sends periodic updates back to the conductor containing:

{
  id: "j12345",
  state: "active",
  progress: 0.75,
  cpu: { current: 45.2, min: 12.1, max: 87.3, total: 2341.5, count: 52 },
  mem: { current: 524288000, min: 104857600, max: 629145600, total: 27262976000, count: 52 },
  procs: { /* process tree */ },
  conns: [ /* network connections */ ]
}

Job Output

Log Streaming

Job output is written to a log file at logs/jobs/{job.id}.log and streamed in real-time to users watching the job detail page.

// Source: lib/job.js:235-245
appendJobLog(job, msg, data) {
  msg = '' + msg;
  var log_file = Path.resolve( Path.join( 
    this.config.get('log_dir'), 'jobs', job.id + '.log' 
  ));
  fs.appendFileSync( log_file, msg );
  job.log_file_size += Buffer.byteLength(msg);
  
  this.doPageBroadcast( 'Job?id=' + job.id, 'log_append', { text: msg } );
}

Meta Log (Activity)

Jobs maintain a separate activity log for internal events and action results:

// Source: lib/job.js:247-265
appendMetaLog(job, msg, data) {
  if (!info.activity) info.activity = [];
  var row = { 
    ...data, 
    id: Tools.generateShortID('m'), 
    epoch: Tools.timeNow(), 
    msg: ''+msg,
    server: 'm:' + this.hostID
  };
  
  info.activity.push(row);
  
  // Keep under 1000 entries
  if (info.activity.length > 1000) info.activity.shift();
}

Job Data and Files

Jobs can produce structured output:

data: JSON object containing job results
files: Array of file objects with metadata
html: Custom HTML content to display
table: Tabular data with headers and rows

{
  "data": {
    "records_processed": 1543,
    "errors": 2,
    "duration_ms": 45231
  },
  "files": [
    {
      "id": "f123abc",
      "filename": "output.csv",
      "path": "files/jobs/j12345/output.csv",
      "size": 2048576,
      "date": 1735689600
    }
  ]
}

Job Completion

When a job completes:

Set Completion Time

job.completed is set to current timestamp, job.elapsed calculated

Process Log File

User-generated log is prepared for upload and compressed if needed

Check Retry

If job failed and retry limit is configured, job may be retried

Run Completion Actions

Actions are fired based on job result (success, error, warning, critical, abort, or tag-based)

Index in Database

Job is stored in database for history and searching (unless ephemeral)

Job Exit Codes

Jobs complete with a code that determines which actions fire:

Code	Meaning	Actions Fired
`0` or `false`	Success	`complete`, `success`
`"warning"`	Warning state	`complete`, `error`, `warning`
`"critical"`	Critical error	`complete`, `error`, `critical`
`"abort"`	Aborted	`complete`, `abort`
Any other value	User-defined error	`complete`, `error`, `user`

Job Limits

Limits are continuously monitored while jobs run:

Active Monitoring

// Source: lib/job.js:469-541
checkJobActiveLimits(job) {
  var now = Tools.timeNow();
  
  // Time limits
  Tools.findObjects( job.limits, { type: 'time', enabled: true } )
    .forEach( function(limit) {
      if (now - job.started > limit.duration) {
        self.triggerActiveJobLimit(job, limit);
      }
    });
  
  // Log file size limits
  Tools.findObjects( job.limits, { type: 'log', enabled: true } )
    .forEach( function(limit) {
      if (job.log_file_size > limit.amount) {
        self.triggerActiveJobLimit(job, limit);
      }
    });
  
  // Memory limits (with sustain period)
  Tools.findObjects( job.limits, { type: 'mem', enabled: true } )
    .forEach( function(limit) {
      if (job.mem.current > limit.amount) {
        if (!limit.when) limit.when = now;
        if (now - limit.when > limit.duration) {
          self.triggerActiveJobLimit(job, limit);
        }
      } else if (limit.when) {
        delete limit.when; // reset sustain timer
      }
    });
  
  // CPU limits (with sustain period)
  // Similar to memory...
}

Start-Time Limits

Before a job starts, the system checks:

Max Concurrent Jobs: Prevents exceeding parallel job limit per event
Max Queue Limit: Determines if job can queue when limits are reached
Max File Limit: Prunes input files based on count, size, and extensions

Job Queuing

When a job cannot start immediately:

// Source: lib/job.js:688-729
if (jobs.length >= job_limit.amount) {
  var queue_limit = Tools.findObject( job.limits, { type: 'queue', enabled: true } );
  
  if (queue_limit && queue_limit.amount) {
    var queued = this.findSimilarJobs(job, { state: 'queued' });
    
    if (queued.length < queue_limit.amount) {
      // Room in queue
      this.appendMetaLog(job, "Moving job state from {ready} to {queued}");
      job.state = 'queued';
      job.position = queued.length + 1;
      return false;
    } else {
      // Queue is full
      this.abortJob(job, "Maximum number of concurrent jobs reached, and queue is maxed out.");
      return false;
    }
  } else {
    // No queue configured
    this.abortJob(job, "Maximum number of concurrent jobs reached.");
    return false;
  }
}

Queued jobs are monitored every second and automatically moved to ready state when a slot opens.

Job Metrics

Jobs track various performance metrics:

CPU and Memory

{
  "cpu": {
    "current": 45.2,
    "min": 12.1,
    "max": 87.3,
    "total": 2341.5,
    "count": 52
  },
  "mem": {
    "current": 524288000,
    "min": 104857600,
    "max": 629145600,
    "total": 27262976000,
    "count": 52
  }
}

Process Timeline

Process tree snapshots are stored every second for the last 5 minutes, and every minute for up to 24 hours:

// Source: lib/job.js:1092-1116
if (updates.procs) {
  if (!info.timelines) info.timelines = {};
  if (!info.timelines.second) info.timelines.second = [];
  if (!info.timelines.minute) info.timelines.minute = [];
  
  // Keep up to 5 minutes of second data
  info.timelines.second.push({ 
    epoch: now, 
    procs: self.pruneProcsForTimeline(updates.procs) 
  });
  if (info.timelines.second.length > 300) info.timelines.second.shift();
  
  // Keep up to 24 hours of minute snaps
  var cur_min_epoch = Tools.normalizeTime( now, { sec: 0 } );
  if (!info.timelines.minute.length || 
      (info.timelines.minute[info.timelines.minute.length - 1].epoch != cur_min_epoch)) {
    info.timelines.minute.push({
      epoch: cur_min_epoch, 
      procs: updates.procs,
      conns: updates.conns
    });
    if (info.timelines.minute.length > 1440) info.timelines.minute.shift();
  }
}

Job Status

The job object tracks current status information:

{
  "id": "j12345",
  "event": "event100",
  "state": "active",
  "started": 1735689600,
  "updated": 1735689652,
  "progress": 0.75,
  "server": "main",
  "pid": 54321,
  "elapsed": 52,
  "log_file_size": 2048576,
  "code": 0,
  "description": "",
  "complete": false,
  "retried": false,
  "retry_count": 0,
  "invisible": false,
  "ephemeral": false
}

Dead Job Timeout: If a job doesn’t receive updates for 120 seconds (configurable), it’s considered dead and automatically aborted. This prevents zombie jobs from consuming resources.

Special Job Types

Workflow Jobs

Workflow jobs have a type of "workflow" and contain:

workflow.state: State of all nodes
workflow.jobs: Map of completed sub-jobs by node ID
Progress calculated from all sub-jobs
CPU/memory aggregated from active sub-jobs

Ad-Hoc Jobs

Ad-hoc jobs created from workflow job nodes have type of "adhoc" and don’t have an associated event in the database.

Test Jobs

Jobs run in test mode have test: true and can optionally disable actions and limits.

Events

Learn about event configuration

Limits

Configure job resource limits

Actions

Set up job completion actions

Workflows

Build complex job orchestration

Get Started

Core Concepts

Monitoring & Alerts

Organization

Operations

Plugins & Extensions

Advanced

Overview

Job Lifecycle

Job States

Job Creation

Job Execution

Parameter Resolution

Remote Execution

Job Updates

Job Output

Log Streaming

Meta Log (Activity)

Job Data and Files

Job Completion

Job Exit Codes

Job Limits

Active Monitoring

Start-Time Limits

Job Queuing

Job Metrics

CPU and Memory

Process Timeline

Job Status

Special Job Types

Workflow Jobs

Ad-Hoc Jobs

Test Jobs

Events

Limits

Actions

Workflows

Build docs developers (and LLMs) love

Get Started

Core Concepts

Monitoring & Alerts

Organization

Operations

Plugins & Extensions

Advanced

Documentation Index

​Overview

​Job Lifecycle

​Job States

​Job Creation

​Job Execution

​Parameter Resolution

​Remote Execution

​Job Updates

​Job Output

​Log Streaming

​Meta Log (Activity)

​Job Data and Files

​Job Completion

​Job Exit Codes

​Job Limits

​Active Monitoring

​Start-Time Limits

​Job Queuing

​Job Metrics

​CPU and Memory

​Process Timeline

​Job Status

​Special Job Types

​Workflow Jobs

​Ad-Hoc Jobs

​Test Jobs

​Related Documentation

Events

Limits

Actions

Workflows

Build docs developers (and LLMs) love

Overview

Job Lifecycle

Job States

Job Creation

Job Execution

Parameter Resolution

Remote Execution

Job Updates

Job Output

Log Streaming

Meta Log (Activity)

Job Data and Files

Job Completion

Job Exit Codes

Job Limits

Active Monitoring

Start-Time Limits

Job Queuing

Job Metrics

CPU and Memory

Process Timeline

Job Status

Special Job Types

Workflow Jobs

Ad-Hoc Jobs

Test Jobs

Related Documentation