System Hooks

System Hooks let you run background actions in response to any activity across xyOps—user transactions, system events, job completions, alert triggers, and notices. Configure hooks in config.json to send web requests, execute shell commands, send emails, or create tickets.

Overview

System Hooks differ from regular Actions:

Global scope - Fire for all activity, not just specific events or alerts
Configuration-based - Defined in config.json, not in the UI
Background execution - Run asynchronously without blocking the main activity
Firehose capability - Can capture every transaction with a wildcard hook

Configuration

Define hooks in config.json under the hooks object:

"hooks": {
  "error": {
    "shell_exec": "/opt/system/bin/notify-error.sh"
  },
  "critical": {
    "shell_exec": "/opt/system/bin/notify-critical.sh",
    "email": "oncall-pager@company.com",
    "ticket": {
      "type": "issue",
      "assignees": ["admin"]
    }
  },
  "secret_access": {
    "web_hook": "mkd0t84o7hwasu0g"
  }
}

From ~/workspace/source/docs/syshooks.md:11-37.

Activity types

System Hooks can trigger on any activity type ID. Common examples:

User transactions

user_create, user_update, user_delete - User account changes
event_create, event_update, event_delete - Event management
workflow_create, workflow_update, workflow_delete - Workflow changes
alert_create, alert_update, alert_delete - Alert definitions
secret_create, secret_update, secret_delete - Secret management
secret_access - When a secret is accessed by a job or plugin

System events

server_add, server_update, server_remove - Server changes
conductor_primary - Conductor became primary
conductor_secondary - Conductor became secondary

Log levels

notice - Informational notices
warning - Non-critical warnings
error - Recoverable errors
critical - Critical failures requiring attention

Job and alert conditions

All action conditions fire system hooks with a job_ or alert_ prefix:

job_start, job_complete, job_success, job_error, job_warning, job_critical, job_abort
job_tag:TAGID - Jobs with specific tags
alert_new, alert_cleared

See the full list at Activity.action in the source documentation.

Firehose hook

Capture all activity with a wildcard * key:

"hooks": {
  "*": {
    "shell_exec": "/opt/system/bin/firehose.sh"
  }
}

Firehose hooks execute for every activity. Use with caution—high-volume systems can generate thousands of events per minute.

Hook action types

System Hooks support four action types. Combine multiple actions in a single hook—they all run in parallel.

Web hook

Two options for sending HTTP requests: 1. Simple URL - POST activity payload as JSON:

"hooks": {
  "alert_new": {
    "url": "http://alerts.company.com/api/new-alert/v1"
  }
}

2. Configured web hook - Reference a web hook ID managed in xyOps UI:

"hooks": {
  "alert_new": {
    "web_hook": "wmkd2yx4yw4ihh7lu"
  }
}

Configured web hooks support full templating, custom headers, and all features of regular Web Hooks.

Shell exec

Execute a command on the primary conductor server. Activity payload is sent to STDIN as JSON:

"hooks": {
  "critical": {
    "shell_exec": "/path/to/script.sh"
  }
}

The payload follows the xyOps Wire Protocol format (single-line JSON).

Shell exec runs on the primary conductor only. Use with extreme caution. Scripts have full system access.

Debugging shell hooks

Search for System Shell Hook in /opt/xyops/logs/Action.log. Set debug_level: 9 in config.json for verbose logging.

Send email

Send a generic activity summary email:

"hooks": {
  "critical": {
    "email": "oncall-pager@company.com"
  }
}

Multiple recipients can be comma-separated:

"email": "admin@company.com, oncall@company.com"

The email includes:

Activity summary
Full JSON payload
Any details provided (common for notice, warning, error, critical)

For alert emails, use alert_universal_actions instead. It generates alert-specific emails with better formatting.

Create ticket

Automatically create tickets for activity:

"hooks": {
  "critical": {
    "ticket": {
      "type": "issue",
      "assignees": ["admin"],
      "tags": ["automated", "critical"]
    }
  }
}

The ticket is populated with activity details and includes a link to the related resource (job, alert, user, etc.).

Payload format

All System Hooks receive an Activity payload:

{
  "action": "event_create",
  "username": "admin",
  "ip": "192.168.1.10",
  "summary": "Created event: Daily Backup",
  "date": 1763256572,
  "details": {
    "event": {
      "id": "emi11ejdlde",
      "title": "Daily Backup",
      "enabled": true
    }
  }
}

Key fields:

action - Activity type ID
username - User who triggered the action (or system for automated actions)
summary - Human-readable description
date - Unix timestamp
details - Activity-specific data (varies by action type)

Examples

Alert all critical errors to Slack

"hooks": {
  "critical": {
    "web_hook": "wmkd2yx4yw4ihh7lu"
  }
}

Where wmkd2yx4yw4ihh7lu is a Slack web hook configured in the xyOps UI.

Log all secret access to a security log

"hooks": {
  "secret_access": {
    "shell_exec": "/opt/system/bin/log-secret-access.sh"
  }
}

Script example:

#!/bin/bash
# Read JSON from STDIN
read -r payload
echo "$(date -Iseconds) $payload" >> /var/log/xyops-secrets.log

Create tickets for all job failures

"hooks": {
  "job_error": {
    "ticket": {
      "type": "issue",
      "assignees": ["ops-team"],
      "tags": ["job-failure", "automated"]
    }
  }
}

Email admins when servers disconnect

"hooks": {
  "server_remove": {
    "email": "admins@company.com"
  }
}

Multi-channel critical alert

"hooks": {
  "critical": {
    "email": "oncall@company.com",
    "web_hook": "wmkd2yx4yw4ihh7lu",
    "ticket": {
      "type": "incident",
      "assignees": ["admin"],
      "tags": ["critical", "automated"]
    },
    "shell_exec": "/opt/system/bin/page-oncall.sh"
  }
}

This fires email, Slack, creates a ticket, and executes a paging script—all in parallel.

Best practices

Use configured web hooks

Reference web hook IDs instead of inline URLs for better control and templating support.

Avoid shell exec in production

Shell scripts run with full system privileges. Prefer web hooks and tickets for security.

Filter carefully

Firehose hooks execute constantly. Use specific activity types to reduce overhead.

Monitor hook performance

Check /opt/xyops/logs/Action.log for slow hooks that block activity processing.

Test with debug level 9

Enable verbose logging temporarily to troubleshoot hook execution.

Combine actions sparingly

Multiple actions per hook run in parallel. Ensure they don’t conflict or duplicate work.

Comparison: System Hooks vs Actions

Feature	System Hooks	Actions
Scope	Global (all activity)	Scoped to events/alerts
Configuration	config.json	UI or API
Conditions	Activity type IDs	Job outcomes, alert states
Web hooks	Simple URL or ID reference	Full template control
Shell exec	Yes (conductor only)	No
Tickets	Yes	Yes
Email	Generic summary	Templated with context
Firehose	Yes (`*` wildcard)	No

Get Started

Core Concepts

Monitoring & Alerts

Organization

Operations

Plugins & Extensions

Advanced

System Hooks

Overview

Configuration

Activity types

User transactions

System events

Log levels

Job and alert conditions

Firehose hook

Hook action types

Web hook

Shell exec

Debugging shell hooks

Send email

Create ticket

Payload format

Examples

Best practices

Use configured web hooks

Avoid shell exec in production

Filter carefully

Monitor hook performance

Test with debug level 9

Combine actions sparingly

Comparison: System Hooks vs Actions

See also

Build docs developers (and LLMs) love

Get Started

Core Concepts

Monitoring & Alerts

Organization

Operations

Plugins & Extensions

Advanced

Documentation Index

​Overview

​Configuration

​Activity types

​User transactions

​System events

​Log levels

​Job and alert conditions

​Firehose hook

​Hook action types

​Web hook

​Shell exec

​Debugging shell hooks

​Send email

​Create ticket

​Payload format

​Examples

​Best practices

Use configured web hooks

Avoid shell exec in production

Filter carefully

Monitor hook performance

Test with debug level 9

Combine actions sparingly

​Comparison: System Hooks vs Actions

​See also

Build docs developers (and LLMs) love

Overview

Configuration

Activity types

User transactions

System events

Log levels

Job and alert conditions

Firehose hook

Hook action types

Web hook

Shell exec

Debugging shell hooks

Send email

Create ticket

Payload format

Examples

Best practices

Comparison: System Hooks vs Actions

See also