Skip to main content

​
Overview

Worker management provides visibility into the distributed extraction workers that execute data extraction jobs. Operators can list all registered workers, inspect individual worker details including version information, and monitor worker health through heartbeat timestamps.

​
Key Concepts

  • Worker: A process running ampd worker that executes extraction jobs
  • Node ID: Unique identifier for a worker instance (format: ^[a-zA-Z][a-zA-Z0-9_\-\.]*$)
  • Heartbeat: Periodic health signal sent by workers to the metadata database (every 1 second)
  • Worker Info: Build metadata including version, commit SHA, and build date

​
Architecture

Workers register with the metadata database and maintain liveness through periodic heartbeats: 
Worker → Heartbeat (1s) → Metadata DB ← Admin API ← ampctl

​
Commands

​
ampctl worker list

List all registered workers. 
ampctl worker list [OPTIONS]
Aliases: ls Examples: 
# Basic listing
ampctl worker list
ampctl worker ls  # alias

# JSON output
ampctl worker list --json
Output: 
worker-01h2xcejqtf2nbrexx3vqjhp41 (last heartbeat: 2025-01-15T17:20:15Z)
indexer-node-1 (last heartbeat: 2025-01-15T17:18:45Z)
eu-west-1a-worker (last heartbeat: 2025-01-15T17:20:10Z)

​
ampctl worker inspect

Get detailed information about a specific worker. 
ampctl worker inspect <NODE_ID>
Aliases: get
​
NODE_ID
string
required
The unique identifier of the worker node
Shows:
  • Node ID
  • Creation timestamp
  • Registration timestamp
  • Last heartbeat timestamp
  • Worker build information (version, commit SHA, build date)
Examples: 
# Get detailed worker information
ampctl worker inspect worker-01
ampctl worker get worker-01  # alias

# JSON output
ampctl worker inspect worker-01 --json

# Extract specific fields with jq
ampctl worker inspect worker-01 --json | jq -r '.info.version'
Output: 
Node ID: worker-01h2xcejqtf2nbrexx3vqjhp41
Created: 2025-01-01T12:00:00Z
Registered: 2025-01-15T16:45:30Z
Heartbeat: 2025-01-15T17:20:15Z

Worker Info:
  Version: v0.0.22-15-g8b065bde
  Commit: 8b065bde9c1a2f3e4d5c6b7a8e9f0a1b2c3d4e5f
  Commit Timestamp: 2025-01-15T14:30:00Z
  Build Date: 2025-01-15T15:45:30Z

​
Monitoring Worker Health

​
Check Worker Liveness

Workers send heartbeats every 1 second. A stale heartbeat indicates the worker may have crashed or lost connectivity. 
# List all workers and check heartbeat times
ampctl worker list

# Inspect a specific worker's heartbeat
ampctl worker inspect worker-01 --json | jq -r '.heartbeat_at'

​
Monitor Worker Fleet

# Get all workers as JSON
ampctl worker list --json

# Extract worker IDs and heartbeat times
ampctl worker list --json | jq -r '.workers[] | "\(.node_id): \(.heartbeat_at)"'

# Find workers with stale heartbeats (older than 5 seconds)
ampctl worker list --json | jq '.workers[] | select(
  (now - (.heartbeat_at | fromdateiso8601)) > 5
)'

​
Check Worker Versions

# List all workers with version information
for worker in $(ampctl worker list --json | jq -r '.workers[].node_id'); do
  echo "$worker: $(ampctl worker inspect $worker --json | jq -r '.info.version')"
done

​
Workflow Examples

​
Identify Healthy Workers

# List all workers
ampctl worker list

# Check if a specific worker is responding
ampctl worker inspect worker-01

# Verify heartbeat is recent (within last 5 seconds)
ampctl worker inspect worker-01 --json | jq -r '.heartbeat_at'

​
Debug Worker Issues

# Check when worker was last seen
ampctl worker inspect problematic-worker

# Compare worker versions across fleet
ampctl worker list --json | jq -r '.workers[] | 
  "\(.node_id): \(if .info then .info.version else "unknown" end)"'

# Identify workers that haven't sent heartbeats recently
ampctl worker list --json | jq '.workers[] | 
  select((now - (.heartbeat_at | fromdateiso8601)) > 10)'

​
Capacity Planning

# Count active workers
ampctl worker list --json | jq '.workers | length'

# List workers by creation date
ampctl worker list --json | jq -r '.workers[] | 
  "\(.created_at) - \(.node_id)"' | sort

​
Direct API Access

Query the Admin API directly using curl: 
# List all workers
curl http://localhost:1610/workers

# Get worker details
curl http://localhost:1610/workers/worker-01

# Format with jq
curl http://localhost:1610/workers | jq

​
Worker ID Format

Valid worker node IDs must match the pattern: ^[a-zA-Z][a-zA-Z0-9_\-\.]*$ Valid examples:
  • worker-01
  • indexer-node-1
  • eu_west_1a_worker
  • worker.production.01
Invalid examples:
  • 1-worker (cannot start with a number)
  • worker@01 (@ is not allowed)
  • worker#01 (# is not allowed)

​
JSON Output

All worker commands support JSON output for scripting: 
# List workers as JSON
ampctl worker list --json

# Inspect worker as JSON
ampctl worker inspect worker-01 --json

# Extract specific fields with jq
ampctl worker list --json | jq -r '.workers[] | "\(.node_id): \(.heartbeat_at)"'
ampctl worker inspect worker-01 --json | jq -r '.info.version'

Build docs developers (and LLMs) love

Get started for freeTalk to us