Understanding Logs - Conway Automaton

Log System Overview

Conway Automaton uses structured JSON logging for all operations. Logs are written to stdout in real-time and can be consumed by log aggregation systems.

Log Format

Every log entry is a JSON object:

{
  "timestamp": "2026-03-03T10:15:30.123Z",
  "level": "info",
  "module": "heartbeat.scheduler",
  "message": "Heartbeat task completed",
  "context": {
    "taskName": "check_inbox",
    "durationMs": 234,
    "shouldWake": false
  }
}

Fields

timestamp: ISO 8601 timestamp with milliseconds
level: Log level (debug, info, warn, error, fatal)
module: Hierarchical module name (e.g., agent.tools, conway.credits)
message: Human-readable message
context: Structured data (optional)
error: Error details (optional, for error/fatal levels)

Error Log Example

{
  "timestamp": "2026-03-03T10:20:15.456Z",
  "level": "error",
  "module": "conway.topup",
  "message": "Credit topup failed",
  "context": {
    "amountUsd": 25,
    "address": "0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb1"
  },
  "error": {
    "message": "Insufficient USDC balance",
    "code": "INSUFFICIENT_FUNDS",
    "stack": "Error: Insufficient USDC balance\n    at topupCredits..."
  }
}

Log Levels

debug

Detailed diagnostic information for development and troubleshooting. Examples:

Tool argument parsing
Database query execution
Context token counting
Heartbeat tick details

When to use: Investigating specific issues, tracing execution flow

info

General informational messages about normal operation. Examples:

Agent turn completed
Heartbeat task success
Credit topup successful
Skill loaded

When to use: Default log level for production

warn

Warning messages for potentially problematic situations that don’t prevent operation. Examples:

Treasury policy limit reached
Failed requirement check for skill
High context token usage
Approaching credit threshold

When to use: Monitoring for issues that may need attention

error

Error messages for operations that failed but didn’t crash the system. Examples:

Tool execution error
API request failure
x402 payment rejected
Database transaction rollback

When to use: Diagnosing failures and errors

fatal

Critical errors that require system shutdown. Examples:

Database corruption
Wallet file missing
Unrecoverable API errors
Configuration invalid

When to use: System-level failures

Viewing Logs

CLI Commands

# Stream all logs (live)
automaton-cli logs --follow

# Last 100 lines
automaton-cli logs --tail 100

# Filter by level
automaton-cli logs --level error
automaton-cli logs --level warn --level error  # Multiple levels

# Filter by module
automaton-cli logs --module heartbeat
automaton-cli logs --module "conway.*"  # Wildcards

# Search message content
automaton-cli logs --grep "credit"
automaton-cli logs --grep "topup|transfer"  # Regex

# Time range
automaton-cli logs --since "2026-03-03T10:00:00Z"
automaton-cli logs --until "2026-03-03T12:00:00Z"
automaton-cli logs --since "1 hour ago"

# JSON output (for processing)
automaton-cli logs --format json

# Combine filters
automaton-cli logs --level error --module conway --since "1 day ago"

Direct File Access

If running as a daemon, logs may be written to a file:

# Follow logs
tail -f ~/.automaton/logs/automaton.log

# Search logs
grep "error" ~/.automaton/logs/automaton.log

# Parse JSON logs with jq
cat ~/.automaton/logs/automaton.log | jq 'select(.level == "error")'

Log Modules

From the source code structure:

Core Modules

config: Configuration loading and validation
database: SQLite operations
wallet: Wallet management

Agent Modules

agent.loop: Main agent loop
agent.tools: Tool execution
agent.context: Context building
agent.policy: Policy engine

Conway Modules

conway.client: Conway API client
conway.credits: Credit balance checks
conway.topup: x402 credit purchases
conway.inference: Model inference routing

Heartbeat Modules

heartbeat.scheduler: Task scheduling
heartbeat.daemon: Daemon process
heartbeat.tasks: Task execution

Skills Modules

skills.loader: Skill discovery and loading
skills.registry: Skill management

Memory Modules

memory.episodic: Episodic memory
memory.semantic: Semantic memory
memory.working: Working memory
memory.compression: Context compression

Observability Modules

observability.logger: Logging system
observability.metrics: Metrics collection
observability.alerts: Alert evaluation

Replication Modules

replication.spawn: Child creation
replication.health: Health monitoring
replication.messaging: Inter-agent messaging

Reading Common Log Patterns

Agent Turn Execution

// Turn started
{
  "level": "info",
  "module": "agent.loop",
  "message": "Starting agent turn",
  "context": {
    "turnId": "01HWQK2G9MXYZ",
    "inputSource": "heartbeat",
    "state": "running"
  }
}

// Tool called
{
  "level": "debug",
  "module": "agent.tools",
  "message": "Executing tool",
  "context": {
    "toolName": "check_balance",
    "turnId": "01HWQK2G9MXYZ"
  }
}

// Turn completed
{
  "level": "info",
  "module": "agent.loop",
  "message": "Agent turn completed",
  "context": {
    "turnId": "01HWQK2G9MXYZ",
    "durationMs": 1234,
    "tokenUsage": 567,
    "costCents": 12
  }
}

Credit Topup Flow

// Topup initiated
{
  "level": "info",
  "module": "conway.topup",
  "message": "Attempting credit topup: $25 USD"
}

// x402 payment
{
  "level": "debug",
  "module": "conway.x402",
  "message": "Signing USDC transfer authorization",
  "context": {
    "amount": 25,
    "validAfter": "2026-03-03T10:00:00Z",
    "validBefore": "2026-03-03T10:05:00Z"
  }
}

// Topup success
{
  "level": "info",
  "module": "conway.topup",
  "message": "Credit topup successful: $25 USD → 2500 credits cents"
}

Heartbeat Task Execution

// Task started
{
  "level": "debug",
  "module": "heartbeat.scheduler",
  "message": "Executing heartbeat task",
  "context": {
    "taskName": "check_inbox",
    "tickId": "01HWQK3H7PQRS"
  }
}

// Task result
{
  "level": "info",
  "module": "heartbeat.tasks",
  "message": "Inbox check completed",
  "context": {
    "unreadCount": 3,
    "shouldWake": true
  }
}

// Wake event created
{
  "level": "info",
  "module": "heartbeat.scheduler",
  "message": "Wake event created",
  "context": {
    "source": "heartbeat",
    "reason": "Unread messages in inbox"
  }
}

Policy Denial

{
  "level": "warn",
  "module": "agent.policy",
  "message": "Policy denied tool execution",
  "context": {
    "toolName": "transfer_credits",
    "reasonCode": "EXCEEDS_DAILY_LIMIT",
    "humanMessage": "Transfer denied: daily limit of $250 reached",
    "riskLevel": "dangerous",
    "currentDailySpend": 25000
  }
}

Skill Loading

// Skill discovered
{
  "level": "debug",
  "module": "skills.loader",
  "message": "Parsing skill file",
  "context": {
    "path": "/home/user/.automaton/skills/web-scraper/SKILL.md"
  }
}

// Requirements check
{
  "level": "debug",
  "module": "skills.loader",
  "message": "Checking skill requirements",
  "context": {
    "skillName": "web-scraper",
    "requiredBins": ["curl", "jq"]
  }
}

// Skill loaded
{
  "level": "info",
  "module": "skills.loader",
  "message": "Skill loaded successfully",
  "context": {
    "skillName": "web-scraper",
    "autoActivate": true
  }
}

Log Analysis

Performance Analysis

Find slowest turns:

automaton-cli logs --module agent.loop --format json | \
  jq -r 'select(.message == "Agent turn completed") | "\(.context.durationMs)\t\(.context.turnId)"' | \
  sort -rn | \
  head -10

Error Rate Calculation

# Count errors by module
automaton-cli logs --level error --since "1 day ago" --format json | \
  jq -r '.module' | \
  sort | \
  uniq -c | \
  sort -rn

Cost Analysis

# Sum inference costs
automaton-cli logs --module agent.loop --format json | \
  jq -r 'select(.message == "Agent turn completed") | .context.costCents' | \
  awk '{sum+=$1} END {print "Total cost: $" sum/100}'

Timeline Reconstruction

# Show activity timeline
automaton-cli logs --since "1 hour ago" --format json | \
  jq -r '"\(.timestamp) [\(.level | ascii_upcase)] \(.module): \(.message)"'

Configuring Log Level

Set log level in automaton.json:

{
  "logLevel": "info"
}

Or via environment variable:

export AUTOMATON_LOG_LEVEL=debug
automaton-cli start

Log Level Hierarchy

From src/types.ts:

export const LOG_LEVEL_PRIORITY: Record<LogLevel, number> = {
  debug: 0,
  info: 1,
  warn: 2,
  error: 3,
  fatal: 4,
};

Setting level to warn will show warn, error, and fatal messages.

Log Rotation

If using file-based logging, configure rotation:

# Using logrotate
sudo tee /etc/logrotate.d/automaton <<EOF
/home/user/.automaton/logs/automaton.log {
  daily
  rotate 30
  compress
  delaycompress
  missingok
  notifempty
  create 0600 user user
}
EOF

Integration with Log Aggregation

Shipping to Elasticsearch

automaton-cli logs --follow --format json | \
  filebeat -c /etc/filebeat/filebeat.yml

CloudWatch Logs

automaton-cli logs --follow --format json | \
  aws logs put-log-events \
    --log-group-name /conway/automaton \
    --log-stream-name $(hostname)

Datadog

automaton-cli logs --follow --format json | \
  datadog-agent run

Debugging Techniques

Tracing a Specific Turn

# Get turnId from status
TURN_ID=$(automaton-cli status --format json | jq -r '.lastTurnId')

# Show all logs for that turn
automaton-cli logs --format json | \
  jq --arg tid "$TURN_ID" 'select(.context.turnId == $tid)'

Correlation by Timestamp

# Find what happened around a specific time
automaton-cli logs \
  --since "2026-03-03T10:15:00Z" \
  --until "2026-03-03T10:16:00Z"

Following Module Activity

# Watch specific module in real-time
automaton-cli logs --follow --module conway.topup

Extracting Error Context

# Show errors with full context
automaton-cli logs --level error --format json | \
  jq '{timestamp, module, message, context, error}'

Common Log Messages

Startup Sequence

[info] config: Configuration loaded successfully
[info] database: Database opened: ~/.automaton/state.db
[info] wallet: Wallet loaded: 0x742d...
[info] skills.loader: Loaded 3 skills
[info] heartbeat.daemon: Heartbeat daemon started
[info] agent.loop: Automaton started

Normal Operation

[info] heartbeat.scheduler: Heartbeat tick started
[info] heartbeat.tasks: Balance check completed
[info] heartbeat.tasks: Inbox check completed
[info] agent.loop: Starting agent turn
[info] agent.loop: Agent turn completed

Low Credit Warning

[warn] conway.credits: Credit balance below threshold
[info] conway.topup: Attempting credit topup: $5 USD
[info] conway.topup: Credit topup successful

Skill Installation

[debug] skills.loader: Parsing skill file
[debug] skills.loader: Checking skill requirements
[info] skills.loader: Skill loaded successfully

Error Scenarios

[error] conway.client: API request failed: 503 Service Unavailable
[error] agent.tools: Tool execution error: timeout after 30s
[error] database: Transaction rollback: constraint violation

Best Practices

Use appropriate log levels: Don’t log everything at info
Include context: Add relevant data to context objects
Never log secrets: Redact API keys, private keys, etc.
Use structured context: Prefer context fields over string interpolation
Consistent module naming: Follow the hierarchical pattern
Log boundaries: Start/end of important operations
Log decisions: Why did the agent do something?
Keep messages concise: Details go in context

Troubleshooting with Logs

Agent not waking up

automaton-cli logs --module heartbeat --since "1 hour ago"
# Look for: wake event creation, shouldWake: true/false

High costs

automaton-cli logs --module agent.loop --format json | \
  jq 'select(.context.costCents > 100)'
# Identify expensive turns

Tool failures

automaton-cli logs --level error --module agent.tools
# Review error messages and stack traces

Policy denials

automaton-cli logs --grep "Policy denied"
# Check reasonCode and humanMessage

Overview

Getting started

Core concepts

Features

Guides

Architecture

Conway Cloud

Documentation Index

​Log System Overview

​Log Format

​Fields

​Error Log Example

​Log Levels

​debug

​info

​warn

​error

​fatal

​Viewing Logs

​CLI Commands

​Direct File Access

​Log Modules

​Core Modules

​Agent Modules

​Conway Modules

​Heartbeat Modules

​Skills Modules

​Memory Modules

​Observability Modules

​Replication Modules

​Reading Common Log Patterns

​Agent Turn Execution

​Credit Topup Flow

​Heartbeat Task Execution

​Policy Denial

​Skill Loading

​Log Analysis

​Performance Analysis

​Error Rate Calculation

​Cost Analysis

​Timeline Reconstruction

​Configuring Log Level

​Log Level Hierarchy

​Log Rotation

​Integration with Log Aggregation

​Shipping to Elasticsearch

​CloudWatch Logs

​Datadog

​Debugging Techniques

​Tracing a Specific Turn

​Correlation by Timestamp

​Following Module Activity

​Extracting Error Context

​Common Log Messages

​Startup Sequence

​Normal Operation

​Low Credit Warning

​Skill Installation

​Error Scenarios

​Best Practices

​Troubleshooting with Logs

​Agent not waking up

​High costs

​Tool failures

​Policy denials

Build docs developers (and LLMs) love