Logging Integration

AWX can send detailed logs to third-party log aggregation services for analysis, monitoring, and compliance. Logs are transmitted in JSON format over HTTP, TCP, or UDP connections.

Overview

AWX logging integration provides:

Real-time streaming of job events and activity
Structured JSON log format
Support for multiple log aggregation platforms
Configurable log sources and levels
Secure transmission with TLS/SSL support

Supported Services

Officially Supported

Splunk - Enterprise log aggregation and analysis
Elastic Stack (ELK) - Elasticsearch, Logstash, Kibana

Tested and Compatible

Sumologic - Cloud-based log management
Loggly - Cloud logging and monitoring

Potentially Compatible

Datadog - Infrastructure monitoring and logging
Red Hat Common Logging - Via Logstash connector

Log Sources

AWX provides several specialized log sources:

Job Event Logs

Logger: awx.analytics.job_events Captures detailed output from Ansible playbook execution:

Task execution results
Play and task metadata
Host-level event data
Ansible callback module output

Activity Stream Logs

Logger: awx.analytics.activity_stream Records all changes to AWX objects:

User actions and operations
Object creation, modification, deletion
Association and disassociation events
Audit trail for compliance

System Tracking Logs

Logger: awx.analytics.system_tracking Data from Ansible fact gathering and scan jobs:

System facts and configuration
Package inventories
Service states
File system information

AWX Application Logs

Standard AWX application logs with configurable levels:

ERROR - Error messages with tracebacks
WARNING - Warning messages
INFO - Informational messages
DEBUG - Detailed debugging information

Configuration

Settings

Configure log aggregation through the AWX Settings UI or API at /api/v2/settings/logging/:

{
  "LOG_AGGREGATOR_ENABLED": true,
  "LOG_AGGREGATOR_HOST": "logs.example.com",
  "LOG_AGGREGATOR_PORT": 514,
  "LOG_AGGREGATOR_TYPE": "logstash",
  "LOG_AGGREGATOR_USERNAME": "",
  "LOG_AGGREGATOR_PASSWORD": "",
  "LOG_AGGREGATOR_LOGGERS": [
    "awx",
    "activity_stream",
    "job_events",
    "system_tracking"
  ],
  "LOG_AGGREGATOR_INDIVIDUAL_FACTS": false,
  "LOG_AGGREGATOR_TOWER_UUID": "a1b2c3d4-e5f6-g7h8-i9j0-k1l2m3n4o5p6",
  "LOG_AGGREGATOR_PROTOCOL": "https",
  "LOG_AGGREGATOR_TCP_TIMEOUT": 5,
  "LOG_AGGREGATOR_VERIFY_CERT": true,
  "LOG_AGGREGATOR_LEVEL": "INFO"
}

Connection Types

HTTPS:

Most common for cloud services
Port specified in URL or PORT field
SSL/TLS encryption by default

TCP:

Direct TCP connection
Requires hostname and port
Optional TLS encryption

UDP:

Lightweight, fire-and-forget
No delivery guarantee
Best for high-volume, low-criticality logs

Splunk Integration

Configuration

{
  "LOG_AGGREGATOR_ENABLED": true,
  "LOG_AGGREGATOR_HOST": "splunk.example.com",
  "LOG_AGGREGATOR_PORT": 8088,
  "LOG_AGGREGATOR_TYPE": "splunk",
  "LOG_AGGREGATOR_USERNAME": "",
  "LOG_AGGREGATOR_PASSWORD": "<hec-token>",
  "LOG_AGGREGATOR_PROTOCOL": "https",
  "LOG_AGGREGATOR_VERIFY_CERT": true
}

Splunk HTTP Event Collector (HEC)

Enable HEC in Splunk:
- Settings → Data Inputs → HTTP Event Collector
- Click “Global Settings” and enable HEC
- Configure default source, source type, and index
Create Token:
- Click “New Token”
- Name: “AWX Integration”
- Set allowed indexes
- Copy the token value
Configure AWX:
- Use token as LOG_AGGREGATOR_PASSWORD
- URL format: https://splunk.example.com:8088/services/collector

Splunk Queries

# View all AWX logs
index=main sourcetype=awx

# Job events only
index=main sourcetype=awx logger_name="awx.analytics.job_events"

# Failed jobs
index=main sourcetype=awx job_status=failed

# Activity stream for user actions
index=main sourcetype=awx logger_name="awx.analytics.activity_stream" actor=*

Elastic Stack (ELK) Integration

Logstash Configuration

Add JSON filter to Logstash config:

# /etc/logstash/conf.d/awx.conf
input {
  tcp {
    port => 5514
    codec => json
  }
}

filter {
  json {
    source => "message"
  }
  
  # Parse timestamp
  date {
    match => [ "@timestamp", "ISO8601" ]
  }
  
  # Add tags based on logger
  if [logger_name] == "awx.analytics.job_events" {
    mutate { add_tag => [ "job_event" ] }
  }
  
  if [logger_name] == "awx.analytics.activity_stream" {
    mutate { add_tag => [ "activity" ] }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "awx-%{+YYYY.MM.dd}"
  }
}

AWX Configuration

{
  "LOG_AGGREGATOR_ENABLED": true,
  "LOG_AGGREGATOR_HOST": "logstash.example.com",
  "LOG_AGGREGATOR_PORT": 5514,
  "LOG_AGGREGATOR_TYPE": "logstash",
  "LOG_AGGREGATOR_PROTOCOL": "tcp"
}

Kibana Dashboards

Create index pattern in Kibana:

Index pattern: awx-*
Time field: @timestamp

Example visualizations:

Job success/failure rates over time
Top users by activity
Job execution duration trends
Error rate by job template

Elasticsearch Queries

// Get failed jobs
{
  "query": {
    "bool": {
      "must": [
        { "match": { "logger_name": "awx.analytics.job_events" }},
        { "match": { "status": "failed" }}
      ]
    }
  }
}

// Activity by user
{
  "query": {
    "match": { "logger_name": "awx.analytics.activity_stream" }
  },
  "aggs": {
    "users": {
      "terms": { "field": "actor.keyword" }
    }
  }
}

Sumologic Integration

Configuration

{
  "LOG_AGGREGATOR_ENABLED": true,
  "LOG_AGGREGATOR_HOST": "https://collectors.sumologic.com/receiver/v1/http/<token>",
  "LOG_AGGREGATOR_TYPE": "sumologic",
  "LOG_AGGREGATOR_PROTOCOL": "https"
}

Sumologic Setup

Create HTTP Source in Sumologic
Copy the endpoint URL
Use full URL as LOG_AGGREGATOR_HOST
Configure source category for filtering

Loggly Integration

Configuration

{
  "LOG_AGGREGATOR_ENABLED": true,
  "LOG_AGGREGATOR_HOST": "logs-01.loggly.com",
  "LOG_AGGREGATOR_PORT": 514,
  "LOG_AGGREGATOR_TYPE": "loggly",
  "LOG_AGGREGATOR_PASSWORD": "<customer-token>",
  "LOG_AGGREGATOR_PROTOCOL": "tcp"
}

Log Schema

Common Fields

All logs include these fields:

{
  "@timestamp": "2026-03-04T12:00:00.000Z",
  "cluster_host_id": "awx-host-1",
  "level": "INFO",
  "logger_name": "awx.analytics.job_events",
  "path": "/awx/main/models/jobs.py",
  "message": "Job completed successfully"
}

Job Event Schema

{
  "@timestamp": "2026-03-04T12:00:00.000Z",
  "logger_name": "awx.analytics.job_events",
  "level": "INFO",
  "cluster_host_id": "awx-1",
  "job_id": 123,
  "job_name": "Deploy Application",
  "event": "runner_on_ok",
  "event_host": "web-server-1",
  "task": "Install packages",
  "role": "webserver",
  "playbook": "site.yml",
  "event_data": {
    "res": {
      "changed": true,
      "msg": "Package installed"
    }
  }
}

Activity Stream Schema

{
  "@timestamp": "2026-03-04T12:00:00.000Z",
  "logger_name": "awx.analytics.activity_stream",
  "level": "INFO",
  "cluster_host_id": "awx-1",
  "actor": "admin",
  "operation": "create",
  "object1": "job_template 'Deploy App'",
  "object2": null,
  "changes": {
    "name": [null, "Deploy App"],
    "playbook": [null, "deploy.yml"]
  }
}

System Tracking Schema

{
  "@timestamp": "2026-03-04T12:00:00.000Z",
  "logger_name": "awx.analytics.system_tracking",
  "level": "INFO",
  "cluster_host_id": "awx-1",
  "host": "web-server-1",
  "inventory_id": 5,
  "packages": {
    "nginx": {"version": "1.18.0"},
    "python3": {"version": "3.9.7"}
  }
}

Selective Logging

Configure Log Sources

Enable specific loggers:

{
  "LOG_AGGREGATOR_LOGGERS": [
    "awx",              // AWX application logs
    "activity_stream",  // Activity stream
    "job_events",       // Job events
    "system_tracking"   // System tracking
  ]
}

Log Levels

Set minimum log level:

{
  "LOG_AGGREGATOR_LEVEL": "INFO"  // DEBUG, INFO, WARNING, ERROR
}

Individual Facts

Control system tracking detail:

{
  "LOG_AGGREGATOR_INDIVIDUAL_FACTS": false  // true for per-fact logging
}

Performance Considerations

Asynchronous Processing

Logs are sent asynchronously to avoid blocking job execution. A timeout on the log aggregator will not cause AWX operations to hang.

Message Threading

Log messages are sent in threaded mode to improve performance and prevent backlog.

Network Optimization

Use TCP or HTTPS for reliable delivery
UDP for high-volume, non-critical logs
Configure appropriate timeout values
Consider log aggregator proximity to AWX

Troubleshooting

Connection Issues

Verify connectivity:

# Test TCP connection
telnet logs.example.com 514

# Test HTTPS endpoint
curl -X POST https://logs.example.com:8088/services/collector \
  -H "Authorization: Splunk <token>"

Check AWX logs:

# View AWX logs for aggregator errors
docker logs awx_task | grep -i log_aggregator

No Logs Appearing

Verify LOG_AGGREGATOR_ENABLED is true
Check selected loggers include desired sources
Verify log level allows messages through
Test with a simple job to generate events
Check firewall rules between AWX and aggregator

SSL/TLS Errors

{
  "LOG_AGGREGATOR_VERIFY_CERT": false  // Disable for testing only
}

For production, add CA certificate to AWX trust store.

Elasticsearch Performance

Index Management:

Use date-based indices (e.g., awx-2026.03.04)
Configure index lifecycle management
Set appropriate retention policies
Consider hot/warm/cold architecture

Optimize Mappings:

{
  "mappings": {
    "properties": {
      "event_data": { "enabled": false },
      "@timestamp": { "type": "date" },
      "job_id": { "type": "integer" }
    }
  }
}

API Configuration

Get Current Settings

curl https://awx.example.org/api/v2/settings/logging/ \
  -H "Authorization: Bearer <token>"

Update Settings

curl -X PATCH https://awx.example.org/api/v2/settings/logging/ \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "LOG_AGGREGATOR_ENABLED": true,
    "LOG_AGGREGATOR_HOST": "logs.example.com",
    "LOG_AGGREGATOR_PORT": 514
  }'

Best Practices

Security

Use HTTPS/TLS for transmission
Rotate aggregator credentials regularly
Restrict access to log data
Enable certificate verification in production
Use dedicated service accounts

Data Management

Set appropriate retention policies
Archive old logs to cold storage
Implement log rotation and compression
Monitor aggregator storage capacity

Monitoring

Alert on log aggregator connectivity failures
Monitor log volume and rates
Track parsing errors in aggregator
Set up dashboards for key metrics

Compliance

Enable activity stream for audit trail
Configure appropriate retention for compliance requirements
Implement access controls on log data
Document log collection and retention policies

​Overview

​Supported Services

​Officially Supported

​Tested and Compatible

​Potentially Compatible

​Log Sources

​Job Event Logs

​Activity Stream Logs

​System Tracking Logs

​AWX Application Logs

​Configuration

​Settings

​Connection Types

​Splunk Integration

​Configuration

​Splunk HTTP Event Collector (HEC)

​Splunk Queries

​Elastic Stack (ELK) Integration

​Logstash Configuration

​AWX Configuration

​Kibana Dashboards

​Elasticsearch Queries

​Sumologic Integration

​Configuration

​Sumologic Setup

​Loggly Integration

​Configuration

​Log Schema

​Common Fields

​Job Event Schema

​Activity Stream Schema

​System Tracking Schema

​Selective Logging

​Configure Log Sources

​Log Levels

​Individual Facts

​Performance Considerations

​Asynchronous Processing

​Message Threading

​Network Optimization

​Troubleshooting

​Connection Issues

​No Logs Appearing

​SSL/TLS Errors

​Elasticsearch Performance

​API Configuration

​Get Current Settings

​Update Settings

​Best Practices

​Security

​Data Management

​Monitoring

​Compliance

​See Also

Overview

Supported Services

Officially Supported

Tested and Compatible

Potentially Compatible

Log Sources

Job Event Logs

Activity Stream Logs

System Tracking Logs

AWX Application Logs

Configuration

Settings

Connection Types

Splunk Integration

Configuration

Splunk HTTP Event Collector (HEC)

Splunk Queries

Elastic Stack (ELK) Integration

Logstash Configuration

AWX Configuration

Kibana Dashboards

Elasticsearch Queries

Sumologic Integration

Configuration

Sumologic Setup

Loggly Integration

Configuration

Log Schema

Common Fields

Job Event Schema

Activity Stream Schema

System Tracking Schema

Selective Logging

Configure Log Sources

Log Levels

Individual Facts

Performance Considerations

Asynchronous Processing

Message Threading

Network Optimization

Troubleshooting

Connection Issues

No Logs Appearing

SSL/TLS Errors

Elasticsearch Performance

API Configuration

Get Current Settings

Update Settings

Best Practices

Security

Data Management

Monitoring

Compliance

See Also