Capacity Planning

Overview

The AWX capacity system determines how many jobs can run on an instance based on available memory and CPU resources. Capacity management ensures efficient resource utilization while preventing system overload.

Capacity Fundamentals

Capacity is calculated based on:

Memory capacity (mem_capacity): Available system memory
CPU capacity (cpu_capacity): Available CPU cores
Forks: Number of simultaneous connections Ansible maintains

How Capacity Works

Each instance has a calculated capacity based on hardware resources
Jobs consume capacity based on their “impact” (primarily fork count)
The task manager assigns jobs to instances with sufficient capacity
When capacity is exhausted, jobs wait until resources free up

Capacity is not a zero-sum system. If only one instance is available, AWX allows jobs to run even if they exceed capacity, ensuring jobs don’t become permanently blocked.

Capacity Algorithms

Memory-Relative Capacity (Default)

Calculates capacity based on available memory, allowing CPU overcommit:

capacity = (total_memory_mb - 2048) / mem_per_fork

Example: 4GB system

(4096 - 2048) / 100 = ~20 forks

Key Points:

Reserves 2GB for AWX services
Default: 100MB per fork (SYSTEM_TASK_FORKS_MEM)
Best for I/O-bound workloads
Protects against out-of-memory conditions

Configuration:

# /etc/tower/conf.d/capacity.py
SYSTEM_TASK_FORKS_MEM = 100  # MB per fork

CPU-Relative Capacity

Calculates capacity based on CPU cores:

capacity = cpu_cores * forks_per_cpu

Example: 4-core system

4 * 4 = 16 forks

Key Points:

Default: 4 forks per core (SYSTEM_TASK_FORKS_CPU)
Best for CPU-bound workloads
Reduces contention for compute resources

Configuration:

# /etc/tower/conf.d/capacity.py
SYSTEM_TASK_FORKS_CPU = 4  # Forks per CPU core

Capacity Adjustment

Balance between memory and CPU capacity using capacity_adjustment:

final_capacity = min_capacity + (max_capacity - min_capacity) * capacity_adjustment

Values:

0.0: Use minimum (most conservative)
0.5: 50/50 balance
1.0: Use maximum (most aggressive)

Example: CPU=16, Memory=20, adjustment=0.5

16 + (20 - 16) * 0.5 = 18 forks

Set via API:

curl -X PATCH https://awx.example.org/api/v2/instances/1/ \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"capacity_adjustment": 0.5}'

Job Impact

Job Types and Impact

Jobs have two impact types:

Control Impact

Fixed: AWX_CONTROL_NODE_TASK_IMPACT (default: 1)
Applied to: The instance controlling the job

Execution Impact

Variable: Based on job type

Job Type	Execution Impact	Formula
Job Templates	`forks + 1`	`min(forks, host_count) + 1`
Ad-hoc Commands	`forks + 1`	`min(forks, host_count) + 1`
Project Updates	`1`	Fixed
Inventory Updates	`1`	Fixed
System Jobs	`5`	Fixed

The +1 accounts for the Ansible parent process that coordinates execution.

Impact Examples

Example 1: Hybrid Node (Control + Execution)

Settings: AWX_CONTROL_NODE_TASK_IMPACT=1, forks=5, hosts=3

Total Impact = Control Impact + Execution Impact
             = 1 + (min(5, 3) + 1)
             = 1 + 4
             = 5

Example 2: Container Group Job

Settings: AWX_CONTROL_NODE_TASK_IMPACT=1

Controller Node Impact = 1 (control only)
Execution Node Impact = 0 (external to cluster)

Example 3: Project Update

Settings: AWX_CONTROL_NODE_TASK_IMPACT=1

Total Impact = Control Impact + Execution Impact
             = 1 + 1
             = 2

Control Node Task Impact

The AWX_CONTROL_NODE_TASK_IMPACT setting controls how much capacity controlling jobs consumes.

When to Adjust

Increase (AWX_CONTROL_NODE_TASK_IMPACT = 2 or higher):

Control plane CPU/memory usage is high
Many concurrent container group jobs
Job event processing is slow
Need to throttle concurrent jobs

Decrease (AWX_CONTROL_NODE_TASK_IMPACT = 0.5 or lower):

Control plane is underutilized
Most jobs run on execution nodes
Want more concurrent job control

Configuration:

# /etc/tower/conf.d/capacity.py
AWX_CONTROL_NODE_TASK_IMPACT = 2  # More conservative

Container groups have effectively infinite capacity. Without proper control plane throttling, you can overwhelm your control nodes with too many concurrent jobs.

Instance Groups

Instance Group Capacity

Instance groups aggregate capacity from member instances. Configure group-wide limits:

max_concurrent_jobs

Maximum concurrent jobs across entire group:

curl -X PATCH https://awx.example.org/api/v2/instance_groups/1/ \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"max_concurrent_jobs": 50}'

max_forks

Maximum total forks across entire group:

curl -X PATCH https://awx.example.org/api/v2/instance_groups/1/ \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"max_forks": 200}'

Container Group Capacity Planning

Calculate max_concurrent_jobs

Based on pod resource requests:

max_concurrent_jobs = (node_memory_mb * 1024) // pod_memory_mb

Example: 8GB node, 100MB pod

(8 * 1024) // 100 = 81 jobs

Calculate max_forks

Based on Ansible memory usage (100MB per fork):

max_forks = (node_memory_mb * 1024) // 100

Example: 8GB node

(8 * 1024) // 100 = 81 forks

With max_forks=81:

81 jobs with 1 fork each, OR
40 jobs with 2 forks each, OR
2 jobs with 40 forks each

Configure Container Group:

curl -X PATCH https://awx.example.org/api/v2/instance_groups/2/ \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "max_concurrent_jobs": 81,
    "max_forks": 81
  }'

Capacity Monitoring

Check Instance Capacity

# View all instances
curl https://awx.example.org/api/v2/instances/ \
  -H "Authorization: Bearer <token>" | jq '.results[] | {
    hostname: .hostname,
    capacity: .capacity,
    consumed_capacity: .consumed_capacity,
    percent_used: ((.consumed_capacity / .capacity) * 100)
  }'

Monitor Running Jobs

# Check jobs by status
curl https://awx.example.org/api/v2/jobs/?status=running \
  -H "Authorization: Bearer <token>" | jq '.count'

# View jobs waiting for capacity
curl https://awx.example.org/api/v2/jobs/?status=pending \
  -H "Authorization: Bearer <token>" | jq '.results[] | {
    id: .id,
    name: .name,
    status: .status
  }'

Instance Group Status

# Check instance group capacity
curl https://awx.example.org/api/v2/instance_groups/ \
  -H "Authorization: Bearer <token>" | jq '.results[] | {
    name: .name,
    capacity: .capacity,
    consumed_capacity: .consumed_capacity,
    jobs_running: .jobs_running,
    jobs_total: .jobs_total
  }'

Capacity Optimization

Recommendations by Workload

I/O-Bound Workloads

(Network operations, cloud APIs, service calls)

# Prefer memory capacity
SYSTEM_TASK_FORKS_MEM = 100
SYSTEM_TASK_FORKS_CPU = 6  # Higher allows more overcommit

On instances:

curl -X PATCH https://awx.example.org/api/v2/instances/1/ \
  -d '{"capacity_adjustment": 1.0}'  # Use memory capacity

CPU-Bound Workloads

(Computation, template rendering, encryption)

# Prefer CPU capacity
SYSTEM_TASK_FORKS_MEM = 150  # Higher than default
SYSTEM_TASK_FORKS_CPU = 4

On instances:

curl -X PATCH https://awx.example.org/api/v2/instances/1/ \
  -d '{"capacity_adjustment": 0.0}'  # Use CPU capacity

Mixed Workloads

# Balanced settings
SYSTEM_TASK_FORKS_MEM = 120
SYSTEM_TASK_FORKS_CPU = 5

On instances:

curl -X PATCH https://awx.example.org/api/v2/instances/1/ \
  -d '{"capacity_adjustment": 0.5}'  # Balanced

Instance Sizing Guidelines

Instance Type	vCPU	Memory	Est. Capacity	Workload
Small	2	4 GB	~8-20 forks	Dev/test
Medium	4	8 GB	~24-60 forks	Light production
Large	8	16 GB	~56-140 forks	Production
XLarge	16	32 GB	~120-300 forks	Heavy production

Dedicated Instance Groups

Create dedicated groups for specific workloads:

# Create high-memory group for large inventories
curl -X POST https://awx.example.org/api/v2/instance_groups/ \
  -H "Authorization: Bearer <token>" \
  -d '{
    "name": "high-memory",
    "max_concurrent_jobs": 10,
    "max_forks": 100
  }'

# Create CPU-optimized group for compute-heavy jobs
curl -X POST https://awx.example.org/api/v2/instance_groups/ \
  -d '{
    "name": "cpu-optimized",
    "max_concurrent_jobs": 50,
    "max_forks": 50
  }'

Troubleshooting

Jobs Stuck in Pending

Cause: Insufficient capacity

# Check instance capacity
curl https://awx.example.org/api/v2/instances/ | jq '.results[] | {
  hostname, capacity, consumed_capacity
}'

# Check instance group assignments
curl https://awx.example.org/api/v2/jobs/123/ | jq '.instance_group'

Solutions:

Add more instances to the group
Increase instance capacity (add memory/CPU)
Adjust capacity_adjustment to 1.0
Reduce job fork counts
Add fallback instance groups

Capacity Calculations Seem Wrong

# Check capacity settings
awx-manage shell -c "
from django.conf import settings
print(f'SYSTEM_TASK_FORKS_MEM: {settings.SYSTEM_TASK_FORKS_MEM}')
print(f'SYSTEM_TASK_FORKS_CPU: {settings.SYSTEM_TASK_FORKS_CPU}')
print(f'AWX_CONTROL_NODE_TASK_IMPACT: {settings.AWX_CONTROL_NODE_TASK_IMPACT}')
"

# Force instance health check
curl -X POST https://awx.example.org/api/v2/instances/1/health_check/ \
  -H "Authorization: Bearer <token>"

High Memory Usage

Symptoms: Jobs failing with OOM, system slowness Solutions:

Reduce SYSTEM_TASK_FORKS_MEM (more conservative)
Use CPU capacity (capacity_adjustment=0.0)
Reduce JOB_EVENT_WORKERS
Limit concurrent jobs on instance group
Add more memory to instances

High CPU Usage

Symptoms: Slow job execution, high load average Solutions:

Reduce SYSTEM_TASK_FORKS_CPU
Use memory capacity (capacity_adjustment=1.0)
Add more CPU cores
Reduce job forks in templates
Use dedicated execution nodes

Best Practices

Monitor continuously: Track capacity metrics and adjust as needed
Start conservative: Begin with lower capacity and increase gradually
Separate workloads: Use dedicated instance groups for different job types
Test under load: Simulate production load before going live
Document changes: Keep records of capacity adjustments and their effects
Plan for growth: Size instances with 20-30% headroom
Use execution nodes: Offload job execution from control plane
Throttle container groups: Set appropriate max_concurrent_jobs limits

Configuration - Configure capacity settings
Metrics - Monitor capacity usage
Clustering - Instance groups and clustering

​Overview

​Capacity Fundamentals

​How Capacity Works

​Capacity Algorithms

​Memory-Relative Capacity (Default)

​CPU-Relative Capacity

​Capacity Adjustment

​Job Impact

​Job Types and Impact

​Control Impact

​Execution Impact

​Impact Examples

​Example 1: Hybrid Node (Control + Execution)

​Example 2: Container Group Job

​Example 3: Project Update

​Control Node Task Impact

​When to Adjust

​Instance Groups

​Instance Group Capacity

​max_concurrent_jobs

​max_forks

​Container Group Capacity Planning

​Calculate max_concurrent_jobs

​Calculate max_forks

​Capacity Monitoring

​Check Instance Capacity

​Monitor Running Jobs

​Instance Group Status

​Capacity Optimization

​Recommendations by Workload

​I/O-Bound Workloads

​CPU-Bound Workloads

​Mixed Workloads

​Instance Sizing Guidelines

​Dedicated Instance Groups

​Troubleshooting

​Jobs Stuck in Pending

​Capacity Calculations Seem Wrong

​High Memory Usage

​High CPU Usage

​Best Practices

​Related Resources

Overview

Capacity Fundamentals

How Capacity Works

Capacity Algorithms

Memory-Relative Capacity (Default)

CPU-Relative Capacity

Capacity Adjustment

Job Impact

Job Types and Impact

Control Impact

Execution Impact

Impact Examples

Example 1: Hybrid Node (Control + Execution)

Example 2: Container Group Job

Example 3: Project Update

Control Node Task Impact

When to Adjust

Instance Groups

Instance Group Capacity

max_concurrent_jobs

max_forks

Container Group Capacity Planning

Calculate max_concurrent_jobs

Calculate max_forks

Capacity Monitoring

Check Instance Capacity

Monitor Running Jobs

Instance Group Status

Capacity Optimization

Recommendations by Workload

I/O-Bound Workloads

CPU-Bound Workloads

Mixed Workloads

Instance Sizing Guidelines

Dedicated Instance Groups

Troubleshooting

Jobs Stuck in Pending

Capacity Calculations Seem Wrong

High Memory Usage

High CPU Usage

Best Practices

Related Resources