Container Groups

Container Groups enable AWX to provision job environments on-demand as Pods that exist only for the duration of the playbook run, providing a clean, isolated execution environment for every job.

Overview

In a traditional AWX installation, jobs (ansible-playbook runs) are executed directly on cluster members. Container Groups introduce an ephemeral execution model that ensures a clean environment for every job run.

Execution Models

Ephemeral

Pods created on-demand, exist only during job execution

Always-On

Manually created instances that persist beyond individual jobs

Configuration

A ContainerGroup is an InstanceGroup that has an associated Credential for connecting to an OpenShift or Kubernetes cluster.

Prerequisites

Kubernetes/OpenShift Access

Ensure AWX can reach your Kubernetes or OpenShift cluster API

Container Credential

Create credentials with appropriate authentication method

Namespace Permissions

Verify permissions to create pods in target namespace

Create Container Credential

A Credential must be created where the associated CredentialType is one of:

OpenShift User/Pass

openshift_username_password

OpenShift Token

openshift_token

Kubernetes Bearer Token

kubernetes_bearer_token

Example: Creating a Kubernetes Bearer Token Credential

curl -Lk --user 'admin:password' \
     -X POST \
     -d '{
       "name": "Kubernetes Cluster",
       "credential_type": 17,
       "inputs": {
         "host": "https://kubernetes.example.com:6443",
         "bearer_token": "eyJhbGciOiJSUzI1NiIs...",
         "verify_ssl": true
       }
     }' \
     -H 'Content-Type: application/json' \
     https://localhost:8043/api/v2/credentials/

Create a Container Group

Once a Credential has been associated with an InstanceGroup, the InstanceGroup.kubernetes property will return True. Via API:

curl -Lk --user 'admin:password' \
     -X POST \
     -d '{
       "name": "Kubernetes Container Group",
       "credential": 42,
       "is_container_group": true
     }' \
     -H 'Content-Type: application/json' \
     https://localhost:8043/api/v2/instance_groups/

Once created, you can associate the Container Group with Job Templates, Inventories, or Organizations just like regular Instance Groups.

Pod Customization

Default Pod Spec

AWX provides a simple default pod specification in code. This includes:

Container image (execution environment)
Resource requests/limits
Volume mounts
Security context

Custom Pod Specifications

A custom YAML document may be provided which will be merged on top of the default pod spec. This allows you to customize:

Container Image

Specify custom execution environment image

Namespace

Target specific Kubernetes namespace

Resources

Set CPU and memory limits

Node Selection

Use node selectors or affinity rules

Security Context

Configure pod security settings

Volumes

Mount additional volumes

Example: Custom Pod Spec Override

Using Custom Image:

cat > api_request.json <<EOF
{
  "pod_spec_override": "spec:\n  containers:\n    - image: my-custom-image"
}
EOF

curl -Lk --user 'admin:password' \
     -X PATCH \
     -d @api_request.json \
     -H 'Content-Type: application/json' \
     https://localhost:8043/api/v2/instance_groups/2/

Full Pod Customization Example:

spec:
  serviceAccountName: awx-jobs
  automountServiceAccountToken: true
  containers:
    - image: quay.io/ansible/awx-ee:latest
      name: worker
      args:
        - ansible-runner
        - worker
        - '--private-data-dir=/runner'
      resources:
        requests:
          cpu: 250m
          memory: 512Mi
        limits:
          cpu: 1000m
          memory: 2Gi
      env:
        - name: MY_CUSTOM_VAR
          value: "custom_value"
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "awx"
      effect: "NoSchedule"

The pod spec override is merged with the default spec. Be careful not to override critical fields that AWX requires for job execution.

Advanced Configuration

Namespace Configuration

Specify the namespace where pods should be created:

spec:
  namespace: awx-jobs

Ensure the credential has permissions to create pods in the specified namespace.

Resource Limits

Set appropriate resource requests and limits:

spec:
  containers:
    - resources:
        requests:
          cpu: "500m"
          memory: "1Gi"
        limits:
          cpu: "2"
          memory: "4Gi"

Resource Guidelines:

Requests: Minimum resources guaranteed to the pod
Limits: Maximum resources the pod can consume
Set requests lower than limits to allow bursting
Consider your playbook requirements and cluster capacity

Image Pull Secrets

For private container registries:

spec:
  imagePullSecrets:
    - name: private-registry-secret
  containers:
    - image: private-registry.example.com/awx-ee:latest

Security Context

Configure security settings:

spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  containers:
    - securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL

Node Selection and Affinity

Target specific nodes:

spec:
  nodeSelector:
    disktype: ssd
    size: large
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-role.kubernetes.io/worker
                operator: Exists

Job Execution Flow

Job Submitted

User launches a job targeting a Container Group

Pod Creation

AWX creates a pod in the Kubernetes cluster using the custom spec

Job Execution

Ansible playbook runs inside the container

Results Collection

Job output is streamed back to AWX

Pod Deletion

Pod is automatically deleted after job completion

Always-On Instances

For persistent execution capacity, manually create instances through the AWX API or UI:

curl -Lk --user 'admin:password' \
     -X POST \
     -d '{
       "hostname": "persistent-worker-01",
       "node_type": "execution",
       "node_state": "installed"
     }' \
     -H 'Content-Type: application/json' \
     https://localhost:8043/api/v2/instances/

Kubernetes API Reference

For a full list of customizable pod options, refer to the Kubernetes documentation: Pod v1 API Reference

Troubleshooting

Pods Not Starting

Check Credentials

Verify the Container Group credential can authenticate to the cluster

Verify RBAC

Ensure the service account has permissions to create pods

Review Pod Logs

Check Kubernetes events and pod logs for errors

Validate Pod Spec

Ensure custom pod spec override is valid YAML and doesn’t conflict with defaults

Jobs Failing in Containers

Common issues:

Insufficient resources (CPU/memory limits too low)
Missing image pull secrets for private registries
Network connectivity issues from pods
Volume mount permissions

Performance Issues

Optimize Container Group Performance:

Use local container image caches
Set appropriate resource requests/limits
Consider node affinity to reduce pod scheduling time
Use persistent volumes for shared data

Best Practices

Resource Planning

Size resource requests based on typical playbook requirements

Image Optimization

Use optimized execution environment images to reduce startup time

Namespace Isolation

Use separate namespaces for different teams or environments

Monitor Capacity

Watch cluster capacity and scale nodes as needed

Benefits of Container Groups

Isolation: Each job runs in a fresh, clean environment
Scalability: Dynamically scale job capacity with cluster resources
Flexibility: Use different execution environments per job
Resource Efficiency: Only consume resources during job execution
Security: Leverage Kubernetes security features and isolation

​Overview

​Execution Models

Ephemeral

Always-On

​Configuration

​Prerequisites

​Create Container Credential

OpenShift User/Pass

OpenShift Token

Kubernetes Bearer Token

​Create a Container Group

​Pod Customization

​Default Pod Spec

​Custom Pod Specifications

Container Image

Namespace

Resources

Node Selection

Security Context

Volumes

​Example: Custom Pod Spec Override

​Advanced Configuration

​Namespace Configuration

​Resource Limits

​Image Pull Secrets

​Security Context

​Node Selection and Affinity

​Job Execution Flow

​Always-On Instances

​Kubernetes API Reference

​Troubleshooting

​Pods Not Starting

​Jobs Failing in Containers

​Performance Issues

​Best Practices

Resource Planning

Image Optimization

Namespace Isolation

Monitor Capacity

​Benefits of Container Groups

Overview

Execution Models

Configuration

Prerequisites

Create Container Credential

Create a Container Group

Pod Customization

Default Pod Spec

Custom Pod Specifications

Example: Custom Pod Spec Override

Advanced Configuration

Namespace Configuration

Resource Limits

Image Pull Secrets

Security Context

Node Selection and Affinity

Job Execution Flow

Always-On Instances

Kubernetes API Reference

Troubleshooting

Pods Not Starting

Jobs Failing in Containers

Performance Issues

Best Practices

Benefits of Container Groups