Run a Benchmark with Cua Bench: cb run Command Guide

The cb run command is the primary way to execute benchmarks in Cua Bench. It supports running a single task or an entire dataset, connecting different agent implementations, and storing trajectories for later analysis or RL training. This page covers every major flag and shows you what the output looks like.

Run commands overview

Command	Purpose
`cb run task <task-path>`	Run a single task, optionally with an agent or oracle.
`cb run dataset <dataset>`	Run all tasks in a dataset, in parallel.
`cb run list`	List all runs and their current status.
`cb run info <run-id>`	Show details for a completed or active run.
`cb run watch <run-id>`	Follow a run’s progress in the terminal.
`cb run logs <run-id>`	Print session logs from a run.
`cb run stop <run-id>`	Stop an active run immediately.

Running a single task

cb run task ./my-task \
  --agent cua-agent \
  --variant-id 0 \
  --max-steps 50

When --agent is omitted, the task runs setup only and exits without taking any agent actions. Add --oracle to use the task’s built-in reference solution instead of an agent.

Running a dataset

cb run dataset datasets/cua-bench-basic \
  --agent cua-agent \
  --max-parallel 4 \
  --output-dir ./results/run-001

The runner discovers all task directories in the dataset, expands their variants, and distributes them across the requested number of parallel workers.

Agent options

The --agent flag accepts built-in agent names or a path to a custom agent module.

Built-in agents
Custom agent module
Oracle mode

Cua Bench ships several reference agent implementations:

Agent name	Description
`cua-agent`	The default Cua computer-use agent backed by the `cua-agent` SDK.
`opencua-agent`	An open-weights alternative using the OpenCua model.
`gemini`	Google Gemini multimodal agent adapter.
`qwen3vl`	Qwen 3 Vision-Language agent adapter.

cb run dataset datasets/cua-bench-basic --agent cua-agent

Point --agent at a Python file that exposes an agent function with the signature (screenshot: bytes, task: Task) -> Action:

cb run dataset datasets/my-dataset \
  --agent ./agents/my_custom_agent.py

# agents/my_custom_agent.py
from cua_bench import Task
from cua_bench.types import Action, ClickAction, DoneAction
import random

def agent(screenshot: bytes, task: Task) -> Action:
    """Minimal random agent — replace with your model call."""
    if random.random() > 0.9:
        return DoneAction()
    return ClickAction(
        x=random.randint(0, 1920),
        y=random.randint(0, 1080),
    )

Run all tasks using their built-in reference solutions instead of an agent. Useful for validating a dataset before deploying it to agent evaluation:

cb run dataset datasets/cua-bench-basic --oracle

Parallel execution with `--max-parallel`

The --max-parallel flag controls how many task variants run concurrently. Each worker gets its own isolated environment session.

cb run dataset datasets/cua-bench-workflows \
  --agent cua-agent \
  --max-parallel 8 \
  --max-steps 100

Start with --max-parallel 4 and increase it based on available CPU and memory. Each native-provider worker requires a container instance; simulated-provider workers are lighter and support higher concurrency.

Output directory and trajectory storage

Use --output-dir to control where results and trajectories are written:

cb run dataset datasets/cua-bench-basic \
  --agent cua-agent \
  --max-parallel 4 \
  --output-dir ./results/experiment-01

The output directory receives:

results/experiment-01/
├── run-summary.json          # Aggregate scores and metadata
├── click-button/
│   └── variant-0/
│       ├── trajectory.json   # Full observation–action sequence
│       └── screenshots/      # Per-step screenshots
├── fill-form/
│   └── variant-0/
│       ├── trajectory.json
│       └── screenshots/
└── ...

Complete flag reference

Flag	Default	Description
`--agent`	none	Agent name or path to agent module.
`--model`	agent default	Model identifier to pass to the agent.
`--oracle`	false	Use the task’s built-in reference solution.
`--max-steps`	100	Maximum agent steps per task variant.
`--max-parallel`	4	Maximum number of concurrent workers.
`--output-dir`	current directory	Directory for results and trajectories.
`--image`	platform default	Container image to use for native-provider tasks.
`--platform`	auto	Execution platform (e.g., `docker`, `lume`).
`--split`	`train`	Dataset split to run.
`--task-filter`	none	Glob pattern to filter tasks by name.
`--variant-id`	all	Run only a specific variant index.

Scoring and evaluation output

After every run, the terminal prints a summary table followed by the aggregate score:

Run ID: run-a1b2c3d4
─────────────────────────────────────────────────────
 Task               Variant  Agent steps  Reward
─────────────────────────────────────────────────────
 click-button       0        3            1.0
 color-picker       0        7            1.0
 date-picker        0        9            0.0
 drag-drop          0        12           1.0
 fill-form          0        8            1.0
 right-click-menu   0        5            1.0
 select-dropdown    0        4            1.0
 toggle-switch      0        2            1.0
 typing-input       0        6            1.0
─────────────────────────────────────────────────────
 Total: 9   Success: 8   Failed: 1   Avg reward: 0.89
 Duration: 74.2 s   Output: ./results/run-a1b2c3d4

The machine-readable equivalent lives in run-summary.json:

{
  "run_id": "run-a1b2c3d4",
  "total_tasks": 9,
  "success_count": 8,
  "failed_count": 1,
  "avg_reward": 0.889,
  "duration_seconds": 74.2,
  "output_dir": "./results/run-a1b2c3d4",
  "task_results": [
    {
      "task_path": "click-button",
      "variant_id": 0,
      "success": true,
      "reward": 1.0,
      "steps": 3
    }
  ]
}

Exporting trajectories for RL training

Each trajectory.json stores the complete observation–action sequence in the format expected by Cua Bench’s reinforcement-learning pipeline:

{
  "task_path": "click-button",
  "variant_id": 0,
  "reward": 1.0,
  "steps": [
    {
      "observation": { "screenshot_b64": "..." },
      "action": { "type": "click", "x": 342, "y": 215 }
    }
  ]
}

Load trajectories programmatically using the cb.trace API or with the CLI:

# View a trajectory in the terminal
cb trace view run-a1b2c3d4

# Open the agent trajectory viewer in the browser
cb trace open run-a1b2c3d4

Trajectories produced by the oracle are particularly useful as demonstration data for behavior cloning, because the oracle always takes the known-correct action sequence.

Infrastructure benchmarking

To measure the throughput of the worker infrastructure independently of task logic, use the built-in benchmark script:

uv run python -m cua_bench.scripts.benchmark_workers \
  --num_workers 16 \
  --num_steps 10

The script reports average reset time, average step time, step throughput (steps/sec), and average finish time — useful for capacity planning before a large dataset run.

Get Started

Concepts

Cua Driver

Cua Sandbox

Lume

Cua Bench

Use Cua With

Run a Benchmark with Cua Bench: cb run Command Guide

Run commands overview

Running a single task

Running a dataset

Agent options

Parallel execution with `--max-parallel`

Output directory and trajectory storage

Complete flag reference

Scoring and evaluation output

Exporting trajectories for RL training

Infrastructure benchmarking

Build docs developers (and LLMs) love

Get Started

Concepts

Cua Driver

Cua Sandbox

Lume

Cua Bench

Use Cua With

Documentation Index

​Run commands overview

​Running a single task

​Running a dataset

​Agent options

​Parallel execution with --max-parallel

​Output directory and trajectory storage

​Complete flag reference

​Scoring and evaluation output

​Exporting trajectories for RL training

​Infrastructure benchmarking

Build docs developers (and LLMs) love

Run commands overview

Running a single task

Running a dataset

Agent options

Parallel execution with `--max-parallel`

Output directory and trajectory storage

Complete flag reference

Scoring and evaluation output

Exporting trajectories for RL training

Infrastructure benchmarking