Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/trycua/cua/llms.txt

Use this file to discover all available pages before exploring further.

Cua Bench is a framework for defining, running, and evaluating verifiable computer-use tasks. A task packages the prompt, simulated or real desktop environment, setup function, oracle solution, and evaluator into a single reusable unit. In this tutorial you create a small task, inspect its lifecycle, run it with the reference oracle, and then attempt it yourself — all without Docker, a VM, or an API key, because the task uses a simulated desktop.
Prerequisites: Python 3.12 or 3.13 and uv.

​
Steps

1

Install Cua Bench

Install the CLI and browser support needed for simulated tasks:
uv tool install 'cua-bench[browser]'
uv tool run --from 'cua-bench[browser]' playwright install chromium
Confirm the CLI is available:
cb --help
You should see command groups including run, interact, task, trace, and dataset.
2

Create a task

Create a working directory, enter it, and run the task scaffolder:
mkdir cua-bench-tutorial
cd cua-bench-tutorial
cb task create first-task
Use these values when prompted:
Author name: Your name
Author email: you@example.com
License [MIT]:
Task description: Click the Submit button
Task difficulty (easy|medium|hard) [easy]:
Task category (e.g., grounding, software-engineering) [grounding]:
Tags (comma-separated): button,simulated
The scaffolder generates the following structure:
first-task/
├── main.py
├── pyproject.toml
└── gui/
    └── index.html
main.py contains the task definition with a setup, solve (oracle), and evaluate function. gui/index.html is the small HTML interface the agent interacts with.
3

Inspect the task

Ask Cua Bench to load and display the task definition:
cb task info first-task
The output reports a simulated provider, one macOS-themed variant, and check marks for the setup, solve, and evaluate functions. This confirms the CLI can discover the complete task lifecycle.
4

Run the oracle

Run the task with its reference (oracle) solution:
cb interact first-task --variant-id 0 --oracle --no-wait
A desktop window opens while Cua Bench sets up the interface, clicks the Submit button, and evaluates the final state. The final lines should include:
✓ Solution complete
✓ Evaluation result: [1.0]
✓ Task completed successfully!
A reward of 1.0 means the evaluator observed the state that the oracle produced — the task was solved correctly.
5

Try the task yourself

Run the same variant without the oracle:
cb interact first-task --variant-id 0
Click Submit in the task window. Return to the terminal and press Enter to evaluate and close the task. You should see the same 1.0 reward if you clicked the right button.

​
What you built

You created a task with a prompt, simulated desktop, setup function, oracle solution, and evaluator. You then used the same evaluator to verify both the reference oracle solution and your own manual attempt. That evaluator is the single source of truth — it makes the task score reproducible regardless of who (or what model) solves it.

​
Running a dataset with an agent

Once you have tasks, you can run them in parallel against a real agent: 
# Run a dataset with the built-in cua-agent, up to 4 tasks in parallel
cb run dataset datasets/cua-bench-basic --agent cua-agent --max-parallel 4
Cua Bench records a trajectory for each run. You can inspect, score, or export trajectories as training data.

​
Supported benchmarks

OSWorld

A large-scale benchmark for evaluating desktop GUI agents across Windows, macOS, and Ubuntu tasks.

ScreenSpot

A grounding benchmark that tests the ability to locate UI elements from natural language descriptions.

Windows Arena

A benchmark for Windows desktop task automation with verifiable outcomes.

Custom tasks

Define your own tasks with custom environments, prompts, and evaluation logic.

​
Next steps

What is Cua Bench?

Understand the task lifecycle, verifiable tasks, agent adapters, and trajectory export.

Run and validate a task

Learn how to run, inspect, and validate existing tasks from the registry.

Task definition reference

Complete specification for task setup, variants, solve, and evaluate functions.

Task registry

Browse published tasks and the applications they cover.

Build docs developers (and LLMs) love

Get started for freeTalk to us