Cua Bench Tasks: Lifecycle, Variants, and Custom Tasks

A Cua Bench task is a repeatable experiment, not only a natural-language instruction. It combines the state an agent starts from, the interface it can operate, and the evidence used to score the result. This page explains the task lifecycle, shows how to write a custom task in Python, and covers dataset formats for grouping multiple tasks into a benchmark run.

The four lifecycle phases

Every task module describes four distinct phases, each decorated with the corresponding cua_bench decorator:

Phase	Decorator	Purpose
Configuration	`@cb.tasks_config`	Returns one or more `Task` objects describing the variants available for a dataset split.
Setup	`@cb.setup_task`	Prepares the selected variant — opens applications, loads files, positions windows.
Solve	`@cb.solve_task`	Optional oracle: a known procedure that exercises the task so the evaluator can confirm it works.
Evaluate	`@cb.evaluate_task`	Inspects final state and returns a list of reward values between `0.0` and `1.0`.

Configuration is shared across the dataset split, while setup, solve, and evaluate each operate on a single selected variant and its session.

Task lifecycle diagram

dataset
  └── task definition
        └── variant (via tasks_config)
              ├── setup_task   → environment ready
              ├── agent loop   → actions taken
              └── evaluate_task → reward(s) returned

The oracle (solve_task) replaces the agent loop during validation runs. Keeping the oracle and evaluator separate means the evaluator can score a human attempt, the oracle, or an agent trajectory with the same code.

Writing a custom task

The following example creates a task that asks an agent to click a Submit button in a simulated HTML desktop window.

# my-task/main.py
from pathlib import Path
import cua_bench as cb


@cb.tasks_config(split="train")
def load():
    """Return one variant per button label we want to test."""
    return [
        cb.Task(
            description='Click the "Submit" button on the page.',
            metadata={"button_text": "Submit"},
            computer={
                "provider": "simulated",
                "setup_config": {
                    "os_type": "macos",
                    "width": 800,
                    "height": 600,
                },
            },
        )
    ]


# Module-level state shared between lifecycle functions
_pid = None


@cb.setup_task(split="train")
async def start(task_cfg: cb.Task, session: cb.DesktopSession):
    global _pid
    _pid = await session.launch_window(
        html=(Path(__file__).parent / "gui/index.html").read_text(),
        title="Task Window",
        width=400,
        height=300,
    )


@cb.solve_task(split="train")
async def solve(task_cfg: cb.Task, session: cb.DesktopSession):
    """Oracle: click the submit button by CSS selector."""
    global _pid
    if _pid is not None:
        await session.click_element(_pid, ".btn")


@cb.evaluate_task(split="train")
async def evaluate(task_cfg: cb.Task, session: cb.DesktopSession) -> list[float]:
    """Check whether the submit button was clicked."""
    global _pid
    if _pid is None:
        return [0.0]
    submitted = await session.execute_javascript(_pid, "window.__submitted")
    return [1.0] if submitted is True else [0.0]

Every decorator must use the same split value (e.g., "train") so the runner can match lifecycle functions when it loads the module.

Task variants

Variants let a benchmark test the same skill across different inputs, layouts, or operating systems without duplicating setup and evaluation code. Return multiple Task objects from tasks_config to generate variants:

@cb.tasks_config(split="train")
def load():
    os_types = ["macos", "linux"]
    labels = ["Submit", "OK", "Confirm"]
    return [
        cb.Task(
            description=f'Click the "{label}" button.',
            metadata={"button_text": label},
            computer={
                "provider": "simulated",
                "setup_config": {"os_type": os_type, "width": 800, "height": 600},
            },
        )
        for os_type in os_types
        for label in labels
    ]

This produces six variants. The runner assigns each variant its own environment session and evaluates them independently.

Provider types

The provider field in the computer configuration controls where the task runs:

Simulated
Native (container)

A lightweight desktop rendered through Playwright in a browser. No Docker or VM required. Ideal for fast iteration and CI.

"computer": {
    "provider": "simulated",
    "setup_config": {
        "os_type": "macos",  # or "linux", "win11"
        "width": 1280,
        "height": 800,
    },
}

A full operating-system environment running inside a Docker container. Required for tasks that need real applications.

"computer": {
    "provider": "native",
    "setup_config": {
        "os_type": "linux",
        "width": 1920,
        "height": 1080,
        "background": "#2d2d2d",
    },
}

Verifier patterns

The evaluator function can use a range of strategies to check whether the agent succeeded.

JavaScript check
Screenshot comparison
Shell command
Accessibility tree

Read DOM state or a JavaScript global set by the page after a successful action:

@cb.evaluate_task(split="train")
async def evaluate(task_cfg, session):
    value = await session.execute_javascript(pid, "window.__result")
    return [1.0 if value == "success" else 0.0]

Capture a screenshot and compare it to a reference image:

@cb.evaluate_task(split="train")
async def evaluate(task_cfg, session):
    screenshot = await session.screenshot(pid)
    reference = (Path(__file__).parent / "expected.png").read_bytes()
    similarity = compute_similarity(screenshot, reference)
    return [1.0 if similarity > 0.95 else 0.0]

Run a shell command inside the environment and inspect its output:

@cb.evaluate_task(split="train")
async def evaluate(task_cfg, session):
    result = await session.run_shell("cat /tmp/output.txt")
    return [1.0 if "expected content" in result.stdout else 0.0]

Query the accessibility tree to verify UI state:

@cb.evaluate_task(split="train")
async def evaluate(task_cfg, session):
    tree = await session.get_accessibility_tree(pid)
    # Check that a specific element is in the expected state
    found = any(
        el.get("role") == "checkbox" and el.get("value") == "checked"
        for el in tree.get("elements", [])
    )
    return [1.0 if found else 0.0]

Dataset format

A dataset is a directory containing task subdirectories, each with a main.py. The runner discovers all tasks in the directory and expands their variants:

datasets/my-dataset/
├── click-button/
│   └── main.py
├── fill-form/
│   ├── main.py
│   └── gui/
│       └── index.html
└── drag-drop/
    └── main.py

Run the full dataset with:

cb run dataset datasets/my-dataset \
  --agent cua-agent \
  --max-parallel 4

Inspect and validate a task

Before adding a task to a dataset, validate it with the built-in inspection and oracle commands:

# Inspect provider, variants, and lifecycle functions
cb task info ./my-task

# Run the oracle to confirm the evaluator returns the expected reward
cb interact ./my-task --variant-id 0 --oracle --no-wait

A successful oracle run should report:

✓ Solution complete
✓ Evaluation result: [1.0]
✓ Task completed successfully!

An oracle reward below the expected value indicates a task or environment problem. Do not use that variant in a benchmark until the oracle passes reliably.

Scaffold a new task

Use the interactive scaffolder to generate a task skeleton:

cb task create my-new-task

The scaffolder prompts for metadata (author, description, difficulty, category) and writes a main.py plus a minimal gui/index.html you can customize.

Get Started

Concepts

Cua Driver

Cua Sandbox

Lume

Cua Bench

Use Cua With

Cua Bench Tasks: Lifecycle, Variants, and Custom Tasks

The four lifecycle phases

Task lifecycle diagram

Writing a custom task

Task variants

Provider types

Verifier patterns

Dataset format

Inspect and validate a task

Scaffold a new task

Build docs developers (and LLMs) love

Get Started

Concepts

Cua Driver

Cua Sandbox

Lume

Cua Bench

Use Cua With

Documentation Index

​The four lifecycle phases

​Task lifecycle diagram

​Writing a custom task

​Task variants

​Provider types

​Verifier patterns

​Dataset format

​Inspect and validate a task

​Scaffold a new task

Build docs developers (and LLMs) love

The four lifecycle phases

Task lifecycle diagram

Writing a custom task

Task variants

Provider types

Verifier patterns

Dataset format

Inspect and validate a task

Scaffold a new task