Computer use is the ability of an AI agent to operate a real computer and complete work across applications. A computer-use agent can inspect the computer’s current state, choose an action, observe the result, and continue until it reaches a goal. It works through code, structured tool calls, or the same graphical interfaces people use. Cua extends this with Computer-Use 2.0 — a broader model where an agent moves freely across all three action surfaces during a single task, rather than being limited to screenshot-driven clicks.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/trycua/cua/llms.txt
Use this file to discover all available pages before exploring further.
Computer-use models vs. agents vs. harnesses
Understanding who does what in a computer-use system avoids confusion about what Cua provides. A computer-use model interprets an observation and proposes an action. Depending on the model, the observation may include screenshots, accessibility information, text, or structured application state. The proposed action may be a mouse movement, a keystroke, a tool call, or a program to run. A computer-use agent is the running system around that model. It supplies the computer, tools, memory, instructions, permissions, and feedback loop needed to turn proposed actions into completed work. It also decides when to stop, retry, ask for confirmation, or switch action surfaces. An agent harness is the runtime that hosts this loop. A coding agent such as Claude Code or Codex supplies instructions, memory, and MCP connections. Cua Driver is the UI tool layer connected to that harness — it does not select or run the model.Three action surfaces
Computer-Use 2.0 treats the GUI as one of three complementary action surfaces. A capable agent chooses the surface that fits the next piece of work.Coding surface
The agent writes and runs code as the action itself. This is the right choice when:- The task is text-native (transforming data, processing files, writing output)
- The same operation must run many times
- A small program can cover more state than a human would want to click through manually
Tool-use surface
The agent makes typed calls to tools, functions, APIs, and MCP servers. Each call has named inputs and defined outputs. This surface reaches external systems for structured operations without requiring a script or GUI navigation.UI automation surface
The agent clicks, types, scrolls, and presses keys on the interface a person would use. This surface is the right choice when:- The work depends on visual state
- The application exposes no useful API
- The task involves signed-in sessions or machine state that already exists
The observe-decide-act loop
The observe-decide-act loop belongs specifically to the UI automation surface. Each iteration has three phases:- Observe — capture the current state through a screenshot, an accessibility tree, or both. The screenshot gives the model what a person would see. The accessibility tree gives structured semantic information from the operating system.
- Decide — the model uses the observation to select the next action. For GUI work, this includes grounding: translating a target (a button, a field, a menu item) into on-screen coordinates that input events can hit.
- Act — deliver the action to the target application. Then observe again.
How computer use differs from RPA and browser automation
Computer use overlaps with APIs, browser automation, and robotic process automation, but each has a different interface and tradeoff.| Approach | Best fit | Main limitation |
|---|---|---|
| API or structured tool | Stable, well-defined operations with typed inputs and outputs | Cannot reach behavior the API does not expose |
| Browser automation | Repeatable workflows contained within web pages | Does not cover native applications or OS-level UI |
| Traditional RPA | Known, predefined business processes | Depends on fixed selectors, coordinates, and workflow branches |
| Computer-use agent | Adaptive work across applications and interface types | Requires careful observation, recovery, permissions, and verification |
Where Cua fits
You bring the agent, which already handles coding and tool-use on its own. Cua gives that agent the UI automation surface and a real computer to act in. Cua Driver drives the GUI of a real machine you already have — macOS, Windows, or Linux. It is the right choice when the agent needs local applications, signed-in accounts, existing files, or machine state that already lives on that computer. Cua Sandbox is a fresh isolated computer running on local Docker or VM infrastructure. In a sandbox, all three action surfaces share one filesystem and OS state. An agent can click through an app and then run Python against the files that app produced — or set up state in code and then automate the UI over that state.Safety and isolation
Computer-use agents can affect the same files, accounts, and applications as a person, so environment and permissions are part of the system design. Best practices:- Use the least privilege needed for the task
- Require confirmation for consequential or irreversible actions
- Keep sensitive or untrusted work isolated from a personal computer
Further reading
Drive your first app
Give an agent access to a real desktop app on macOS, Windows, or Linux via Cua Driver.
Start a local sandbox
Create an isolated Linux computer where the agent has full code and GUI access.
How sandboxes work
Understand the isolation model, runtime backends, and image composition.
How Cua Driver works
Learn about MCP over stdio, the Rust runtime, permission modes, and the no-foreground contract.