Cua Driver’s default operation mode keeps the agent working in the background while you continue using your machine. Clicks and keystrokes reach a target window without raising it, without moving your real mouse pointer, and without switching your frontmost application. The driver renders a synthetic cursor overlay so you can watch what the agent is doing — but your actual cursor stays exactly where you left it. This is described as a best-effort contract rather than an absolute promise. Most automation stays background through accessibility actions, routed input, and window-specific capture. A small number of apps and OS surfaces only accept foreground input, so the driver reports that limit and lets the caller choose a foreground escalation for that specific action.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/trycua/cua/llms.txt
Use this file to discover all available pages before exploring further.
Why background matters
Traditional GUI automation assumes the automated app owns the desktop. It activates a window, moves the pointer, and repeats. That works fine for unattended jobs on disposable machines, but breaks down the moment a person is using the same desktop — every activation interrupts their flow. Cua Driver’s default path lets an agent operate a background app while you keep coding, reading logs, or using another tool entirely. The agent and the person share the desktop without stepping on each other.What the agent observes
get_window_state(pid, window_id) returns both the accessibility tree and a screenshot in a single call — no capture mode to pick, no extra round trip. The two signals arrive together because each catches what the other misses:
- The accessibility tree is the ground truth for what is clickable: roles, labels, advertised actions, and an
element_indexhandle on every actionable element. - The screenshot tells you which one. It disambiguates repeated or empty labels and shows captions, colors, and layout that the tree omits — common in Chromium and Electron.
include_screenshot: false to skip the screen grab for that call.
capture_mode is deprecated and ignored. Both the accessibility tree and screenshot are always returned regardless of what you pass.Action rungs
Two rungs deliver actions. The one you use is selected by how you address the target — not by a separate mode flag:| Rung | Address with | Delivered through | Properties |
|---|---|---|---|
| Element AX action | element_index / element_token | Accessibility API (UIA on Windows, AXPerformAction on macOS, AT-SPI on Linux) | Backgroundable, z-order-independent, driver-verifiable |
| Element PX action | x, y coordinates | Pixel-routed input from the screenshot in the get_window_state response | Best-effort; caller confirms effect off the screenshot |
suspected_noop, or when the tree disagrees with the pixels.
Delivery modes
Setdelivery_mode per action call on all input tools (click, double_click, right_click, drag, scroll, type_text, press_key, hotkey):
delivery_mode | Behavior |
|---|---|
background (default) | Input is routed to the target process/window/element directly. The user’s frontmost app, real cursor, and window z-order are untouched when the target surface supports it. |
foreground | The target is briefly raised for that action, input lands on the now-active window, then the prior frontmost is restored. Use this when a background attempt did not land, or when the app only accepts events while foregrounded. |
element_index) address an element instead of the focused window, so they hold the background path without any delivery_mode flag. The delivery axis matters most for pixel-rung actions, where background routes the event to the target and foreground raises the window first.
How fallback works
When background delivery is not possible, the driver returns a structured refusal rather than silently disturbing your desktop. The recommended escalation ladder is:Act by element_index in the background
The accessibility rung is backgroundable and driver-verifiable. Use it by default.
Fall back to x, y from the window screenshot
If the element path is unavailable or unverifiable, act by pixel coordinate using the screenshot already present in the
get_window_state response.effect: "confirmed"— the driver verified the result through AX read-back.effect: "unverifiable"— delivery ran, but the driver cannot prove the application applied it.effect: "suspected_noop"— an AX action ran but almost certainly did not change the target.escalation— machine-readable hint:"px"to act off the screenshot,"foreground"to front the target, or"page"to use the browser-tab DOM path.
Browser targeting and background delivery
Browsers present two identities at once: the desktop knows a native process and window; the browser runtime knows DevTools targets and tabs. Acting safely in the background requires proof that both identities describe the same surface.get_browser_state correlates a native (pid, window_id) with a browser target and mints opaque target and tab capabilities. Raw DevTools target identifiers are not part of the public contract. Selected-tab state is deliberately tri-state: active: true or false is reported only when the native window title uniquely identifies one DevTools tab. Duplicate titles produce active: null for every candidate instead of guessing.
The Chrome DevTools Protocol adds a full-background rung for page-aware work. Page navigation, text insertion, and explicit synthetic DOM clicks can address an occluded tab without moving the real cursor or borrowing keyboard focus.
Platform mechanisms
Each platform exposes different background capabilities. Cua Driver chooses the most background-capable path available for the OS and application surface.- macOS
- Windows
- Linux
The Accessibility API can press buttons, set values, and read semantic state even when an app is not frontmost. ScreenCaptureKit captures a specific window without requiring it to be raised or visible on the active Space. Cua Driver also uses scoped CoreGraphics and SkyLight delivery for routed input when an app responds better to pointer-like events than accessibility actions.Some macOS surfaces still need foreground: SwiftUI windows parked on another Space can lose their detailed accessibility tree, and game/canvas surfaces may reject routed input. Those cases return structured refusals.
Capture scope
Capture scope controls what coordinate space actions use:capture_scope | Coordinate space | Capture surface |
|---|---|---|
window (default) | Per-window. Actions carry pid + window_id; coordinates are window-relative or addressed by element_index. | get_window_state |
desktop | Screen-absolute. Window-less actions land at absolute screen coordinates via hit-testing. | get_desktop_state with full display |
Next steps
Permission Modes
Control which tools and resources an agent may reach.
Windows & Linux
Platform-specific limits, SSH mode on Windows, and Wayland notes.