Documentation Index
Fetch the complete documentation index at: https://mintlify.com/trycua/cua/llms.txt
Use this file to discover all available pages before exploring further.
cua-driver exposes 54 MCP tools through a single stdio server started with cua-driver mcp. Every tool is also callable directly from the shell:
snake_case and identical between MCP and the CLI. Responses are MCP CallTool.Result envelopes: a text content block prefixed with a ✅ summary on success (or the error reason on failure), plus optional image or structured-content blocks for tools that produce them.
The shared parameter contract —
session, delivery_mode, capture_mode,
element_index, element_token, modifier, button — applies uniformly
across platforms. See the cross-cutting notes below each section for delivery
rung details.Runtime ownership. On Windows and Linux, bare
cua-driver mcp owns the
SDK runtime directly and shuts it down on stdin EOF. On macOS it proxies to
the installed CuaDriver.app daemon so AX and Screen Recording grants retain
the app-bundle identity. Pass --socket to select an explicit daemon endpoint
on any platform.Inspection Tools
These tools read desktop, window, and accessibility state without performing any action.list_apps
List macOS apps — both running and installed-but-not-running — with per-app state flags. Filters to NSApplicationActivationPolicyRegular apps only; background helpers and system agents are excluded.
Arguments: none.
Returned fields per app:
| Field | Description |
|---|---|
running | Whether a live process exists for this app. |
active | Whether it is the system-frontmost app (implies running). |
pid | Process ID when running; 0 otherwise. |
launch_path | Filesystem path to the .app bundle when known. |
kind | "desktop" for .app bundles on macOS. |
last_used | RFC 3339 timestamp from the bundle’s filesystem mtime, or null. |
list_windows
List all layer-0 top-level windows currently known to WindowServer, including minimized, off-Space, and hidden-launched windows.
When
true, drop windows not on the current Space. Default false.Optional PID filter — return only this process’s windows.
window_id, pid, app_name, title, bounds (x/y/width/height, top-left origin), z_index, is_on_screen, space_ids, current_space_id, on_current_space.
get_window_state
Walk a running app’s accessibility tree and return both a structured elements array and a Markdown rendering of the same tree. Every actionable element is tagged with [element_index N] in the markdown and as element_index in the structured array.
Target process ID.
Target window ID from
list_windows.Set
false to skip the screenshot and return the tree only — the fast path
when re-indexing before an element AX action.Case-insensitive substring filter applied to
tree_markdown and elements.
Returns matching rows plus their ancestor chain without renumbering indices.Cap on the total AX nodes walked (default 2 000). Lower for large Electron or
Obsidian trees.
Cap on the AX-tree walk depth (default 25).
Write the PNG to this file path (tilde-expanded) instead of embedding base64
in the response.
Optional session ID for the agent cursor and per-session state.
get_accessibility_tree
Return a lightweight desktop snapshot: running regular apps and on-screen visible windows with their bounds, z-order, and owner PID. Faster than get_window_state; requires no TCC grants.
Arguments: none.
get_desktop_state
Capture a full-display screenshot at native screen pixels for scope="desktop" GUI loops.
Write PNG here instead of base64.
Optional session ID.
get_screen_size
Return the logical size of the main display in points and its backing scale factor. Requires no TCC permissions.
Optional session ID.
get_cursor_position
Return the current mouse cursor position in screen points (origin top-left).
Optional session ID.
get_config
Return the current cua-driver configuration.
Arguments: none.
get_recording_state
Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder that will be written. Counter increments on every recorded action tool call and resets to 1 each time recording is (re-)enabled. Pure read-only.
Arguments: none.
get_agent_cursor_state
Return the session cursor’s theme, semantic playback, position, visibility, and motion.
Session ID to inspect.
Mouse Tools
All pointer tools share thedelivery_mode parameter: "background" (default) injects without stealing focus; "foreground" briefly fronts the target, acts, then restores the prior frontmost.
click
Click against a target PID. Prefer element_token over pixel coordinates — the token works on backgrounded, minimized, hidden, and off-Space windows and identifies exactly which element you are clicking.
Target process ID.
Target window ID. Required for
element_index.Opaque per-snapshot element handle from
structuredContent.elements[].element_token.
Preferred over element_index.Element index from
get_window_state. Requires the matching snapshot_id.X in window-local screenshot pixels (top-left origin of the
get_window_state PNG). Use for canvas, WebGL, or custom surfaces absent from the AX tree.Y in window-local screenshot pixels.
Mouse button:
"left", "right", or "middle".Click count (pixel path only).
Modifier keys:
cmd, shift, option/alt, ctrl."background" or "foreground".double_click
Double-click at (x, y) or on an AX element identified by element_index. The AX path performs AXOpen when the element supports it; otherwise falls back to a pixel double-click at the element’s on-screen center.
Target process ID.
Preferred element handle from the last snapshot.
Pixel path X coordinate.
Pixel path Y coordinate.
right_click
Right-click against a target PID. element_index path performs AXShowMenu; x, y path synthesizes rightMouseDown/rightMouseUp CGEvents.
Target process ID.
Preferred element handle.
X in window-local screenshot pixels. Must be provided together with
y.Y in window-local screenshot pixels.
scroll
Scroll the target PID. Two paths:
- Targeted wheel path — pass
element_token/element_indexor window-localx, y. Synthesizes a real mouse-wheel event hit-tested by the renderer. Required for nestedoverflow:autoregions in web views. - Keystroke path — pass only
pidanddirection. Drives the focused/page scroller via PageDown/PageUp or arrow keys.
"up", "down", "left", or "right".Wheel notches (targeted path) or keystroke repetitions (keystroke path). Range 1–50.
"line" or "page".Target process ID.
Preferred element handle for targeted wheel path.
Window-local screenshot X for targeted wheel path.
Window-local screenshot Y for targeted wheel path.
drag
Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels. Use for marquee selection, drag-and-drop, slider scrubbing, and panel repositioning.
Drag-start X in window-local screenshot pixels.
Drag-start Y in window-local screenshot pixels.
Drag-end X.
Drag-end Y.
Wall-clock duration of the drag path. Range 0–10 000.
Number of intermediate
mouseDragged events. Range 1–200.Modifier keys held across the entire gesture.
move_cursor
Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer.
X coordinate.
Y coordinate.
"window" for the agent overlay; "desktop" for the real OS pointer.zoom
Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixel coordinates, with 20% padding added on each side. Output is at most 500 px wide. After a zoom, pass from_zoom=true to click or type_text to auto-translate coordinates back to full-window space.
CGWindowID from
list_windows.Left edge of the region in screenshot pixels.
Top edge of the region in screenshot pixels.
Right edge of the region in screenshot pixels.
Bottom edge of the region in screenshot pixels.
Keyboard Tools
type_text
Insert text into the target PID via AXSetAttribute(kAXSelectedText). Works for standard Cocoa text fields and text views. For Chromium/Electron inputs, falls back to CGEvent character synthesis automatically.
For browser tab content (Chromium/WebKit), the reliable path is the
page
tool (drives the DOM via CDP) or pass x,y (no element_index) to
pixel-click the field, then type in one call.Text to insert at the target’s cursor.
Target process ID.
Preferred element handle for directing the write to a specific field.
Screenshot-pixel X of the field — pixel-clicks to focus, then types. Use for Chromium/Electron inputs.
Screenshot-pixel Y of the field.
Milliseconds between characters in the CGEvent fallback path. Range 0–200.
"background" or "foreground".press_key
Press and release a single key. Does not raise the window by default. Confirmed only when a bounded native AX value/selection read-back changes.
Key names: return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1–f12, plus any letter or digit.
Key name (e.g.
"return", "escape", "a").Modifier keys:
cmd, shift, option/alt, ctrl, fn.Target process ID.
"background" or "foreground". Use "foreground" with window_id when
the key must go to a Chromium content area.hotkey
Press a key combination, e.g. ["cmd", "c"] for Copy. Order: modifiers first, one non-modifier last.
Modifier(s) and one non-modifier key. Minimum 2 items (e.g.
["cmd", "c"]).Target process ID.
"background" (auth-message envelope, Chromium-safe) or "foreground"
(fronts briefly for NSMenu key-equivalents).set_value
Set a value on a UI element. For AXPopUpButton/select dropdowns, finds the matching child option and presses it without opening the native menu. For all other elements, writes AXValue directly (sliders, steppers, date pickers, native text fields).
New value. AX will coerce to the element’s native type.
Target process ID.
Preferred element handle.
Application Tools
launch_app
Launch a macOS app in the background — the target does not come to the foreground. Provide bundle_id (preferred) or name. Returns the launched app’s pid, bundle_id, name, and a windows array.
App bundle identifier, e.g.
com.apple.calculator. Preferred over name.App display name. Used only when
bundle_id is absent.When
true, forces a new app instance even if one is already running (passes
-n to open). Use for concurrent multi-agent work.File paths or URLs to open with the app (e.g. a folder path for Finder).
Opens a WebKit inspector server on this port. Use for Tauri/WebKit-based apps.
kill_app
Force-terminate a process by PID (kill -9 on macOS/Linux; taskkill /F on Windows). Use as escalation when the cooperative close path failed. Unsaved state is lost.
PID of the process to terminate.
bring_to_front
Persistently activate an app and leave it in the foreground. With window_id, success means the exact window was independently verified as focused and first in WindowServer layer-0 order. This does steal foreground.
Target process ID.
Target window ID for exact verification.
set_window_frame
Set a top-level window’s frame in the desktop-coordinate space and verify the resulting geometry through an independent readback.
Target process ID.
Target window ID.
New X origin.
New Y origin.
New width (≥ 1).
New height (≥ 1).
Browser Tools
Browser tools operate on exactly-bound Chromium/Electron CDP endpoints. Safari and Firefox lack typed mutation routes and fall back to native AX/PX paths.page (legacy compatibility)
page (legacy compatibility)
The
page tool provides legacy browser compatibility across Chrome, Brave, Edge, Safari (via AppleScript on macOS), Electron, and WKWebView/Tauri fallbacks.Supported actions:| Action | Description |
|---|---|
execute_javascript | Run JS and return the result. |
get_text | Extract visible text from the page. |
query_dom | Find elements matching a CSS selector. |
click_element | Click a CSS-selected element and animate the agent cursor to its center. |
insert_text | Insert text at the focused DOM element via CDP Input.insertText. |
type_keystrokes | Type text via per-character keystroke events into the focused element. |
enable_javascript_apple_events | macOS only — patch browser Preferences to allow JS from Apple Events. |
get_browser_state
get_browser_state
Read-only browser inspection. Mode 1 (bind): pass
pid + window_id to classify the browser, correlate it to a CDP target, and mint session-scoped target_id and tab_id handles. Mode 2 (snapshot): pass target_id + tab_id to read DOM state.PID of the browser process (bind mode).
Native window ID owned by
pid (bind mode).Opaque target handle from bind mode.
Opaque tab handle from bind mode.
Versioned snapshot contract.
dom_refs_v1 is the default.Read-only semantic match over role, accessible name, and visible text.
Capture the exact tab viewport as PNG through CDP.
Stable caller session ID.
browser_prepare
browser_prepare
Explicitly prepare an owned DevTools endpoint for a browser PID. Existing endpoints are detected without side effects. Existing-profile attachment follows the runtime’s immutable permission mode.
Browser process ID to prepare.
Allow a separate driver-owned isolated Chromium process to be launched.
Stable caller session ID.
browser_navigate, browser_click, browser_type, browser_dialog, browser_pointer, browser_set_input_files, browser_download
browser_navigate, browser_click, browser_type, browser_dialog, browser_pointer, browser_set_input_files, browser_download
Clipboard Tools
clipboard_read
List available system clipboard types and optionally return plain text. Clipboard content is never retained in telemetry.
Return plain-text clipboard content in addition to the available types.
clipboard_write
Replace the system clipboard with exactly one value: plain text, an image from an absolute local path, or a file URL.
Plain text to place on the clipboard.
Absolute path to a local image.
Absolute path to a local file to place as a file URL.
Recording Tools
start_recording
Start trajectory recording. Every subsequent action tool call writes a turn folder under output_dir containing before.png, after.png, action.json, before_state.json, after_state.json, and click.png.
Absolute or tilde-rooted directory for turn folders and (optionally) the
video file.
Capture the main display to
<output_dir>/recording.mp4. On macOS uses
native ScreenCaptureKit (requires macOS 15.0+). On Windows/Linux requires
ffmpeg on PATH.stop_recording
Stop trajectory recording and finalize any in-progress MP4. Calling stop on an already-stopped session is a no-op.
Arguments: none.
replay_trajectory
Replay a recorded trajectory by re-invoking every turn’s tool call in lexical order.
Element-indexed actions (
click with element_index) will fail during
replay because element indices are per-snapshot. Pixel clicks and all keyboard
tools replay cleanly.Trajectory directory previously written by
start_recording.Milliseconds to sleep between turns. Range 0–10 000.
Stop replay on the first tool-call error. Set
false to best-effort through
the full trajectory.Session and Configuration Tools
start_session
Declare a named, color-coded session identity for an agent run. Enables the agent cursor overlay and scopes capture policy, per-session config, and recording. Idempotent: re-calling with the same ID refreshes its idle TTL.
Stable session ID for this run (e.g.
"research-run-1")."auto" (starts window-only, can escalate), "window", or "desktop".
Immutable for the live session.end_session
End a declared session: removes its agent cursor, stops any recording it owns, and clears per-session config. Idempotent.
The session ID to end.
set_config
Update cua-driver configuration. Changes to max_image_dimension take effect immediately. experimental_pip keys take effect on the next daemon restart.
Max dimension for screenshot resizing.
0 means no limit.Enable the experimental picture-in-picture preview window. Applies on next restart.
set_agent_cursor_enabled
Show or hide the agent cursor overlay owned by a session.
Session ID whose cursor to show or hide.
true to show the cursor; false to hide it.set_agent_cursor_motion
Configure movement physics and visibility timing for a session cursor.
Session ID whose motion settings to update.
Duration of cursor glide animation in milliseconds.
Spring constant for cursor movement physics.
Arc flow parameter for curved path animation.
Arc size parameter for curved path animation.
Turn radius for curved path animation.
Cubic bezier start handle.
Cubic bezier end handle.
Milliseconds of inactivity before the cursor hides itself.
Milliseconds to keep the cursor visible after a click.
set_agent_cursor_theme
Select an already-installed cursor theme for a session.
Session ID whose cursor theme to change.
Installed cursor theme ID (e.g.
cua.default).Animation preference:
"auto" (follow OS setting), "on" (still frames), or "off" (allow animation).escalate_session
Unlock the desktop phase of an auto capture-scope session after the window action ladder has been exhausted and verified. Escalation is permanent for that session and disables window-scoped tools. To recover window scope, call end_session then start_session with a new session ID.
Session ID to escalate.
Machine-readable reason for escalation (e.g.
"ax_tree_pixel_mismatch").Optional bounded diagnostic detail. Never use secrets or page content.
get_session_state
Read the live session’s capture policy and effective scope.
Session ID to inspect.
Maintenance Tools
check_permissions
Report TCC permission status for Accessibility and Screen Recording. By default also raises system permission dialogs for missing grants.
Raise the system permission prompts for missing grants. Only a trusted host
setup route may set
true.When prompting and Screen Recording is granted, also run the live
ScreenCaptureKit probe.
health_report
Single-call end-to-end driver diagnostics. Returns a stable JSON structure with schema_version, platform, driver_version, overall (ok/degraded/failed), and a checks array.
Only run these checks (canonical names). Wins over
skip.Skip these checks. Ignored when
include is set.check_for_update
Check whether a newer cua-driver release is available on GitHub. Returns current_version, latest_version, update_available, install one-liner, and release notes URL. Mirror of cua-driver check-update --json.
Arguments: none.
verify_state
Deterministically verify bounded predicates against one exact window. Returns satisfied, unsatisfied, or unknown per predicate.
Exact process whose window is observed.
Exact native window identifier.
One to eight predicates combined with logical AND.
Bounded wait. Zero performs one sample. Range 0–10 000.
invoke_menu
Resolve an exact application-menu path one live native level at a time and invoke its final item through accessibility APIs. Missing, ambiguous, disabled, or structurally mismatched segments fail closed; this tool never falls back to pixels.
Target process ID.
Target window ID.
Menu path segments from root to item, e.g.
["File", "Save As…"]. 1–16 items.Optional session ID.
install_ffmpeg
Install the ffmpeg binary used by start_recording’s video capture on Linux and Windows. (macOS records natively and does not need ffmpeg.) Called without confirm, only reports the exact install command for this platform’s package manager. Pass confirm: true to actually run it. No-op if ffmpeg is already on PATH.
Run the install command. Without it, only the planned command is reported.
Action Response Shape
All action tools (click, double_click, right_click, drag, scroll, type_text, press_key, hotkey, set_value) return these structured fields:
Delivery rung that ran:
"ax", "cgevent", "cgevent_fg", "key_events",
"key_events_fg", "pixel", "x11_atspi", "x11_pixel", "x11_pixel_fg",
or "msaa".Action confidence signal:
"confirmed", "unverifiable", or
"suspected_noop".AX read-back result.
true = driver confirmed the effect through AX; false
= ran but unconfirmed; absent = not applicable for this tool.Present only when the driver recommends climbing the delivery ladder.
Shape:
{ recommended: "px" | "foreground" | "page", reason: string }.