Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/trycua/cua/llms.txt

Use this file to discover all available pages before exploring further.

Cua Driver supports local model inference through any agent harness that speaks Anthropic’s API format. The tested architecture uses Ollama or llama.cpp as the inference server, Claude Code as the agent harness, and a small MCP tool filter that reduces context cost before the model sees the tool catalog. This page covers both serving paths, context budget considerations, and notes on Apple Silicon and MLX models.

​
Architecture overview

Local computer use involves three components: 
Local model (Ollama or llama.cpp)
        ↓  Anthropic-compatible API
Claude Code (agent harness)
        ↓  filtered stdio MCP
Cua Driver (computer-use tools)
        ↓
Desktop (macOS, Linux, or Windows)
The model never connects to Cua Driver directly. Claude Code acts as the agent harness: it receives tool call results from Cua Driver and sends the model’s next action back as another tool call. The filter sits between Claude Code and Cua Driver, trimming the tools/list response so the model sees only the tools your task needs.
This is a tested configuration for Meta’s Muse Glimmer 30B on macOS with Apple Silicon. It is not a compatibility claim for every local model or inference server.

​
Context budget considerations

Local models pay a context cost for every exposed tool schema, every screenshot, and every accessibility tree read. The 54 tools in Cua Driver’s full catalog add significant token overhead. Key rules for keeping context under control:
  • Expose only the tools the task needs. A Calculator task needs launch_app, get_window_state, click, type_text, and press_key — not the full catalog.
  • Bound accessibility reads. Use max_elements=25 and max_depth=3 for routine window-state reads. Increase them only when a specific task requires deeper inspection.
  • Request window state instead of desktop state when the target window is already known.
  • Batch deterministic text entry. One type_text action can replace several inference turns when the input is known in advance.
  • Use a context window of at least 64K. Computer-use tasks with screenshots and accessibility trees consume context quickly. The tested configuration uses 128K.
In verified Calculator runs, filtering tools, bounding state reads, and batching input reduced uncached input tokens from 71,088 to 12,251 and elapsed time from 660 seconds to 224 seconds.

​
Install the MCP schema filter

The filter intercepts tools/list responses and returns only the tools you name. Install it: 
mkdir -p "$HOME/.local/bin"
curl -fsSL \
  https://cua.ai/docs/examples/local-models/cua-mcp-filter.py \
  -o "$HOME/.local/bin/cua-mcp-filter"
chmod +x "$HOME/.local/bin/cua-mcp-filter"
Create muse-cua-mcp.json in your working directory: 
{
  "mcpServers": {
    "cua-computer-use": {
      "command": "cua-mcp-filter",
      "args": [
        "--allow",
        "start_session,end_session,launch_app,list_apps,list_windows,get_window_state,move_cursor,click,type_text,press_key,hotkey,invoke_menu"
      ]
    }
  }
}
Adjust the --allow list to the minimum set of tools your task requires.
The filter changes the tools/list response but does not enforce an authorization boundary. It does not block a caller that already knows another tool name. Use permission policies when you need enforcement.

​
Path 1: Ollama on Apple Silicon

Ollama serves local models through an Anthropic-compatible API. Pull the official Muse Glimmer MLX model: 
ollama pull muse-glimmer:30b-mlx
From the directory containing muse-cua-mcp.json, launch Claude Code: 
CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \
ollama launch claude \
  --model muse-glimmer:30b-mlx \
  --yes \
  -- \
  --bare \
  --strict-mcp-config \
  --mcp-config ./muse-cua-mcp.json \
  --tools ""
The muse-glimmer:30b-mlx model has a 128K context window, supports images, and supports tool calling. --bare avoids loading unrelated project instructions. --tools "" removes Claude Code’s built-in tools, keeping only the filtered Cua Driver tools.
Ollama’s Claude Code integration page has up-to-date information on supported models and context flags: docs.ollama.com/integrations/claude-code.

​
Path 2: llama.cpp with an Unsloth GGUF

llama.cpp gives you direct control over inference settings. This path uses Unsloth’s quantized GGUF variant of Muse Glimmer 30B and matches the configuration used for the recorded macOS runs. Start the llama.cpp server in a separate terminal: 
llama-server \
  --hf-repo "unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL" \
  --alias "muse-glimmer-local" \
  --host 127.0.0.1 \
  --port 8001 \
  --ctx-size 131072 \
  --parallel 1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --jinja \
  --mmproj-auto \
  --fit on \
  --no-webui
The first start downloads the GGUF and vision projector. Keep the endpoint bound to 127.0.0.1. Wait for the server to finish loading, then verify it is healthy: 
curl -fsS http://127.0.0.1:8001/health
From the directory containing muse-cua-mcp.json, start Claude Code pointed at the local server: 
ANTHROPIC_BASE_URL=http://127.0.0.1:8001 \
ANTHROPIC_API_KEY=local-no-key-required \
CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \
claude \
  --bare \
  --strict-mcp-config \
  --mcp-config ./muse-cua-mcp.json \
  --tools "" \
  --model muse-glimmer-local

​
MLX models on Apple Silicon

Ollama’s muse-glimmer:30b-mlx variant uses Apple’s MLX framework for native Silicon acceleration. If you want to serve a custom MLX model through a different harness, any server that exposes an Anthropic-compatible /v1/messages endpoint and supports multimodal inputs and tool calls will work with the same filter and Claude Code configuration.

​
Run a bounded smoke task

Start with a short task that has an observable outcome: 
Use only the cua-computer-use MCP tools. Launch Calculator, calculate 2 + 3,
and verify from fresh state that the display shows 5. Take a fresh window state
before every action and verify the result after every action. Keep
get_window_state calls to max_elements=25 and max_depth=3 unless a deeper
accessibility tree is required.
The model should launch Calculator, act through Cua Driver, read fresh final state, and report the observed value. If it reports success without the final observation, ask it to verify again.

​
Troubleshooting

Confirm the session uses both --strict-mcp-config and the filtered muse-cua-mcp.json. A global Cua Driver registration can load alongside the filtered server if --strict-mcp-config is missing.
Reduce max_elements and max_depth, remove unused tools, and shorten the task. With Ollama, confirm the selected model has at least a 64K context window. If llama.cpp reports a smaller usable context than expected, restart with a --ctx-size value the machine can hold reliably.
Use a real MCP connection. Shell wrappers that flatten MCP image blocks into text remove the visual input that a multimodal model needs for pixel grounding.
Run cua-driver permissions status on the machine being operated. Grant Accessibility and Screen Recording to the Cua Driver identity using cua-driver permissions grant.

Build docs developers (and LLMs) love

Get started for freeTalk to us