LLM Providers: Bring Your Own AI for Ghostly

Ghostly is a bring-your-own-LLM system. It doesn’t hardwire a specific AI vendor — instead, it exposes a clean LlmProvider interface and lets you plug in any model you have access to, from OpenAI’s hosted API to a locally running Ollama instance to the Cursor Agent already installed on your developer machine. The LLM is used in three places inside every assisted run: the Strategist (planning step horizons), the Healer (proposing selector replacements), and the Observer (interpreting the page accessibility tree). Choosing the right provider for your environment directly affects run speed, cost, and reliability.

Two Provider Kinds

HTTP (OpenAI-Compatible)

Any endpoint that accepts chat completions in the OpenAI request format (POST /v1/chat/completions with messages, model, and optionally response_format: { type: "json_object" }). This covers:

OpenAI — gpt-4o, gpt-4o-mini, gpt-4.1-mini
Anthropic — via OpenRouter or LiteLLM proxy
Mistral — mistral-small-latest, mistral-large-latest, codestral-latest
OpenRouter — multi-model aggregator supporting dozens of providers
Ollama — local inference at http://127.0.0.1:11434/v1/chat/completions

Configuration requires: providerId, model, apiKey (if the endpoint demands one), and baseUrl.

CLI (Cursor Agent)

Uses the local agent binary installed with Cursor — no API key required. Ghostly invokes it as a subprocess in headless mode:

agent -p --output-format json --trust --mode ask --model composer-2.5

The prompt is passed via stdin to avoid shell injection and Windows argv length limits. The response is a JSON envelope from which Ghostly extracts the result field.

Flag	Purpose
`-p` / `--print`	Headless script mode, no interactive TUI
`--output-format json`	Machine-parseable response envelope
`--trust`	Trust the workspace without a confirmation prompt
`--mode ask`	Read-only — the agent never edits files or runs shell commands
`--model <id>`	Explicit model selection for deterministic behavior

Each Cursor CLI call spawns a new subprocess. Expect 7–35 seconds of latency per LLM call. A run with 5 horizons and 3 steps each could involve 15+ LLM calls — that’s potentially 3–8 minutes of AI time alone. Use an HTTP provider for CI environments where turnaround time matters.

Provider Catalog

The full provider catalog is defined in apps/api/src/llm/catalog.ts:

`providerId`	Kind	Default model	API key required
`cursor-cli`	CLI	`auto`	No — uses `agent login` session
`openai`	HTTP	`gpt-4o-mini`	Yes
`mistral`	HTTP	`mistral-small-latest`	Yes
`anthropic`	HTTP	`anthropic/claude-sonnet-4` (via OpenRouter)	Yes
`ollama`	HTTP	`llama3`	No
`openrouter`	HTTP	`openai/gpt-4o-mini`	Yes

Configuration Methods

1. Interactive Wizard

Run the guided setup wizard from your terminal. It prompts for provider, model, API key, and base URL, then saves everything to ~/.ghostly/auth.json:

ghostly config

For non-interactive CLI flags:

ghostly config \
  --llm-provider http \
  --llm-model gpt-4o \
  --llm-api-key sk-... \
  --llm-base-url https://api.openai.com/v1/chat/completions

2. Dashboard Settings Panel

Navigate to Settings → LLM Settings in the Ghostly web dashboard. Settings are stored per-user in the user_llm_settings table and take precedence over environment variables.

GET  /v1/settings/llm   → returns current LLM settings for the authenticated user
PUT  /v1/settings/llm   → saves providerId, model, apiKey, baseUrl

3. Environment Variables

Environment variables are the fallback when no user settings are configured in the database:

Variable	Description	Default
`ASSIST_LLM_PROVIDER`	`http` or `cursor-cli`	`http`
`ASSIST_LLM_API_URL`	Full endpoint URL	—
`ASSIST_LLM_API_KEY`	Bearer token / API key	—
`ASSIST_LLM_MODEL`	Model ID	`assist-fallback-v1`
`ASSIST_LLM_TIMEOUT_MS`	Per-call timeout in ms	`45000`

Provider Configuration Examples

OpenAI (HTTP)
Cursor CLI
Ollama (Local)
OpenRouter

Direct OpenAI API access using gpt-4o. Best balance of speed and capability for most E2E flows.

# Via CLI wizard
ghostly config \
  --llm-provider http \
  --llm-model gpt-4o \
  --llm-api-key sk-proj-... \
  --llm-base-url https://api.openai.com/v1/chat/completions

# Via environment variables
export ASSIST_LLM_PROVIDER=http
export ASSIST_LLM_MODEL=gpt-4o
export ASSIST_LLM_API_KEY=sk-proj-...
export ASSIST_LLM_API_URL=https://api.openai.com/v1/chat/completions

// Via PUT /v1/settings/llm
{
  "providerId": "openai",
  "model": "gpt-4o",
  "apiKey": "sk-proj-...",
  "baseUrl": "https://api.openai.com/v1/chat/completions"
}

Uses your local Cursor Agent session. No API key needed — authenticate once with agent login.

# Authenticate your Cursor session (one-time)
agent login

# Configure Ghostly to use the CLI provider
ghostly config \
  --llm-provider cursor-cli \
  --llm-model composer-2.5

# Via environment variables
export ASSIST_LLM_PROVIDER=cursor-cli
export ASSIST_LLM_MODEL=composer-2.5
# Optional: bypass agent login on machines where it doesn't persist
export CURSOR_API_KEY=your-cursor-api-key

Available models for Cursor CLI (from the catalog fallback list):

Model ID	Label
`auto`	Auto (Cursor selects)
`composer-2.5`	Composer 2.5
`claude-sonnet-5-thinking-high`	Sonnet 5 Thinking
`gpt-5.3-codex`	Codex 5.3

Fully offline inference using a locally running Ollama server. No API key or internet connection required.

# Start Ollama (separate terminal)
ollama run llama3

# Configure Ghostly
ghostly config \
  --llm-provider http \
  --llm-model llama3 \
  --llm-base-url http://127.0.0.1:11434/v1/chat/completions

# Via environment variables
export ASSIST_LLM_PROVIDER=http
export ASSIST_LLM_MODEL=llama3
export ASSIST_LLM_API_URL=http://127.0.0.1:11434/v1/chat/completions
# No API key needed for Ollama

Access hundreds of models through a single API key. Useful for switching between Claude, GPT, and Mistral without managing multiple keys.

ghostly config \
  --llm-provider http \
  --llm-model anthropic/claude-sonnet-4 \
  --llm-api-key sk-or-... \
  --llm-base-url https://openrouter.ai/api/v1/chat/completions

// Via PUT /v1/settings/llm
{
  "providerId": "openrouter",
  "model": "anthropic/claude-sonnet-4",
  "apiKey": "sk-or-...",
  "baseUrl": "https://openrouter.ai/api/v1/chat/completions"
}

Per-User Settings in the Database

User LLM settings are stored in the user_llm_settings table and scoped to the authenticated user. They override environment variables for that user’s runs:

-- From apps/api/prisma/schema.prisma

model UserLlmSettings {
  userId     String   @id
  providerId String           -- e.g. "openai", "cursor-cli"
  model      String           -- e.g. "gpt-4o", "composer-2.5"
  apiKey     String?          -- API key (store securely in production)
  baseUrl    String?          -- Custom endpoint URL
  updatedAt  DateTime @updatedAt
}

The settings-to-config resolution order is:

User database settings — highest priority; overrides everything
Environment variables — fallback when no user settings exist
Catalog defaults — model and endpoint defaults per provider

For CI environments, use an HTTP provider (OpenAI, Mistral, or OpenRouter) with a direct API key set via environment variable. This avoids the subprocess startup overhead of Cursor CLI and gives you predictable, low-latency LLM calls that fit comfortably within typical CI job timeouts.

Get Started

CLI Reference

Core Concepts

Integrations

Configuration

LLM Providers: Bring Your Own AI for Ghostly

Two Provider Kinds

HTTP (OpenAI-Compatible)

CLI (Cursor Agent)

Provider Catalog

Configuration Methods

1. Interactive Wizard

2. Dashboard Settings Panel

3. Environment Variables

Provider Configuration Examples

Per-User Settings in the Database

Build docs developers (and LLMs) love

Get Started

CLI Reference

Core Concepts

Integrations

Configuration

Documentation Index

​Two Provider Kinds

​HTTP (OpenAI-Compatible)

​CLI (Cursor Agent)

​Provider Catalog

​Configuration Methods

​1. Interactive Wizard

​2. Dashboard Settings Panel

​3. Environment Variables

​Provider Configuration Examples

​Per-User Settings in the Database

Build docs developers (and LLMs) love

Two Provider Kinds

HTTP (OpenAI-Compatible)

CLI (Cursor Agent)

Provider Catalog

Configuration Methods

1. Interactive Wizard

2. Dashboard Settings Panel

3. Environment Variables

Provider Configuration Examples

Per-User Settings in the Database