Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/simonw/LLM/llms.txt

Use this file to discover all available pages before exploring further.

Beyond the built-in OpenAI support, LLM can use virtually any model through its plugin system — including models that run entirely on your own hardware. Plugins add new model IDs to llm models and work seamlessly with all of LLM’s features: logging, templates, fragments, and conversations.

​
Installing Local Model Plugins

LLM plugins can provide local models that run on your machine without sending data to any external service.

llm-ollama

Run any model available in Ollama, including Llama, Mistral, Phi, and Gemma families.

llm-mlx

Run Apple MLX models directly on Apple Silicon.

llm-llamafile

Run self-contained llamafile executables.

llm-gpt4all

17+ models from the GPT4All project.
Install any plugin with llm install: 
llm install llm-ollama
After installation, llm models will show the newly available models. Run prompts against them the same way as any other model: 
llm -m orca-mini-3b-gguf2-q4_0 'What is the capital of France?'
The model is downloaded and cached the first time you use it. Subsequent runs are faster.
Check the plugin directory for the full list of available model plugins.

​
OpenAI-Compatible Models

Many services and local inference servers expose an API that mirrors OpenAI’s interface. LLM can connect to any of them by adding entries to extra-openai-models.yaml.

​
Finding the Config Directory

dirname "$(llm logs path)"
On macOS this is typically ~/Library/Application Support/io.datasette.llm. Create a file called extra-openai-models.yaml there.

​
Basic Configuration

- model_id: orca-openai-compat
  model_name: orca-mini-3b.ggmlv3
  api_base: "http://localhost:8080"
  • model_id — the name LLM uses internally and on the command line
  • model_name — the exact model name sent to the API (may differ to avoid clashes)
  • api_base — the root URL of the OpenAI-compatible server
When api_base is set, LLM does not send the configured openai API key by default.

​
Using a Stored API Key

Set api_key_name to the name of a key you have stored with llm keys set: 
llm keys set openrouter

- model_id: claude-via-openrouter
  model_name: anthropic/claude-2
  api_base: "https://openrouter.ai/api/v1"
  api_key_name: openrouter

​
Capability Flags

KeyDescription
completion: trueUse /completion endpoint instead of /chat/completion
supports_tools: trueModel supports function / tool calling
can_stream: falseDisable streaming for models that don’t support it
supports_schema: trueModel supports JSON structured schema output
vision: trueModel can accept images as attachments
audio: trueModel can accept audio attachments

​
Verifying the Setup

llm models --options -m orca-openai-compat
llm -m orca-openai-compat 'What is the capital of France?'
llm logs -n 1

​
Extra HTTP Headers

Some providers (such as openrouter.ai) require custom HTTP headers on every request. Add them with the headers: key: 
- model_id: claude
  model_name: anthropic/claude-2
  api_base: "https://openrouter.ai/api/v1"
  api_key_name: openrouter
  headers:
    HTTP-Referer: "https://llm.datasette.io/"
    X-Title: LLM
Any keys under headers: are passed verbatim on every API request to that model.

Build docs developers (and LLMs) love

Get started for freeTalk to us