LLM Council is a local web application that replaces single-model queries with a structured deliberation process. Submit a question, and the council—GPT, Gemini, Claude, Grok, or any OpenRouter-supported model—answers independently, ranks each other’s work under anonymous labels, and lets a designated Chairman synthesize the collective wisdom into a single final response.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/karpathy/llm-council/llms.txt
Use this file to discover all available pages before exploring further.
Quickstart
Install dependencies, set your OpenRouter API key, and have the app running in under five minutes.
Configuration
Customize council members, the Chairman model, and storage settings in a single config file.
How It Works
Understand the 3-stage deliberation pipeline—from parallel queries to peer rankings to final synthesis.
API Reference
Explore the FastAPI REST endpoints that power the backend, including SSE streaming.
The 3-Stage Process
Stage 1 — First Opinions
Your query is sent to every council model in parallel via OpenRouter. Each model answers independently, and all responses are displayed side-by-side in a tab view so you can inspect them.
Stage 2 — Anonymous Peer Review
Each council model receives all Stage 1 responses with model identities stripped (Response A, Response B, …). It evaluates every response and provides a
FINAL RANKING: in a structured format. The app computes aggregate rank scores across all evaluators.Why LLM Council?
Reduce single-model bias
No single model’s blind spots dominate. Peer review surfaces weaknesses before the final answer is written.
Full transparency
Every raw response and every raw ranking evaluation is inspectable. Nothing is hidden behind an abstraction.
OpenRouter-powered
Access GPT, Gemini, Claude, Grok, and hundreds of other models through a single API key—no juggling multiple provider accounts.
Local & private
The app runs entirely on your machine. Conversations are stored as JSON files locally—no cloud sync, no telemetry.
LLM Council was built as a Saturday hack by Andrej Karpathy to explore multi-model evaluation. It is provided as-is for inspiration. See the README for context.