LLM Council: Run Your Own Multi-Model Debate Panel with Karpathy’s 400-Line App

In a world where every AI startup asks the same question — which model should I build on? — Andrej Karpathy built a different answer: why not all of them?

LLM Council is a 23.8k-star local web app that looks like ChatGPT but works like a boardroom. Your query goes to four frontier models at once. They answer independently, then anonymously review and rank each other’s work, and finally a designated Chairman model compiles a single synthesized verdict.

It’s a 400-line “fun Saturday hack” that Karpathy calls 99% vibe coded — and it’s become one of the most discussed projects in AI, because it sits at the intersection of two seismic shifts: ensemble architectures and vibe coding.

The problem: the yes-man fallacy

Stanford research showed that models like Claude agree with the user’s implicit opinion about 49% more than a human would — and hallucinate anywhere from 3-48% of the time. Ask a single AI for advice on a strategic decision and you get your own opinion back “in a smarter font.”

Relying on one model creates what analysts call Model Monoculture Syndrome: you treat a probabilistic engine’s output as gospel, blind to its training gaps, biases, and failure modes. Gemini is concise but risk-averse. Claude is literary but overly cautious. GPT is confident but wrong on niche topics.

A board of directors with one member isn’t a board — it’s a mirror. The council replaces the mirror with a boardroom.

The 3-stage pipeline

graph LR
    A[Your Query] --> B[Stage 1: Fan-Out]
    B --> C1[GPT-5.1]
    B --> C2[Gemini 3 Pro]
    B --> C3[Claude Sonnet 4.5]
    B --> C4[Grok 4]
    C1 --> D[Anonymized Responses]
    C2 --> D
    C3 --> D
    C4 --> D
    D --> E[Stage 2: Peer Review]
    E --> F[Ranking Matrix]
    F --> G[Stage 3: Chairman Synthesis]
    G --> H[Final Answer]

Stage 1: First opinions (divergence)

The query is fanned out in parallel to every council member via asyncio.gather(). Each model answers independently — no groupthink, no seed bias. The UI shows all responses in a tab view so you can compare perspectives side by side.

Latency here is bounded by the slowest model, not the sum — a critical UX decision.

Stage 2: Review (convergence)

This is the architectural genius. All responses are anonymized — model identities stripped, labeled simply “Response A,” “Response B,” etc. — and sent back to every model, which ranks them on accuracy and insight.

Why anonymize? LLMs exhibit documented self-preference bias (favoring their own family’s output) and length bias (favoring verbose answers). Forcing blind evaluation makes the judges assess semantic quality only. As the theory goes, discriminators are easier than generators: it’s easier to recognize a good image than to paint one.

This phase costs O(N²) in context — every model evaluates every response — which is where the token budget goes.

Stage 3: Final response (synthesis)

The designated Chairman model (default: Gemini 3 Pro, chosen for its long context window) receives:

• The original question
• All candidate responses from Stage 1
• The full ranking matrix from Stage 2

It resolves contradictions — spotting that “Model A made a calculation error, which Model C noticed” — and synthesizes one definitive answer. A Map-Reduce for intelligence: map across models, reduce to a verdict.

Setup: from zero to council in 5 minutes

# 1. Clone
git clone https://github.com/karpathy/llm-council.git && cd llm-council

# 2. Dependencies
uv sync
cd frontend && npm install && cd ..

# 3. API key
echo "OPENROUTER_API_KEY=sk-or-..." > .env

# 4. Optional: customize the council (backend/config.py)
# COUNCIL_MODELS = ["openai/gpt-5.1", "google/gemini-3-pro-preview", "anthropic/claude-sonnet-4.5", "x-ai/grok-4"]
# CHAIRMAN_MODEL = "google/gemini-3-pro-preview"

# 5. Run
./start.sh   # or: uv run python -m backend.main + cd frontend && npm run dev

Open http://localhost:5173 — a ChatGPT-like interface where your question goes to the whole council.

Tech stack: FastAPI (Python 3.10+, async httpx) · React + Vite + react-markdown · JSON storage in data/conversations/ · uv + npm. OpenRouter acts as a unified facade — swap any model by editing config, no code changes.

The honest tradeoffs

The decision framework: if the cost of being wrong exceeds the cost of querying multiple models, use a council. Most teams find 5-10% of queries are council-worthy — the high-value questions where the best possible answer justifies the expense.

Skip it for: real-time chat, simple lookups, code generation (inconsistent styles), creative writing (diluted voice), fine-tuned specialists, tight budgets.

The ecosystem: 2026’s multi-model wave

LLM Council wasn’t a one-off — it landed at the start of a genuine industry shift toward model orchestration:

• OpenRouter Fusion (March 2026): multi-model response synthesis at the gateway level — Deep Research agents preferred fused output to their own
• Maestro / RouteLLM: cheap-first routing that escalates only when needed
• LiteLLM / Portkey: the unified plumbing underneath
• Community forks: llm-council-plus (modern UI, multi-provider, Ollama support, up to 8 models), n8n rebuilds wired into Claude via MCP, Dockerized deployments

Karpathy’s 400 lines anticipated the direction: the “model” you interact with is becoming a dynamic ensemble, not a single endpoint.

The philosophy: why ’libraries are over'

The README’s Vibe Code Alert is the real artifact:

“Code is ephemeral now and libraries are over. Ask your LLM to change it in whatever way you like.”

Karpathy treats main.py not as sacred text but as temporary scaffolding. The assets are the system prompts and the interaction protocol — the Stage 1→2→3 pipeline design. The Python glue is disposable, rewritable by an LLM on demand.

This is the shift from the Clean Code paradigm to vibe coding: work at the conceptual level, let the agent handle the implementation, iterate freely. It sends shivers down QA engineers’ spines — and it’s where the industry is heading.

Conclusion

LLM Council is deceptively simple: 400 lines, one weekend, zero maintenance intent — yet it demonstrates a pattern that’s reshaping AI product design. When you need an answer where being wrong is expensive, one model is a mirror; a council is a boardroom.

Run it yourself, watch four frontier models debate your hardest question in anonymized peer review, and see what a Chairman synthesis produces. The code is ephemeral — the pattern isn’t.