When choosing between an open model you can self-host and a proprietary API, the honest question is: how big is the gap, and on what tasks?
What the Chatbot Arena measures
The LMSYS Chatbot Arena collects human votes in blind pairwise comparisons: two anonymous models answer the same prompt, and the user picks the better response. Results are converted to Elo ratings.
This is a measure of human preference, not task accuracy. It rewards responses that are helpful, well-formatted, and engaging. It does not directly measure coding correctness, factual accuracy, or safety.
What the leaderboard shows (broadly)
- Proprietary models (GPT-4 class, Claude) have historically led the overall ranking.
- Open models (Llama, Mistral, Qwen families) have closed much of the gap and now compete in the top tier on many categories.
- The gap is larger on complex reasoning and coding tasks than on casual conversation.
- Category-specific leaderboards (coding, math, hard prompts) tell a more useful story than the overall score.
How to use this for a decision
| Your priority | What to check |
|---|---|
| Cost and data control | Open model on your own infra |
| Peak capability on hard tasks | Top proprietary model |
| Latency and throughput | Benchmarks on your hardware, not Arena |
| Privacy | Self-hosted open model |
Caveats
- Arena voters are not your users. Preference does not equal fitness for your task.
- Elo ratings shift as new models are added; a snapshot is not a permanent ranking.
- Always run a small eval on your own prompts before committing.
VERDICT
Open models are now competitive on many tasks. Choose on cost, control, and your own evals — not on a single leaderboard number.