When choosing between an open model you can self-host and a proprietary API, the honest question is: how big is the gap, and on what tasks?

What the Chatbot Arena measures

The LMSYS Chatbot Arena collects human votes in blind pairwise comparisons: two anonymous models answer the same prompt, and the user picks the better response. Results are converted to Elo ratings.

This is a measure of human preference, not task accuracy. It rewards responses that are helpful, well-formatted, and engaging. It does not directly measure coding correctness, factual accuracy, or safety.

What the leaderboard shows (broadly)

How to use this for a decision

Your priorityWhat to check
Cost and data controlOpen model on your own infra
Peak capability on hard tasksTop proprietary model
Latency and throughputBenchmarks on your hardware, not Arena
PrivacySelf-hosted open model

Caveats

Open vs. proprietary: decision factors Open models control + cost, narrower gap Proprietary peak capability, managed Arena = preference, not accuracy always eval on your own tasks

VERDICT

Open models are now competitive on many tasks. Choose on cost, control, and your own evals — not on a single leaderboard number.

Sources