MMLU was a strong benchmark in 2020. In 2026, most frontier models score in the high 80s or 90s, and the spread between them is small. The benchmark has saturated.
What MMLU measures
MMLU (Hendrycks et al., 2021) is a multiple-choice benchmark across 57 subjects, from elementary maths to professional law. It tests factual recall and reasoning on short, self-contained questions.
It was valuable because it covered breadth. It is now limited because the top models are clustered near the ceiling, and small score differences are not meaningful.
Why saturation is a problem
- A benchmark that cannot separate models stops being useful for selection.
- Optimising for MMLU can distort training toward trivia rather than capability.
- High MMLU scores are routinely cited as proof of “intelligence” that does not transfer to real tasks.
What to measure instead
| Goal | Better evaluation |
|---|---|
| Code capability | SWE-bench (real GitHub issues, hidden tests) |
| Reasoning | Task-based evals with verifiable answers |
| Tool use | Agent benchmarks with real environments |
| Safety | Red-team suites and refusal evaluations |
The pattern: prefer benchmarks where the model must produce an artifact (code, a file, a decision) that can be checked, over benchmarks where it picks a letter.
Practical advice
- When comparing models for your use case, build a small eval from your own tasks.
- Treat published benchmark scores as a filter, not a ranking.
- Re-run evals when you change prompts, tools, or retrieval.
VERDICT
MMLU is a filter, not a ranking. For real capability, use task-based benchmarks and your own evals.