3 articles.
Benchmarks like HumanEval measure small, self-contained problems. Real software tasks are harder, and coding agents score far lower on benchmarks built from real GitHub issues.
MMLU and similar multiple-choice benchmarks no longer discriminate between strong models. Task-based, real-world evaluations like SWE-bench give a more honest picture of capability.
SWE-bench evaluates whether models can resolve real GitHub issues with tests. Top systems solve a minority of tasks, and the jump from benchmark scores to dependable autonomous bug-fixing remains large.