4 articles.
Apple researchers rebuilt 100 grade-school math problems into templates and generated 5,000 variants, then ran 25 state-of-the-art models against them. The problems needed identical reasoning, yet accuracy swung by more than 12 points between easy and hard versions of the same question, and adding one sentence that changes nothing cost some models over 65% of their accuracy. Few-shot examples did not fix it.
Benchmarks like HumanEval measure small, self-contained problems. Real software tasks are harder, and coding agents score far lower on benchmarks built from real GitHub issues.
MMLU and similar multiple-choice benchmarks no longer discriminate between strong models. Task-based, real-world evaluations like SWE-bench give a more honest picture of capability.
The LMSYS Chatbot Arena ranks models by human preference in blind pairwise comparisons. Open models have closed much of the gap, but proprietary models still lead on complex tasks. The metric measures preference, not accuracy.