6 articles.
Benchmarks like HumanEval measure small, self-contained problems. Real software tasks are harder, and coding agents score far lower on benchmarks built from real GitHub issues.
USENIX Security 2025's Distinguished Paper ran 16 models across 576,000 code samples and counted 2.23 million recommended packages, of which 440,445 were hallucinated and 205,474 were distinct names that exist nowhere in PyPI or npm. Closed models hallucinated at 5.2% against 21.7% for open models. When researchers repeated 500 prompts ten times each, 43% of the hallucinated names reappeared in every single run, which is what makes the attack possible.
SWE-bench evaluates whether models can resolve real GitHub issues with tests. Top systems solve a minority of tasks, and the jump from benchmark scores to dependable autonomous bug-fixing remains large.
Coding agents multiplied the code security teams have to review, and the vendors selling AI review tools all report spectacular accuracy. Here is what one vendor's own benchmark against frontier models shows, and why the agent attack surface is the part worth acting on.
A new benchmark ran 520 whole-repository migrations through 8 frontier models. Only 28 finished the job. Here's what the failures tell you about handing an agent your upgrade.
I compared Cursor, Claude Code, GitHub Copilot, and Devin Desktop on real multi-file work. Here's which one actually saves time without creating more review debt.