Coding assistants, code review tools, and data tools — tested against real codebases.
A concrete look at Anthropic's October 2026 'mods' addition: what it lets a team actually block, and why agent permissions matter more than raw model capability.
GitHub Copilot, Cursor, and Codeium compared for beginners — real benchmark numbers, pricing that actually changed in 2026, and which one to install first.
METR ran a randomised controlled trial where 16 experienced open-source developers worked on issues in repositories they knew, with AI tools freely available in half the tasks. Implementation took 19% longer with AI. The developers predicted a 24% speedup before starting and still estimated a 20% speedup afterwards. Economics experts had forecast 39% and ML experts 38%. No confidence interval was published for the headline number, and the study has since been superseded.
Studies across multiple languages and scenarios show GitHub Copilot can produce insecure code suggestions at nontrivial rates. The risk is highest when developers accept suggestions uncritically without security review.
Benchmarks like HumanEval measure small, self-contained problems. Real software tasks are harder, and coding agents score far lower on benchmarks built from real GitHub issues.
MMLU and similar multiple-choice benchmarks no longer discriminate between strong models. Task-based, real-world evaluations like SWE-bench give a more honest picture of capability.
USENIX Security 2025's Distinguished Paper ran 16 models across 576,000 code samples and counted 2.23 million recommended packages, of which 440,445 were hallucinated and 205,474 were distinct names that exist nowhere in PyPI or npm. Closed models hallucinated at 5.2% against 21.7% for open models. When researchers repeated 500 prompts ten times each, 43% of the hallucinated names reappeared in every single run, which is what makes the attack possible.
SWE-bench evaluates whether models can resolve real GitHub issues with tests. Top systems solve a minority of tasks, and the jump from benchmark scores to dependable autonomous bug-fixing remains large.
AGENTS.md won, and it won by committee: it is stewarded under the Linux Foundation, used by over 60,000 projects, and read by Cursor, Copilot, Codex, Gemini CLI, Aider, and opencode. But the three tools that host it publish size limits that differ by an order of magnitude, resolve conflicting instructions in four different ways, and one of them will silently stop reading your file. Here is the actual state of each.
Two 2026 results about AI-written tests look contradictory and are not. Given a test prefix and the methods called, models reach 43% mutation score against 45% for human-written oracles on a contamination-free dataset. Asked to write the whole test for a real project, the same models cover about a third of the lines. Here is where the pipeline breaks, and why coverage is the wrong metric to check.
Coding agents multiplied the code security teams have to review, and the vendors selling AI review tools all report spectacular accuracy. Here is what one vendor's own benchmark against frontier models shows, and why the agent attack surface is the part worth acting on.
A new benchmark ran 520 whole-repository migrations through 8 frontier models. Only 28 finished the job. Here's what the failures tell you about handing an agent your upgrade.
CodeRabbit, Greptile, and GitHub Copilot Code Review compared on price, depth, and the kind of cross-file bug a diff-only reviewer walks right past.
I compared Cursor, Claude Code, GitHub Copilot, and Devin Desktop on real multi-file work. Here's which one actually saves time without creating more review debt.
I ran the same messy CSV through the tools that claim you can skip Python. Here's what actually returned usable charts and numbers in 2026.