SWE-bench is the most credible measure of whether an AI agent can actually fix real software problems. It takes genuine GitHub issues and their tests, then asks a system to produce a patch that passes. It is hard on purpose.

What the numbers mean

SWE-bench tasks come from real repositories (django, sympy, scikit-learn, etc.). Scoring uses the project’s own tests, so a “pass” means the fix actually works and doesn’t break existing behaviour.

SettingWhat is measuredReality
SWE-bench (full)Resolve rate on ~2,294 issuesEven top systems solve a minority
SWE-bench VerifiedCurated subset, human-validatedProgress is real but incomplete
SWE-bench LiteSmaller, cheaper subsetGood for iteration, easy to over-read

Two traps: high scores on curated subsets do not transfer directly to arbitrary repos, and a benchmark issue is cleaner than a real bug report, which is often under-specified.

Practical implications

Benchmark pass vs production reliability SWE-bench task clean & testable Real bug report under-specified Production change needs review Gap widens as context gets messier keep humans in the loop

VERDICT

Treat coding agents as capable assistants, not autonomous maintainers. Benchmark success rates understate the difficulty of real, underspecified work.

Sources