SWE-bench is the most credible measure of whether an AI agent can actually fix real software problems. It takes genuine GitHub issues and their tests, then asks a system to produce a patch that passes. It is hard on purpose.
What the numbers mean
SWE-bench tasks come from real repositories (django, sympy, scikit-learn, etc.). Scoring uses the project’s own tests, so a “pass” means the fix actually works and doesn’t break existing behaviour.
| Setting | What is measured | Reality |
|---|---|---|
| SWE-bench (full) | Resolve rate on ~2,294 issues | Even top systems solve a minority |
| SWE-bench Verified | Curated subset, human-validated | Progress is real but incomplete |
| SWE-bench Lite | Smaller, cheaper subset | Good for iteration, easy to over-read |
Two traps: high scores on curated subsets do not transfer directly to arbitrary repos, and a benchmark issue is cleaner than a real bug report, which is often under-specified.
Practical implications
- Agents are useful as first-draft fixers and codebase navigators, not as autonomous maintainers.
- Keep a human in the loop for anything touching production, auth, or money.
- Cost and latency matter: multi-step agents can retry many times before one pass.
- Test coverage is the bottleneck: if your repo lacks tests, the agent has no signal.
VERDICT
Treat coding agents as capable assistants, not autonomous maintainers. Benchmark success rates understate the difficulty of real, underspecified work.