A high score on a coding benchmark does not mean an AI agent can fix real bugs. The gap between benchmark problems and real repository work is large, and it shows up in the numbers.

What HumanEval measures

HumanEval (Chen et al., 2021) consists of 164 hand-written Python problems. Each problem is a short function with a docstring and unit tests. Models are scored by pass@k: the fraction of problems solved within k samples. Codex-12B reached 28.8% pass@1 on this set.

That is a useful signal for code generation on small, self-contained tasks. It is not a signal for editing a real codebase, running a test suite, or navigating a repository.

What real-world tasks look like

SWE-bench (Jimenez et al., 2023) was built from real GitHub issues and their gold patches across popular Python repositories. An agent must reproduce the issue, locate the relevant files, edit the code, and make the hidden tests pass. On the full SWE-bench set, a strong baseline (Claude 2) resolved only 4.8% of tasks.

The difference is not just difficulty. HumanEval problems are designed to be solvable in isolation. SWE-bench tasks require understanding an existing codebase, its conventions, and its test harness.

Why the gap matters

How to evaluate agents honestly

Benchmark vs. real-world gap HumanEval small isolated problems SWE-bench real repo issues + tests High HumanEval does not imply high real-world fix rate

VERDICT

HumanEval is a useful but narrow signal. For coding agents, real-repository benchmarks and your own test suite are the honest measure of readiness.

Sources