A high score on a coding benchmark does not mean an AI agent can fix real bugs. The gap between benchmark problems and real repository work is large, and it shows up in the numbers.
What HumanEval measures
HumanEval (Chen et al., 2021) consists of 164 hand-written Python problems. Each problem is a short function with a docstring and unit tests. Models are scored by pass@k: the fraction of problems solved within k samples. Codex-12B reached 28.8% pass@1 on this set.
That is a useful signal for code generation on small, self-contained tasks. It is not a signal for editing a real codebase, running a test suite, or navigating a repository.
What real-world tasks look like
SWE-bench (Jimenez et al., 2023) was built from real GitHub issues and their gold patches across popular Python repositories. An agent must reproduce the issue, locate the relevant files, edit the code, and make the hidden tests pass. On the full SWE-bench set, a strong baseline (Claude 2) resolved only 4.8% of tasks.
The difference is not just difficulty. HumanEval problems are designed to be solvable in isolation. SWE-bench tasks require understanding an existing codebase, its conventions, and its test harness.
Why the gap matters
- A model can ace HumanEval while failing on the issues you actually file.
- Benchmark saturation (scores approaching 100%) on HumanEval has not translated to proportional gains on repository tasks.
- Evaluating an agent on HumanEval alone overestimates production readiness.
How to evaluate agents honestly
- Prefer repository-level benchmarks (SWE-bench and its Verified subset) for agent claims.
- Run the agent on your own codebase with your own tests before trusting it.
- Track not just pass rate but also the cost, number of attempts, and regressions introduced.
VERDICT
HumanEval is a useful but narrow signal. For coding agents, real-repository benchmarks and your own test suite are the honest measure of readiness.