There is exactly one credible randomised controlled trial on whether AI coding tools speed up working developers, and its result was that experienced developers got measurably slower while believing they had got faster.

METR gave 16 developers with moderate AI experience a set of real issues in mature open-source projects they had contributed to for about five years on average. The design was randomised at the issue level: 136 issues were worked on with AI allowed, 110 with AI disallowed. Developers could use any AI tool they liked in the allowed condition, including none, and in the disallowed condition no generative tooling was permitted at all, though search engines stayed available. Primarily they used Cursor Pro and Claude 3.5 or 3.7 Sonnet.

What was measuredResult
Implementation time, AI allowed vs disallowed19% slower
Raw difference in times34%
Measured with screen-recording time instead of self-report25% slower
Developers’ forecast before starting24% faster
Developers’ estimate after finishing20% faster
Economics experts’ forecast39% faster
Machine learning experts’ forecast38% faster

Two things about that table matter more than the headline. The first is that everyone was wrong in the same direction. The economists and the machine learning researchers were not the naive group here; they were the furthest off. The developers were also wrong after the fact, by about 39 points, which means self-report from a developer using a coding assistant is close to worthless as a measurement.

The second is the confidence interval, which does not exist. METR’s own write-up says the intervals were computed with clustered standard errors accounting for the number of developers and were not reported in the released paper. With 16 developers on a 246-task study, treat 19% as a well-motivated finding rather than a precisely estimated constant.

Why experienced developers in familiar repositories is the worst case

The authors were careful about what this does and does not show, and they are the source of the caveat, not a bystander to it.

They state the results do not imply AI systems are not useful in realistic, economically relevant settings. They found that high familiarity with the repositories, and the size and maturity of those repositories, both contributed to the slowdown, and that those factors do not apply to many software development settings. They name small greenfield projects and work in unfamiliar codebases as settings where their results are consistent with substantial speedup.

They also say future models may speed up developers in this exact setting, and that better prompting, agent scaffolding or domain-specific fine-tuning could flip the result. And they explicitly warn readers against overgeneralizing.

They are right about the mechanism, and the mechanism is worth stating plainly. The slowdown concentrates in the part of the work that requires knowing what the codebase already does before you can write anything. The 19% figure is not a statement about generating code. It is a statement about the cost of reconstructing context that an experienced person already had for free.

That framing also explains why the finding is specific to repositories averaging five years of ownership. Nobody has five years of context in a new project.

Do not use this as an argument either way

Two things to be careful about. The study is dated: METR’s own page now carries a banner saying the results are out of date and superseded by a continuation study covering early 2026. This article describes a measurement of early-2025 tools, not the current frontier.

Second, the population is 16 developers on mature open-source repositories, chosen because they were experienced in exactly the setting studied. That is the population most likely to be slowed down by a tool that does not know the codebase. METR says outright that they do not claim their developers or repositories represent a majority or plurality of software development work.

Who was measured, and who was not 16 developers 246 real issues 5 years average repo experience Cursor Pro Claude 3.5 / 3.7 Sonnet Cursor Pro plus Claude 3.5 / 3.7 Sonnet Randomised 136 issues AI on 110 issues AI off no genAI when off Slowdown likely here familiar, mature, large repositories May not transfer here greenfield, unfamiliar codebases

VERDICT

Do not run a productivity argument on self-report, because the only randomised controlled trial on this question found that experienced developers were 19% slower with AI while estimating they had been 20% faster, and that is a 39-point error in perception made by people who were being watched. That is the finding to carry, more than the 19%. It also means your own impression that the tool is speeding you up is not evidence, in the same way that a developer's confidence in generated code is not a test suite. What the study supports is narrower and more specific than either the enthusiasts or the sceptics claim: 16 developers with about five years of history in mature open-source repositories took 19% longer across 246 issues when AI tooling was allowed, measured at 34% on raw times and 25% using screen recordings rather than self-reported effort. The mechanism is context reconstruction, not code generation, which is why the number is largest precisely where a person already knew the codebase and smallest where nobody does. There is no published confidence interval for the headline estimate, and the authors say so themselves, so treat 19% as well-motivated rather than precise. Two limits to carry with the number: this measures the early-2025 frontier, and METR now labels the result out of date and superseded by a continuation study, and the sample is 16 people chosen for expertise in the exact repositories studied, a population with a structural reason to be slowed by a tool that cannot read what they already know. The practical use of this study is not a verdict on AI coding tools. It is a design rule: give the tool work where the repository is small, new or unfamiliar, where the 19% penalty has no context to reconstruct, and measure elapsed time yourself rather than asking how it felt.

Sources