What studies show about the quality of AI-generated code

01

How do you read a study on AI-generated code?

With three questions: what exactly was measured, lab tasks, code in repositories, survey answers or working time in an experiment? Who published the study, and does the publisher sell something that fits the result? And does the design support a claim about causes, or only about correlations? Without that context, any percentage is only half understood.

The four studies covered here measure very different things and cannot simply be weighed against each other. They complement one another: one tests code samples for security flaws, one analyses change patterns across hundreds of millions of lines of code, one surveys thousands of practitioners, one measures working time in an experiment. How these findings carry over to the path from prototype to production is set out in From prototype to production-ready application.

02

How secure is code written by language models?

According to Veracode 2025, 45% of generated code samples failed security testing and contained a vulnerability from the OWASP Top 10. Larger, newer models performed no better. The spring 2026 update shows the same picture: nearly all the code is syntactically correct, but only just over half of it is secure.

The tasks tested covered Java, Python, C# and JavaScript, each solvable securely or insecurely. In the March 2026 update, Veracode describes the test set as 80 tasks across four vulnerability classes. The result there: over 95% syntactically correct, around 55% secure. Models guarded against SQL injection in 82% of cases, against cross-site scripting in only 15%, and against log injection in 13%. Java came in at 29% secure solutions (Veracode, 2026).

Limits: what gets tested are individual tasks covering a handful of vulnerability classes, not complete applications, and not code that has since gone through review. Veracode sells security-testing tools, so the result suits its business model. The core finding still holds up, though: accepting generated code without review means accepting a measurable risk. What the same mechanism looks like in concrete tools is shown in Built with Lovable, Bolt or Replit.

03

Does code become harder to maintain with AI assistants?

Across 211 million changed lines of code, GitClear finds a clear trend: since AI assistants became widespread, more code gets copied and less gets refactored. In 2024, copied code exceeded moved, or restructured, code for the first time, and the 2026 analysis continues the trend. That is a correlation across the whole dataset, not proof of a cause.

Why this matters: copied code has to be maintained in several places with every change, while refactored code keeps logic in one place. A falling share of refactoring suggests that new code tends to sit alongside existing code rather than being worked into it. The 2026 analysis covers 623 million changes from 2023 to 2026 and points in the same direction (GitClear, 2026).

Limits: GitClear measures all changes in the dataset, not code flagged as AI-generated, and infers refactoring from moved lines, an approximation. The publisher sells development-productivity analytics. What such patterns mean over time for a single application is covered in Spotting technical debt in AI-generated code.

04

Does AI make teams faster or more stable?

DORA 2025 does not yet show both at once. The report finds, for the first time, a positive correlation between AI use and software delivery throughput, but still a negative one with its stability. The core claim: AI does not fix a team, it amplifies what is already there, strong practices as much as weak ones.

The report State of AI-assisted Software Development 2025 draws on a survey of just under 5,000 practitioners worldwide and over 100 hours of qualitative data. 90% of respondents use AI at work, more than 80% report higher productivity, and 30% trust AI-generated code little or not at all (DORA, 2025). Stack Overflow's developer survey fits the same picture: 66% name their biggest frustration as AI solutions that are almost right, but not quite (Stack Overflow, 2025).

For production systems, the stability finding matters more. More speed without safeguards just shifts errors to after delivery. DORA therefore describes capabilities that amplify the benefit of AI, including clear rules for AI use and high-quality internal platforms. Limits: these are self-reported answers and correlations, not measured causes.

05

Are experienced developers really slower with AI?

In the 2025 METR study, 16 experienced open-source developers using AI tools took 19% longer on their tasks, yet afterwards believed themselves 20% faster. The second part is the more solid finding: self-assessment is a poor measure. METR revised the study design in 2026 because of selection effects and reassessed the strength of the evidence.

The study was a randomised experiment: 246 real tasks in large repositories familiar to the participants, averaging over a million lines of code, with tasks taking roughly two hours on average, mostly using Cursor Pro and Claude 3.5 and 3.7 Sonnet. Beforehand, the developers expected a 24% time saving. What was measured was 19% more time, with a confidence interval of plus 2 to plus 39% (METR, 2025). METR itself stresses that the study does not show that AI fails to help most developers.

In February 2026, METR reported on follow-up rounds from late 2025 and on a problem: developers increasingly declined to work on tasks without AI, and dropped tasks where they expected AI to help a lot. The new estimates point more towards a time saving, with confidence intervals that also include no effect at all. Because of these selection effects, METR calls its own data only very weak evidence and is overhauling the design (METR, 2026).

06

Which numbers circulate but do not hold up?

Some numbers keep reappearing in pieces about AI code and AI projects without holding up. Usually the number is not invented, just pulled out of context: a forecast becomes a measurement, a US study of enterprise pilots becomes a claim about German mid-sized companies (Mittelstand), a vendor scan becomes a neutral survey. Tap a card for the context.

07

What do these studies mean for your project?

None of the studies argue against AI tools. They argue against accepting output unchecked. Security testing, review and automated tests are where the measured weaknesses get caught. And anyone passing a number along should first verify it at the source; a short checklist can help with that.

How these findings translate into concrete rules, such as which AI tools a development team uses under which review obligations, is covered in Approving AI coding tools for a development team. A shorter take on where vibe coding helps and where it does not is offered in Vibe coding: where it helps, where it bites.