Nine widespread beliefs about AI codebases, checked one by one

Nine claims about controlling an AI-built codebase keep circulating right now. We checked each one against our own gate register and against outside sources. Five held up cleanly, two held with a correction, one did not survive the check and one splits apart once the source's own interest gets named.

Advice on keeping an AI-built codebase under control circulates widely right now. We checked nine claims that repeat most often across sources, each one against our own repository and against outside sources.

Table 1: Nine claims, our verdict on each.

1. Claim: AI codebases degrade fast if nobody watches

Figure 1: Duplication up, refactoring down: the wall behind the first claim.

Image generated by AI

This one holds, with numbers behind it now. GitClear analysed 211 million lines of code across the shift to AI-assisted coding. The share of copy-pasted lines rose from 8.3% of all changed lines to 12.3% of all changed lines. The share of refactored lines fell from 25% of all changed lines in 2021 to under 10% of all changed lines in 2024. [1]

Code churn nearly doubled over the same years. Less refactoring, more duplication, more churn, confirmed now with real numbers behind it. That is the wall the advice describes.

More evidence points the same way, one study against it:

• A controlled study found that AI tooling added 19 percent of extra time to task completion. [2]
• A large annual survey found that AI adoption lifts individual output and still harms delivery stability. [3]
• One 2026 measurement found more churn in AI-touched code within 30 and 90 days than in human code. [4]
• The same measurement found less duplication in AI code than in human code, a real point against the wall. [4]
• The share of developers who distrust AI code accuracy rose from 31% of respondents to 46% of respondents in one year. [5]

2. Claim: code is the only documentation

Figure 2: Documentation ages. So does the code it describes.

Image generated by AI

The reasoning is sound at first glance. External documentation goes stale. A stale document is worse than none: its content stopped being true. An empirical study bears that out at a large scale. Tan, Wagner and Treude scanned over 3,000 GitHub projects for stale code references. Function names and file names, still named in READMEs and wikis, already deleted from the source.

"Outdated documentation is a pervasive problem in software development, preventing effective use of software, and misleading users and developers alike." [6]

Most of the scanned projects carried at least one such stale reference at some point in their history.

Here is where the claim overreaches. Code itself is not immune to the same rot. A comment inside a function drifts from what the function now does. A README drifts from the code it describes, for the same reason. Neither stays in sync when the other changes. Deleting the external document does not remove that failure mode. It just moves the stale text one file closer to the logic it misdescribes.

Our own repository keeps a written record on purpose. One machine readable spec file per piece of content states the single claim it makes, plus every source behind a number in it. We do not trust it by default. A spec drift gate reads the spec on every push. It reads the artifact too. It fails the build the moment the two disagree. That is a different answer to the same problem the claim tries to solve. A machine checks this record every time. It cannot go stale unnoticed like those 3,000 scanned projects.

The claim overreaches in the wider literature too:

• Inconsistent code comments are roughly 1.5 times more likely to sit inside a bug-introducing change. [7]
• Even the source most cited for "code as documentation" rejects the word "only" in that claim. [8]
• Consumer-driven contracts enforce a written agreement against the real code on every single build. [9]
• Google's own style guide calls unmaintained documentation dead. Dead docs actively mislead engineers. [10]

3. Claim: schedule a mandatory cleanup every two weeks

Figure 3: A calendar date is not a control.

Image generated by AI

We tried a close relative of the fourteen day sweep first. It was more careful than a calendar habit. A hand-picked subset of our own automated checks, chosen by a person, ran locally before a push. That subset came back green twice. The full set of checks ran on the same change. It came back red both times. Not carelessness. The register had simply grown past what a person reliably holds in memory while picking a subset.

That growth is not an accident either. Ward Cunningham introduced the debt metaphor for exactly this dynamic in 1992: "Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite" [11]. A fixed two week interval is a decision. Let the interest accrue on a schedule instead of paying it as it is incurred. Our case shows what that delay costs: two real gaps opened in exactly the days between one hand-run check and the next.

So the control moved out of a person's memory. It moved into the merge itself. The register we check against currently lists 43 automated checks. A git hook runs the full set on every push. It blocks the push the moment one check is red. A red check does not go on a list for the next sweep. It gets fixed on the spot. That happens before the branch merges. No two week window lets a defect sit unrepaired.

The research on both halves of that mechanism agrees:

• Projects running continuous integration shipped releases more than twice as often as projects without it. [12]
• Detection accuracy in a controlled watch task declined significantly the longer the watch went on. [13]
• Debt mounts faster, not slower, on projects that skip automated testing, a later study on Cunningham's own metaphor found. [14]
• The field's own definition of continuous integration is a build and a test run on every commit. [15]
• Across 2.6 million real builds, one extra automated check caught 10 percent more failures. [16]

4. Claim: fix incrementally, leave every file a little cleaner

Figure 4: Small repairs, sewn in as they happen.

Image generated by AI

This claim holds here too. We wrote the same rule for ourselves in plain words:

"wir brauchen eine regel, dass bei roten gates das repariert und nicht einfach aufgehört wird." [17]

A finding is work for the current change, never a note for later. The difference against the widespread version of the claim is what carries it. The habit does not rest on a person choosing to leave a file cleaner. A red gate blocks the merge until someone does exactly that.

The claim has old, well tested company:

• A full rewrite from scratch was the single worst mistake a company could make, one famous case study concluded. [18]
• Reviewers catch 70 to 90 percent of defects when a change stays under 400 lines. [19]
• Smaller code changes get measurably more effective review than large ones. [20]

5. Claim: keep it lean, ship only what works cleanly

Figure 5: One tool, well used, beats a drawer full of them.

Image generated by AI

No correction needed here. Fewer moving parts fail less often. A 43 check register only works because each check tests one narrow thing well. We agree. The idea is old:

• Write programs that do one thing and do it well, the Unix philosophy said in 1978. [21]
• Build something only when it is actually needed, never for a future you merely expect. [22]
• Simplicity is the most important consideration in a design, one classic essay on software argued. [23]

6. Claim: AI coding delivers ten times the output of a human

Figure 6: Ten times and ten percent, measured on two different scales.

Image generated by AI

This claim does not survive the check. It traces back to real, on-the-record statements, not just internet folklore.

"I'm less interested in that question than when a coder becomes 10 times more productive. I think that can happen this year, or next year." [24]

That is one leading AI lab's own chief executive, speaking about coding output. Measured studies land far below it.

• A management consultancy measured real engineering teams and found efficiency gains of roughly 10 to 15 percent of output, not ten times. [25]
• An engineering analytics study across more than 400 companies found pull request throughput rose by under 8 percent of its prior rate. [26]
• Our own earlier finding fits the same picture: a controlled study found AI tooling added 19 percent of extra time to task completion, a slowdown, not a speedup. [2]

A ten times claim and an 8 percent measurement describe two different worlds. Only one of them comes from a repeated, independent measurement.

7. Claim: the agent that writes the code should not also approve it

Figure 7: A second, separate set of eyes.

Image generated by AI

This one holds. It matches the shape of our own gate register: a check that a human wrote lives outside the agent it watches.

"Code review functions as the final checkpoint before production, and that checkpoint can't be trusted when the same model both produces the code and signs off on it." [27]

The mechanism behind that claim is measured, not just argued.

• Code passed review nine to seventeen percentage points more often when the reviewer came from the same model family as the writer, one vendor measurement found. [27]
• Language models rate outputs closer to their own style higher, regardless of who actually wrote them. [28]
• Self-recognition and self-preference rise together: a model that can tell its own output apart from others also rates that output higher. [29]
• The pattern predates AI by decades. NASA ran independent verification teams alongside shuttle software developers for exactly this reason. [30]

8. Claim: AI code carries security vulnerabilities at a high, stubborn rate

Figure 8: A crack that a vendor's own number alone will not show.

Image generated by AI

Partly right. The number quoted most often, 45 percent, comes from one vendor with a stake in the answer. Independent tests do not agree with each other either.

"AI-generated code introduces security vulnerabilities in 45 percent of cases." [31]

The same vendor retested a year later and found little movement.

"Security pass rates remain stubbornly stuck at approximately 55 percent, virtually identical to where they stood two years ago." [32]

An earlier, independent academic study on a previous model generation landed close by.

• Roughly 40 percent of tested completions from an early code assistant carried a known vulnerability class, a peer-reviewed study found. [33]

One more independent study complicates the picture rather than confirming it.

• A large-scale analysis of AI-generated code already merged into real GitHub projects found identifiable vulnerabilities in only about 12 percent of cases. [34]

Benchmark prompts designed to probe for weaknesses and real code that already passed a human's review are not the same test. The rate depends heavily on which one you run.

9. Claim: AI-authored pull requests carry more bugs and take longer to review

Figure 9: The queue AI-written code waits in.

Image generated by AI

This one holds on the numbers, with two of the four sources selling the fix.

• AI-authored pull requests carried 10.83 issues per request against 6.45 issues per human-only request, a vendor study of 470 real pull requests found. [35]
• AI-generated pull requests waited 4.6 times longer for review than human-written ones, a second vendor's benchmark found. [36]
• An independent, non-commercial study of over 500,000 code samples found AI-generated code carries more high-risk vulnerabilities than human-written code. [37]
• Our own earlier finding applies here too: AI tooling added measurable extra time to a task, not less. [2]

Two of those four sources sell AI code review tools. The pattern still holds once you set those two aside: an independent academic study points the same way.

10. The takeaway: what to actually watch for when you code with AI

Five claims out of nine held up cleanly. Two more held with a real correction. One did not survive the check at all. The ninth splits apart once the source's own interest gets named. Turn the pattern behind that into a short list for your own repository:

• Watch the ratio ahead of the total. A rising share of duplicated lines and a falling share of refactored ones warn you before a defect count does. [1]
• Never let a hand-picked subset of checks stand in for the full set. Ours agreed with itself twice and still missed what the full register caught.
• Bind the check to the merge, not to a date. A rule that fires later fires too late for at least one change in between.
• Keep a written spec if you want one, but only if a gate reads it against the code on every push. An unchecked spec decays exactly like an unchecked comment. [7]
• Fix the small thing now. A red result blocks the one change in front of you, not a future sprint.
• Add a check only where a concrete failure already happened once. A register nobody can explain earns the same distrust the input side already has. [5]
• Never let the agent that wrote the code be the only one who checks it. Self-review bias is measured, not just suspected. [28]
• Treat a headline number from a vendor as a lead, not a verdict, until an independent source lands near it too. [31]

11. Conclusion

None of this was really about nine claims. It was about where trust comes from. A number without an independent check is a claim like any other, ten times the output or forty five percent of cases alike. Our register never settles whether AI helps in general. It settles one narrower question, on every single push: does the code in front of it do what it claims, no matter who or what wrote that code.

12. Sources

[1] GitClear, "AI Copilot Code Quality: 2025 Data Suggests 4x Growth in Code Clones", GitClear research report, 2026. https://www.gitclear.com/ai_assistant_code_quality_2025_research

[2] METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 2025, arXiv:2507.09089. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

[3] DORA / Google Cloud, "Accelerate State of DevOps Report 2024", 2024. https://dora.dev/research/2024/dora-report/

[4] "A Large-Scale Comprehensive Measurement of AI-Generated Code in Real-World Repositories", arXiv preprint, 2026, arXiv:2603.27130. https://arxiv.org/abs/2603.27130

[5] Stack Overflow, "2025 Developer Survey", 2025. https://stackoverflow.co/company/press/archive/stack-overflow-2025-developer-survey/

[6] W. S. Tan, M. Wagner and C. Treude, "Detecting Outdated Code Element References in Software Repository Documentation", Empirical Software Engineering, 2023, arXiv:2212.01479. https://arxiv.org/abs/2212.01479

[7] T. Wu et al., "Investigating the Impact of Code Comment Inconsistency on Bug Introducing", arXiv preprint, 2024, arXiv:2409.10781. https://arxiv.org/abs/2409.10781

[8] M. Fowler, "Code As Documentation", martinfowler.com bliki, 2005. https://martinfowler.com/bliki/CodeAsDocumentation.html

[9] M. Fowler, "Consumer-Driven Contracts: A Service Evolution Pattern", martinfowler.com, 2006. https://martinfowler.com/articles/consumerDrivenContracts.html

[10] Google, "Documentation Best Practices", Google Style Guide, 2026. https://google.github.io/styleguide/docguide/best_practices.html

[11] W. Cunningham, "The WyCash Portfolio Management System", Addendum to the Proceedings of OOPSLA '92, ACM SIGPLAN OOPS Messenger, vol. 4, no. 2, pp. 29 to 30, 1992. https://c2.com/doc/oopsla92.html

[12] M. Hilton, T. Tunnell, K. Huang, D. Marinov and D. Dig, "Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects", ASE 2016, pp. 426 to 437. https://mir.cs.illinois.edu/marinov/publications/HiltonETAL16ContinuousIntegration.pdf

[13] R. A. Grier, J. S. Warm, W. N. Dember, G. Matthews, T. L. Galinsky, J. L. Szalma and R. Parasuraman, "The Vigilance Decrement Reflects Limitations in Effortful Attention, Not Mindlessness", Human Factors, 45(3), 2003, pp. 349 to 359. https://sciences.ucf.edu/psychology/perl/wp-content/uploads/sites/29/2013/01/Grier-et-al-2003-vigilance-is-effortful-not-mindless.pdf

[14] P. Kruchten, R. L. Nord and I. Ozkaya, "Technical Debt: From Metaphor to Theory and Practice", IEEE Software, 29(6), 2012, pp. 18 to 21. https://www.sei.cmu.edu/documents/360/2012_019_001_58818.pdf

[15] DORA / Google Cloud, "Continuous Integration", capability page, dora.dev, 2026. https://dora.dev/capabilities/continuous-integration/

[16] M. Beller, G. Gousios and A. Zaidman, "Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub", MSR 2017, pp. 356 to 367. https://research.tudelft.nl/en/publications/oops-my-tests-broke-the-build-an-explorative-analysis-of-travis-c

[17] "wir brauchen eine regel, dass bei roten gates das repariert und nicht einfach aufgehört wird", internal house rule A13, AGENTS.md (this repository), Owner-Entscheidung 2026-08-08. https://github.com/betteryieldsgmbh/marketing/blob/main/AGENTS.md

[18] J. Spolsky, "Things You Should Never Do, Part I", Joel on Software, 2000. https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/

[19] J. Cohen / SmartBear Software, "Best Practices for Peer Code Review", 2026, summarising the Cisco code review study (roughly 2,500 reviews, 3.2 million LOC). https://smartbear.com/learn/code-review/best-practices-for-peer-code-review/

[20] G. Kudrjavets and A. Rastogi, "Does Code Review Speed Matter for Practitioners?", arXiv preprint, 2023, arXiv:2311.02489. https://arxiv.org/abs/2311.02489

[21] M. D. McIlroy, "UNIX Time-Sharing System: Foreword", The Bell System Technical Journal, 57(6), 1978. https://inigomedina.co/library/work/mcilroy-unix-foreword

[22] R. Jeffries, "You're NOT gonna need it!", ronjeffries.com, 2026. https://ronjeffries.com/xprog/articles/practices/pracnotneed/

[23] R. P. Gabriel, "The Rise of Worse Is Better", from Lisp: Good News, Bad News, How to Win Big, dreamsongs.com, 2026. https://www.dreamsongs.com/RiseOfWorseIsBetter.html

[24] S. Altman, remarks on the Varun Mayya Podcast, quoted in "AI Could Make Coders 10x Productive This Year: OpenAI CEO Sam Altman", OfficeChai, 2025. https://officechai.com/ai/ai-could-make-coders-10x-productive-this-year-openai-ceo-sam-altman/

[25] Bain and Company, "Beyond Code Generation: More Efficient Software Development", Technology Report 2024, 2024. https://www.bain.com/insights/beyond-code-generation-more-efficient-software-development-tech-report-2024/

[26] J. Reock, "AI productivity gains are 10%, not 10x", DX Research, 2026. https://getdx.com/blog/ai-productivity-gains-are-10-percent-not-10x/

[27] Y. Xu and L. Ye, "AI-Written Code Needs Independent Review", CodeRabbit blog, 2026. https://www.coderabbit.ai/blog/code-review-needs-independence

[28] K. Wataoka, T. Takahashi and R. Ri, "Self-Preference Bias in LLM-as-a-Judge", arXiv preprint, 2024, arXiv:2410.21819. https://arxiv.org/abs/2410.21819

[29] A. Panickssery, S. R. Bowman and S. Feng, "LLM Evaluators Recognize and Favor Their Own Generations", arXiv preprint, 2024, arXiv:2404.13076. https://arxiv.org/abs/2404.13076

[30] National Research Council, "An Assessment of Space Shuttle Flight Software Development Processes", National Academies Press, ch. 2, 1993. https://www.nationalacademies.org/read/2222/chapter/4

[31] Veracode, "2025 GenAI Code Security Report", 2025. Commercial source: Veracode sells application security testing tools. https://www.veracode.com/blog/genai-code-security-report/

[32] Veracode, "Spring 2026 GenAI Code Security Update", 2026. Same commercial interest as [31]. https://www.veracode.com/blog/spring-2026-genai-code-security/

[33] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt and R. Karri, "Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions", IEEE S&P 2022, 2022, arXiv:2108.09293. https://arxiv.org/abs/2108.09293

[34] M. Schreiber and P. Tippe, "Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories", arXiv preprint, 2025, arXiv:2510.26103. https://arxiv.org/abs/2510.26103

[35] D. Loker, "State of AI vs Human Code Generation Report", CodeRabbit, 2025. Commercial source: CodeRabbit sells AI code review tooling. https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report

[36] Opsera, "2026 AI Coding Impact Benchmark Report", 2026. Commercial source: Opsera sells AI and DevOps coding oversight tooling. https://opsera.ai/newsroom/new-opsera-report-reveals-how-ai-is-transforming-software-delivery-and-driving-business-outcomes/

[37] D. Cotroneo, C. Improta and P. Liguori, "Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity", arXiv preprint, 2025, arXiv:2508.21634. https://arxiv.org/abs/2508.21634