9 Limits and Risks
Behind the promise of vibe coding lies a bill to be paid. This chapter lays that bill out honestly: the hollowing-out of understanding (codebases turning into black boxes), security, quality debt, the breakdown of the pathway to expertise, and the question of accountability. The standard here is evidence, not optimism or fear.
9.1 The Codebase Becoming a Black Box
Pre-draft. Check TOPICS.md at the repository root for owner and status.
Brief. Code that AI writes quickly piles up just as quickly, in a state where no one understands the whole. This section addresses the problem of hollowed-out understanding. It examines the process by which individual commits work but the people who grasp the overall architecture disappear from the organization, and what happens when AI layers more code on top of such a codebase again. Establishing criteria for when “code that works without being understood” becomes a problem and when it does not is the core argument of this section. This serves as the introduction to the whole of Chapter 9 (Limits and Risks).
Seed Questions
- Are there reported cases where “code that works without being understood” actually led to accidents or outages?
- What do empirical studies on the relationship between codebase comprehension and maintenance cost show?
- When AI layers more code on top of code that AI already wrote, is the accumulation of errors or style inconsistencies confirmed?
- How do organizations try to measure or mitigate the state where “no one knows the whole”? Have documentation requirements or onboarding practices actually changed?
- How does this problem play out differently in personal projects (solo development) versus team codebases?
Research Pointers
- Prioritize finding academic papers on program comprehension and codebase complexity.
- Collect field reports from practice, such as incident postmortem reports and maintenance experience accounts.
- Cross-reference with §11.1, which addresses Parnas’s (1972) module decomposition argument.
9.2 Security: A New Attack Surface
Not yet drafted. Check TOPICS.md in the repository root for owner and status.
Brief. Does AI-generated code create vulnerabilities of a different kind, or at a different frequency, than human-written code? This section covers empirical research on security vulnerabilities in generated code, prompt injection where an agent reads external content (search results, documents, issue comments) and ends up executing malicious instructions embedded in them, and slopsquatting, where a model invents a package name that doesn’t exist and an attacker registers that name first. Figures and cases must be verified against primary sources before being cited.
Seed questions
- What empirical studies compare vulnerability rates in AI-generated code with human-written code, and is their methodology reliable?
- Concretely, what attack paths does prompt injection create in the context of coding agents?
- Are there actual reported cases of slopsquatting, and how large is the estimated damage?
- How do the execution privileges granted to coding agents (shell commands, file writes, network access) change the security model?
- What mitigations do vendors (Anthropic, GitHub, etc.) present in official documentation for this attack surface?
Research pointers
- Prioritize searching for academic papers that quantitatively analyze security vulnerabilities in generated code
- Check empirical reports from security firms and researchers covering prompt injection and slopsquatting
- Cross-reference with the slopsquatting entry (G14) in the glossary, but divide roles to avoid duplication
9.3 Quality Debt and Maintenance
Not yet drafted. Check assignment and status in TOPICS.md at the repository root.
Brief. Rapidly generated code often works but is hard to fix. This section covers how quality debt is similar to and different from the existing concept of technical debt, how AI’s generation speed changes the rate of debt accumulation, and when rewriting from scratch becomes the cheaper option. If there are attempts to calculate rewrite costs financially, they will be checked and introduced. This pairs with §11.2 (technical debt, from a rediscovery perspective), but this section emphasizes the practical and financial perspective.
Seed Questions
- Is there empirical research showing that AI-generated code has different maintenance costs compared to human-written code?
- When applying the technical debt metaphor to AI-generated code, what fits and what doesn’t?
- What concrete criteria (complexity metrics, review time, etc.) are used in practice to identify “code that works but cannot be fixed”?
- What financial frameworks have been proposed for judging whether a rewrite or incremental refactoring is cheaper?
- Is the increase in code generation speed outpacing the growth rate of review and testing capacity, and is this confirmed by data?
Research Pointers
- Check Cunningham’s (1992) original presentation of technical debt (OOPSLA experience report) in the original text.
- Prioritize searching academic papers on software maintenance costs and refactoring-versus-rewrite decisions.
- Divide roles with §11.2: this section takes the practical/financial perspective, §11.2 takes the conceptual history perspective.
9.4 The Junior Paradox: The Ladder Disappears
Pre-draft. Check ownership and status in TOPICS.md at the repository root.
Brief. The work that AI substitutes for best is often the work that juniors used to take on in order to grow. This section addresses the concern that automating entry-level work cuts off the ladder to expertise. It looks for data on junior developer hiring and changes in internships and entry-level postings, to determine whether this problem is actually confirmed by data or is an exaggerated narrative. It pairs with §13.3 (The Ladder of Expertise), but this section weighs more heavily toward labor market and hiring data. Figures must be verified against original data before being cited.
Seed Questions
- How have the number of junior developer job postings and the scale of internships actually changed recently, and what are reliable data sources for this?
- Is the observation that “juniors have become faster with AI” and the observation that “junior hiring has declined” different facets of the same phenomenon, or are they separate phenomena?
- Have companies ever officially acknowledged that the decline in entry-level hiring is due to AI, or do other factors such as the economy or corrections for prior overhiring play a larger role?
- What research refutes the concern that “the ladder is disappearing,” or offers a different explanation?
- Are there reported cases of company- or industry-level responses to the decline in entry-level hiring (such as maintaining junior-dedicated projects, redesigning mentoring, etc.)?
Research Pointers
- Prioritize searching empirical labor economics research that deals with job postings and employment statistics; figures must be verified against original data before being cited.
- Distinguish between industry surveys (such as developer hiring trend reports) and academic research, and treat their reliability differently.
- Divide roles with §13.3 (The Ladder of Expertise) and §14.2 (Changes in the Job Landscape): this section is limited to the coding ladder.
9.5 Accountability and Sign-off: Who Signs Off on This Code
Not yet drafted. Check ownership and status in TOPICS.md at the repository root.
Brief. When an accident occurs in code written by AI, who is responsible? This section covers how the traditional sign-off practices of code review and deployment approval are being redefined in the face of AI-generated code, what AI code governance policies (review requirements, labeling of generated code, designation of accountable parties) real companies and institutions have actually adopted, and how responsibility has actually been distributed in incident cases. As the final section of Chapter 9, it asks whose responsibility the preceding problems (the hollowing-out of understanding, security, and quality debt) ultimately become at the organizational level.
Seed Questions
- In outages or incidents caused by AI-generated code, how has responsibility actually been attributed, and are there any publicly documented cases?
- What do the AI code governance policies formalized by companies and institutions specifically require regarding review obligations, approval procedures, and labeling of generated code?
- Is the traditional practice of code review approval being maintained only in form, while its substance hollows out in the face of AI-generated code?
- What labeling and review rules are open source projects introducing for AI-generated contributions?
- How far have discussions of legal and contractual liability (such as analogies to product liability) progressed?
Research Pointers
- Collect AI code governance policy documents published by companies and government agencies as primary sources.
- Check actual cases in open source projects’ AI contribution policies (contribution guidelines).
- Cross-reference with §10.3, which covers the teleology of code review, but divide the roles: this section is limited to accountability and governance.