- What: An article discusses the limitations of using AI to secure its own generated code
- Impact: Raises concerns about the reliability of AI-driven security verification
AI can tell you what your software does. It cannot tell you what your software must never do. So you should not rely on it If I wrote my own financial audit report and handed it to you, would you give me a loan? Most people would not. A self-certified audit tells you nothing because the auditor and the auditee are the same person. This is similarly happens when you ask AI to verify the security of code that AI wrote. The conflict is identical. Paying for more audits does not fix it. Every audit cycle costs more tokens, produces more confidence, and leaves the underlying liability untouched. This is the structure of a Ponzi scheme. Early returns look real. The liability grows in the dark. The collapse comes when someone external comes to collect. In a financial Ponzi scheme, that is the investor who wants their money back. They were never inside the loop. They do not care about the audit report. They just want to see the money. In software security, that someone is the adversary. What Vibe Coding Actually Delivers # When you vibe code, you describe a feature in natural language, the AI generates the code, and it works. For functional requirements, this is a legitimate productivity gain. A functional requirement defines what software should do. The user authenticates, the payment processes, the API returns the expected response. These requirements are bounded and testable. You can write a test, run it, and get a pass or fail. AI handles this well. The problem is that software has another category of requirements. Non-functional requirements define how software should behave under load, under failure, and under attack. Security sits here. It is not about what the software does. It is about what the software must never do. That distinction matters because the verification methods are fundamentally different. Functional requirements have a known expected output you can test against. Security requirements have no finite expected output. You cannot enumerate every way a system can be exploited. The Ponzi Mechanics # Imagine a fund manager who promises high returns on your investment. You invest, and shortly after you receive a return. You are satisfied, so you invest more and tell your friends. They invest too. What you do not know is that your return was not generated from any real profit. It was paid using the money your friends just deposited. The fund has never actually made anything. As long as new investors keep joining, the manager can keep paying returns and the scheme stays alive. The moment new money stops coming in, there is nothing left to pay anyone. That is how a Ponzi scheme works. The early returns are real. The mechanism behind them is not. Using AI to review vibe coded software follows the same pattern. You prompt the AI to generate code, the tests pass. You pay for an AI security scan, it surfaces some issues, the AI fix them and ship the feature. Each AI cycle produces a report that looks reasonable. The AI vendor profits on every iteration because more code means more tokens to generate it, more tokens to scan it, and more findings means more tokens to fix. The security reports keep looking good, confidence keeps growing, and the loop continues until an adversary finds a vulnerability and tells you otherwise. The underlying attack surface has never been independently verified. It has only been assessed by the same system that created it. That is not a security audit. That is a self-certified statement. Ponzi schemes collapse when someone outside the loop comes to collect and discovers the money was never there. In software security, that outside party is the adversary. They are not part of your development cycle, they did not read your security report, and they only need to find one thing to prove your software is not secure. Functional and Non-Functional Requirements # To understand why security behaves this way, we need to look at how software requirements are defined. Software requirements are divided into two categories. Functional requirements define what a system should do. A user can register an account, upload a file, or place an order. These are concrete, observable behaviours. You can write a test for each one, run it, and confirm whether the software satisfies it. Non-functional requirements define how a system should behave across all conditions. Performance, reliability, maintainability, and security all live here. They are not about specific features. They are about the properties of the system as a whole. Security is a non-functional requirement, but it has a characteristic that makes it different from the others. Performance has measurable thresholds. Reliability can be tested under simulated failure conditions. Security cannot be defined the same way because it is expressed as a negative requirement. It is not about what the software should do. It is about what the software must never do. In software engineering, this is modelled through misuse cases. A use case describes a le...