Informa TechTarget | SearchSecurity Cybersecurity Dive InformationWeek Channel Dive Explore our brands Dark Reading Resource Library Black Hat News Omdia Cybersecurity Advertise NEWSLETTER SIGN-UP Cybersecurity Topics World The Edge DR Technology Events Resources APPLICATION SECURITY CYBERSECURITY OPERATIONS CYBER RISK VULNERABILITIES & THREATS NEWS AI-Generated Patches Fail Half the Time A study of more than 6,000 patches found that even working patches can introduce new bugs, break something else, or are open to bypass. Robert Lemos,Contributing Writer August 7, 2026 5 Min Read SOURCE: NNATTALLI VIA SHUTTERSTOCK As developers turn to AI models to generate an increasing amount of code, they are also relying on the systems to find vulnerabilities and generate patches. Unfortunately, the models just aren't very good at their jobs. Recent research suggests that even the latest AI systems only produce effective patches about half the time, according to a report published on Aug. 6 by identity management firm 1Password. To test their efficacy, the company's Off-By-1 research team generated 540 patches for six vulnerabilities disclosed since March using two different large language models (LLMs): OpenAI's ChatGPT-5.5 with Trusted Access for Cyber, and Anthropic's Opus 4.8 with Cyber Verification Program. Of those 6,080 patches, only 46% of them solved the underlying vulnerability, and many introduced new vulnerabilities into the mix. Even the working patches were often "brittle," fixing only a narrow set of cases or were bypassed with a fairly easy workaround, says Keith Hoodlet, director of security research for 1Password. Related:No Perfect Fix for AI Browser Prompt Injection Flaws "Based on our research for [AI-generated patches for flaws] that are novel or not part of the training data set for the AI labs, the patch success rate has gone negative, in terms of the outcomes," he says. AI systems appear to have an easier time exploiting flaws than patching them, an unfortunate asymmetry in capabilities, Hoodlet points out. In July, OpenAI's latest research model escaped its sandboxed environment and attacked open-model repository Hugging Face. The escape is not unique: Anthropic disclosed that its Claude AI model had escaped more than once as well, and Meta's AI this week also went rogue, though details are scarce. Meanwhile, attackers are increasingly using AI to find vulnerabilities and automated attacks in ways that pressure defenders to keep up. "Because of AI and agent capabilities, [attackers] are expanding their ability to attack more companies, more individuals, more codebases at scale in a way that I don't think we've ever seen before," he says. "Because of that, the best thing I think defenders can do is apply time and energy to truly burning down vulnerability backlogs if they have them." Why AI-Generated Patches Are "FLAWED" 1Password's shows that for six major open source flaws, a variety of patching pipelines, differing prompts, and two major models resulted in less than half of the issues actually being patched. The research, dubbed "Fix-Like Artifacts with Embedded Defects" or FLAWED, classified patches into five scenario categories: fixed; fixed by altering behavior; not fixed; fixed but opened a new flaw; and not fixed and added a new flaw. Only a quarter of patches (26%) fixed the original vulnerability without changing application behavior, while 20% fixed the vulnerability by modifying how the code works. The largest volume of patches (49%) did not fix the vulnerability, 1Password stated in its research report. Related:AI Notetaker Lets Hackers Spy on Government, Corporate Video Calls The results closely resembled research conducted by application security firm Veracode, which found that, across more than 100 models and 80 coding tasks, the average security pass rate for AI-generated code was 56%. Nearly half of the generated code (44%) introduced detectable OWASP Top 10 vulnerabilities, the company stated in its research brief. The research "shows that, given a choice of implementation, LLMs are very inconsistent about whether they choose secure or insecure strategies," says Sam Guyer, principal architect at Veracode. "In both cases, the output can look complete and functional while remaining unreliable from a security standpoint." The results have a direct bearing on efforts to patch vulnerabilities using AI before attackers gain access to frontier models and find the vulnerabilities themselves. OpenAI's June "Patch the Planet" announcement that it would work with Trail of Bits, a cybersecurity consultancy, to find and close vulnerabilities has already run into problems with the quality of their patches, 1Password's Hoodlet says. Related:AI Harnesses Burst With Potential Exploit Opps "I can confirm for you, there were definitively scenario three, four, and five outcomes from the weekend patch set, and we've reported them," Guyer says, referring to FLAWED classifications; a patch either does not fix the original vulnerability or introduces a new vulnerability — or sometimes both. Another problem is that developers are increasingly accepting code from agentic AI without critical analysis of the codebase, Hoodlet says. While companies should try to hunt down vulnerabilities as fast as possible and eliminate their backlogs, they need to make sure that they are verifying and validating the patches, he says. A study of developer habits by development tools maker Cursor found that 36% of changes are accepted automatically without a manual review of the code changes. While the measurement is related to code volume, not the number of developers shifting to that practice, the agentic AI code-generation loop seems to be accelerating without a critical analysis component or even some sort of validator, Hoodlet says. For many developers, "if you go through that loop and ... the patch worked — why would you continue?" he says. Burning Down the Backlog of Vulnerabilities Development teams can still benefit greatly from AI, but must take more care in how they manage the process, says Robert Coles, senior cybersecurity engineer at Black Duck, an application security firm. "The takeaway isn't that AI-generated patches don't work — it's that they still require human validation," he says. "AI can accelerate development and help identify potential fixes, but security teams and developers need to verify that a patch actually resolves the underlying vulnerability, doesn't introduce regressions, and meets secure coding standards." Companies can continue to capture the productivity gains from AI coding, but verification must scale with generation, Veracode's Guyer says. "Developers should treat AI-generated code and patches as proposed changes, rather than verified fixes," he says. "They should run functional and regression tests, require human review for sensitive or complex changes, and backstop the process with whole-program static analysis that can trace data across files, dependencies, and call paths." AI systems are getting better, and combined with improved harnesses, the problem could be fixed relatively soon. Developers may reach the point where humans produce more rigorous specifications, and then AI systems formally verify that the code meets those specifications, Hoodlet says. "When you look at the specification creation process, that still belongs to a human, but the models themselves are getting very good at writing formal verification code," he says. "And so as long as you have good specifications, you can actually formally verify code in ways that are both far easier and perhaps more performant than what you would be able to do, say, three to five years ago." Read more about: CISO Corner About the Author Robert Lemos Contributing Writer Rob is an award-winning, veteran technology journalist of more than 30 years, reporting on global cybersecurity issues, the latest offensive and defensive technologies, malware incidents, cyber conflict, and AI's impact on software and cybersecurity. A former research engineer, Rob has written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. He has received five awards for journalism, including Best Deadline Journalism (Online) in 2003 for his coverage of the Blaster worm. Rob also analyzes data on various trends using Python and R for both his reporting and his clients. Recent reports include analyses of the shortage in cybersecurity workers, annual vulnerability trends, and annual threat reports. Rob holds degrees from Cornell University in Electrical Engineering and Computer Science (double major). Want more Dark Reading stories in your Google search results? ADD US NOW More Insights Industry Reports The State of Cloud Security: The Latest Challenges How Organizations Are Managing Incident Response How Enterprises Are Developing Secure Applications Inside RSAC 2026: security leaders reveal the risks redefining your defense strategy Essential News & Insights from Black Hat USA 2025 Access More Research Webinars Building a Secure AI Strategy for the Enterprise Is your AppSec program Mythos Ready? Experts Explain How to Develop a Framework for Cyber-Fraud Fusion Prevention at Machine Speed: Hunting Beyond Known Detections 0-Day to 10x Discovery: Security at the Speed of Mythos More Webinars You May Also Like APPLICATION SECURITY Supply Chain Attack Secretly Installs OpenClaw for Cline Users by Rob Wright FEB 19, 2026 APPLICATION SECURITY Chinese Hackers Hijack Notepad++ Updates for 6 Months by Jai Vijayan FEB 02, 2026 APPLICATION SECURITY Trump Administration Rescinds Biden-Era Software Guidance by Alexander Culafi JAN 29, 2026 APPLICATION SECURITY Microsoft Fixes Exploited Zero Day in Light Patch Tuesday by Jai Vijayan DEC 09, 2025 Black Hat USA Coverage CYBERATTACKS & DATA BREACHES The Coordination Gap: How Attackers Are Outpacing Law Enforcement byArielle Waldman AUG 6, 2026 4 MI