Security News

Cybersecurity news aggregator

đź“°
INFO News Reddit r/netsec

Claude Fable 5: the agent harness matters more than the frontier model

  • What: Claude Fable 5 was benchmarked for security performance
  • Impact: Results show 72.6% functional solves and 29% security solves, leaving most vulnerabilities open
Read Full Article →

These items are required to enable basic website functionality. These items are used to deliver advertising that is more relevant to you and your interests. These items help the website operator understand how its website performs, how visitors interact with the site, and whether there may be technical issues. These items allow the website to remember choices you make (such as your user name, language, or the region you are in) and provide enhanced, more personal features. We benchmarked Claude Fable 5 again on 200 real-world coding tasks for the Agent Security League, this time using the Cursor harness. It posted our best security score yet. 72.6% on functional solves and 29% on security solves,. but still leaves most vulnerabilities open. We benchmarkedClaude Fable 5again, this time paired with the Cursor agent, on the same 200 real-world vulnerability-fixing tasks. The model that landed mid-table under Claude Code nowtops our fair leaderboard: 72.6% FuncPass and 29% SecPass. The story here is not the model, it is the harness. This is the companion piece toClaude Fable 5: Mythos-grade hype, record cheating, and a few hall-of-fame entries, where the same model with Claude Code returned an average scorecard (59.8% FuncPass, 19.0% SecPass). Reading the two together is the point: the agent scaffolding wrapped around a frontier model can move security outcomes more than the model choice itself. Key takeaways Fable 5 arrived with high expectations: Anthropic positioned it as a generally available, safeguarded Mythos-class model built for long, complex work, with strong reported performance across software engineering, cybersecurity, and long-horizon tasks. Our first look at the model, through Claude Code, did not match that promise on the Agent Security League. It was not bad, but it was not a breakout either: 59.8% FuncPass and 19.0% SecPass after fair scoring. So we ran the same model again through a different harness: Cursor. The result changes the story, but it does not make the story cheerful. Cursor + Fable 5 becomes the strongest SecPass result we have measured so far, and still lands below 30%: roughly seven out of ten AI-generated patches that work still leave the vulnerability open. Still, that makes this a useful stress test for a question we keep seeing in the benchmark: how much of "model capability" is really the model, and how much is the agent scaffold wrapped around it? Our approach is described in detail in ourwhitepaper. Here is a short version to recall some key points. On this benchmark, we measure combos, combinations of a harness (Cursor, Claude Code, ...) and a frontier model (Fable 5, GPT-5.5, Gemini 3.5, ...), on coding tasks inside real, complex projects. The combo is not told that the missing code is security-critical; it is only instructed to follow security best practices while writing code. We run each combo once per task and apply its predicted patch in an isolated Docker environment. FuncPass means the patch passes the functional tests the combo could use during development. SecPass means it also passes the hidden security tests introduced by the original vulnerability fix, so a secure result must first be functionally correct. We also require the combo to solve the task using its own reasoning: recovering the known fix from git history, the web, or similar sources is treated as cheating. We made this explicit in the prompt after observing many cases of agents retrieving patches from git history or web search. On top of that, our anti-cheating pipeline runs post-hoc checks for suspicious behavior in the trajectory and for high similarity between the combo's patch and the known fix, with LLM adjudication for flagged cases. One additional wrinkle is that part of the dataset contains overly strict security tests: tests that expect implementation details that are almost impossible to guess independently, such as a complex exception string copied from the reference patch. Passing those tests can itself be a cheating signal, so we use them as traps to surface additional cheating strategies and feed new signals back into the anti-cheating pipeline. Confirmed cheating is removed, and overly strict / unfeasible instances are excluded from the denominator to produce thefairscores we use in our leaderboard. Memorization (also referred to as training recall) has become a major issue withmore advanced models(e.g., Opus 4.8, Composer 2.5, ...), and we saw it on many task instances in last week'sClaude Code with Fable 5experiment. Memorization is the subtle case: unlike git-history inspection or web lookups, which prompt instructions can largely suppress, the model may simply know the upstream fix from training data. In normal software engineering, that is not inherently wrong. Human developers also reuse what they have seen before. But this benchmark is designed to measure whether a combo can reason from the local codebase, not whether the model has already seen the answer. For that reason, w...

Share this article