Snyk Blog In this article Summary Introduction How we measure: the Golden Test benchmark The fixtures The scoring What we evaluated How does the Snyk Agent Fix benchmark compare to prior benchmarks? Results What the numbers mean Raw model size has plateaued on secure-and-functional fixing Security context, not a bigger model, is the lever The lift is largest where the model is weakest The result holds up to a second look Up next: Snyk VulnBench and vulnerability detection with coding agents Limitations What's next Appendix: methodology and aggregation Benchmarking Secure-and-Functional Remediation and How Snyk Agent Fix Lifts Frontier-Model Fix Rates by over 14% Written by Stephen Thoemmes August 18, 2026 0 mins read Summary We benchmarked how well leading models produce vulnerability fixes that are both secure and functional, across ~150 real vulnerable code samples in JavaScript, Java, and Python. We ran each model on its own and with Snyk Intelligence (the new agentic Agent Fix architecture). The headline findings: Out-of-the-box frontier models cluster at 72–75%. Gemini 3.1 Pro, Claude Sonnet 4.6, and Claude Opus 4.6 all land within a few points of each other. On secure-and-functional fixing, model choice barely moves the number. Snyk Intelligence moves the same models clear of that cluster. Opus 4.6 rises from 74.6% to 85.4% , a 10.8-point gain ( 14.48% more samples fixed). The variable is the security context, not the model. The lift is largest where the model is weakest. Opus alone fixes only 64% of Python samples; with Snyk Intelligence, that reaches 88%. Introduction Coding-agent benchmarks exist for unit-test generation, SWE-bench-style bug fixing, and code completion. There is no widely used public benchmark for the task a security team actually cares about: take code with a known vulnerability and produce a fix that removes the vulnerability and keeps the code working. Those are two independent bars, and clearing one while failing the other is a common, expensive failure mode. A fix that removes a SQL injection but changes what the query returns has not helped anyone. A fix that looks clean but leaves the injection in place is worse, because it reads like a solution. So we built a benchmark that scores both bars on every fix, and ran frontier models through it twice: on their own, and equipped with Snyk's security intelligence . The question we wanted answered in plain terms: how much does Snyk's security context change what a frontier model can fix, and where? The short version: out-of-the-box models have plateaued around 72–75%, and Snyk Intelligence is what moves them past it. The rest of this post is the data and the method behind it. How we measure: the Golden Test benchmark Most code benchmarks check whether the code runs or whether it resolves a bug report . Neither is enough for security remediation, which has to clear a security bar and a functional bar at once. Our evaluation set, the Golden Tests, is built to measure both. The design is inspired by SWE-bench , adapted for security. The fixtures The set is ~150 real vulnerable code samples: 50 in Python, 54 in JavaScript, and 39 in Java. Each sample is a piece of code with exactly one vulnerability, found by Snyk Code and confirmed by a human security expert, and chosen so it is fixable from the code in front of the model without missing external context. The scoring Each sample ships with two human-verified unit tests: A test that fails because the vulnerability is present, and A test that passes when the code's original functionality is preserved (for example, a helper that should echo 'hello world' still returns 'hello world' after the fix). To score a pass, the model must fix the code so that both tests pass, on the first try , without ever seeing either test . The model never sees the unit tests themselves, so a pass reflects a genuinely secure and functional fix rather than output fitted to a known test. Take a Python sample with a SQL injection: the code builds a query by concatenating user input. The security test sends a SQL injection payload and asserts that the database does not leak every row; the vulnerable code fails the test. The functional test sends an ordinary username and asserts the correct record comes back; the original code passes it. A fix counts only if the model's rewrite makes the security test pass while keeping the functional test green, on the first attempt, tests unseen. What we evaluated Six configurations: Snyk's previous, internal StarCoder-based Agent Fix model, three out-of-the-box frontier models (Gemini 3.1 Pro, Claude Sonnet 4.6, Claude Opus 4.6), and Sonnet 4.6 and Opus 4.6 each running with Snyk Intelligence under the new agentic Agent Fix architecture. "Snyk Intelligence" here refers to dynamic few-shot prompting: at a fix time, we inject the most relevant expert-written fixes for that specific weakness, drawn from Snyk's database of more than 35,000 vulnerabilities . (This builds on earlier work where the same idea improved the performance of off-the-shelf LLMs .) How does the Snyk Agent Fix benchmark compare to prior benchmarks? The Golden Test set sits in a lineage of work that has steadily raised the bar for evaluating AI on code. SWE-bench established the pattern of scoring models against hidden, real tests rather than self-reported plausibility. On the security side, Vul4J introduced reproducible vulnerabilities paired with proof-of-vulnerability tests and a functional regression suite, the closest precedent to our FAIL-to-PASS plus PASS-to-PASS design. More recent work, like BaxBench and SEC-bench , reinforces the premise we built around: code that is functionally correct is frequently still insecure, so a credible remediation benchmark has to score both properties at once. What distinguishes the Golden Test set is applying both gates together, on human-verified real samples, with the tests withheld from the model, across three production languages. Results The headline metric is the share of Golden Tests where the fix was both secure and functional. Configuration Functional & secure fix rate StarCoder (previous Agent Fix model) 72.4% Gemini 3.1 Pro 74.2% Claude Sonnet 4.6 72.4% Claude Opus 4.6 74.6% Claude Sonnet 4.6 + Snyk Intelligence 82.5% Claude Opus 4.6 + Snyk Intelligence 85.4% CHART 1: Functional & secure fix rate. The out-of-the-box models sit within a three-point band. Adding Snyk Intelligence opens an 8-to-11-point gap on the same model: Opus 4.6 goes from 74.6% to 85.4%. Breaking down the Opus comparison by language shows that the gain is not an averaging artifact. It holds in every language we tested, and it is the largest where the out-of-the-box model is weakest. CHART 2: Per-language lift Python is the clearest case: Opus alone fixes 64.0% of samples, and with Snyk Intelligence, that jumps to 88.0%. JavaScript and Java, where Opus already starts strong, each gain five to six points. What the numbers mean Raw model size has plateaued on secure-and-functional fixing The three out-of-the-box models span 72.4% to 74.6%, a 2.2-point spread across two vendors. If a bigger or newer model were the lever for this task, we would expect to see it here. We do not. The task is hard in a way that more general capability does not directly address: the model has to know what a secure fix for this weakness looks like, not just write plausible code. Security context, not a bigger model, is the lever The same Opus 4.6 gains 10.8 points (14.48% more samples) purely from the security examples injected at fix time. Because the approach is model-agnostic, every gain in the underlying frontier models compounds with that context rather than competing with it. The durable asset is the 35,000-plus expert fixes; the model is a component we can swap as the field moves, which is why production Agent Fix now pairs Snyk Intelligence with Claude Opus 4.7. The lift is largest where the model is weakest Opus alone fixed only 64% of Python samples, its worst language. With Snyk Intelligence, it reached 88%, its best gain of the three. The security context does not just raise the average; it lifts the floor. The result holds up to a second look The Opus-plus-Snyk number is an average across runs (84.6% and 86.0% on the two runs we aggregated), so the headline 85.4% reflects consistent performance across runs. Run-to-run variance on this set is roughly a point, which is worth keeping in mind when comparing configurations that are within a point or two of each other. Up next: Snyk VulnBench and vulnerability detection with coding agents Fixing a vulnerability presumes you found it. In June 2026, we published the Snyk VulnBench JS 1.0 paper , which aimed at benchmarking Snyk Code as a deterministic and fast SAST engine vs LLMs powered by coding agents (the Claude Code harness) to detect vulnerabilities in code to begin with, before they need remediation. Our Snyk VulnBench findings discovered that even frontier large language models such as Claude Opus 4.7 on its maximum reasoning level (aka max ), and even when powered by an advanced coding agent harness ( Claude Code itself), had faced challenges in and around repeatability and determinism. Some headline findings: In one case, the model and coding agent reported ~50% of findings that did not repeat in four out of the next five executions, producing a backlog of false positives and vulnerability fatigue for agentic developers and AI security engineers. In other cases, 13% of the coding agents reported that unmatched vulnerabilities appeared in all 5 executions, proving an even more confusing and cognitive load situation in terms of the security backlog of issues that could prove false positives. We invite you to investigate and explore through the Snyk VulnBench dataset, which has been made online and public for you to review at https://vulnbench.com/ Limitations Four limits a reader should weigh before leaning on these numbers: Set size: ~150 samples (50 Python