Sybil Finds More Real Vulnerabilities, With Less Noise, Than Frontier Models

Table of contents

In a head-to-head benchmark, Sybil found 28% more vulnerabilities than the best frontier models and reported roughly a third fewer false positives.

We built a test set of ~70 zero-day and n-day vulnerabilities across three large, production-grade open-source web applications. We then tested Sybil against models including Anthropic’s Mythos and OpenAI’s Daybreak Red, each running in a general-purpose coding agent such as Claude Code or Codex. We measured two things: how many real vulnerabilities each found, and how many false positives each reported.

That accuracy gap matters more with every model release. As models improve, producing a long list of potential bugs gets cheaper every month, but that isn’t what security teams are short on. What they need is to get them triaged, find out which findings carry real business impact, and get them fixed. And because developers never stop shipping, teams need to keep watching for new gaps in their attack surface. 

In this blog, we'll explain why this matters with a quick look under Sybil’s hood, then dig into the experiment our research team conducted.

Discovery is Cheap. Triage Isn’t.

As frontier models make discovery cheaper, the number of reported issues grows faster than any team can verify and fix them. 2026 already has 24,568 more findings than all of 2025 (72,812 vs. 48,244).

Sorting through that deluge costs time and money, and the gaps that matter stay open longer.

Figure 1: The number of publicly disclosed vulnerabilities grows every year, and is on track to multiply as shown by the count of CVEs assigned
https://www.cve.org/about/Metrics

For security teams with access to unguardrailed models, money to burn on tokens, and time to spend poking agents along, discovery is no longer the bottleneck but triage and remediation is. Each candidate finding still has to be reproduced against the running application, deduplicated, rated for business impact, and routed to an engineer who can fix it. A tool that reports 50 candidate issues has created 50 triage tasks. So it matters how many real, exploitable vulnerabilities a system confirms, and how much noise it hands the team along the way.

A model can’t do this downstream work on its own. It needs a harness, the agent loop that lets it run tools, send requests, and decide what to try next. It also needs a system around the harness that validates, deduplicates, and prioritizes what it finds. Sybil provides both, and it uses whichever model performs best for each stage of a test. This experiment shows how much difference a well-built harness and system make. (We cover what goes into that system in Build, Buy, or Both: How to Choose Your AI Pentesting Path.)

The Experiment

One of the things our research team evaluates is recall and precision. We look at how many real vulnerabilities Sybil finds and how much noise it produces. 

// The question we asked

Our evaluation asked a specific question: does a purpose-built testing system find more real vulnerabilities than frontier models running in general-purpose harnesses, even when those models have more access to the target?

Targets and Ground Truth

We tested against three widely used large, production-grade open-source web applications, focusing on zero-day or n-day issues. The ground truth set is a list of 69 real vulnerabilities each tool was scored against. Of those, 38 were publicly disclosed in the last six months, and 31 are not yet patched, so we are not naming the applications or publishing the dataset until the 30-day disclosure period has ended. 

Configurations

We compared Sybil with five frontier model and harness combinations.  In Codex, we tested two versions of Daybreak Red: GPT-5.6-Cyber and GPT-6-Cyber. In Claude Code, we tested Opus 5.0, Opus 5.5, and Mythos 5.2. In this evaluation, Sybil ran on Opus 4.8 xhigh.

This experiment measured vulnerabilities found, as well as precision.

Results

+28%Known vulns found

Sybil found 28% more of the known vulnerabilities than the two strongest coding agent configurations.

Claude Code with Opus 5.0, and Codex with Daybreak Red.

‍

Figure 2: Sybil found 28% more known vulnerabilities than the best frontier model configurations
82%Precision

Sybil also had higher precision than every frontier model configuration.

82% of the issues it reported were true positives, compared with 49% for Claude Code with Opus 5.5, and 23% for Codex with Daybreak Red, which tied for the most vulnerabilities found among the five model configurations 

Figure 3: Sybil reported 33% fewer false positives than the best frontier model configurations

// TakeawayNewer models did not reliably find more.

Claude Code with Opus 5.5 found fewer vulnerabilities than the same harness running the older Opus 5.0, though it was more precise. 

This is why Sybil is model-agnostic. It uses whichever model performs best for each stage of a test. Our research team evaluates each new model as soon as it’s released to decide where it fits. 

A model upgrade is not automatically an improvement, and the only way to know that is to measure it!

Measure What Reaches Your Team

The speed at which vulnerabilities are being discovered is increasing, but reliability isn’t improving as quickly when using models in generic harnesses.

In our benchmark, Sybil found 28% more of the known vulnerabilities than the best frontier model configurations, with 33% fewer false positives. For this experiment, Sybil also ran on Opus 4.8, an older model than most of the models it outperformed. The difference comes from the harness and system Sybil is built with.

For security teams, that difference shows up after discovery: in how much time goes to triaging noise, and how quickly real vulnerabilities get fixed. Sybil gives security teams measurable coverage and faster remediation by automating the intuition of an experienced hacker, continuously.

Want to see how Sybil performs on your own applications? Request a demo

Subscribe to RunSybil Blog

By clicking Sign Up you're confirming that you agree with our Terms and Conditions.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.