Testing Jev on Security Triage and Classification

In order to find out if TypeSafe AI's new classification engine Jev would be better than the frontier models in our own agent, we evaluated it against heavy hitters, including Claude Opus 5.5, Kimi K3, and GPT-6 Astra. We found the new model was 8x to 120x faster at 1% to 3% of the cost. On accuracy, it came close to or beat them, depending on the task.

It wasn't a clean sweep though.

Out of the box, Jev was far more permissive than any of the frontier LLMs, blocking 71% of invalid findings compared with 91% to 96% for the LLMs. On validated platform data, it got only 10% of OWASP labels exactly right.

In this blog, we’ll give you a high-level introduction to what makes Jev different, and talk about what it might mean for decision making, accuracy, and speed in Sybil. Then we’ll walk through the four experiments where we thought Jev could improve Sybil’s agents that filter out speculative and low-impact findings, deduplicate findings, and assign OWASP categories.

Jev in 60 Seconds: Why Jev Isn't an LLM

TypeSafe AI, a San Francisco startup, released Jev in September. Jev is touted as a System 1 model, a reference to how psychologists differentiate fast, instinctive decision-making (System 1) and slow, deliberate reasoning (System 2) in the human mind.

Unlike generative LLMs, Jev doesn't generate text. You send it up to 64,000 tokens of context along with a set of typed questions, and it returns an answer to each one in a single pass:

• Noul: Asks a yes/no question and returns a probability score between 0 (no) and 1 (yes), letting you set the decision threshold.
• Choice: Selects a single option from a predefined list and returns the selection, confidence level, and probability distribution across all options.
• Score: Rates content against ordered, descriptive levels, such as ranking severity or quality tiers.

TypeSafe's pitch is architectural. Questions run in parallel and independently, so adding more questions barely changes response time. That lets an agent hand off its classification, routing, and guardrail calls to Jev and save expensive LLM calls for work that needs long-form reasoning or text generation.

We put TypeSafe’s claims that Jev has lower latency and cost without sacrificing quality to the test with Sybil's agents.

When Sybil finds a potential vulnerability during testing, its agents handle the finding through a series of steps, which require an agent to make a decision. First up is an experiment that tests how Jev handles filtering out false positive or low impact vulnerabilities.

Experiment 1: Filtering Speculative and Low Impact Findings

Bottom Line

After tuning, Jev was the best-balanced filter we tested: It persisted 1.3x-2.7x more impactful findings than untuned LLMs while still blocking nearly as many invalid ones. It did that 8x to 18x faster and at less than 3% of the cost.

Setup

We evaluated Jev, both out of the box and tuned, against untuned versions of Claude Opus 5.5 xhigh, Kimi K3, and GPT-6 Astra xhigh on a validation task in Sybil’s pipeline. We evaluated each model's ability to classify candidate findings across a dataset of 117 valid findings and 45 speculative or low impact findings. All models shared the same initial prompt.

Models were scored on two key metrics for this step of our pipeline: True Positive Rate, or TPR, for approving valid findings, and True Negative Rate, or TNR, for blocking speculative or low impact ones. These metrics reflect a single classification task and do not reflect Sybil’s end-to-end performance.

We configured Jev to use a set of Noul questions that mirror those posed in our LLM prompts. For a finding to make it through our filter, it had to pass each Noul question with a probability >0.5. We used TypeSafe’s autoresearch loop to tune Jev, turning the set of Noul questions into structured question-score pairs to feed a CatBoost classifier. Below are Jev’s post-autoresearch results, evaluated on an independent holdout dataset to prevent overfitting.

Results

Highlighted blue cells mark the top-performing model for each metric.
‍

Jev out of the box was cheap, fast, and highly permissive. This made it a competitive alternative to existing LLM-based approaches. After tuning, Jev became even more effective at identifying genuine findings while still retaining significant cost and latency savings. Tuning also changed why Jev rejected findings: roughly 22% of rejected findings were dropped due to nuanced, multi-factor signals captured by the learned classifier, rather than a single, strict Noul probability cutoff.

Jev as a Dial

If avoiding speculative and low impact findings is critical, rival models perform better at filtering out invalid findings. For users who cannot afford to miss valid findings, Jev proved best at avoiding aggressive over-filtering.

Previously, finding the right balance between how many valid findings and speculative findings an LLM shows required iterative tuning of the LLM. Jev offers a dial, where LLMs that directly assign categories don't. Uncalibrated, Jev returns a probability for every answer, while a tuned Jev provides a single score. That means we can move the reporting threshold and predict how many extra speculative findings we'd admit in this stage for every additional valid finding we surface.

Below is an ROC curve diagram that shows how adjusting Jev’s classification threshold impacts system performance. It compares tuned and baseline Jev on how many valid findings pass through versus how many speculative findings get filtered out. Positions in the diagram closer to the top left are ideal; they represent all TPs surviving and all unsupported and low impact findings being filtered out. Within this diagram, dots show the settled threshold values. From this, we can quickly understand the tradeoffs we’re making as we adjust our thresholds. This enables us to more quickly and dynamically balance noise tolerance against the need to catch every real finding. The diagonal dashed line shows how randomly guessing valid and speculative findings performs.

Experiment 2: Jev as a Screener for Root Cause De-duplication

Bottom Line

More accurate, 44x faster, at about 1.5% the cost: In replays from our evaluation environment, 50.6% of Jev's judged candidates were confirmed matches, vs. 44.8% for Sybil’s current screener. Where the two disagreed: Jev's picks were about 10 times more likely to be real.

Setup

Sybil tests an application and reports what it finds. However, the same underlying flaw can surface multiple times, such as:

• A single missing authorization check reached across three different endpoints.
• The same injection point appeared again on a subsequent test run.

Merge deduplication is the pass that catches that. It evaluates whether two findings share a single root cause match, so one fix resolves both. When a match occurs, the new finding merges into the existing one instead of creating a new report.

A cheap LLM screener shortlists the existing findings that are worth a closer look.

For this experiment, we replayed thousands of past screening decisions across tens of thousands of findings from our evaluation environment, comparing Jev's shortlists against those produced by our current screener. We measured precision by how often a screener agrees with a judge on whether two findings share the same root cause.

Results

Highlighted blue cells mark the top-performing model for each metric.
‍

Jev's shortlist was more precise overall. The difference is sharpest where the two screeners disagreed. Of the judged pairs flagged exclusively by the current screener, only 3.9% were actual root-cause matches. Of the pairs flagged exclusively by Jev, 38.5% were valid matches. Most of what Jev left out was noise, and more than a third of what it added were true positive findings. It was also far faster. At the p95, a screening decision dropped from nearly five minutes to four seconds.

Experiment 3: OWASP Classification on Synthetic Findings

Bottom Line

Jev achieves near-frontier accuracy out of the box with default LLM prompts: Jev classified 98.6% of synthetic findings correctly while Kimi K3 and GPT-5.6 Sol achieved 100%. Jev finished the batch 8x to 48x faster.

Setup

As part of our product, we map Sybil’s actions to the OWASP Top 10 and determine whether each test was completed successfully and revealed a vulnerability. Using historical trace data as a seed, we generated 350 synthetic findings with Claude Opus 5.0. We then ran Jev, Kimi K3, and GPT-5.6 Sol medium across this dataset using identical prompts to compare performance.

Results

Highlighted blue cells mark the top-performing model for each metric.

All three models completed with no failures or retries. Jev ran the full batch about 8x faster than GPT-5.6 Sol and about 48x faster than Kimi K3.
‍

Experiment 4: Evolving OWASP Prompt on Real Data

Bottom Line

Cheap iteration fixed a weak start: Out of the box, Jev labeled only 10% of our dev-set findings exactly right. A handful of threshold and prompt changes took that to 95%, and to 10 out of 10 on a small holdout.

Setup

We also tried Jev on OWASP classification (mirroring the same process as experiment 3) using a snapshot of real platform data. We split the data into a 20-sample development set for prompt iteration and a 10-sample holdout set to check final performance on unseen data.

Results

Out of the box, Jev really leaned toward labeling everything “Insecure Design,” which is technically not wrong for many findings, but it wasn't the answer we wanted either. Nearly any vulnerability can be traced back to “Insecure Design,” so using that category for everything doesn't tell a developer what actually went wrong or how to fix it.

Using prompt optimization techniques, we iterated through multiple tests and rapidly increased accuracy over the course of an hour.

Scoring rubric: First value is percentage of time it exactly matched, while second value is partial credit percentage when close (micro-F1).
‍

Our Noul variants ask a yes/no question per OWASP category, and the “@” value is the probability threshold for counting a yes. “Choice” asks a single question that picks one category from the set. Raising the baseline threshold from 0.5 to 0.9 already helped as confidence plays an important role. Using a mix of confidence and prompt tuning, the evolved v3 prompts at a 0.6 threshold did the best on both the original set used for tuning and on the holdout.

What's Next

What These Experiments Mean for Sybil

We found that while untuned Jev tends to be less performant than frontier LLMs, easy-to-tune Jev-like classification models are competitive with frontier LLMs across different stages of Sybil’s pipeline. We're working on bringing Jev out of testing and into Sybil's live pipeline where it makes sense.

Here's what that will mean for you:

• Accelerated triage speed: Executes classification steps in a fraction of the time, getting findings into your hands even faster.
• Enhanced finding precision: Further sharpens deduplication and finding verification, surfacing real vulnerabilities with even greater consistency.
• Granular threshold control: Pairs every evaluation with an explicit confidence score, giving you direct control over filtering thresholds to optimize for strictness or coverage.

This reflects a broader principle behind Sybil: use the right model for the job. Fast classifiers answer the closed questions; frontier LLMs take on the work that demands deep reasoning.

We continue to evaluate new technologies for the benefit of Sybil and the security community as a whole. Check out more of our research.