Jev and the Future of AI Pentesting: Why the Harness Matters More Than Ever

Security teams now use agents for a wide range of work, from finding vulnerabilities in code to investigating and triaging issues. Many of these tasks need complex reasoning and an understanding of how an application works, so the shift makes sense. We're also seeing more security use cases where an LLM may be more than the job needs.

Triage is a good example. We ask an LLM whether a finding is a false positive, give it the context it needs, extract its decision as JSON, and use that to trigger the next step in the workflow. A model built as a classifier, designed to make decisions instead of writing responses, could do the same job faster and for less money.

That is the idea behind Jev, the System One model from TypeSafe AI, and the company's recent $870M Series A has drawn a lot of attention to it. Security could gain more than most fields. We make decisions across thousands of lines of code, API endpoints, and findings, and cutting the cost and time of each one could change how we approach AI-powered security.

Security isn't always a yes-or-no question, though. Finding a vulnerability can depend on how different parts of an application interact, and deciding whether a finding is a false positive can depend on the application's business logic. A faster decision isn't necessarily a better one.

We wanted to know how far Jev could go in AI pentesting, so we tested it on vulnerability detection and false-positive filtering and looked at how application context changed its decisions. The results raised a broader question: how much of a model's effectiveness comes from the model, and how much from the harness around it?

This article covers our experiments, where Jev did well and where it didn't, and why pairing specialized models like Jev with the right context, tools, and orchestration could make AI pentesting much more efficient.

1. How Jev differs from an LLM

Jev makes decisions instead of generating text. An LLM writes an answer that we then have to interpret, while Jev classifies, scores, or picks between options directly. It was built as a classifier, so nobody has to coax a generative model into acting like one.

Jev can also attach a confidence score to each decision. In an AI pentesting workflow, that would let us handle high-confidence classifications automatically, send uncertain ones to a more powerful LLM, and escalate others for deeper investigation. A high score doesn't guarantee the answer is right, but it gives us one more signal to work with.

Then there's speed and, more importantly, cost. Cost has become a big topic in AI. Coinbase recently reported cutting its AI spending nearly in half while its token usage kept growing, by moving to cheaper open-weight models, routing between models more carefully, and caching better. Companies are learning that not every task needs the most powerful LLM. Even when they switch to smaller, cheaper models, though, they're still using LLMs. The bigger saving may come from asking whether a task needs an LLM at all.

That question matters in security, where budgets are limited and attack surfaces keep growing. When you're analyzing thousands of API endpoints, vulnerabilities, or scanner findings, the cost of each decision adds up. If those decisions got much cheaper and faster without losing accuracy, AI pentesting tools could work quite differently. Before picturing Jev at scale, we wanted to see how it handled actual security tasks.

2. Testing Jev on security tasks

Speed and low cost don't help if the security decisions are wrong, so we started with the most obvious use case: finding vulnerabilities in code.

Can Jev find vulnerabilities in code?

We gave Jev 20 code snippets, 10 with real vulnerabilities and 10 benign, covering XSS, SSRF, IDOR, and other types. For each one, we asked whether it was vulnerable.

We made the task fairly easy. Each snippet was about 20 lines long, picked so the relevant code was already isolated. Jev didn't have to explore a repository, trace functions, or work out where a vulnerability might be hiding, as a real security agent would. It only had to classify what we handed it.

Jev caught 9 of the 10 vulnerabilities, and inference for all 20 snippets cost less than $0.001 in total. That's impressive for the price. The benign snippets went worse: Jev flagged 5 of the 10 as vulnerable, a 50% false-positive rate on a task we had already simplified a lot. Spotting potentially vulnerable code is useful, but if half the safe code gets flagged, security teams end up with more to review.

This was a small experiment, not a benchmark of how well Jev finds vulnerabilities in real applications. Its design is worth a second look, though. By picking the relevant code in advance, we did part of the work a real AI pentesting system has to do itself. That made us wonder what would happen if we changed the information the model had.

What about false positives and business context?

Finding vulnerabilities is one problem. Working out which findings matter is another, and business context plays a big part in it.

Take an authorization vulnerability. An API endpoint can look vulnerable from its code alone when access is already restricted somewhere else in the application, or when the business rules explain why the behavior is expected. Without that information, even a powerful model could get the decision wrong.

So we tested whether more information about the application would change how Jev judged findings. We asked it to evaluate a set of known false-positive findings, first on their own and then with access to the application's business wiki. With the wiki, Jev recognized enough of them to cut the number of those findings left for review by 20.7%, at the classification threshold we tested. Knowing how the application was supposed to behave helped it spot findings that weren't real vulnerabilities.

A production security system couldn't safely dismiss 20.7% of its findings on this basis. We only tested known false positives, and we would still need to check that real vulnerabilities don't get filtered out with them. The result does suggest that better security decisions can come from giving a model the right information as well as from making the model smarter. That's worth keeping in mind when thinking about where smaller, specialized models fit in AI pentesting.

What we learned

Jev found most of the vulnerabilities in our simplified benchmark at a very low cost, and business context helped it recognize false positives. In both experiments, what surrounded the model mattered. In the first, we chose the relevant code before Jev saw it. In the second, we supplied the application's business logic.

This matters for AI pentesting because a model doesn't know on its own which code to inspect, which application behavior is expected, or how to check that a vulnerability can actually be exploited. Those capabilities have to come from somewhere, and in practice they come from the harness.

3. Why the harness matters as much as the model

Jev can make some security decisions very cheaply, and the right context improves its results. Testing an entire application is a much bigger problem than making individual decisions, and no model, however powerful, can realistically cover a whole attack surface by itself.

An AI pentesting agent has to deal with thousands of API endpoints, several authentication mechanisms, user roles, business logic, and vulnerabilities spread across the application. As it investigates, it piles up API responses, code snippets, earlier tests, failed attempts, and discovered vulnerabilities. Eventually that becomes a problem. Context windows are limited, processing all of it gets more expensive, and important details get lost along the way.

Delegating some of the work to subagents helps. If the main agent still has to keep track of everything happening across the application, though, the context management problem has only moved somewhere else. The challenge is letting agents investigate the whole attack surface while keeping the context and coverage they need to find real vulnerabilities.

What a harness does

A harness is everything around the model that lets it do a task well, and AI pentesting needs a lot beyond reasoning. Coverage comes first: the system has to explore the application's attack surface methodically, so that endpoints, features, and possible attack paths all get investigated. The harness also manages context, keeping track of what the assessment has collected and giving each model the relevant information when it needs it. Agents need security tooling to interact with the application, run tests, and gather evidence. Orchestration and validation keep investigations moving and make sure potential vulnerabilities get verified instead of only predicted.

Each of these affects how well an AI pentesting tool works. A well-designed harness lets models run at scale, cover an application's attack surface systematically, and turn what they can do into real, validated findings. That gets especially interesting with smaller, specialized models.

What does a strong harness mean for smaller models?

Comparisons of AI systems tend to focus on the model: which one reasons best, which scores highest on benchmarks, which is most powerful. A model's performance in isolation says little about how effective a security assessment will be.

Picture a frontier model with excellent reasoning but limited application context, poor attack surface coverage, and no reliable way to validate its findings. Then picture a smaller, specialized model inside a harness that supplies the right context, relevant security tools, and systematic coverage. For some security tasks, the second system could do better despite the less capable model.

We haven't benchmarked these two approaches against each other, and frontier models' reasoning is still essential for complex security investigations. What counts is how well the harness uses each model's capabilities.

This is where Jev fits. Our experiments showed it can make some security decisions at very low cost, and that application context improves its classifications. With a strong harness feeding it the right information, a model like Jev could take on high-volume security decisions while more powerful reasoning models handle complex investigations.

The benefit could go beyond inference cost. Cheaper decisions mean more of an application's attack surface can be tested on the same budget. They could also free up resources to explore more attack paths, run assessments more often, and spend more time validating complex vulnerabilities.

Those gains would still need to be measured across complete AI pentesting assessments. A cheap classification is only worth something if it helps find real vulnerabilities without hurting coverage or accuracy. For security teams, the numbers that count are how much of the application gets tested, how many real vulnerabilities are found, and what it costs to find and validate them.

Bringing it all together

Our experiments with Jev showed what specialized decision models could do for application security, especially with application context behind them. Harness engineering already makes a clear difference in AI pentesting today.

At Escape, this is how we approach AI pentesting with Cascade. It combines autonomous security agents, application context, deep reasoning, and security testing to explore applications and validate real vulnerabilities. Specialized models like Jev could make individual security decisions inside a system like this more efficient. The harness is what lets those models work at scale, keep coverage systematic, and turn their output into real security results.

Want to see how AI pentesting can improve your security coverage and help you find real vulnerabilities? Book a demo of Escape's AI Pentesting.