RunSybil CEO Talks AI Hacking, Model Distillation Attacks, and the Future of Security

Ari Herbert-Voss: All right. So, I am a man on a mission where we're trying to automate hacker intuition. I guess I can start with a brief intro to myself. I was the first security hire at OpenAI back in 2019. I was a grad student at Harvard doing my machine learning PhD, and I saw GPT-2 come out, and I was like, "Wow, this would have been really useful back when I was a miscreant teenager doing insane operations on the internet."

So I ended up bundling up a couple of demos of things that I would have made as a miscreant, and I sent them to Sam Altman and to Jack Clark, who was the head of comms at the time. And then the rest is history. They liked it enough that they invited me to come join, and I was there for three years. I was a core researcher on GPT-3 and on the Codex model. I also built our first monitoring system for when we started offering the API, to make sure that customers were following our terms of service.

I left the company in part because we just didn't have a good answer for when the bad guys have access to everything. The black-pill moment for me was when we were doing an anonymized review of model outputs. I saw somebody trying to lock a file system. That could be totally benign — educational, like how does encryption work? Or it could be somebody futzing around with malware, ransomware specifically, and there's really no way of telling that particular intentionality.

At the time, our thought was, well, what if we focus on the monitoring? We just block people that are doing bad stuff. But realistically, when you have something that's as explosive as language models have become, you're not going to be able to play whack-a-mole. You have to get offensive with it. So I started this company. I'm very blessed to work with my co-founder, Vlad Ionescu, who built the red team at Meta. And I have a team of really strong engineers that we've pulled from some of the top security engineering teams in the industry. We're focused on building something that will make the internet broadly more safe, and it's really rewarding to see it pay off.

TBPN host: Are you seeing more danger and risk from large-scale state actors, or the script kiddies who are just trying to wreak havoc, or is it both? Some of the new AI security threats come with new capabilities, but there's also a lot more cost than just running some PHP script that guesses WordPress passwords like back in the old days.

Ari Herbert-Voss: I'd say it's actually a combination of the two. You have two types of threats, and oftentimes it also depends on the type of organization that you are. For our customers that are smaller startups, they're not really seeing any of this kind of stuff. But for our larger enterprises, we've been asked by a lot of them if they want to replace their bug bounty with us.

TBPN host: Oh, they want you to win all of the bug bounties. That's great. Just don't make mistakes. That's your immediate TAM, and I'm sure much further. Talk to me about model distillation. There's been a lot of news about it. It's not as serious of a threat — it feels more like a business threat — but I've always been shocked by the stories about different open-source companies where it feels like they trained on an American lab, and that seems like something the lab should be able to detect. Is that hard? Is that something you can help with?

Ari Herbert-Voss: So, that's not what we do, but it is something that I've dealt with previously. It is something that everybody does and everybody knows about, as a bit of a dirty secret. But I'll also say that when you do distillation, the model that you get out of it is going to be net less good than the model that you're distilling from. That's just information theory 101. So it's somewhat of a business threat, but it's not as big of a business threat as, say, stealing the actual model itself.

TBPN host: Yeah. Just actually breaking into the system. Can you give us your pitch, from a startup or a scale-up on the customer side all the way up to a lab — how you sell the product right now? I understand the opportunity at a high level: all these companies are distributing intelligence, it's hard to understand how it's going to be used, a lot of people are going to use it for things they shouldn't, and you want to stop that. But what is the specific pitch in this moment in time?

Ari Herbert-Voss: For one thing, security means a bunch of different things to different people, and right now what we're seeing is that smaller teams need different things from larger teams. The benefit of the way that we've built our product is that if you have more attack surface, it's just much more interesting for the type of things that we can find for you. So we've been moving up into enterprises, and we have a lot of strong response from large enterprise teams. They have a lot of old code bases that go back 40 years. There's a lot of cursed things in their environments. Trying to get additional coding tools in is kind of tricky. And I actually have an interesting hot take for you, if you'll take it.

TBPN host: Please. Love it.

Ari Herbert-Voss: Okay. So obviously there's a lot of movement in security in terms of the markets, especially when Anthropic dropped some of their news about their vulnerability discovery work. I think a lot of people are concerned about whether or not language model labs are just going to solve security. What I think is interesting is: if code gets so much better in terms of security, the main question is, does that mean that hacking gets harder? And my answer is no, because I think that speed is what's going to kill us.

The space of these large possible attack vectors requires a lot more data than simply the code. If you look at the code, I like to think about it as looking at the bones of a dinosaur. You're going to find a lot of interesting things about structure, but you're going to miss a ton about things like muscles, and whether or not they have feathers, and also behavior and broader things like that, which are also very important for understanding the ecosystem. And that's true of code as well, and it's true of computers. It's easier to find bugs with the code. Having the bones is very helpful for us, even knowing that these dinosaurs exist. But you miss so much other stuff, and that is where the real delta lies here.

Authentication, for example, is famously difficult to suss out. I don't think there are any good authentication scanners out there. But the way in which we've built our product, it's very good at finding auth bugs. In fact, one of our strengths that we've heard continuously is that we're very good at finding these weird, esoteric things that have existed in bug bounties for the last 10 years, and we get pretty nice payouts from that, which is always fun.

TBPN host: That's really cool. Talk to me about your take on the forward-deployed engineer pattern — model, trend, boom, whatever you want to call it. Are you in favor of that? Are you employing that at this time?

Ari Herbert-Voss: That's a good question. I think forward-deployed is important if you're working with enterprise, because there's a lot of a human factor. In startups, what we learned is that people just want to solve the problem. The CTO comes to us and says, "Hey, I have this deal blocked by SOC 2. I really need to get a pentest." And we're like, "On it, got you," and they're on their merry way.

But with these enterprise companies, it's a lot more of a political process. Security in general is partly the people, and of course the software too. What you can do with a forward-deployed engineer is provide more of that layer of trust. You can communicate a lot more with the proper stakeholders so they don't feel like their jobs are going to be taken away. There's a lot more of the human factor that you're able to introduce, and I think that's why it's so popular. We do something somewhat similar, and we find it to be particularly helpful with bringing people on board and being able to serve them faster.

TBPN host: I'm looking at some of the customers here: Cursor, Turbopuffer, Notion, Baseten, Thinking Machines. Congratulations. The business sounds great and makes a ton of sense, and obviously you're raising money. But I'm curious if there's almost a direct-to-consumer play at some point, because everyone is going to be vibe coding. We are a 10-person team, we have three systems, and we're sort of security second. Maybe we're working on that.

Ari Herbert-Voss: I have a black pill for you, then.

TBPN host: Tell me.

Ari Herbert-Voss: I don't think so, because people don't like paying for security.

TBPN host: Okay. Isn't there another way that you can make it so cheap, or bake it in, or partner with a lab, where I'm vibe coding something and I get RunSybil installed, or it's built into the system? The SOC 2 example is relevant, because that's somebody saying, "I have to do this because I'm selling software. It's hurting my revenue." But we've heard so many things about somebody using OpenClaw, vibe coding something, running their business on it, and increasingly it's turning into a system, and at some point they need to think about security. We bought hundreds of thousands of dollars of camera equipment before we bought cameras to secure the office, right?

Ari Herbert-Voss: I think there's some truth to that. But let's also think about the economics of who buys these tools. If you're paying 20 bucks or 200 bucks for a month-long subscription, how much are you going to pay for additional security on top of that? That's something you'd have to sell direct to that company, and there aren't a lot of companies that really offer code-security-related stuff there. For companies making the bet in that space, they're focused a bit more on that ecosystem. We're focusing more on the overall ecosystem within an enterprise that has a ton of ancient code that is going to require a lot more in order to fix, and they also have these enormous attack surfaces that need some help.

TBPN host: Yeah, that makes a ton of sense. Well, how much did you actually raise? Tell me about the funding round. We want to ring the gong.

Ari Herbert-Voss: Oh yeah. So we raised 40 million, which we love.

TBPN host: Congratulations.

Ari Herbert-Voss: Khosla led the round. We had participation from S32, Conviction, and Elad Gil. We also had a bunch of angels as well, including [unclear], who I know has been on the show, Jeff Dean, and Ian Goodfellow too.

TBPN host: Wow. That's incredible. Congratulations. Thank you so much for taking the time to come break it down for us. [unclear] Thank you for securing the American software ecosystem. We appreciate that as well. Get every bug bounty that is out there. You deserve it. We'll talk. Great to meet you. Very, very bullish. Cheers.

Ari Herbert-Voss: Goodbye.