Black Hat Asia 2026 Keynote: From Prompt Tricks to Autonomous Hackers

Black Hat host: Good morning, everyone, and welcome to day two of Black Hat Asia. It's fantastic to see so many of you here again, and a very warm welcome to those joining us today. One of the things I love most about Black Hat is that it spans the full career journey. In fact, this morning's keynote speaker, Ari Herbert-Voss, first came to Black Hat as a student attendee, and today he's back on stage to share his work with all of you. That progression from learning to contributing to leading is exactly what this community is about.

Black Hat exists to share what's really happening in security and to make sure that knowledge is challenged, built on, and passed forward. As part of that, we're continuing to expand the community globally, and later this year we'll be launching Black Hat India in Bengaluru, from the 27th to the 30th of October.

This week, we set the context. We looked at a landscape being reshaped by AI, expanding attack surfaces and, increasingly, geopolitics. Today you'll see that play out in more depth, particularly as we explore how offensive capabilities are evolving. Across the program, that comes through in the research being shared and the techniques being tested in our Briefings, from emerging vulnerabilities and AI security to real-world attack methods coming directly from the community. Beyond the Briefings, there are plenty of ways to engage: the stages to continue learning and connecting, the Arsenal for hands-on access to tools and techniques, and the lounges — spaces to play and experiment, but also, critically, to connect. So I'd encourage you to explore, ask questions, and make the most of being here. With that, it's my pleasure to welcome the founder of Black Hat and president of DEF CON, Jeff Moss.

Jeff Moss: Hey, good morning, everyone. Thank you for coming to our second day of the 22nd Black Hat Asia. Just a couple of comments before we introduce our day-two keynote, focused on the subject of AI. There's so much misconception, so many hopes and dreams wrapped up in AI, and I've been thinking about how quickly it's moved — the comments I would have made a year or two years ago on AI, and how they might have changed today.

One of the observations is that great power competition hasn't changed. It always takes different forms, and there's a bunch of great power competition going on around AI. It's basically falling into China, the US and Europe for a little bit, but primarily I would say the US and China, with the energy, the compute, the mathematicians, the scientists and the national giant hyperscalers all racing. One thing I tell people is that different nations want different things. They have different ambitions, they're on different trajectories, with different budgets, and so their national interests sometimes might look irrational to you, but to them it's rational.

I'll give you a little glimpse of this. Years ago, when I was at ICANN, China really wanted a root name server of their own — a 14th root name server. And we said, well, it doesn't really work that way. They really wanted it, and they'd wanted it for a long time. So to show us how ready they were to have a root server, they were going to fly in a hundred of the leading DNS security experts from around the world. They had a hundred-million-dollar building built to hold the root server, with a thousand engineers ready to operate it. That is a huge commitment. And all I'm thinking is, our root server is like a 1U server in a rack — here's the root server. They were allocating hundreds of millions of dollars to this for national pride, and the root was working fine in China. Think about how much more scaled up that is now in the era of AI, especially now that there are starting to look like there are some actual economic benefits.

One final thing on great power competition: we see a lot of theft of model data, training data, poisoning of training data, and weights. It could be from competitors, it could be from nation states. So all the normal things that you might see in a military or telecom setting are happening in AI. You might be heads down and not think that what you're doing is of any interest — maybe not to you, maybe not to your government, but maybe to another government or another competitor. Compared to two years ago, the intensity has ratcheted up, and so has the spend.

Also, if great power competition sometimes leads to irrational-looking behavior, greenfield opportunities also sometimes lead to irrational-looking behavior. The spend on AI data centers doesn't make much sense, but the greenfield opportunity in the eyes of some business leaders is unbelievable. If AI is the new electricity, of course you would spend unlimited amounts of other people's money to try to capture electricity in a bottle. The opportunity is perceived as unbounded. Look at how much money search has made, and how search disintermediated us from the original source — now imagine how much AI summarization will disintermediate us even further. On the way over, I was reading an article that said organic search traffic from an end user to a site is now down about 90%. End users never get to the source site anymore. They get the summarization and leave, and it's destroying ad revenue.

Rational behavior, on the other hand, looks something like what happened with extreme ultraviolet chip manufacturing. It took decades to master extreme ultraviolet lithography, and there were three companies doing it, all spending unlimited money — a Japanese company, an American company and a Dutch company. In the end, the three companies came together and said, "We're all spending money trying to do the same thing." They became a consortium, and that's how the machine was finalized. That seems rational, but we're not in a rational mode right now.

If I look at the trajectory of what's worked and what hasn't: how quickly was it obvious to you that the World Wide Web was going to be a winner? You saw a web page in a browser with an image and you were like, "This is it." Within months there were guides online to every website. Faxes pretty quickly were a winner. Email, winner. Cloud caught on really quickly — we were making fun of it, and then it ruled the world. Web3, blockchain, NFTs, the metaverse, VR — they sounded interesting but didn't catch on. AI picture generation of cats — okay, kind of interesting. But now agents seem to be the thing that's taking off. My prediction is that agents might be the thing that actually launches AI efficiencies and unlocks new behaviors beyond rendering pictures or predictive autocomplete. Agents look like where the money will be, and that's why I think it's accelerating so quickly.

At DEF CON 3, Curtis Karnow, who became a superior court judge in California, gave a talk on agents in the telco context. That's 31 years ago. He asked: what's the liability if agents on a telecommunications network start trading or shopping for their owners and operators, and they make a mistake? Where does the liability lie? So none of this is new. Lawyers and judges have been thinking about this for 30 years. The difference is that in the last six months, it's become possible. So you're going to see a huge amount of thinking from the last three decades unearthed, and now people can actually try to apply it and see what happens.

That's why I'm super fascinated and interested in what Ari is going to talk about in his keynote today. Ari was a student awardee, then a speaker multiple times at Black Hat, and now he's a Black Hat keynote. He came to Beijing when we did DEF CON China in 2018 and 2019 with the AI Village, and he came to AI before it was called AI — maybe it was machine learning or expert systems, but more from a neuroscience and mathematics perspective — almost 10 years ago, in 2016, and has been in the field ever since. Now he runs his own company, focused on offense. So, super fascinating. Glad you're here for the keynote, and it's my pleasure to welcome Ari Herbert-Voss.

[Music]

Ari Herbert-Voss: Hello. I'm very excited to be here today. It's my extreme pleasure to talk about this very important topic, because, as I'm sure you're aware, over the past couple of weeks you've probably seen a lot of headlines about AI finding vulnerabilities and writing exploits. A lot of this is real, and some of it is also aspirational. So this talk is about separating those two and getting down to what it means for our industry.

By way of introduction, I happen to have been the first security hire and a core contributor to GPT-3 and to Codex back at OpenAI. Currently I run a company where we automate offensive security at scale, for everyone from hot Silicon Valley companies to Fortune 1000 enterprises and financial institutions.

In this talk, I want to get four things across. What are the AI fundamentals behind the capability jumps that we're seeing in the news? What things are getting better with scale? What hasn't gotten better with scale? And what does this mean for security? I'm also going to reserve some time at the end for live Q&A.

So let's get into the fundamentals. LLMs are basically deep learning models that were originally designed to generate text. Many of the AI models we've seen applied to security in the past were trained for what we call discriminative problems — malware classification is a classic example. LLMs are different because they're generating rather than classifying data.

A good model for thinking about LLMs is as an adaptive lookup table, or a compressed representation of training data. The lookup occurs when you ask the model to predict the next word in a sequence, much like your smartphone keyboard. When we say words, we really mean tokens. Tokens are integers that represent parts of words, and LLMs are just fancy lookup tables for tokens. During training is when we build that table. We create dynamic key-value pairs for each input token by generating key, query and value vectors for each position in the input token sequence. These attention weights represent how much focus or attention the model should place on each element in the sequence when processing the current element. When you hear about weights in conjunction with models, this is what people are referring to. Weights, or parameters, can be thought of as a compressed representation of the input data that we sample from in order to complete the input sequence.

So how do you get from building a lookup table to solving math equations and the other advanced reasoning capabilities we're seeing in the news? You might be thinking, "Ari, sure, there's probably more going on than simply predicting the next token in a sequence." But no, really, that's all it is. Broadly speaking, we call this few-shot learning. It's called few-shot because you don't need a lot of examples. You start with a base model.

Here's an example of few-shot learning in terms of how humans think about things. Throughout our lifetime, we see enough data to learn how to read and make connections between concepts. So somebody can ask you to perform a calculation — in this example, translating hex to decimal — and, using what you know about the meaning of "translate," "hex" and "decimal," you can infer that you need to transform one sequence of characters into another. I didn't need to tell you mathematically which operations to do, because you've already had enough training information to do that. As hackers, we've seen enough examples of hexadecimal, and maybe read enough Hitchhiker's Guide to the Galaxy, to know that in this example 2A is also 42 in decimal.

One of the most important things happening right now is what we call the scaling hypothesis. More data, plus more compute, plus more parameters, gives you better performance across a variety of tasks, and this has held surprisingly well over the last seven-plus years. What has happened recently is that it turns out reasoning capabilities are scaling what we call superlinearly rather than linearly. In this graph, you see linear goes like this, and superlinear goes like this. This means that when you train a model that is twice as big, for twice as long, on twice as much data, you can get a model that is four times as capable. This is the difference between the latest generation of models from Anthropic and OpenAI — Mythos, and GPT-5.5, which just came out literally today — and the previous generation, such as Opus and the previous GPT models.

Additionally, as models get better at writing software, they get better at breaking it too. Reasoning capabilities are a proxy for vulnerability finding, because this is a task that requires pulling in different sources of information and synthesizing them in a new way in order to accomplish a goal. So it moves beyond basic pattern matching.

Let's talk about what is getting better with scale. The capability ceiling is rising, which means there's less need for complex scaffolding to get the models to do what we want. Opus 4.6 had near-zero exploit success with the same setup that Anthropic used for experimenting with Mythos in the press release and research they released a couple of weeks ago. Simpler agentic loops can go further than ever before, which means the model itself can now handle a lot more of the process than previously.

The speed at which vulnerabilities are being discovered is also increasing. We have this handy graph, courtesy of the folks who put together the Zero Day Clock. Most notably, at the very right-hand side, we see that between 2023 and 2026 the average time to exploit has dropped from five months to 10 hours. Early reports from Project Glasswing partners indicate meaningful impact as well. Palo Alto Networks said they accomplished a year's worth of pentesting in less than three weeks. And if you run a bug bounty program, chances are you've felt this change in the volume of reports you're seeing, as bug bounty hunters incorporate models into their bug-finding practices.

Let's also talk about what hasn't gotten better with scale. The capability floor isn't rising as quickly. What this means is that improvements are nonlinear across different classes of vulnerabilities. Take the OSS-Fuzz evaluation in the Anthropic report. They have a tiering system for how they weight the different vulnerabilities they find. For prior models, there were about 200 to 300 tier-one and tier-two crashes and about one tier three, with nothing at tiers four or five. For Mythos, there are almost 400 tier-one and tier-two crashes, which is a huge jump, a handful of tier threes and fours, and 10 tier fives from control-flow hijacks. This indicates massive gains at low severity, around shallow bugs; modest gains at mid-tier bugs; and relatively sparse gains at deep exploitation. That's not to say this isn't worth writing home about, because it's very important. But it does provide some evidence that these gains are nonlinear and that the floor isn't rising as quickly as the ceiling.

Models also continue to struggle with certain bug classes that require holding state. This has improved dramatically as context window sizes have increased, but again it points to this nonlinear improvement. These are bugs that usually involve some amount of time or other types of state. Concurrency is also a pretty big one.

I also mentioned bug bounty earlier as a bit of subtle foreshadowing, because the volume of slop reports has also meaningfully increased. That's improving, but not as fast as the ceiling of what is possible to find. Improvements are also nonlinear across exploit steps. Validation, exploitability and reliability all improve, but much more slowly. So for now, meaningfully assessing impact still requires more than simply asking a model to tell you what it thinks is the highest impact.

The net result is that better models find more bugs, but that also means more outputs for you to sift through. Models do not guarantee that those findings are going to be worth your time. What this means for attackers relying on LLMs is that reliability and targeting are still a bit of a game of chance. Individual attackers need to get lucky when they rely on models to find exploits, and many iterations are required if you want specific impacts on specific targets. Anthropic's experiments with Mythos still boil down to 198 human-reviewed findings that sit behind a much larger pool of automated outputs. The model generates volume, but for end-to-end bug finding, humans are still doing a large amount of filtering, validation and impact determination. Defenders are unfortunately going to get hit by millions of monkeys with typewriters, and some of those monkeys will write very good exploits. Defenders are going to have to win every time, whereas attackers only have to get lucky once.

So what does this mean for security? It is possible to get similar performance with enough scaffolding on open-source models, which is very interesting. It also indicates that expert guidance is still quite important here, and this has been true since 2023. The amount of expert scaffolding required is decreasing over time, but it's still possible to get Mythos-level results with enough effort and enough expertise. Most bugs that cause serious damage are relatively simple and don't require Mythos-level reasoning, which means we're in a very interesting period where the bugs being found are a bit more shallow, but we're finding them at much higher volumes. And open source is enabling just about anybody to start using models to find vulnerabilities.

We can think about fuzzing as a mirror for the future. When fuzzing first entered the scene, the early consensus was that maybe we'd solved bug finding. But the hard part wasn't generating findings. It ended up being understanding which bugs were exploitable and whether they could have impact in real-world environments. LLMs are following that same trajectory.

There are also some implications for defense in depth. Running multiple models on the same target leads to a surprisingly low overlap in findings. In the near future, it's unlikely that one model will find everything, because when you combine the different experience across different types of models, you can usually find a lot more and get higher coverage. There is currently no single best model that gives full coverage, and the best results still come from layering models and approaches.

In closing: the pace of vulnerability discovery is going to get faster. The accessibility of open-source and frontier models is going to continue to increase. There are extreme economic pressures in the AI industry to broaden access to these capabilities, and that is true for both good and bad use cases. This is probably the best thing that could have happened to our industry, because it's the forcing function we need to do the things we probably should have already been doing. It's now extremely easy to find high volumes of basic bugs, which happen to be the types that also cause the biggest headaches.

I'm personally very excited, because the ceiling is now higher on the interesting and complex vulnerabilities that LLMs can find. Multi-chain vulnerabilities that previously took months to build now appear literally overnight, on an industrial scale. The floor is not rising as quickly, which means there's still a lot of work that will go into prompting, validation and proper context to get maximally informative results. The hard work of prioritizing and patching will be even more important in the coming months. Offensive capabilities will continue to scale, but the ergonomics around how you build with them are going to be core to deploying them effectively. There's still a lot to build, and our work is cut out for us. So don't let the initial panic over Mythos or any of the other models coming out distract you, because we're just getting started. Now that we've set the stage, I'd like to welcome Jeff back up, and we're going to open things up for Q&A.

Jeff Moss: Okay. I've got one or two questions to get things going, and then we're going to move to the microphones. There's one on each side. Remember to ask a question in the form of a question, and we'll take it from there. First off, since we're in Singapore, in Southeast Asia: how do you think about the researchers or the work you're seeing from the US, Europe and China? How do you contextualize what they're focusing on? Are we all pursuing the same thing, or are we pursuing different things for different reasons?

Ari Herbert-Voss: That's a fantastic question. When I first got off the plane here, I was greeted by a giant ad for Qwen. We don't have any ads for Qwen in the US. Open source is not as big of a topic in the US. There's Hugging Face and a couple of other companies really carrying the torch there, but for the most part in the US, a lot of the focus is on the reasoning capabilities coming out of the large language models from Anthropic, OpenAI, Google and Meta. In this market, it seems like there's a lot more effort on the scaffolding I was talking about earlier, where you take smaller models, run them more locally, and get Mythos-level performance out of a collection of smaller models. In Europe, there are similar dynamics, because Mistral, the French model, is out there and open source. The US is unique in some ways because it has the majority of the large models that are most capable at reasoning. So everyone else is figuring out different strategies for how to squeeze out that same kind of capability when they don't have the same resources as the US companies.

Jeff Moss: And then China — maybe they have a higher emphasis on inference because they're doing more manufacturing, and that might be more important to them than training. Do you see different countries balancing training and inference differently?

Ari Herbert-Voss: That's a good question. One thing to note is that the GPUs you use for training are not the same GPUs you use for inference. The cluster design has to be very different. So if you're going to focus more on using language models, you need to focus on building out clusters for inference, and that mirrors what we're starting to see in China too.

Jeff Moss: So it's interesting — they'll all start differentiating. If there's less manufacturing in America, they might use less inference, but if there's a lot of training, they'll buy more training GPUs. Okay, fantastic. The other question was maybe a little bit of a hint toward Mythos and OpenAI: marketing has now entered the chat. Does anything change when marketing is involved, or are you still heads down and it's just overhead buzz? It seems like a certain level of maturity has been achieved when marketing decides it's time to get involved.

Ari Herbert-Voss: That's right. We're in a very interesting period, because so much capital has gone into developing these models that we need to figure out how we're going to get returns, and so marketing can't help but enter the chat. Since we're in an industry that is traditionally a cost center, we have slightly different dynamics than regular SaaS companies, because the way we generate value is by reducing risk and showing that you can get more from less. A lot of that means we tend to rely on, unfortunately, fear-based marketing. I do think there's a lot of concern over how fast things are moving, and it's something we definitely need to pay attention to. But there's also a lot more opportunity to focus on building and patching, and to use this energy and momentum to do a lot of the things we probably should have been doing the first time around. The economic incentives haven't really been there, so this is our opportunity to take advantage of this and fix a lot of the stuff we should have fixed the first time.

Jeff Moss: Okay, so I lied — I have one last question, because you just mentioned economics. One of the tenets of economics is that if something gets cheaper, you do more of it. I know there have been some 10x improvements in efficiency, and after each one, it's not that we clap and say, "Oh, 10 times less energy consumption." It's, "No, we're going to build more, because now you can do it everywhere. Instead of it being expensive and only IBM can do it, it's cheaper and anybody can do it." What are your observations over the last decade as certain things have gotten cheaper?

Ari Herbert-Voss: There are two concepts worth exploring here: hedonic adaptation, and the Red Queen race dynamics we're experiencing. By hedonic adaptation, I mean that every time something interesting comes out in terms of capability, people get very excited about it for a number of months, and then over time the excitement tends to fade. GPT-3 is not nearly as exciting as GPT-5.5. Opus is not nearly as exciting as Mythos. It's not that the capabilities have dropped off; it's more that we've gotten used to a certain expectation of what we should be seeing. It's going to be very interesting to see how long we can keep that up. I'm pretty optimistic, but I'm very interested to see how these dynamics play out.

I also think the Red Queen race dynamics are pretty important, because as we find more vulnerabilities and patch them faster, the expectation might be that this is the new normal — we're just finding tons and tons of stuff much faster. That's what I was alluding to with fuzzing. I'm sure a lot of us have friends in the room who really like running fuzzers and love finding tons of bugs, but running a fuzzer all the time doesn't mean we naturally have more secure software. There's still a lot of human process in figuring out which bugs matter. And honestly, a lot of the bugs that cause the huge issues are ones that are really not that sexy — ones that people forgot. One nice thing about LLMs is that they're very good at finding a lot of the very basic things, the basic checks we probably should have been running the entire time. So I think that's going to change the game a little bit, but we're going to find a lot of the same dynamics. The more things change, the more they stay the same.

Jeff Moss: Earlier, when we were talking, you said there's a 10x improvement in some efficiency, and then another 10x improvement, so 100 times more efficiency. Is this just going to keep occurring? When CPUs got smaller and smaller, at some point the features were about as small as they could get, so we started competing on power per watt or cycles per watt. Is there a trajectory where all of a sudden the game changes — right now it's raw speed, but in the future it's energy? How do you think about the future?

Ari Herbert-Voss: It's a very interesting and good question that I don't have a very good answer to. I tend to think about this in terms of Moore's law. Moore's law continues to hold, and scaling laws continue to hold, but at some point we're all kind of holding on and wondering why it's still working. I don't have a very good answer for this one.

Jeff Moss: I didn't know if there was an equivalent of Moore's law for training.

Ari Herbert-Voss: That's the scaling laws. Let me go back to the graph. The most interesting thing about why we're seeing the behaviors we have now is the superlinearity I mentioned. When you train a model that is twice as big, for twice as long, on twice as much data, the thinking used to be that you'd get a model that was twice as capable. But today it seems like it's probably closer to four times as capable, which is crazy. So we're starting to see this insane takeoff. Granted, we only have about two data points: Mythos and GPT-5.5, compared with the previous generation of Opus and GPT models — current models as of this month versus models that came out not too long ago. So right now we're starting to see a bit of a takeoff, which is exciting. It's terrifying.

Jeff Moss: Any idea how much it costs to do twice as much, for twice as long? Is it four times as expensive? Sixteen times as much compute?

Ari Herbert-Voss: These are all questions that I don't think any of us can answer, because of NDAs.

Jeff Moss: Oh, so people know the answer. They just can't say it out loud. Okay.

Ari Herbert-Voss: Yeah. But it's also what makes these economic questions so interesting, because there is so much capital going into this kind of stuff. As long as the music keeps playing, we're going to continue finding some crazy stuff like this, but the music needs to keep playing for us to find it. There are a lot of incentives — from the scientists, from the venture capitalists, from the equity holders — to keep doing this, in part because people just want to see how far it goes. But the cost of doing it is also starting to get expensive.

Jeff Moss: Okay, let's open up to audience questions. I'll start on the left, because yesterday I started on the right.

Audience member: Thank you for the insightful keynote. Here's my question: in your assessment, who is currently ahead on the autonomous offensive curve? Is it the commercial defenders, or the financially motivated and state-nexus threat actors?

Ari Herbert-Voss: Let me make sure I understand your question. You're asking who is ahead right now, attackers or defenders?

Jeff Moss: Commercial defense versus commercially motivated criminal groups.

Ari Herbert-Voss: That's a really good question. I think the dynamics of open source make this very interesting. From everything I've seen in the market, talking to bug bounty hunters and to the various people who use language models to improve their workflows, I think we're neck and neck at the moment. I still think the commercial folks are focused a bit more on the practical side of what you can squeeze out of these models, and there are a lot of economic incentives to find as much as possible for your customers. But these criminal groups are catching up pretty quickly, too, because they also have a lot of these incentives. It's kind of a non-answer, but I think we're neck and neck, which is very interesting.

Jeff Moss: Thank you. Gentleman over here.

Audience member: Good morning, Ari. Thanks for your keynote. Training data is obviously going to be a big, important thing, as we need twice as much data for twice-larger models. At some point, won't that data run out? Or are we just getting a lot smarter at using it?

Ari Herbert-Voss: That's a great question. This last generation of models has used so much data, but there's also a lot of synthetic data starting to end up in training. There are a lot of problems with synthetic data, because sometimes you get learning signals that are not nearly as rich, but there's still a lot you can get out of it. There's also a lot of focus on alternative methods to squeeze the models intelligently. You may have heard of Thinking Machines, another ex-OpenAI company. A lot of their methods focus on intelligently squeezing as much as possible through post-training techniques, to get more of this learning signal into the larger pre-trained models. I think we're going to see a lot more of that. If you're looking for particular types of capabilities, you need to figure out, first, what learning signal you're looking for and where you get data that represents it. And if you can't get that easily or at a volume that makes sense, you need to figure out how to generate it synthetically while mitigating the downsides of synthetic data.

Audience member: Thank you for sharing this today. I wanted to go back to your roots as a student. I work with a lot of students on a regular basis, and I'm curious: with everything moving as quickly as it has over these last few months, do you have any words of wisdom or suggestions for students — people aspiring to go into computer science or cybersecurity who don't know what to do with everything moving so fast? Anything I could pass along to them, I would really appreciate.

Ari Herbert-Voss: Oh, that's a question close to my heart. I love that. We're in very interesting times right now. I can't imagine what it's like to be a student right now. It's got to be very scary from a job market perspective, but in terms of being able to learn whatever you want and accelerate how much you can learn in a very short time, it's kind of a golden renaissance for a lot of people. In order to be successful, I think a lot of it comes down to figuring out what you're very passionate about, finding a mentor in that space, and then leveraging all the AI tools available to you to learn and really grow in a direction you're excited about. Passion will take you very far, but preparation will take you even further, and having mentorship is what has really served me well. I think it will continue to serve people well in this new era.

Audience member: Hello, I'm a data engineering student. A lot of the discourse whenever new frontier models come out is about whether the previous models get worse, and we've heard a bit of that with Claude. With your experience of seeing under the hood, what does that resource allocation look like?

Ari Herbert-Voss: That's a good question. Part of why that happens comes down to the difficulties of shipping a large language model and deploying it across a bunch of different GPUs. It's not an intentional "now that we have this big thing, we can forget about this one." It's more: how do we serve customers the next best thing while also handling the fact that there isn't an infinite number of GPUs out there, or an infinite amount of compute or other resources? It is a known effect in the industry. Working on the inside, it's not one that people are particularly happy exists, and people are always trying to figure out how to mitigate it. But it is one of the growing pains of building fantastic new technology.

Jeff Moss: You have to tell them your story about when you were at OpenAI, running the job, and then you looked at the billing.

Ari Herbert-Voss: Oh yeah. Working at a large company like this, at least when I was there, between 2019 and about 2023, the economics were a little different. At the time, we were working with Microsoft, and I had kicked off this giant training run. It was very expensive, and it turns out I messed it up. So I looked in my console to see how much it cost, because I was thinking, "Oh man, I've got to go to my manager and figure out what I'm going to do about this." And I kid you not, it was about a $5 million job that I had absolutely botched. I was like, "Oh my god, I'm going to get fired." But because research was at such a premium and I was on a research team, they looked at it and said, "Well, I guess we're just going to write that one off." The capital cost that goes into making these large language models is quite high, but it's an R&D cost for a lot of these places, because the expectation is that you're going to mess up a few times, but when you really hit it big, you're going to hit it big in terms of the capabilities you can squeeze out.

Jeff Moss: So welcome to the $5 million screw-up club, and all the other researchers say, "Hey, I've got six million."

Audience member: I'm a researcher in automotive security. Your talk mentioned attacks running continuously. For connected automotive systems, how do we mitigate the risk of resource exhaustion on the gateway when running AI-driven continuous simulation cycles? [Partly unclear]

Ari Herbert-Voss: I'm not sure I fully got the question — resource exhaustion in autonomous vehicles when an AI simulation is running continuously?

Audience member: Yes. In automotive and IoT, we run AI on the gateway. How do we mitigate the risk when we run these continuous AI simulation cycles on the gateway? [Partly unclear]

Jeff Moss: Maybe come on up afterward. It's a little hard to understand the nature of the question.

Ari Herbert-Voss: I think it might just be the audio. If you want to find me after, I'm happy to talk.

Jeff Moss: While we're waiting for the next question: I've heard that Mythos and similar models destroy capture-the-flag contests, because CTFs are so well studied — they're bounded problems. Other people have said they don't perform so well in debugging situations, so people are speculating that CTFs might move toward debugging challenges. You talked about certain types of problems it's not so good at. If it's good at attack and not good at debugging, why? And are there other classes of problems where you'd say, "No, that's a tomorrow problem"?

Ari Herbert-Voss: Let me find the slide — I'm going to use this as notes. Going back to the OSS-Fuzz evaluation included in Anthropic's Mythos report, I think this tells the story best. Where we see the most massive gains for a model like Mythos, which is arguably at the edge, is among these low-severity, shallow bugs, which you could also describe a lot of CTF bugs as, because they're pretty well articulated. They're not very natural. We're seeing modest gains at the mid-tier bugs and still sparse gains at deeper exploitation. That's not to say we're not seeing interesting stuff happening at the deeper exploitation level, but a lot of it is like shooting a shotgun in the dark and then figuring out how to roughly stitch things together, and we're getting pretty lucky on some of that. Over time, models are going to get better at this, but right now the most success is in shallow, basic bugs that have a lot of documentation and are pretty easy to validate.

The other thing is that models tend to struggle with anything that requires holding state — things like concurrency, understanding what's happening across different machines and in different levels of memory. They really struggle with that, because fundamentally language models don't have a model of state being held. And the issues we have with language models around hallucinations will still plague us regardless of whether you have a larger context window. We're seeing improvements here because context windows have really exploded in the last three years, but these issues with the complexity of temporal state analysis, and other bugs like that, still persist. I anticipate that will probably continue to be true for as long as we use transformer-based LLM architectures.

Jeff Moss: So, to improve debugging, does that mean that as a developer you want your crash dumps and error logs to be as detailed as possible? The more detailed the error output, the better these models do — so there'd be a competitive advantage if your tooling generates better errors than the next guy's?

Ari Herbert-Voss: Yes, absolutely. Even with Mythos and these other large models, we find we get better performance if you know what you're doing. I think that's going to continue to be a theme across this generation of AI: if you know what you're doing, you know what kind of information you need to make a decision, and that's the kind of information you can provide to a model and instrument it with.

Jeff Moss: So maybe there's hope for us. Instead of getting an "error 17254" pop-up, we'll actually get more meaningful, contextual errors, because these companies will want to train on them. But as long as it's for end users, maybe they're not so interested.

Ari Herbert-Voss: Yeah, the economic incentives have to be at play for this to happen.

Audience member: Hello, I'm from India. The next big thing I see is quantum computing. Do you foresee quantum computing significantly disrupting the AI landscape, especially in terms of infrastructure and security?

Ari Herbert-Voss: No, and I'll tell you why. I think of quantum as being orthogonal to AI. The things quantum is very good at are things that I think mostly help or break cryptography. But the type of operations you do on a computer for AI-based tasks are not the same. What underlies a lot of AI capability is just being able to multiply matrices together, and with quantum, it's actually pretty expensive to multiply matrices. So you're not really going to see the same gains if quantum continues to develop.

Jeff Moss: Is that vector math?

Ari Herbert-Voss: Exactly. It's a vector thing, and quantum is not so good at vectors — not in the ways it would need to be. If you've ever had a classical mathematics education, there are two branches they put you into: either you go down the algebra path, or the real analysis and calculus path. Algebra is a mindset that's much more helpful for solving quantum-related problems in cryptography, and the calculus or analysis-based path tends to be more helpful for solving AI problems.

Audience member: You discussed how these models struggle with concurrency and temporal analysis in bug hunting. Is there any evidence that the code they generate is more prone to those types of errors?

Ari Herbert-Voss: That's a good question. I don't have an immediate answer for you, so let me reason through it. I haven't seen that. I have seen a lot more hallucinations, but as for true errors, a lot of this is shooting in the dark and hoping you hit something. I'll admit that's a great question, and I don't have a good answer for you.

Audience member: As the capabilities of models in offensive spaces continue to improve, we've seen with Project Glasswing that access has been restricted to a select set of companies, and the model may not be released publicly while they work on improving alignment and guardrails. Do you think alignment and guardrails will improve in a way that allows public access, or for other companies to get access? Or will access continue to be restricted as models improve, and will governments possibly get involved?

Ari Herbert-Voss: That's a good question. What we're seeing right now with the restrictions rhymes with what happened with GPT-2. There's a paper I wrote with colleagues who are now at Anthropic, and some who are still at OpenAI, about what we called staged release. The idea is that if you have a capability where you're not quite sure how it's going to land when it goes public, you want to be very thoughtful about how you do the deployment. So we built up a trust and safety team, monitoring, all those sorts of things — I built the first versions of some of that. It's been very interesting to see a lot of the same arguments we made back then come back with Mythos.

I think we're in a very different regime than we were in 2019, and candidly, I feel like Anthropic may need to think a little differently about how they handle this staged release, because I don't know if it's as helpful as it would be if we were able to give more access to more people. The other thing with a very large language model is that you can't serve as many customers that way, because it's so expensive to serve a much larger model than we've seen before. So naturally, you end up restricting use anyway, which gives you a bit more time to make sure your monitoring systems and mitigations are up to snuff. I think there's definitely a lot more incentive for governments to step in and figure out how to be helpful. But because of how hard it is to deploy the technology, I'm not confident that artificially gating it will be meaningfully different from just naturally having a waiting list.

Jeff Moss: Hitting on that a little: you mentioned earlier that you can achieve sort of Mythos-level results with open source if you have a higher skill level and a better team. Talk a little about how much open source is lagging, or whether open source is doing things differently — exploring spaces that don't have immediate economic returns. How do you think about the frontier models versus the open-source community?

Ari Herbert-Voss: That's a great question. Open source is going to lag in terms of reasoning capabilities, because it doesn't make a lot of sense to release something open source that was very expensive to train, required a lot of expensive data, and used a lot of GPUs you could otherwise be using to train other things that give you a competitive advantage in the AI race. So we're going to continue seeing meaningful lag time. That said, if you're an expert, that doesn't really matter. You can get a lot of the benefit of a Mythos-level system if you understand how to instrument a whole bunch of smaller systems. I think this means there will be more of a focus in the open-source community on expert-led instrumentation, versus the more consumer-focused capabilities that come out of the classical foundation model labs.

Jeff Moss: I'm curious, because at some point, to the earlier question, somebody says, "Well, it's too powerful, we can't release it," and some developer in Kraków says, "Here's my open-source version of it." If things continue to improve on both sides, open source will lag, but it'll still probably be pretty powerful in a couple of years. And maybe there'll be an Internet Archive for open-source training data, so not everybody has to collect their own, or community efforts to pool national data. So I was curious where this goes.

Ari Herbert-Voss: I think it's worth digging into a little more. Interesting dynamics come out of things like OpenClaw. People may remember BabyAGI, or some of the other harness technologies that people got similarly excited about. These are all solving the same problem of figuring out more intelligent ways of instrumenting open-source models, so you can try to get the same performance out of them as you would from a larger language model from a foundation model lab. Economically, it's kind of impossible to get exactly the same level of performance, but you can get an approximation. And I think we'll see more focus on outputs: what are you trying to get out of a language model? If you can get what you need out of it, people will naturally figure out ways to instrument models much more intelligently, and you'll end up with these more interesting open-source communities around things like OpenClaw.

Audience member: I'm from Nepal, and I have a question. As offensive security evolves into autonomous agents, the speed and scale of initial access attacks like AI-driven spear phishing will inevitably increase. How will SecOps teams adapt their defensive posture to catch these agent-driven campaigns early in the kill chain?

Jeff Moss: Let me repeat the question: AI-driven spear phishing attacks are increasing right now, so how will SecOps teams throughout the world adapt?

Ari Herbert-Voss: I think it's important that people get a bit more proactive about running these tools. There are a whole bunch of tools starting to pop up — including my own, though I'm not here to sell — that let you find problems before the bad guys do. Being more thoughtful about how to shift left within your own organization is really how you win here, because it's going to move whether or not you move. So you should probably start looking for things and figure out how to build this into your cycle much faster. I would think about this as an opportunity. This is a moment in time where people are finally looking at security and thinking, "Maybe we should give it a bigger budget, maybe we should give it a bit more attention and care." So let's take advantage of this moment and figure out how to encourage people to find vulnerabilities faster, patch them faster, and build up the processes we really should have been running in the first place, when the economic incentives weren't necessarily aligned with us being able to do that.

Jeff Moss: All right, we have time for one more question. Final question. Okay, go for it.

Audience member: Maybe it's not your direct specialization, but I do cyber-physical security, which means degrading or destroying physical objects by means of cyber attacks. Not many people do it, because it requires a huge amount of interdisciplinary knowledge. You need to be able to read a lot of engineering documentation and books, consume a large amount of information, and correlate it correctly. This type of attack is on the rise among state-sponsored threat actors, at least in terms of intention and motivation. So how realistic is it to teach AI bots to become very intelligent, creative engineers in a multidisciplinary domain?

Jeff Moss: Cyber-physical interaction — OT systems have a lot of engineering and interconnections and reliance on many other parts. Is that further down the road for AI attacks, because it's not the same as reading source code and iterating? How realistic is it for AI to get good at cyber-physical?

Ari Herbert-Voss: That's a great question. Yes, it is further down the road, because it's straight up not in the training data set, and nobody at these foundation model labs is particularly aware of how to put it in, because it's complex data to collect. The payoff is not necessarily high for a company that needs the returns required to keep the music playing as it moves toward AGI-type systems. So this is an area where I think open source and that expert-based community could really shine, in ways you're not going to see from the traditional foundation model lab approaches.

Audience member: I actually see this becoming a large market, because this is the type of AI automation or trained bots that state-sponsored threat actors would love to buy.

Ari Herbert-Voss: Oh, absolutely. I'm right there with you. We've been thinking about that too.

Black Hat host: All right. That wraps up our keynote session. I just have a couple of announcements, and then I'll release you into the vortex of Black Hat. Jeff, Ari, thank you very much.