alt.hn

3/30/2026 at 6:43:38 PM

Vulnerability research is cooked

 https://sockpuppet.org/blog/2026/03/30/vulnerability-research-is-cooked/

by pedro84

3/30/2026 at 8:29:34 PM

I don't understand why the takeaway here is (unless I'm missing something), more or less "everything is going to get exploited all the time". If LLMs can really find a ton of vulnerabilities in my software, why would I not run them and just patch all the vulnerabilities, leading to perfectly secure software (or, at the very least, software for which LLMs can no longer find any new vulnerabilities)?

by stavros

3/30/2026 at 8:43:55 PM

When did we enter the twilight zone where bug trackers are consistently empty? The limiting factor of bug reduction is remediation, not discovery. Even developer smoke testing usually surfaces bugs at a rate far faster than they can be fixed let alone actual QA.

To be fair, the limiting factor in remediation is usually finding a reproducible test case which a vulnerability is by necessity. But, I would still bet most systems have plenty of bugs in their bug trackers which are accompanied by a reproducible test case which are still bottlenecked on remediation resources.

This is of course orthogonal to the fact that patching systems that are insecure by design into security has so far been a colossal failure.

by Veserv

3/31/2026 at 6:15:49 AM

Bugs are not the same as (real) high severity bugs.

If you find a bug in a web browser, that's no big deal. I've encountered bugs in web browsers all the time.

You figure out how to make a web page that when viewed deletes all the files on the user's hard drive? That's a little different and not something that people discover very often.

Sure, you'll still probably have a long queue of ReDoS bugs, but the only people who think those are security issues are people who enjoy the ego boost if having a cve in their name.

by bawolff

3/31/2026 at 7:14:15 AM

Eh, with browsers you can tell the user to go to hell if they don't like a secure but broken experience. The problem in most software is that you commit to bad ideas and then have to upset people who have higher status than the software dev that would tell them to go to hell.

by kackerlacker

3/30/2026 at 9:18:53 PM

That might have been true pre LLMs but you can literally point an agent at the queue until it’s empty now.

by reactordev

3/30/2026 at 9:27:04 PM

You literally cannot, since ANY changes to code tend to introduce unintended (or at least not explicitly requested) new behaviors.

by batshit_beaver

3/30/2026 at 9:32:49 PM

Eventual convergence? Assuming each defect fix has a 30% chance of introducing a new defect, we keep cycling until done?

by lll-o-lll

3/30/2026 at 9:49:00 PM

Assuming you can catch every new bug it introduces.

Both assumptions being unlikely.

You also end up with a code base you let an AI agent trample until it is satisfied; ballooned in complexity and redudant brittle code.

by saintfire

3/30/2026 at 10:23:02 PM

You can have an AI agent refactor and improve code quality.

by charcircuit

3/31/2026 at 12:22:17 AM

But, have you any code that has been vetted and verified to see if this approach works? This whole Agentic code quality claim is an assertion, but where is the literal proof?

by abakker

3/31/2026 at 4:22:49 PM

Did we have code quality before llms?

by wredcoll

3/31/2026 at 7:34:05 AM

If it can be trained with reinforcement learning then it will happen

by WithinReason

3/31/2026 at 8:01:42 AM

Funnily enough I've literally never seen anyone demo this, despite all the other AI hype. It's the one thing that convinces me they're still behind.

by lmm

3/31/2026 at 2:27:15 PM

Just today I had an agent add a fourth "special case" to a codebase, and I went back and DRY'd three of them.

Now I used the agent to do a lot of the grunt work in that refactor, but it was still a design decision initiated by me. The chatbot, left unattended, would not have seen that needed to be done. (And when, during my refactor, it tried to fold in the fourth case I had to stop it.)

(And for a lot of code, that's ok - my static site generator is an unholy mess at this point, and I don't much care. But for paid work...)

by flir

3/31/2026 at 5:20:12 AM

It’s agents all the way down - until you have liability. At some point, it’s going to be someone’s neck on the line, and saying “the agents know” isn’t going to satisfy customers (or in a worst case, courts).

by intended

3/31/2026 at 4:24:58 PM

> until you have liability

And are you thinking this going to start happening at some point or what?

The letters I get every other month telling me I now have free credit monitoring because of a personal info breach seems to suggest otherwise.

by wredcoll

3/31/2026 at 7:26:10 PM

A firm has very different amounts of time, ability and money to spend on following up on broken contracts.

by intended

3/31/2026 at 7:45:38 AM

Sure it can. It's not like humans aren't already deflecting liability or moving it to insurance agencies.

by charcircuit

3/31/2026 at 11:13:25 AM

> It's not like humans aren't already deflecting liability

They attempt to, sure, but it rarely works. Now, with AI, maybe it might, but that's sort of a worse outcome for the specific human involved - "If you're just an intermediary between the AI and me, WTF do I need you for?"

> or moving it to insurance agencies.

They aren't "moving" it to insurance companies, they are amortising the cost of the liability at a small extra cost.

That's a big difference.

by lelanthran

3/31/2026 at 12:14:07 PM

At some point, the risk/return calculus becomes too expensive for insurance companies.

Usually thats after the premiums become too high for most people to pay.

by intended

3/31/2026 at 5:44:48 PM

The chance of a defect fix introducing a new defect tends to grow linearly with the size of the codebase, since defects are usually caused by the interaction between code and there's now more code to interact with.

If you plot this out, you'll notice that it eventually reaches > 100% and the total number of defects will eventually grow exponentially, as each bugfix eventually introduces more bugs than it fixes. Which is what I've actually observed in 25 years in the software industry. The speed at which new bugs are introduced faster than bugfixes varies by organization and the skill of your software architects - good engineers know how to keep coupling down and limit the space of existing code that a new fix could possibly break. I've seen some startups where they reach this asymptote before bringing the product to market though (needless to say, they failed), and it's pretty common for computer games to become steaming piles of shit close to launch, and I've even seen some Google systems killed and rewritten because it became impossible to make forward progress on them. I call this technical bankruptcy, the end result of technical debt.

by nostrademons

3/31/2026 at 8:36:40 AM

That's assuming that each fix can only introduce at most one additional defect, which is obviously untrue.

by majewsky

3/30/2026 at 9:42:58 PM

Why would it converge?

by Kinrany

3/31/2026 at 12:38:00 PM

Except they don't converge. You see that if you use agents to evolve a codebase. We also saw exactly that in the failed Anthropic experiment to create a C compiler.

by pron

3/31/2026 at 12:27:53 PM

As long as we're inventing numbers, what if it's a 90% chance?

What if it's a 200% chance, and every fix introduces multiple defects?

by bluefirebrand

3/30/2026 at 9:31:06 PM

I’ve had mine on a Ralph loop no problem. Just review the PR..

by reactordev

3/30/2026 at 9:31:26 PM

Which still means a single person with Claude can clear a queue in a day versus a month with a traditional team.

by k_roy

3/31/2026 at 5:41:00 AM

Your example must have incredible users or really trivial software.

by worthless-trash

3/30/2026 at 10:26:55 PM

The fact that KiCad still has a ton of highly upvoted missing features and the fact that FreeCAD still hasn't solved the topological renumbering problem are existence proofs to the contrary.

by bsder

3/30/2026 at 11:58:27 PM

Shouldn't be down voted for saying this. There are active repo's this is happening in.

"BuT ThE LlM iS pRoBaBlY iNtRoDuCiNg MoRe BuGs ThAn It FiXeS"

This is an absurd take.

by rybosworld

3/31/2026 at 2:23:42 AM

It probably is introducing more bugs because I think some people dont understand how bugs work.

Very, very rarely is a bug a mistake. As in, something unintentional that you just fix and boom, done.

No no. Most bugs are intentional, and the bug part is some unintended side effects that is a necessary, but unforseen, consequence of the main effect. So, you can't just "fix" the bug without changing behavior, changing your API, changing garauntees, whatever.

And that's how you get the 1 month 1-liner. Writing the one line is easy. But you have to spend a month debating if you should do it, and what will happen if you do.

by array_key_first

3/31/2026 at 10:44:57 AM

So, you have already fixed all the bugs and now just cruising through life?

by missingdays

3/31/2026 at 11:32:53 AM

I wonder whether people like you have actually used Claude for any length of time.

I use it all day. I consider it a near-miracle. Yet I correct it multiple times daily.

by jen729w

3/30/2026 at 8:40:31 PM

The pressure to do so will only happen as a consequence of the predicted vulnerability explosion, and not before it. And it will have some cost, as you need dedicated and motivated people to conduct the vulnerability search, applying the fixes, and re-checking until it comes up empty, before each new deployment.

The prediction is: Within the next few months, coding agents will drastically alter both the practice and the economics of exploit development. Frontier model improvement won’t be a slow burn, but rather a step function. Substantial amounts of high-impact vulnerability research (maybe even most of it) will happen simply by pointing an agent at a source tree and typing “find me zero days”.

by layer8

3/30/2026 at 8:49:17 PM

I feel like the dream of static analysis was always a pipe.

When the payment for vulns drops i'm wondering where the value is for hackers to run these tools anymore? The LLMs don't do the job for you, testing is still a LOT OF WORK.

by cartoonworld

3/30/2026 at 9:19:20 PM

Breaking something is easier than fixing it.

by joatmon-snoo

3/30/2026 at 9:22:05 PM

People have said that for decades and it wasn't true until recently.

by tptacek

3/31/2026 at 12:33:18 AM

Hmm: can you elaborate?

I've never been on a security-specific team, but it's always seemed to me that triggering a bug is, for the median issue, easier than fixing it, and I mentally extend that to security issues. This holds especially true if the "bug" is a question about "what is the correct behavior?", where the "current behavior of the system" is some emergent / underspecified consequence of how different features have evolved over time.

I know this is your career, so I'm wondering what I'm missing here.

by joatmon-snoo

3/31/2026 at 12:37:16 AM

It has generally been the case that (1) finding and (2) reliably exploiting vulnerabilities is much more difficult than patching them. In fact, patching them is often so straightforward that you can kill whole bug subspecies just by sweeping the codebase for the same pattern once you see a bug. You'd do that just sort of as a matter of course, without necessarily even qualifying the bugs you're squashing are exploitable.

As bugs get more complicated, that asymmetry has become less pronounced, but the complexity of the bugs (and their patches) is offset by the increased difficulty of exploiting them, which has become an art all its own.

LLMs sharply tilt that difficulty back to the defender.

by tptacek

3/31/2026 at 8:20:42 AM

In a sense, breaking a vulnerability is easier than fixing it up to be an exploit.

by saagarjha

3/30/2026 at 9:32:56 PM

Specifically in software vulnerability research, you mean.

Fixing vulnerable code is usually trivial.

In the physical world breaking things is usually easier.

by underdeserver

3/31/2026 at 10:09:24 AM

That's why you simply make the LLM part of the CI checks on PRs.

by stavros

3/30/2026 at 9:44:34 PM

A proper fix maybe. But LLMs can easily make it no longer exploitable in most cases.

by charcircuit

3/30/2026 at 8:32:23 PM

That might be one outcome, especially for large, expertly-staffed vendors who are already on top of this stuff. My real interest in what happens to the field for vulnerability researchers.

by tptacek

3/30/2026 at 8:53:49 PM

Perhaps a meta evolution, they become experts at writing harnesses and prompts for discovering and patching vulnerabilities in existing code and software. My main interest is, now that we have LLMs, will the software industry move to adopting techniques like formal verification and other perhaps more lax approaches that massively increase the quality of software.

by lifty

3/31/2026 at 12:00:30 PM

> Perhaps a meta evolution, they become experts at writing harnesses and prompts

Harnesses, maybe, but prompts?

There's still this belief amongst AI coders that they can command a premium for development because they can write a prompt better than Bob from HR, or Sally from Accounting.

When all you're writing are prompts, your value is less than it was before., because the number of people who can write the prompt is substantially more than the number of people who could program.

by lelanthran

3/31/2026 at 1:22:52 PM

I agree with this take. Nothing changes, everything just evolves. Been happening for 60 years, will (likely) continue to happen for the next 60 years.

by sputknick

3/31/2026 at 2:49:57 AM

Also, synthetic data and templates to help them discover new vulnerabilities or make agents work on things they're bad at. They differentiate with their prompts or specialist models.

Also, like ForAllSecure's Mayhem, I think they can differentiate on automatic patching that's reliable and secure. Maybe test generation, too, that does full coverage. They become drive by verification and validation specialists who also fix your stuff for you.

by nickpsecurity

3/31/2026 at 12:36:51 AM

Testing exists.

> formal verification

Outside of limited specific circumstances, formal verification gives you nothing that tests don't give you, and it makes development slow and iteration a chore. People know about it, and it's not used for lot of reasons.

by habinero

3/30/2026 at 8:57:00 PM

True, but I already am curious to see what happens in a multitude of fields, so this is just one more entry in that list.

by stavros

3/30/2026 at 9:45:22 PM

Just wanted to point out that tptacek is the blog post's author (and a veteran security researcher).

by underdeserver

3/31/2026 at 11:51:53 AM

I've worked at companies before where they have balked at spending $300 to buy me a second hand thinkpad because I really wanted to work on a Linux machine rather than a mac. I don't see them throwing $unlimited at tokens to find vulnerabilities, at least until after it's too late.

by weavie

3/30/2026 at 8:49:43 PM

Attackers only have to be successful once while defenders have to be successful all the time?

by htrp

3/31/2026 at 9:15:37 AM

Yes and no. Good defence is layered and an attacker needs to find a hole in each layer. Even if it is not layered intentionally a locally exploitable vulnerability gives little if you have no access to a remote system. But some asymmetry does exist.

by citrin_ru

3/30/2026 at 8:41:59 PM

My sense is that the asymmetry is non-trivial issue here. In particular, a threat actor needs one working path, defenders need to close all of them. In practice, patching velocity is bounded by release cycles, QA issues / regression risk, and a potentially large number of codebases that need to be looked at.

by zar1048576

3/31/2026 at 12:46:31 AM

Find-then-patch only works if you can fix the bugs quicker than you’re creating new ones.

Some orgs will be able to do this, some won’t.

by cvwright

3/31/2026 at 1:07:16 AM

"Find me vulnerabilities in this PR."

by stavros

3/30/2026 at 8:45:22 PM

> If LLMs can really find a ton of vulnerabilities in my software, why would I not run them and just patch all the vulnerabilities, leading to perfectly secure software?

Probably because it will be a felony to do so. Or, the threat of a felony at least.

And this is because it is very embarrassing for companies to have society openly discussing how bad their software security is.

We sacrifice national security for the convenience of companies.

We are not allowed to test the security of systems, because that is the responsibility of companies, since they own the system. Also, companies who own the system and are responsible for its security are not liable when it is found to be insecure and they leak half the nations personal data, again.

Are you seeing how this works yet? Let's not have anything like verifiable and testable security interrupt the gravy train to the top. Nor can we expect systems to be secure all the time, be reasonable.

One might think that since we're all in this together and all our data is getting leaked twice a month, we could work together and all be on the lookout for security vulnerabilities and report them responsibly.

But no, the systems belong to companies, and they are solely responsible. But also (and very importantly) they are not responsible and especially they are not financially liable.

by Buttons840

3/30/2026 at 9:04:50 PM

>> If LLMs can really find a ton of vulnerabilities in my software, why would I not run them and just patch all the vulnerabilities, leading to perfectly secure software?

>Probably because it will be a felony to do so. Or, the threat of a felony at least.

"my software" implies you own it (ie. your SaaS), so CFAA isn't an issue. I don't think he's implying that vigilante hackers should be hacking gmail just because they have a gmail account.

by gruez

3/31/2026 at 2:46:46 PM

Takeaway is formal software.

by whateverboat

3/31/2026 at 12:21:45 AM

Because not all software gets auto-updated. Most of it does not!

by woeirua

3/31/2026 at 9:42:05 AM

closed source software

deliberate vulnerabilities (thanks nsa)

by lukewarm707

3/31/2026 at 11:09:44 AM

Any patch you ship lands on a moving treadmill of releases and deps, with new code stapled onto old junk and old assumptions leaking into the next version. Attackers can run the same models you do, so the gap between finding and fixing bugs shrinks until your team are doing janitorial work against a machine.

"Perfectly secure" software is a philosophy seminar, not an outcome. You can cut the bug pool down a lot, but the tide keeps coming and the sandcastle still falls over.

by hrmtst93837

3/31/2026 at 11:16:43 AM

Any patch you ship can be vetted by the same models, so you can be sure the same models can't find a vulnerability in the attacker's hands. Then it's just a matter of fixing the old vulnerabilities.

by stavros

3/31/2026 at 4:46:08 PM

"so you can be sure"

Nothing is for sure with LLMs.

by g947o

3/31/2026 at 4:55:09 PM

Nothing is for sure with anything.

by stavros

3/31/2026 at 8:09:04 AM

[dead]

by hrmtst93837

3/31/2026 at 12:10:11 AM

Am I wrong in thinking that an "exploits are free" environment massively favors the defender? Given that real-world exploits usually chain 0days, the attacker has to find the whole chain while the defender only needs to fix the weakest link.

The defender also gets to make the first move by just putting a "run an agent to find vulns" step in their CI pipeline. If LLMs truly make finding exploits free, almost no LLM-findable exploits will ever make it into the codebase.

The only way break the equilibrium is still going to be a smart researcher capable of finding exploits that the commoditized tools alone can't.

by somesortofthing

3/31/2026 at 12:43:08 PM

Right, the article makes it seem that the attackers have the advantage in that they can try to attack everything at once, but the defenders have the advantage in that they each need to defend only one product.

But it is true that if attackers more easily find holes in software that's hard to patch (embedded), that's a problem.

by pron

3/31/2026 at 11:13:55 AM

Yes I agree. After a period of turmoil, there'll be a new equilibrium between attackers and defenders.

Considering the new 'meta' that LLMs encode knowledge about existing software but not new ones, I would expect a side effect that newly written software will be inherently less exploitable by LLMs, even if from an actual security perspective, they're worse in design.

by torginus

3/31/2026 at 3:20:31 AM

You aren't wrong for the most part, but this whole thing of "find me vulns" is not really accurately describing the space.

Finding vulns has almost become sort of like a vibe thing even before LLMs. There would be some security patch that everyone says is critical because it fixes a vulnerability, but the vulnerability is like "under certain conditions, and given physical access to the device, an attacker can craft a special input that crashes the service"... and thats it.

Even stuff like Spectre and Meltdown, which I highly doubt an LLM can find on its own without specifically knowing about speculative execution attacks, are incredibly hard to use. People made a big deal of those being able to be used from javascript, but to actually leak anything of importance you need to know memory layouts, a bunch of other info and so on.

So while an LLM can patch all the up front vulnerabilities, most if not all of those are completely useless to an attacker. Modern systems are incredibly secure.

On the flip side, the stuff that LLM doesn't know about, that can be exploited. For example, assume that log4shell hasn't been found yet, and that log statements by default can pull jni objects from the internet and execute them. The llms would happily write you code with log statements using log4j, and pass it through vulnerability checker, and I would bet that even at the bytecode level it won't ever figure out that vulnerability exists.

And overall, because of Rice theorem, you can't tell if the program is fully exploitable or not without actually running it in some form and way. LLMS can help you with this (but not fully of course) by actually running it and fuzzing inputs and observing memory traces, but even this gets very hard when you introduce things like threading and timed executions which can all affect the result.

And also, the LLMs themselves are an exploit vector now. If you manage to intercept the API calls somehow and insert code or other instruction, you can have the developer essentially put the exploit for you into the code.

So I would say the field is about even.

by ActorNightly

3/31/2026 at 6:21:29 AM

> Even stuff like Spectre and Meltdown, which I highly doubt an LLM can find on its own without specifically knowing about speculative execution attacks, are incredibly hard to use. People made a big deal of those being able to be used from javascript, but to actually leak anything of importance you need to know memory layouts, a bunch of other info and so on.

In fairness, i think part of the reason people made a big deal was the novelty of the attack. It was something new. The defenses weren't clear yet. The attack surface wasn't clear. It was unclear if anyone was going to come up with a novel improvement to the technique. Humans love novelty.

by bawolff

3/31/2026 at 9:49:06 AM

there are tools already like trufflehog that will find leaked secrets

macos still ships with live github private keys in the library folder

oops i put my god level aws key in js on my website

oops i put the supabase key in frontend

oh no! i am the maintainer of a hosted password manager and i took home the cmek key

by lukewarm707

3/31/2026 at 10:16:50 AM

The higher the productivity multiplier towards exploiting software, the more developers would find themselves severely outmatched: exploiting software is someone’s full-time job, whereas the engineers already have one—building it.

To express this in numerical terms, let’s consider developer’s incentive to spend effort learning to find and actually finding vulnerabilities in their software (as oppposed to building it) as D, and attacker’s incentive to spend effort exploiting that software as A.

I would say initially A = D × 5 is fair. On one hand, the developer knows their code better. However, their code is open, and most software engineers by definition prefer building (otherwise they would have been pentesters) so that’s where most of their time is going. This is not news, of course, and has been so since forever. The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can deal; the lack of equivalent pressure on the developer’s side leads me to adjust it to A = D × 10.

×10 is our initial power differential between the attacker and the developer.

Now, let’s multiply that effort by a constant L, reflecting the productivity boost from LLMs. Let’s make it a 10 (I’m sure many would say LLMs make them more tham ×10 more productive in exploit-finding, but let’s be conservative).

Additionally, let’s multiply that by a variable DS/AS that reflects developer’s/attacker’s skill at using LLMs in such particular ways that find the most serious vulnerabilities. As a random guess, let’s say AS = DS × 5, as the attacker would have been exclusively using LLMs for this purpose.

With these numbers substituted in, X would be our new power differential:

    X = (A × L × AS) ÷ (D × L × DS)

    X = (D × 10 × 10 × DS × 5) ÷ (D × 10 × DS)

    X = 50.
If my math is right, the power differential between the attacker and a developer jumps from 10 to 50 in favour of the attacker. If LLMs ×100 the productivity, the new differential would be 500.

I didn’t account for the fact that many (especially smaller) developers may not even have the resources to run the equivalent compute power as a dedicated hacking team.

Some ways to shift the balance back could be ditching the OSS model and going all-in on the so-called “trusted computing”. Both measures would increase the amount of effort (compute) the attacker may need to spend, but both happen to be highly unpopular as they put more and more power and control in the hand of the corporations that build our computers. In this way, the rise of LLMs certainly advances their interests.

by strogonoff

3/31/2026 at 12:47:56 PM

> exploiting software is someone’s full-time job, whereas the engineers already have one—building it.

But the attackers needs to spread their attack over many products, while the engineers only need to defend one.

> The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can deal; the lack of equivalent pressure on the developer’s side leads me to adjust it to A = D × 10.

Except that's true even without LLMs. LLMs improve both sides' capabilities by the same factor (at least hypothetically).

> Additionally, let’s multiply that by a variable DS/AS that reflects developer’s/attacker’s skill at using LLMs in such particular ways that find the most serious vulnerabilities. As a random guess, let’s say AS = DS × 5, as the attacker would have been exclusively using LLMs for this purpose.

I'm not sure that's right, because once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them. So again, we can remove the "LLM factor" from both sides of the equation. If anything, security skills can spread more easily to defenders with LLM because without LLMs, the security skill of the attackers require more effort to develop.

by pron

3/31/2026 at 12:57:11 PM

> > exploiting software is someone’s full-time job, whereas the engineers already have one—building it.

> But the attackers needs to spread their attack over many products, while the engineers only need to defend one.

Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely.

Realistically, you have people whose job or passion is to develop software, who often work not on one but on N projects at the same time (especially in OSS), and who definitely aren’t going to make finding vulnerabilities their full-time job because if they do then there’ll be no one to build the thing in the first place.

> Except that's true even without LLMs.

Of course. That’s why I put it before I started taking into account LLMs. LLMs multiply the pre-existing imbalance.

> once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them

Sure, that’s an interesting point. I’m sure the attackers try to conceal their methods; the way we tend to find out about it is when an exploit is exhausted, stops being worth $xxxxxxxx, and starts to be sold on mass markets, at which point arguably it’s a bit late. Furthermore, you still mention those mystical “defenders”, as if you would expect an average software project to have any dedicated defenders.

(Edited my reply to the latest point, I didn’t read it correctly the first time.)

by strogonoff

3/31/2026 at 7:53:24 PM

> Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely.

No, I'm assuming it has maintainers (they play the role of defenders).

> engineers who work on software are simply not that great and dedicated about finding vulnerabilities in it.

Yes, but LLMs help them more than they help the attackers, because the attackers are already security experts. In other words, the LLMs reduce the skill gap rather than increase it. Becoming good at using AI is much easier than becoming good at security.

by pron

3/31/2026 at 10:47:18 AM

The world runs on legacy software, which simply has no budget for even asking LLMs to find vuls, not to mention fixing them. While you are right that it should favor the defenders, in practice, this is a disaster because, in many critical cases, there are no defenders.

by dandanua

3/30/2026 at 8:30:29 PM

I interned for the author at 18. I assumed security testing worked like this:

1. Static analysis catches nearly all bugs with near-total code coverage

2. Private tooling extends that coverage further with better static analysis and dynamic analysis, and that edge is what makes contractors valuable

3. Humans focus on design flaws and weird hardware bugs like cryptographic side-channels from electromagnetic emanations

Turns out finding all the bugs is really hard. Codebases and compiler output have exploded in complexity over 20 years which has not helped the static analysis vision. Todays mitigations are fantastic compared to then, but just this month a second 0day chain got patched on one of the best platforms for hardware mitigations.

I think LLMs get us meaningfully closer to what I thought this work already was when I was 18 and didn't know anything.

by spr-alex

3/30/2026 at 8:46:57 PM

lots of security issues form at the boundaries between packages, zones, services, sessions, etc. Static analysis could but doesn't seem to catch this stuff from my perspective. Bugs are often chains and that requires a lot of creativity, planning etc

consider logic errors and race conditions. Its surely not impossible for llm to find these, but it seems likely that you'll need to step throught the program control flow in order to reveal a lot of these interactions.

I feel like people consider LLM as free since there isn't as much hand-on-keyboard. I kinda disgree, and when the cost of paying out these vulns falls, I feel like nobody is gonna wanna eat the token spend. Plenty of hackers already use ai in their workflows, even then it is a LOT OF WORK.

by cartoonworld

3/30/2026 at 8:57:50 PM

Catching all bugs with static analysis would involve solving the halting problem, so it's never going to happen.

by Legend2440

3/30/2026 at 9:34:18 PM

A lot of software doing useful work halts pretty trivialy, consuming inputs and doing bounded computation on each of them. You're not going to recurse much in click handlers or keep making larger requests to handle the current one.

by IsTom

3/31/2026 at 12:21:51 AM

I was just very naive at 18 about program analysis. I haven't lost my imagination though. I was a self-taught IOI gold division competitor. I thought every problem had an algorithm. It doesn't work like that. Program analysis is collecting special snowflakes that melt in your hand. There is no end to the ways you can write a bug in C. Ghosts of Semmle, Semgrep, Coccinelle past, be humbled. LLMs saturate test coverage in a way no sane human would. I do not think they can catch all bugs because of the state space explosion though, but they will help all programmers get better testing. At the end of the day I believe language choice can obviate security bugs, and C/C++ is not easy or simple to secure.

by spr-alex

3/31/2026 at 8:26:38 AM

If you start with safety in mind and don't just try to bolt it on, you're in a much better place. With the kind of code you need in typical applications you could force vast majority of it in a shape that passes termination checks in theorem provers without much overhead, especially if you can just put gnarly things in standard libarary and validate (with proofs hopefully) once.

Though starting with C/C++ is a losing proposition in that regard. And I guess any kind of discipline loses to just throwing half-baked javascript at wall, because deadlines don't care about bugs.

by IsTom

3/31/2026 at 2:56:35 AM

You've never seen the full power of static analysis, dynamic analysis, and test generation. The best examples were always silo'd, academic codebases. If they were combined, and matured, the results would be amazing. I wanted to do that back when I was in INFOSEC.

That doesn't even account for lightweight, formal methods. SPARK Ada, Jahob verification system with its many solvers, Design ny Contract, LLM's spitting this stuff out from human descriptions, type systems like Rust's, etc. Speed run (AI) producing those with unsafe stuff checked by the combo of tools I already described.

by nickpsecurity

3/31/2026 at 8:23:44 AM

Silo’d, academic codebases are not under the kind of attacks that commodity software is

by saagarjha

3/31/2026 at 5:23:37 PM

The silo'd codebases I was referring to are verification tools they produce. They're used to prevent attacks. Each tool has one or more capabilities others lack. If combined, they'd catch many problems.

Examples: KLEE test generator; combinatorial or path-bases testing; CPAChecker; race detectors for concurrency; SIF information flow control; symbolic execution; Why3 verifier which commercial tools already build on.

by nickpsecurity

3/31/2026 at 8:02:53 AM

Only if you're using a Turing-complete programming language, and why would you do that?

by lmm

3/31/2026 at 12:45:20 AM

Catching all bugs with static analysis is actually really easy, as long as you don't mind false positives.

by MeetingsBrowser

3/31/2026 at 1:00:39 AM

Conventional static analysis tools come nowhere close to catching all bugs, even accounting for the false positives.

by tptacek

3/31/2026 at 12:01:17 PM

Sorry, it was supposed to be a joke.

If everything is reported as a bug, there will be 0 false negatives but a lot of false positives

by MeetingsBrowser

3/30/2026 at 10:25:52 PM

A hard read for a skeptic like me. A lot of speculation and extrapolation of a trend, not to say outright exaggeration, but very little actual data. Let's not forget that we're at the tip of an economic bubble, and what you're writing about is at the very center of it!

For what it's worth, I read Anthropic's write-up of their recent 0-day hunt that most of this post seems to be based on, and I can't help but notice that (assuming the documented cases were the most "spectacular") their current models mostly "pattern-matched" their ways towards the exploits; in all documented cases, the actual code analysis failed and the agents redeemed themselves by looking for known-vulnerable patterns they extracted from the change history or common language pitfalls. So, most of the findings, if not all, were results of rescanning the entire codebase for prior art. The corporate approach to security, just a little more automated.

Hence I agree with "the smartest vulnerability researcher" mentioned near the end. Yes, the most impactful vulnerabilities tend to be the boring ones, and catching those fast will make a big difference, but vulnerability research is far from cooked. If anything, it will get much more interesting.

by m132

3/31/2026 at 12:10:51 AM

I tend to be skeptical but listening to the linked podcast with Carlini and found him very credible–not a sales guy, not an AI doomer, but someone talking about how little work he had to do to find real exploits in heavily-fuzzed code. I think there’s still a safe bet that many apps will be cumbersome to attack but I think it’s still going to happen faster than I used to think.

https://securitycryptographywhatever.com/2026/03/25/ai-bug-f...

by acdha

3/31/2026 at 12:19:40 AM

Nicholas Carlini is the real deal. He was most recently on the front page for "How to win a best paper award", about his experience winning a series of awards at Big 4 academic security conferences, mostly recently for work he coauthored with Adi Shamir (I'm just namedropping the obvious name) on stealing the weights from deep neural networks. Before all that (and before he got his doctorate), he and Hans Nielsen wrote the back half of Microcorruption.

He's not a sales guy.

by tptacek

3/31/2026 at 2:38:33 AM

Thanks for having him on. It was really nice to hear a sober, experienced voice talking about their work with fellow practitioners.

by acdha

3/31/2026 at 3:48:12 AM

Thank Nicholas! We'll talk to anyone. :)

by tptacek

3/31/2026 at 1:05:25 AM

Thanks. Watched most of this talk and, unless I missed something, it seems to confirm what I was thinking—most of the strength currently comes from the scale you can deploy LLMs at, not them being better at vulnerability research than humans (if you factor out the throughput). And since this is a relatively new development, nobody really knows right now if this is going to have a greater impact than fuzzers and static analyzers had, or if newer models are ever going to get to a level that'd make computer security a solved problem.

by m132

3/31/2026 at 12:24:42 AM

Theres a video of a recent talk Nicolas Carlini gave this past week on Youtube. It’s eye opening. If you don’t believe that LLMs are going to transform the cybersecurity space after watching that I can’t help you.

by woeirua

3/31/2026 at 12:30:45 AM

It's this talk right here:

https://www.youtube.com/watch?v=1sd26pWhfmg

7 minutes in, he shows the SQLI he found in Ghost (the first sev:hi in the history of the project). If I'd remembered better, I would have mentioned in the post:

* it's a blind SQL injection

* Claude Code wrote an exploit for it. Not a POC. An exploit.

by tptacek

3/31/2026 at 9:03:54 AM

> Not a POC. An exploit.

What's the distinction? A proof of concept is just something that demonstrates that a bug is possible to exploit, by doing so.

by streetfighter64

3/31/2026 at 1:44:52 PM

Repeatability and/or an actual negative effect.

POC generally means “you can demonstrate unintentional behavior”.

“Exploit” means you can gain access or do something malicious.

It’s a fine line. Author’s point is that the LLM was able to demonstrate some malfeasance, not just unintended consequence. That’s a big deal considering that actual malicious intent generally requires more knowhow than raw POC.

by cushychicken

3/31/2026 at 5:37:25 PM

Specifically: the exploit extracted the admin's credentials from the database. A blind SQLI POC would simply demonstrate the existence of a timing channel based on a pathological input.

by tptacek

3/30/2026 at 9:12:59 PM

So the intersting question: are we long term safer with "simpler" closer to hardware memory unsafe(ish) environments like Zig, or is the memory safe but more abstract feature set of languages like Rust still the winning direction?

If a hypothetical build step is "look over this program and carfully examine the bounds of safety using your deep knowledge of the OS, hardware, language and all the tools that come along with it", then a less abstract environment might be at an overall advantage. In a moment, I'll close this comment and go back to writing Rust. But if I had the time (or tooling) to build something in C and test it as thoroughly as say, SQLite [1], then I might think harder about the tradeoffs.

[1] https://sqlite.org/whyc.html

by thadt

3/30/2026 at 10:01:47 PM

What about this article raises this question? If anything, this article makes it pretty clear that memory safe languages are a win. It seems like a serious disadvantage to require a nondeterministic program to evaluate your code's safety.

by love2read

3/31/2026 at 12:50:09 AM

In general I agree and suspect that memory safety is a tool that will continue to pay dividends for some time.

But there are tradeoffs and more ways to write correct and 'safe' code than doing it in a "memory safe" language. If frontier models indeed are a step function in finding vulnerabilities, then they're also a step function in writing safer code. We've been able to write safety critical C code with comprehensive testing for a long time (with SQLite presenting a well known critique of the tradeoffs).

The rub has been that writing full coverage tests, fuzzing, auditing, etc. has been costly. If those costs have changed, then it's an interesting topic to try to undertand how.

by thadt

3/31/2026 at 8:08:34 AM

> If frontier models indeed are a step function in finding vulnerabilities, then they're also a step function in writing safer code. We've been able to write safety critical C code with comprehensive testing for a long time (with SQLite presenting a well known critique of the tradeoffs).

More like: a few people have been able to write C code where the vulnerabilities are obscure enough that we mostly don't discover them very often.

The result of the phenomenon described in the article is that the gap between 99.9% secure and 100% secure just got a whole lot wider. Using herculean amounts of testing and fuzzing to catch most of the holes in a language that lacks secure-by-construction qualities is going to be even less viable going forward.

by lmm

3/31/2026 at 3:01:10 AM

They're great at Python and Javascript which have lots of tooling. My idea was to make X-to-safe-lang translators, X initially being Python and Javascript. Let the tools keep generating what they're good at. The simpler translators make it safe and fast.

If translated to C or Java, we can use decades worth of tools for static analysis and test generation. While in Python and Javascript, it's easier to analyze and live debug by humans.

Multiple wins if the translators can be built.

by nickpsecurity

3/31/2026 at 6:06:54 PM

> My idea was to make X-to-safe-lang translators, X initially being Python and Javascript.

Both of those languages are already safe. Then you talk about translating to C, so you're actually doing a safe-to-unsafe translation. I'm not sure what properties you're checking with the static analysis at that point. I think what would be more important is that your translator maintains safety.

by sfink

3/30/2026 at 8:55:16 PM

Just like how fuzzers will find all the bugs, right? Right?? There's definitely infrastructure at these big companies that isn't sitting in a while loop 'fuzzing' right? Why is it news that vulnerability research will continue to get harder, exactly? It has always been this way, exploits will get more expensive, and the best researchers will continue with whatever tools they find useful.

by narginal

3/30/2026 at 9:01:25 PM

It's a good question. Fuzzers generated a surge of new vulnerabilities, especially after institutional fuzzing clusters got stood up, and after we converged on coverage-guided fuzzers like AFL. We then got to a stable equilibrium, a new floor, such that vulnerability research & discovery doesn't look that drastically different after fuzzing as before.

Two things to notice:

* First, fuzzers also generated and continue to generate large stacks of unverified crashers, such that you can go to archives of syzkaller crashes and find crashers that actually work. My contention is that models are not just going to produce hypothetical vulnerabilities, but also working exploits.

* Second, the mechanism 4.6 and Codex are using to find these vulnerabilities is nothing like that of a fuzzer. A fuzzer doesn't "know" it's found a vulnerability; it's a simple stimulus/response test (sequence goes in, crash does/doesn't come out). Most crashers aren't exploitable.

Models can use fuzzers to find stuff, and I'm surprised that (at least for Anthropic's Red Team) that's not how they're doing it yet. But at least as I understand it, that's generally not what they're doing. It something much closer to static analysis.

by tptacek

3/30/2026 at 9:04:13 PM

I suspect we'll see combinations of symbolic execution + fuzzing as contextual inputs to LLMs, with LLMs delegating highly directed tasks to these external tools that are radically faster at exploring a space with the LLM guiding based on its own semantic understanding of the code.

I'm with you, I expected this to be happening already. Funny enough, I guess even a hardened codebase isn't at that level of "we need to optimize this" currently so you can just throw tokens at the problem.

by staticassertion

3/30/2026 at 9:05:52 PM

Right, so that's exactly how I was thinking about it before I talked to Carlini. Then I talked to Carlini for the SCW podcast. Then I wrote this piece.

I don't know that I'm ready to say that the frontier of vulnerability research with agents is modeling, fuzzing, and analysis (orchestrated by an agent). It may very well be that the models themselves stay ahead of this for quite some time.

That would be a super interesting result, and it's the result I'm writing about here.

by tptacek

3/30/2026 at 9:08:42 PM

I have just seen too much infrastructure set up to 'find bugs,' effectively sitting and doing nothing- either the wrong thing gets audited, or tons of compute gets thrown at a code base and nobody ever checks in on or verifies.

This seems like a human/structural issue that an AI won't actually fix - attackers/defenders alike will gain access to the same models, feels a little bit like we are back to square one

by narginal

3/30/2026 at 9:11:32 PM

If that's true, and if patches can effectively be pushed out quickly, then the results of this will be felt mostly by vulnerability researchers, which is the subject of the piece. But those are big "ifs".

by tptacek

3/31/2026 at 5:51:28 AM

The other thing is to remember is that when it comes to complex targets, attacks still are found by using a different fuzzer and/or targeting a different entry point.

It stands to reason that the same will apply for LLMs.

by gsnedders

3/30/2026 at 9:45:36 PM

If everyone is running the same models, does this not favour white hat / defense?

Since many exploits consists of several vulnerabilities used in a chain, if a LLM finds one in the middle and it's fixed, that can change a zero day to something of more moderate severity?

E.g. someone finds a zero day that's using three vulns through different layers. The first and third are super hard to find, but the second is of moderate difficulty.

Automated checks by not even SOTA models could very well find the moderate difficulty vuln in the middle, breaking the chain.

by vibe42

3/31/2026 at 12:49:55 AM

> If everyone is running the same models, does this not favour white hat / defense?

The landscape is turbulent (so this comment might be outdated by the time I submit it), but one thing I’m catching between the lines is a resistance to provide defensive coding patterns because (guessing) they make the flaw they’re defending against obvious. When the flaw is widespread - those patterns effectively make it cheap to attack for observant eyes.

After seeing the enhanced capabilities recently, my conspiracy theory is that models do indeed traverse the pathways containing ideal mitigations, but they fall back to common anti-patterns when they hit the guardrails. Some of the things I’ve seen are baffling, and registered as adversarial on my radar.

by dumpsterdiver

3/30/2026 at 10:42:10 PM

I think our saving grace is the fact that, especially in these hardened environments, it's a lot easier to find exploitable bugs than to actually exploit them.

As the defenders will have access to the same agents as the attackers, everybody will (mostly) find the same bugs. If recent trends continue[1], it's likely that major labs will make new models available to defenders first, making the attackers' jobs even harder.

What really worries me is models quickly developing exploits based on freshly-released patches, before most people had a chance to update. Big cloud vendors will likely have the ability to coordinate and deploy updates before the commits hit Github, smaller enterprise on-prem environments won't have that luxury.

by miki123211

3/31/2026 at 8:48:14 AM

As many are saying in the comments... I understand the point of view but if this is all true, why so many open source projects are giving up because of the staggering amount of fake/useless reports.

This is the situation where a software will be pre-patched until bloated for no particular reason... I still think this will happen, but the eventual findings of real security issues at the moment is not really proof it works. A 1/50 real Vs. false is not acceptable even if the plan is to solve with llms.

by motbus3

3/30/2026 at 8:20:44 PM

I'm suspicious of this prediction given the curl project's experiences...

by nitros

3/30/2026 at 8:30:20 PM

The people spamming curl did step one, "write me a vulnerability report on X" but skipped step two, "verify for me that it's actually exploitable". Tack on a step three where a reasonably educated user in the field of security research does a sanity check on the vulnerability implementation as well and you'll have a pipeline that doesn't generate a ton of false positives. The question then will rather be how cost-effective it is for the tokens and the still-non-zero human time involved.

by jerf

3/30/2026 at 8:21:47 PM

Everybody agrees that idiots were spamming curl with random just-plausible-enough-seeming output from old models.

by tptacek

3/30/2026 at 8:32:30 PM

It sounds like what makes the pipeline in the article effective is the second stage, which takes in the vulnerability reports produced by the first level and confirms or rejects them. The article doesn't say what the rejection rate is there.

I don't think the spammers would think to write the second layer, they would most likely pipe the first layer (a more naive version of it too, probably) directly to the issue feed.

by tomjakubowski

3/30/2026 at 8:42:13 PM

There are at least three differences:

* Carlini's team used new frontier models that have gotten materially better at finding vulnerabilities (talk to vulnerability researchers outside the frontier labs, they'll echo that). Stenberg was getting random slop from people using random models.

* Carlini's process is iterated exhaustively over the whole codebase; he's not starting with a repo and just saying "find me an awesome bug" and taking that and only that forward in the process.

* And then yes, Carlini is qualifying the first-pass findings with a second pass.

by tptacek

3/31/2026 at 6:36:50 PM

Thanks, I hadn't considered the second point.

I guess the broader point I wanted to make is about the people responsible for the deluge of LLM-reported bugs and security vulnerabilities on countless open-source projects (not only on curl): they weren't considerate or thoughtful security researchers, they were spammers looking to raise their profile with fully automated, hands-off open source "contributions". I would expect that the spammers would continue to use whatever lowest common denominator tooling is available, and continue to cause these headaches for maintainers.

That doesn't mean frontier models and tooling built around them aren't genuinely useful to people doing serious security research: that does seem to be the case, and I'm glad for it.

by tomjakubowski

3/30/2026 at 8:33:26 PM

That was then, this is now. The new models are scarily good. If you're skeptical, just take an hour to replicate the strategy the article references. Point Claude at any open-source codebase you find interesting and instruct it to find exploitable vulnerabilities. Give it a well-defined endpoint if you want (e.g., "You must develop a Python script that triggers memory corruption via a crafted request") and see how well it does.

by MajesticHobo2

3/30/2026 at 9:20:03 PM

> That was then, this is now.

No, what we were seeing with curl was script kiddies. It wasn't about the quality of the models at all. They were not filtering their results for validity.

by phyzome

3/30/2026 at 10:02:03 PM

It was definitely partially about model quality. The frontier models are capable of producing valid findings with (reasonably) complex exploit chains on the first pass (or with limited nudging) and are much less prone to making up the kinds of nonsensical reports that were submitted to curl. Compared to now, the old models essentially didn't work for security.

If those script kiddies had been using today's models instead and _still_ didn't do any filtering, a lot more of those bugs would have been true positives.

by MajesticHobo2

3/31/2026 at 5:54:21 AM

To some extent, that’s just the nature of the incentive structure — you pay money for exploits with no penalty for false reports? Just rely on monkeys typing reports till one happens to be correct.

by gsnedders

3/30/2026 at 8:27:08 PM

The problem is that you have all kind of "security spam" in the same way that social media is flooded by automatic, but on-topic, content. This doesn't mean that some very few reports are not correct.

One way to filter that out could be to receive the PoC of the exploit, and test it in some sandbox. I think what XBOW and others are doing is real.

by wslh

3/31/2026 at 9:09:29 AM

'Substantial amounts of high-impact vulnerability research (maybe even most of it) will happen simply by pointing an agent at a source tree and typing “find me zero days”'

Not quite... what is forgotten here is that the developers themselves, with equal ease, _also_ will point agents at the source tree and will type "find me zero days".

by abc123abc123

3/31/2026 at 9:02:17 AM

I can't fully agree with the author's worries about the legislation question. Even if legislation were to forbid using LLMs for vulnerability reports at first, they would quickly learn their lesson and reverse the policies. Not even a politician can fight against real world results.

by siruwastaken

3/30/2026 at 9:47:23 PM

Ya, I tend to believe that (most) human VR will be obsoleted well before human software engineering. Software engineering is a lot more squishy and has many more opportunities to go off the rails. Once a goal is established, the output of VR agents is verifiable.

by anematode

3/30/2026 at 8:53:59 PM

I was distracted by the picture of the ingredients to a Final Ward being at the top of the page.

by rubiquity

3/30/2026 at 8:50:10 PM

> Everything is up in the air. The industry is sold on memory-safe software, but the shift is slow going. We’ve bought time with sandboxing and attack surface restriction. How well will these countermeasures hold up? A 4 layer system of sandboxes, kernels, hypervisors, and IPC schemes are, to an agent, an iterated version of the same problem. Agents will generate full-chain exploits, and they will do so soon.

I think this is the interesting bit. We have some insanely powerful isolation technology and mitigations. I can put a webassembly program into a seccomp'd wrapper in an unprivileged user into a stripped down Linux environment inside of Firecracker. An attacker breaking out of that feels like science fiction to me. An LLM could do it but I think "one shots" for this sort of attack are extremely unlikely today. The LLM will need to find a wasm escape, then a Linux LPE that's reachable from an unprivileged user with a seccomp filter, then once they have kernel control they'll need to manipulate the VM state or attack KVM directly.

A human being doing those things is hard to imagine. Exploitation of Firecracker is, from my view, extremely difficult. The bug density is very low - code quality is high and mitigation adoption is a serious hurdle.

Obviously people aren't just going to deploy software the way I'm suggesting, but even just "I use AWS Fargate" is a crazy barrier that I'm skeptical an LLM will cross.

> Meanwhile, no defense looks flimsier now than closed source code.

Interesting, I've had sort of the opposite view. Giving an LLM direct access to the semantic information of your program, the comments, etc, feels like it's just handing massive amounts of context over. With decompilation I think there's a higher risk of it missing the intention of the code.

edit: I want to also note that with LLMs I have been able to do sort of insane things. A little side project I have uses iframe sandboxing insanely aggressively. Most of my 3rd party dependencies are injected into an iframe, and the content is rendered in that iframe. It can communicate to the parent over a restricted MessageChannel. For cases like "render markdown" I can even leverage a total-blocking CSP within the sandbox. Writing this by hand would be silly, I can't do it - it's like building an RPC for every library I use. "Resize the window" or "User clicked this link" etc all have to be written individually. But with an LLM I'm getting sort of silly levels of safety here - Chrome is free to move each iframe into its own process, I get isolated origins, I'm immune from supply chain vulnerabilities, I'm immune to mostly immune to XSS (within the frame, where most of the opportunity is) and CSRF is radically harder, etc. LLMs have made adoption of Trusted Types and other mitigations insanely easy for me and, IMO, these sorts of mitigations are more effective at preventing attacks than LLMs will be at finding bypasses (contentious and platform dependent though!). I suppose this doesn't have any bearing on the direct position of the blog post, which is scoped to the new role for vulnerability research, but I guess my interest is obviously going to be more defense oriented as that's where I live :)

by staticassertion

3/30/2026 at 8:54:52 PM

> With decompilation I think there's a higher risk of it missing the intention of the code.

I'm not sure but suspect the lack of comments and documentation might be an advantage to LLMs for this use case. For security/reverse engineering work, the code's actual behavior matters a lot more than the developer's intention.

by MajesticHobo2

3/30/2026 at 8:58:04 PM

I think the other side of that is that mismatches between intention and implementation are exactly where you're going to find vulnerabilities. The LLM that looks at closed source code has to guess the intention to a greater degree.

by staticassertion

3/30/2026 at 9:16:01 PM

This is true for a lot of things but for low-level code you can always fall back to "the intention is to not violate memory safety".

by moyix

3/30/2026 at 9:20:00 PM

That's true, but certainly that's limiting. Still, even then, `# SAFETY:` comments seem extremely helpful. "For every `unsafe`, determine its implied or stated safety contract, then build a suite of adversarial tests to verify or break those contracts" feels like a great way to get going.

by staticassertion

3/30/2026 at 9:31:50 PM

It's limiting from the PoV of a developer who wants to ensure that their own code is free of all security issues. It is not limiting from the point of view of an attacker who just needs one good memory safety vuln to win.

by moyix

3/30/2026 at 9:30:59 PM

The pipewire-libs package local address function refers to alsa_output.pci

Driver benchmarking the pipewire script calls three local ports:

local.source.port = 10001

local.repair.port = 10002

local.control.port = 10003

by rkrbaccord94f

3/30/2026 at 10:18:24 PM

> The new models find real stuff. Forget the slop; will projects be able to keep up with a steady feed of verified, reproducible, reliably-exploitable sev:hi vulnerabilities?

If LLMs are as capable as said in the article, there will be an initial wave of security vulnerabilities. But then, all vulnerabilities will be discovered (or at least, LLMs will not find any more), and only new code will introduce new vulnerabilities. And everyone will be using LLMs to check the new code. So, regardless of what they say is correct or not, the problem doesn't really exist.

by GTP

3/30/2026 at 8:26:51 PM

> Now consider the poor open source developers who, for the last 18 months, have complained about a torrent of slop vulnerability reports. I’d had mixed sympathies, but the complaints were at least empirically correct. That could change real fast. The new models find real stuff.

The slop reports won't stop just because real ones are coming in. If the author's right, open source maintainers will still will have to deal with the torrent of slop: on top of triaging and identifying the legit vulnerabilities. Obviously, this is just another role for AI models to fill.

by tomjakubowski

3/30/2026 at 9:08:38 PM

I was doing TryHackMe's "advent of cyber" sidequest last christmas and used a process very much like Carlini's that is outlined in TFA.

>I'm doing a CTF. I popped a shell on this box and found this binary. Here is a ghidra decompilation. Is there anything exploitable in $function?

You can't just ask Claude or ChatGPT to do the binex for you, but even last year's models were really good at finding heap or stack vulns this way.

by stackghost

3/31/2026 at 6:18:11 AM

If this is true, beside state actors who did this from the beginning, there is probably a race between criminal organizations to develop hacking tools and frameworks to efficiently extract and exploit as much data as possible.

by DeathArrow

3/30/2026 at 11:56:10 PM

The token cost will be a limiting factor.

by jazz9k

3/31/2026 at 12:20:03 AM

Will it? Why do you assume that?

by tptacek

3/30/2026 at 9:19:12 PM

I agree AI makes exploits more accessible, it also makes pen-testing and finding vulns more accessible, in both early and late stages of product development.

AI has saved me a ton of money and time auditing. Mostly because I'm tired / lazy.

It's both a black pill & white pill, and if we have the right discipline, a tremendous white pill. Engineers can no longer claim to be "cost effective" by ignoring vulns.

by tonymet

3/30/2026 at 8:47:43 PM

> Is the Linux KVM hypervisor connected to the hrtimer subsystem, workqueue, or perf_event? The model knows.

I asked ChatGPT and it claimed "all three". Any linux wizards who can confirm or deny?

Anyway, in my experience using mainly the Claude chat to do some basic (not security) bug hunting, it usually fixates on one specific hypothesis, and it takes some effort to get it off that wrong track, even when I already know it's barking up the wrong tree.

by streetfighter64

3/30/2026 at 10:03:20 PM

The wonderful thing though is that you can just run the model multiple times (even in parallel). Some instances might get stuck but as long as some find the bug and you have a good way to filter outputs (e.g. with another llm that tries to create concrete exploits) even a very small success rate on stage 1 can lead to reliable exploits

by jgeralnik

3/30/2026 at 8:51:52 PM

It's all three, I just had it on the brain when I was writing this.

by tptacek

3/30/2026 at 8:53:43 PM

Hm, kind of a strange question then, no? Is a car's engine connected to the fuel tank, the wheels or the accelerator pedal?

by streetfighter64

3/30/2026 at 9:03:26 PM

I don't know, maybe it is? My point is just that frontier models start off with latent models of all the interconnectivity in all the important open-source codebases, to a degree that would be infeasible for the people who learned how all the CSS object lifecycles and image rendering and unicode shaping stuff worked well enough to use them in exploits.

by tptacek

3/31/2026 at 7:24:24 AM

> I think this outcome is locked in. That we’re starting to see its first clear indications.

Hardly. The linked anthropic paper is extremely underwhelming. It portrays no tectonic shifts.

> Practitioners will suffer having to learn the anatomy of the font gland or the Unicode text shaping lobe or whatever other “weird machines” are au courant

That's absurd. Do the vulnerability writers _start_ with this knowledge? Of course they don't. They work backwards. Anyone can do this. It just takes time in a category of development that most open source authors don't like to be occupied by.

> You can’t design a better problem for an LLM agent than exploitation research.

Did you read the anthropic article you linked? It found absolutely nothing and then immediately devolved into a search for 'strcat.' That's it. Again, literally _anyone with the time_ could just do this.

> a frontier LLM already encodes supernatural amounts of correlation across vast bodies of source code.

'grep strcat' is "supernatural?"

This starts sprawling very quickly after this. The AI revolution is not real. The cargo cult is headed for a new winter. I only see articles proclaiming the sky is just about to fall any day now, yet, I see no real world evidence any such thing is happening or likely to happen.

by themafia

3/30/2026 at 8:17:44 PM

[dead]

by ronnier

3/30/2026 at 8:30:57 PM

Another boring AI hype article.

“The next model will be the one. Trust me. Just one more iteration.”

by badgersnake

3/30/2026 at 9:33:20 PM

LLMs are expert hackers because: 1) They are expert coders, including a decently comprehensive CVE knowledge 2) They know every programming language/framework/stack 3) They know every human language

They already have super human breadth and attention. And their depth is either super human or getting there.

The state of the security industry through 2025 was expensive appsec human reviewers or primitive scanners. Now you can spend a few dollars and have an expert intelligence scrutinize a whole network.

by samuelknight

3/30/2026 at 9:42:14 PM

So much of the current internet is posts that read as a superposition of sincere and parody, and until that's resolved how do you know how to respond?

by gdulli

3/30/2026 at 10:46:25 PM

If that was a jab it my writing then yes, I am absolutely being sincere because I am an expert on this topic. LLMs went from being ok at one-shoting a function a to being so good at hacking that it's difficult to evaluate them. Prospective customers get back to us after a demo and tell us about the exploits it found on their services that are so vague and technical that they wouldn't think to look for them.

by samuelknight

3/31/2026 at 9:05:58 AM

> Prospective customers get back to us after a demo and tell us about the exploits it found on their services that are so vague and technical that they wouldn't think to look for them.

Um, have you actually verified that those are actual exploits then? Vague and technical sounds exactly like a description of AI slop...

by streetfighter64

3/31/2026 at 1:40:26 PM

Yes, that's how they become customers.

by samuelknight

3/30/2026 at 9:59:27 PM

Just wait until you see the same showing up in compliance realms...

Edit: to be slightly less implicit, consider the cargo cult madness that erupts from people thinking they can address risk management and compliance by auto-generating documentation and avoid really doing the legwork.

by saltcured