8/3/2026 at 12:48:29 PM
We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are.LLM-based “AI” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your LLM-based “AI” gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence. What bothers me about this entire situation is that presumably the folks that relied on the LLM-based “AI” to generate these vulnerabilities knew (or should have known) enough about their tool to know this would happen, but did not.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.
by gortok
8/3/2026 at 3:39:50 PM
Many people with no skills are taking advantage of the LLM craze to artificially inflate their own value. I see it every day on LinkedIn.People that previously have barely any experience in tech, now being hired in AI startups because they are good bullshitters.
by elmer2
8/3/2026 at 3:46:37 PM
Countless directors and managers are now cosplaying as engineers. I've seen so many myself and that's just my tiny slice of this engineering worldby prh8
8/3/2026 at 5:22:02 PM
It is pretty funny to see the shoe on the other foot, since it's usually software engineers with unearned arrogance about other fields.by Blackthorn
8/3/2026 at 5:27:10 PM
What fields do you see devs think they know about? I’ve never personally seen this with other devs I work with but obviously small sample size…by ryan_n
8/3/2026 at 7:06:51 PM
In my experience, product. There’s a pervasive view that product is easy, that anyone working in it is a dunderhead who gets in the way of engineers doing the really worthwhile work. In reality, it would just mean they rewrite the software five times and have zero customers.Oh, design too. I’ve met many engineers who simply don’t see any value in design.
by afavour
8/3/2026 at 7:29:18 PM
This. I'm a former teacher. I left to get a PhD in educational psychology, and now I'm a data scientist for a large school division. The number of ed tech companies I interact with who clearly have little grasp of research-based best practices in education is staggering. But they can all build slick-looking websites and dashboards.by ekholm_e
8/4/2026 at 9:30:54 AM
The software is not the product.You can't get a software person to understand this unless they learn the domain they are working in properly.
by Sammi
8/4/2026 at 1:57:56 AM
Literally everything. Just pick a topic and you'll see SWEs on Hacker News prognosticating about it as if they knew everything while knowing nothing but the surface.by Blackthorn
8/4/2026 at 8:34:54 AM
Internet discussion != real-world professional fakeryby lenkite
8/4/2026 at 5:39:38 PM
Devs like to pretend they know everything about everything.Start a discussion about any topic, look at how many nerds will tell you that you are wrong.
by rolymath
8/4/2026 at 5:19:30 PM
If HN is any indication, many devs have hilariously naive beliefs regarding economics and philosophy and will absolutely refuse to engage with actual arguments or experts within those fields.by emp17344
8/3/2026 at 5:46:38 PM
find any econ related post on hn et voilaby mathisfun123
8/3/2026 at 6:08:42 PM
I don't think it is much of a problem that dabblers get into Internet arguments about things they are not experts in. After all the easiest way to get a correct answer/learn something is to say something wrong.Of course this should be differentiated from a professional setting.
And under no circumstances should we put that stuff into the knowledge bases we base our "AI" on without making sure it takes into account the context.
by luke5441
8/3/2026 at 6:18:21 PM
There have always been people with no tech experience building startups generating spaghetti code. It's just that now it looks a lot more polished.by hparadiz
8/3/2026 at 6:21:35 PM
> It's just that now it looks a lot more polished.Yes, and that is the problem. It used to be if a product looked polished it was fairly polished engineering wise as well if we compare to todays AI slop. You can see that on steam, before AI slop games a game that looked polished mostly worked. But today you can get a game that looks like they put in a lot of effort, but its all AI slop and everything is flaky and broken, I've had way more such experiences the last few months than before that.
A human coder that was capable of coding a complex game typically also was good enough and tested things to make most things work. There were bugs etc, but at least buttons did things.
by Jensson
8/3/2026 at 5:01:55 PM
Engineers cosplay as physicists and mathematicians every day. What's your point? Think of it pragmatically. If they can do the job they can do the role.by 1-6
8/3/2026 at 5:27:04 PM
> If they can do the job they can do the role.Obviously.
Can they do the job? Because right now, government decisions are based on AI generated code, which was verified by nobody who can do that. So the cost of an unsatisfactory answer is quite high.
by ruszki
8/3/2026 at 3:59:18 PM
To be fair, "people with no skills inflating their own value" is what LinkedIn has always been like. But I guess LLMs are uniquely well positioned for that task.by eli
8/3/2026 at 8:04:01 PM
LLMs have brought it to a new level. I have multiple people in my feed posting several page dissertations on a daily basis. More often than not, these posts are about the benefits of AI...by icedchai
8/3/2026 at 5:52:38 PM
People who live and create on LinkedIn? absolutely. For a huge group of people LI was just the most popular job board, though.by skeeter2020
8/3/2026 at 4:11:35 PM
I heard the term "Sloperator" a few times. Quite fitting...by Joe_Cool
8/4/2026 at 8:04:27 AM
Thanks!That's definitely getting added to my vocabulary.by glitchcrab
8/3/2026 at 3:59:16 PM
This isn't new due to AI, just amplified.by Reptur
8/3/2026 at 8:23:16 PM
> People that previously have barely any experience in tech, now being hired in AI startups because they are good bullshitters.I like the symmetry of people being unable to detect "skill" in people who are great at manipulating language and in LLMs who are also great at manipulating language.
by e12e
8/4/2026 at 12:06:09 AM
I mean most startups and every billion dollar company are worth a lot because they are good bullshitters, in a way its kinda refreshing to see more people able to bullshit their way up a different ladder, even if it is a bit scary.by abustamam
8/3/2026 at 2:25:54 PM
Unfortunately people sometimes get defensive against this take. But I think treating the LLM as you described can make you a better LLM user and help get better output. It helps understand the failure modes better, and moderate one’s reliance on them. Just like how we should do for every tool we work with.by geraneum
8/3/2026 at 2:34:22 PM
Yes, I've found that reminding yourself of how they actually work helps keep you on guard against LLM-patterned mistakes. Especially things like carefully considering what parts of the current task likely fall outside the distribution of corpus + RL data (as much as that can be guessed).by gr_norm
8/3/2026 at 5:00:33 PM
> ...Unfortunately people sometimes get defensive against this take. But I think treating the LLM as you described can make you a better LLM user and help get better output. It helps understand the failure modes better, and moderate one’s reliance on them. Just like how we should do for every tool we work with.B...b...but the Anthropic trainer said we'd get the best results if we don't think of it as a tool, but instead give it a name and think of it as our brilliant coworker!
Why should I trust you, internet rando over a stormtrooper-level salesman? /s
by palmotea
8/3/2026 at 7:03:02 PM
> A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.The sort of person that's going to offload their thinking to AI is the exact sort of person that is not going to verify anything because they've already offloaded their thinking to AI.
by bigbuppo
8/3/2026 at 7:07:01 PM
For a long time I was anti-licensure in tech; now with the bar being lowered to next to nothing, it seems as if licensure is more important than ever — not to protect this trade (though it will do that, and that is a benefit), but because the sheer amount of irresponsibility in the usage of LLMs and “AI” in general begs for licensure and adoption of a regulatory body for software in general.by gortok
8/3/2026 at 8:50:28 PM
This is unfortunately a feeling I share. It wasn't until LLMs have become nearly ubiquitous at this point, and there has been zero realistic technological response to the dangers they present. Not to mention I suspect there may be some psychological element to being exposed to interactions with AI models and their nonsense for hours a day. Not all of it is nonsense....but you won't ever know for sure.by rafterydj
8/3/2026 at 10:35:09 PM
100%. I don't think "AI" has changed anything wrt to the responsibility of the tech industry in general. Tech has always been pretty much devoid of ethics or a sense of responsibility at the executive level (and therefore "management" more broadly). But before LLMs it was easier for ethical engineers to surreptitiously steer things towards responsible implementations. I suspect also competence correlates pretty strongly with intellectual honesty--narcissists and other personality defectives generally aren't terribly capable of the kind of introspection necessary to learn deeply. With the deluge of LLM slop, it's much harder for even a responsible manager to differentiate the competent from the incompetent, and now the dishonest, unethical folks have more leverage.So now the problem is highly visible. There's only one solution I can think of, and that's to hold individuals (not corporations) responsible for professional malfeasance.
by 27183
8/4/2026 at 1:53:27 AM
But as you've made clear, the root of the problem is not with the engineer.Personally I think a corporate death penalty strikes a nice balance of holding companies and individuals responsible. If a company exercises sufficient malfeasance, it gets nationalized with no compensation for shareholders. Watch the incentives align real fast.
by andrewflnr
8/4/2026 at 2:12:55 AM
Yeah that sounds great! A company guillotine is absolutely something I could get behind. Problem is I don't think there's any way it could possibly happen politically. Whereas we have a really strong existence proof for e.g. the Professional Engineer certification and other professional coherence mechanisms. That seems to work, albeit not without its own inefficiencies. I doubt anyone, though, would advocate for some kind of laissez-faire situation where the profit incentive aligns with the regulatory one. Historically that hasn't gone well. A lot of people died.by 27183
8/4/2026 at 5:14:36 AM
PEs are demonstrably a partial solution at best. Fields with certified engineers still have financially motivated disasters (boeing goes the door plug).For better or worse, I think we'll see more political support for increased oversight of companies on the next free decades. I only hope it manifests in a useful way instead of a self-defeating way.
by andrewflnr
8/4/2026 at 5:17:45 AM
It'll have to be pretty much all the ways. One way will stick. The rest will be forgotten failures.by 27183
8/4/2026 at 5:43:43 AM
I'm afraid this is a situation where the first halfway plausible solution will "stick" and sap the pressure to change even if it doesn't do a good job.by andrewflnr
8/3/2026 at 4:06:02 PM
> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.Brandolini's principle in action. It takes 10 times more energy to refute BS than to generate it. A related analogy to computing: it is easy to generate propositions, but hard to test if a given proposition is satisfiable or not, which curiously ties to P vs NP.
by bwfan123
8/3/2026 at 4:38:48 PM
> Brandolini's principleI much prefer the alternative name: the Bullshit Asymmetry Principle.
by Sohcahtoa82
8/3/2026 at 5:19:53 PM
Seems like most of it is covered by: Entropy increases.by Jblx2
8/3/2026 at 2:54:00 PM
Apophatic intelligence? "We don't know what intelligence is, but LLMs with CoT are certainly not it despite being Turing-complete."Watching for unexpected failure modes is surely worth it.
by red75prime
8/3/2026 at 3:14:27 PM
Turing-completeness is a necessary pre-requisite for being able to fulfill the requirements of a Turing machine, nothing more. In the same way that cell division is a necessary condition for life, but cell division does not mean a given life form itself is sentient.Intelligent life-forms can generate probabilistic outputs based on inputs, but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.
by gortok
8/3/2026 at 3:34:48 PM
OK. A more pointed question. What do you know about intelligence that allows you to exclude LLMs with CoT from the category of intelligent systems with certainty?by red75prime
8/3/2026 at 4:06:40 PM
How do you know there's not a teapot orbiting the sun?by buttercraft
8/3/2026 at 5:14:05 PM
To my knowledge there are many millions of teapots orbiting the sun.by Ma8ee
8/3/2026 at 6:38:33 PM
No, they're all orbiting the Earth very badly. :pby Terr_
8/3/2026 at 4:53:36 PM
There is no reason to think that teapots were sent into orbit or spontaneously formed there.Likewise, there is no reason to think the brain employs super-Turing or quantum computations that cannot be approximated by LLMs.
by red75prime
8/3/2026 at 4:54:48 PM
The point is that you're asking someone to prove a negative.by buttercraft
8/3/2026 at 6:22:57 PM
Right, it follows naturally that one can never meaningfully discuss that which can only be settled by proving negatives. Hence the total lack of debate on religious topics, as everyone just gave up 1500 years ago and decided that God indeed may or may not exist and that's that.by vouwfietsman
8/3/2026 at 5:05:03 PM
It's a negative in your worldview where "intelligence can't be a computation" is an axiom or something close. To me this worldview is sloppy thinking.by red75prime
8/3/2026 at 7:25:34 PM
I did not express my opinion or "worldview." I was simply responding to this:> exclude LLMs with CoT from the category of intelligent systems with certainty
by buttercraft
8/3/2026 at 8:06:51 PM
Why do you think that intelligence of a Turing-complete system trained on human data can be likened to a Russell's teapot?At least, don't you think that the recent mathematical results of LLMs are a bit like a glimpse of something teapot-shaped in the orbit? (which makes it not a Russell's teapot, which. by definition, can't be observed).
To me, it's an expected progression of ANNs' approximation of human cognitive processes. The universal approximation theorem guaranties the existence of such ANNs barring the super-Turing or quantum superiority of the brain.
by red75prime
8/3/2026 at 9:12:08 PM
> Why do you think that intelligence of a Turing-complete system trained on human data can be likened to a Russell's teapot?I don't? You are presenting opinions as if they are mine, but they are not.
by buttercraft
8/3/2026 at 11:06:52 PM
I'm trying to understand what's going on in the head of people who are certain that LLMs can't be intelligent. And it proves to be a daunting task. Have you tried to reflect on your certainty? Where it comes from?by red75prime
8/4/2026 at 5:24:37 PM
We have reason to believe consciousness cannot be replicated via computation, and reason to suspect consciousness is an important aspect of cognition. Now, what reason do you have for believing intelligence could be replicated computationally?by emp17344
8/3/2026 at 4:30:33 PM
You’re right to point that out. The load-bearing seam of your logical retort is the hidden assumption that the teapot is in outer space, not at peace on a kitchen stovetop here on earth. I am sorry I had not considered this fact. \sby tired-turtle
8/3/2026 at 5:00:25 PM
They were modeled after us, which almost certainly dooms them to stupidity.They could have been great, if trained on datasets from a more sensible species.
by pyrale
8/3/2026 at 3:31:27 PM
> but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.???
Of course it is. The brain is mechanically not capable of doing anything other than that.
Do you believe the brain is something other than a bundle of probabilistic physical interactions? Or are brains not the source of what we call intelligence?
by estearum
8/3/2026 at 5:34:42 PM
May I suggest the excellent & recent book: "The brain. In theory" by Romain Brette?This is going to elevate your thinking on this no end, if you're interested.
by polotics
8/3/2026 at 5:45:51 PM
Looks interesting (added to my list, thank you for the recommendation), but not actually relevant to the topic at hand.We know that the brain is a probabilistic input → output machine because the universe is a probabilistic input → output machine. The brain is made of universe. There are deterministic relationships (which at high sensitivity or complexity become easier to describe as probabilistic), and quantum relationships. That's it. The brain, like every other thing comprised of "universe" is comprised of those two types of relationships.
If Romain's book provides evidence of relationships in the brain that are neither quantum (therefore random) NOR classical (therefore deterministic), then 1) he would have already won at least one Nobel prize, and 2) anyone in this thread would be able to at least gesture toward what relationship that is.
by estearum
8/3/2026 at 11:25:05 PM
This is a vacuous argument. If everything is a “probabilistic input output machine” then what exactly are you claiming?You should look into mathematical definitions of probability. You will be surprised to find not all formal mathematical systems can fit inside probability spaces.
by thesmtsolver2
8/3/2026 at 11:44:44 PM
Formal mathematical systems are not physical objects. The brain – which is presumably the source of natural intelligence – is.By "everything" I was referring (obviously) to things in the universe. Mathematical systems are not things in the universe. They are not made of universe the way every object, including a brain, is.
And no, it's not vacuous at all. GP is trying to argue that a probabilistic input → output machine doesn't produce intelligence. But we have an existence proof of exactly that!
by estearum
8/3/2026 at 6:16:17 PM
You're a fast reader :-DNo quantum bullshittery in there I promise.
Equating "classical" with "deterministic" is however pushing it a bit too far, when no one and nothing can ever do even a very small fraction of said determination...
by polotics
8/3/2026 at 7:31:31 PM
Quantum is not bullshit and yes, classical is deterministic.That's why the brain cannot possibly be anything other than an input → output machine, which is functionally deterministic (with maybe some fully random components), but is easiest to describe as probabilistic.
In the same way that LLMs are functionally deterministic, but easiest to describe as probabilistic.
by estearum
8/3/2026 at 8:43:41 PM
We don’t knowby noncoml
8/3/2026 at 8:47:24 PM
Of course we do. Unless the brain is an object unlike anything else in the known universe, then it's an input → output machine.by estearum
8/4/2026 at 1:51:51 AM
Not what what the parent post claimed.by noncoml
8/4/2026 at 2:22:45 AM
The brain is the source of intelligenceThe brain solely does probabilistic conversion of inputs to outputs
Therefore the thing that makes us intelligent is our probabilistic conversion of inputs to outputs
Ergo GP's assertion is wrong
by estearum
8/3/2026 at 4:11:50 PM
Why is the brain probabilistic instead of deterministic?by beepbooptheory
8/3/2026 at 5:11:44 PM
Probabilistic and deterministic are not opposites.The brain is deterministic at the level of specific interactions, which process inputs in a highly chaotic (but still deterministic) manner into probabilistic outcomes.
The opposite of deterministic is random, i.e. in the quantum sense of truly no relationship between input and output.
There are probably some quantum effects in the brain here and there, but the vast majority of it is just traditional deterministic interactions networked together in such a complex system that the resulting behavior is much, much easier to predict in probabilistic terms than otherwise.
by estearum
8/3/2026 at 5:33:57 PM
Ok right, but then I don't see the original point. Or like, you don't seem to be appealing to a hard science/soft science distinction here if the question revolves around what conceptual model is most "useful" to us. Useful for what? For predicting future actions? That makes sense, but does actually that speak to op's (possibly still fraught) metaphysical/semantic point here? It just feels like you are speaking over it, right? They could just as easily counter that a probabilistic model of the physical brain is not a very useful description of human intelligence, so why use it? You can't appeal to materialism here because you are already practicing some implied pragmatism in your commitments. You pretty much have as much ground to stand on as they do!by beepbooptheory
8/3/2026 at 5:48:46 PM
I am saying that there is no categorical distinction to be made on functional grounds between artificial intelligence and brains.You cannot say an AI model cannot be intelligent because it's a probability machine, when all available evidence points toward natural intelligence also being generated by probability machines (much more complex ones, called brains).
by estearum
8/3/2026 at 6:49:25 PM
But brains do much more than just predict tokens based on previously seen tokens. I think all the other things brains do are probably important for our intelligence.So, LLM are just next token predictors, brains are next token predictors + many other things in addition, and that makes people still feel LLM are dumb even when they solve a lot of problems using tokens.
by Jensson
8/3/2026 at 7:51:01 PM
> brains are next token predictorsOf the vast uncertainties and philosophical exercises that we must face to bridge the chasm between where we are now, and where we will be when we understand intelligence, I can take comfort in claiming, with 100% accuracy, that our biology is not based on technology invented by Google in 2017.
by vouwfietsman
8/3/2026 at 7:34:52 PM
> But brains do much more than just predict tokens based on previously seen tokens. I think all the other things brains do are probably important for our intelligence.Like what?
What specific biological structure in the brain could be doing anything other than producing output as a function of 1) current electrical/chemical/thermal inputs and 2) previous electrical/chemical/thermal inputs?
by estearum
8/3/2026 at 7:41:44 PM
LLM doesn't just produce output as a function, they are much more specific: they predict text based on text they have been trained on.by Jensson
8/3/2026 at 7:51:43 PM
Yes that is the item (2) parameter described above.by estearum
8/3/2026 at 8:07:04 PM
No, brains doesn't just try to mimic pasts signals, LLMs do that but brains doesn't.The way they make LLM solve problems is by adding a lot of logical jumps into its data, or break down different problems etc, and then as it predicts the text it predicts these logical jumps and then solves the problem. That is very different from how humans learn to solve problems, you don't feed them a billion different state transitions they have to encode to be able to navigate math, they learn to become proficient at math from a few hundred to a few thousand examples, that is fundamentally different from how LLM can learn.
That LLM are so slow learners that requires massive amount of data is a big reason its hard to make them smarter, and its caused by them being next token predictors. And the reason humans can learn with so little data is because we are not just next token predictors.
by Jensson
8/3/2026 at 8:18:21 PM
Okay, then what part of the brain is doing anything other than transforming an input into an output based on what inputs that part of the brain has previously been exposed to?by estearum
8/3/2026 at 8:21:59 PM
> transforming an input into an output based on what inputs that part of the brain has previously been exposed toYou changed the definition there, for it to be like an LLM it should be:
> transforming an input into an output trying to mimic inputs that part of the brain has previously been exposed to
Anyone can see how that limits you a lot, and why that makes it so much harder for LLM to learn things properly than it is for humans.
by Jensson
8/3/2026 at 8:53:01 PM
Well no, you changed the definition. LLMs don't just mimic what they've been trained on. They mimic what they're rewarded for. Like a brain!Pre-training is just direct mimicry. A pre-trained LLM is very stupid and mostly useless. To become useful they are post-trained with a reward function.
by estearum
8/3/2026 at 9:30:15 PM
Right, the CoT training step does work a bit more like normal training. But those do produce some very weird results, if you look at the "thoughts" CoT training leads to, I wouldn't say that CoT adds general intelligence, it teaches it to solve specific problems just like how any regular neural net training works.The only general part of this training is the text prediction. All the other steps add specific skills to it and try to not hurt the generality as they do so, but we have no way to add general skill in the second step. So the general part is still the text predictor part. We know that training a neural net on specific tasks doesn't produce generality, we have tried that before, the text predictor is what let us get a glimpse of generality.
So then the text predictor try to solve problems by moving into the different specific skills there.
But yeah, you are right, without the CoT the text predictor wouldn't have any reliable skills to map to, so its still an important part of it. But the CoT doesn't add to the generality of the model, it just adds special skills it can utilize.
But, I agree what I am arguing now is a weaker statement. But I still believe its true, but I probably wont convince you of it.
by Jensson
8/3/2026 at 7:41:35 PM
This seems fine but things being the same "on functional grounds" is just one way things can be the same. Why should we privilege this framing as something definitive versus another conceivable one?On functional grounds my bike has not a lot of distinction from a horse, but just, like, saying that doesn't tell me much about either. Or at least, it seems to leave out a lot of otherwise crucial details and differences..
What does it mean to you, this point of view? Are you truly coming from like a 20th century pragmatism point of view? Where what is most useful is what is right? Or are your trying to make a larger claim about nature? I think being clear about that would help focus your critique here.
by beepbooptheory
8/3/2026 at 4:10:37 PM
Yours is a controversial view. It is lazy and selfish to try to get other people to explain their case that it is not exclusively that, when saying that it is exclusively that is the weaker case, and you back it up with nothing but a snarky proclamation.Are newly born babies reacting due to statistical probabilities that they have derived, or are they using something other than their brains?
by pessimizer
8/3/2026 at 5:09:31 PM
> Are newly born babies reacting due to statistical probabilities that they have derived, or are they using something other than their brains?The answer is obviously yes lol.
The creature is an assemblage of electrical, chemical, and kinetic relationships.
Watching a baby develop is exactly what you'd expect from a system that's predominantly electrical noise triggering behaviors and then gradually refining denoising the relationship between inputs and outputs, with the goal function of achieving more desirable inputs.
Surely you can at least gesture toward one thing in the brain that appears not to be a probabilistic relationship between input and output?
by estearum
8/3/2026 at 4:13:37 PM
This deserves about as large of a "[citation needed]" as one could draw. Are you well-studied in neuroscience?by tadfisher
8/3/2026 at 5:19:09 PM
Nothing in the universe is anything other than a chain of reactions, amigo.The brain is an object in the universe.
The universe has quantum behaviors (fully random, not a source of intelligence) and it has deterministic behaviors (fully non-random). Many of those deterministic behaviors are so complex that they're easier to analyze and describe as probabilistic, which is where most brain input → output relationships land.
Please point to any evidence whatsoever that the brain has some third type of interaction going on that has never been observed anywhere in the entire universe, then we can have a discussion about it.
by estearum
8/3/2026 at 9:15:39 PM
Its baffling to me that people think the brain is somehow "outside" of the universe and somehow "magic" and not tied down by physics just like any other matter is. I don't understand how people come to this conclusion, except in the case of belief in magic or religion (which is belief in magic more or less).by chlorion
8/3/2026 at 9:19:18 PM
Yeah, that's ultimately this whole conversation and people just don't have the courage to acknowledge their own beliefs.The brain is a (very complex, incredible) input → output machine. That's it! It's incredible!
I don't understand why people are so afraid of this that they will believe otherwise with literally zero evidence whatsoever.
by estearum
8/3/2026 at 3:43:57 PM
[dead]by nullsanity
8/3/2026 at 5:55:10 PM
> LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence.I agree that humans must verify LLM-produced facts, but strongly disagree with these kinds of "stochastic parrot therefore dumb" arguments.
Yes, an LLM is a "stochastic parrot". No, that doesn't imply that it is dumb. Enough to look at how Terence Tao asks ChatGPT to help him understand a solution that nobody had ever discussed before [1], or how a random guy asks ChatGPT in a handful of words to disprove a 30-year-old conjecture, with zero technical input [2].
If your parrot in a birdcage with internet access can finish the sentence, "The counterexample to the Dinitz–Garg–Goemans conjecture is...", then it's a pretty smart parrot, by all reasonable definitions of "smart". Just because someone bottled up the formula into matrix multiplications and added some random sampling to the outcome, that doesn't take away from the fact that the parrot said provably correct statements that the biggest experts in the field couldn't imagine.
And no, I'm not implying that the LLMs are correct all the time, or that their intelligence and reasoning works in any way like ours.
[1]: https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed... [2]: https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...
by treszkai
8/3/2026 at 6:00:25 PM
The best way to describe the LLM intelligence is "an expert system that works the way people thought expert systems would work". You can encode a massive amount of skills into an LLM, and then the LLM uses those to navigate problems.But the LLM is still dumb where those skills doesn't have good coverage, since unlike the expert systems it maps fuzzily to its skills, and they are tuned to produce results over rejecting the request when its unclear if coverage is good. As long as that is true you have to treat them as dumb even if they sometimes produce brilliant results.
by Jensson
8/4/2026 at 12:49:57 PM
I’m not looking to argue about your position, but I think the inciting incident in the OP shows evidence that LLMs are “dumb” even when they have the sum total of written language and code at their training disposal. In this case I would expect, for your supposition to be true, that an LLM would not mistake a code comment for an actual vulnerability.by gortok
8/3/2026 at 5:30:36 PM
Wow you just got us a complete nostalgia moment to the good old times when the computer who always beats us at chess became `not artificial intelligence`...by polotics
8/3/2026 at 6:13:55 PM
Does a dog possess intelligence? Does a bird? Does a cricket? An amoeba?I hate AI slop as much as the next guy but the amount of tribalism over AI is taking near-religious forms.
Nobody knows what intelligence is, therefore we don't know what does or does not possess it, therefore we don't know whether LLMs currently, or in the future, possess it.
Yes, LLMs can be stupid, guess what: so can I. That doesn't really change the argument at all.
I feel like I'm on a deja-vu from when DALL-E was released and everybody was fighting over whether AI can be creative yes or no. Same story, different words.
Intelligence, creativity: we have no idea what these words mean, and AI is helping us understand them better. That alone is an achievement of epic proportions. I am not joking here. Any computer scientist before 2015 would be absolutely blown away by what you can now do for 10 cents and an API call, yet somehow because of the tech-bro-iness of it all we get a tribal war over what is plainly visible in front of us:
LLMs are uncomfortably close to what we thought intelligent machines would look like
by vouwfietsman
8/3/2026 at 6:30:36 PM
"Dumb next token predictor" keeps popping up since that is the core way they work. Since they aren't logic engines but prediction engines they will always return a result regardless what you ask it.Some predictions might be the tokens "I don't know", but that is based on the model mapping your text to those tokens by having seen many similar "I don't know" responses to such contexts, it didn't do any introspective logic to produce that "I don't know", and its possible it actually does know if it followed another branch there so "I don't know" is often not even true.
If they had an introspective part that stops the prediction when its too unreliable it would no longer just be token prediction engine, and I believe we need such a part for them to become what I call smart. I don't think LLM will ever stop being dumb without such an introspective part to them.
And no, that introspective part is not a part of the token predictor. At least not in us humans, the feeling of certainty we have is not a prediction, it is bundled with our thoughts, so we get both "answer is a bear" and "certainty is low", we don't get just one of those as a "prediction".
Will LLM become smart as humans with such an introspective part? I don't know, but I think they will never become as smart as humans without one.
Note: The certainty score has to be per conclusion or response, not per token. You can't evaluate a responses validity by aggregating the weight of each token. Meaning its a logic engine, not token engine, that evaluates the certainty of a statement being correct or not instead of a token being correct or not. That is the level human thinking works at and seems to be dramatically more efficient.
by Jensson
8/3/2026 at 6:47:04 PM
I strongly suspect that if you dug deeply enough into how a human brain does reasoning, the answer would boil down to "it tries a bunch of pathways and picks the one that seems right". Pathways weighted by experience, but ultimately still guess-and-check. What would the alternative be?Maybe our ability to reason is not as mystical and special as our ego might hope it to be, and discomfort over LLM's bringing that to light is the root of some people's urge to continuously downplay and discredit them.
(As far as I can tell as a non-neuroscientist, the literature on how our default mode network and prefrontal cortex interact agrees with my assessment - free association and logical verification respectively)
by jimmaswell
8/3/2026 at 6:53:59 PM
You have introspection, you can see a part of your thoughts, you know how that introspective part works since its what we call consciousness, you are conscious about it.That introspection isn't an illusion, what your consciousness see of your thoughts obviously are tings that has been calculated and aggregated by the brain, so we know the brain calculates and aggregates those thoughts and feelings to produce its results. And we know LLM doesn't do that, it doesn't have a side system that does that sort of introspection.
by Jensson
8/3/2026 at 6:58:28 PM
It actually is an illusion though. The narrative the brain makes up after the fact for why we did something is not perfectly correlated with the actual reason. It's just very good at its job of spinning a convincing narrative, even to itself. This is demonstrated in split brain patients: they're given instructions to do something that are shown only to one eye, the one connected to the non-speaking hemisphere, and when asked why they did it, the speaking hemisphere confidently hallucinates a wrong answer that the person seems to fully believe.by jimmaswell
8/3/2026 at 7:04:19 PM
> The narrative the brain makes up after the fact for why we did something is not perfectly correlated with the actual reasonBut it does that introspection, we evolved to make it. If its not useful for anything we wouldn't have evolved it, it can't have been easy to evolve a consciousness so there has to be purpose for it.
Or do you think our consciousness is a magical ghost thing unrelated to the brains workings? I think its pretty obvious that our smarts in part depend on the computations that results in our conscious experience, you need a very strong argument as to why that wouldn't be the case. And broken brains being broken doesn't answer this.
by Jensson
8/3/2026 at 6:36:43 PM
> I don't think LLM will ever stop being dumb without such an introspective part to them.The very second they add this part, it will "just be a dumb token predictor with introspection", mark my words.
by vouwfietsman
8/3/2026 at 6:46:03 PM
If it keeps doing dumb things, yeah. But if that actually solves it then those opinions will quickly disappear when it replaces all human white collar work since it does it cheaper and better and faster.AGI is fairly easy to detect for this reason. Does this system make a majority feel you don't have to hire people anymore? If not its not AGI.
by Jensson
8/3/2026 at 7:33:45 PM
> But if that actually solves it thenSolves what? Chess? No thats not AI, its just a chess bot. Turing test? No, thats not AI, its just a dumb token predictor.
Why would the goalposts stop moving at AGI? I am 100% convinced it will somehow still lack the "gusto" or the "taste" or the "timbre" of real intelligence. You can see it in coding right now, AI has bad "taste" in coding, because really, we can't do a better job critiquing that which is obviously (on occasion) just plain good.
by vouwfietsman
8/3/2026 at 7:43:42 PM
> Solves what? Chess? No thats not AI, its just a chess bot. Turing test? No, thats not AI, its just a dumb token predictor.You are moving the goalpost here if you think the chess AI was AGI. All those problems were evidence AI wasn't as smart as humans, not goalposts determining when AI is as smart as humans. The first turing test winner wasn't even an LLM, it was an expert system, so we already knew that test wasn't enough for AGI.
by Jensson
8/4/2026 at 6:22:28 PM
> You are moving the goalpost here if you think the chess AI was AGIIndeed I would be, but I don't, so I'm not.
What I'm saying is that nobody understands either creativity or intelligence, and people keep listing behavior that they think can only be displayed by something that has those qualities. Yet, whenever we build a system that displays this behavior, all of a sudden, those qualities are NOT attributed because we now understand how to achieve this, the curtain has fallen, the magic is gone, and we do not assign the qualities we believe are human to something that is not magic, lest we reduce ourselves to the machines in front of us, losing the things that make us unique.
That is the shifting of goalposts, its not me who's doing it. There's a wiki page for it, its called "the AI effect": https://en.wikipedia.org/wiki/AI_effect
by vouwfietsman
8/3/2026 at 6:43:10 PM
Necessary is not the same as sufficient.If I want my ox-cart to fly, I need to add wings, but that's probably not all that needs changing.
by Terr_
8/3/2026 at 4:26:32 PM
Every day I wake up and open HN.“LLM has made legitimate mathematical discoveries” —> Wow the rate of progress is amazing. Highly upvoted.
“LLM does something not good” -> Does everyone else not realize LLMs are just dumb next token predictors? Highly upvoted.
So tired of this discourse and this site.
by gbnwl
8/3/2026 at 4:42:30 PM
The rate of progress can be high and they can also be dumb next token predictors. Not sure why that is hard to understand.These models can do a lot of things but they also can't do a lot of things. In order to use these models effectively you have to understand that they are next token predictors and how that allows it to do what they do.
by Jensson
8/3/2026 at 4:47:57 PM
Are they useful or not? Will they continue changing the world or not? People who choose one way or the other for describing them typically fall on one side or the other in these questions imo. What do you think? Will these next token predictors change the world or not?by gbnwl
8/3/2026 at 4:49:56 PM
They are useful. They will continue to change the world. They are still next token predictors with all the problems that comes with that.For them to change the world you have to work with them as next token predictors. Ensure that the next token predictor has enough prediction paths to solve the problems you want and so on. Since when they don't they fail spectacularly. These big companies will continue to add new skills to them, so they will continue to get more useful.
by Jensson
8/3/2026 at 8:18:01 PM
In all fairness humans can also be considered next token predictors. It could be said that’s how we communicate with one another today. Presently LLMs lack other things, like physical presence in the world and continuity of input sensory data.by p2detar
8/3/2026 at 8:35:45 PM
Humans learn to be a next token predictor as a kid when they learn to speak, an LLM cannot learn to be a next token predictor or anything of the sort, we have no clue how you could have an LLM learn human language just based on a thousands conversations with a human.You don't see how that is very different? For an LLM to be as smart as a human it has to be able to learn like a human. Like you don't evaluate how smart a human is based on how much he knows, you evaluate it based on how fast he learns. And LLM are so bad at learning its ridiculous, they lack that part of the brain that lets humans be smart and learn so fast and easily.
by Jensson
8/3/2026 at 10:25:21 PM
> For an LLM to be as smart as a human it has to be able to learn like a human."For a plane to fly as well as a bird it has to be able to flap its wings".
"For a submarine to swim as well as a fish it has to be as light as fish".
by fsflover
8/3/2026 at 10:53:48 PM
> "For a plane to fly as well as a bird it has to be able to flap its wings".> "For a submarine to swim as well as a fish it has to be as light as fish".
These are false equivalences. The post you're responding to defined intelligence as learning rate. LLMs unequivocally do not learn. You can disagree with OP or agree, but what you have done is out of bounds. You're implicitly claiming that LLMs learn, albeit differently from humans. This is categorically false, unless you count training as some kind of "learning".
That's of course absurd, almost no user of an LLM also trains it. Instead they rely on queries submitted to pre-trained LLMs ("inference"). And don't bother yapping about context windows, it's just not anything like learning.
by 27183
8/4/2026 at 7:29:45 AM
> The post you're responding to defined intelligence as learning rate.How is the learning method or rate related to intelligence? LLMs learn during the training, much faster than any human. Yes, they drastically slow down their learning afterwards, but they still can learn a bit from an uploaded document or a web site.
And in the end they may be better at intelligence than any human, depending on the task and time given. I though intelligence is not about history but current performance.
How is this different than my examples, in which a possibility to fly is judged by (learning to) flapping the wings and not by the actual result?
> And don't bother yapping about context windows, it's just not anything like learning.
That's debatable, but I don't see how it's relevant here. You can ignore that part of my reply above, and the argument will remain unchanged.
Nevertheless, I don't understand how this not learning. Without it, no meaningful intelligent task can really be performed. LLM/human must learn the relevant bits from current situation in order to answer meaningfully. Often it requires to actually acquire new knowledge like reading a new piece of text unknown before. Feel free to link to a relevant discussion for me if you find this boring and settled.
by fsflover
8/3/2026 at 5:42:56 PM
Somewhat.Yes, but not because they are useful.
by inigyou
8/3/2026 at 6:34:29 PM
It can be a token predictor and still tell me exactly how my life will proceed from now until the indefinite future, or be the most intelligent conversational entity you have ever witnessed.The issue is of course with using the word "dumb": they are next token predictors, no doubt about it, but whether LLms as a class of system are smart or dumb is entirely unknown and entirely variable in time.
To interact with them effectively you must know how they behave, just like you have to know how humans behave to interact with them effectively. If you disagree, find someone with autism and have a conversation with them.
by vouwfietsman
8/3/2026 at 6:43:34 PM
The "dumb" part comes from how it behaves in contexts where it lacks a lot of data, or where the data is skewed. Since they are tuned to give a prediction anyway and just make something up since sometimes those made up things are useful they will produce dumb results.So people call them dumb since like dumb people they make strong statements about things they don't understand. And it doesn't matter how much smart things you encode them with, they will keep making strong statements about things they don't understand until they are fundamentally changed.
But since LLM are very smart about things where they have extensive data they can still be used to reliable solve many problems and probably in the future where we understand that better almost completely replace most lawyer and doctors work etc, because a lot of what a frontline doctor or basis lawyer work is very repetitive and can be encoded with billions of examples and decision paths into an expert system framework the LLM will follow.
So people say LLM are dumb since LLM will always keep making dumb statements. This is the same way we call Elon Musk dumb for making a lot of dumb statements, he is a smart guy but he makes dumb statements so her is dumb.
by Jensson
8/3/2026 at 7:37:43 PM
> they will keep making strong statements about things they don't understand until they are fundamentally changedIf ever there was a human quality.
Also, your explanation of "dumb" is really favoring the anti-llm side, and its a very generous interpretation. I suspect what is much more likely meant, is that token predictors cannot be smart, not now nor in the future after improvements, because they are token predictors and predicting tokens is not how intelligence works.
All of this is of course unfounded, and hidden behind the word "dumb".
by vouwfietsman
8/3/2026 at 7:54:19 PM
> I suspect what is much more likely meant, is that token predictors cannot be smart, not now nor in the future after improvements, because they are token predictors and predicting tokens is not how intelligence works.Why do you think that? LLM are used as expert systems today, in order to quickly navigate problems by breaking them down and iterating between different well known possible solutions and paths to check etc. That is how they work, they do that by using their next token predictions, and for things they aren't well trained on they will produce dumb results.
LLM has solved enough problems that almost nobody has the view you ridicule here, but there are still many who think LLM are thinking just like humans and that you can trust them just like humans. So its important to remind people these are just token predictors and lack many things humans do.
> If ever there was a human quality.
Humans can avoid doing that by using introspection, LLM can't. That some humans do it by not using introspection doesn't mean humans are incapable of it, we know humans are capable of it, which is why we can point out when the LLM is wrong with certainty, humans as a group make extremely good predictions.
by Jensson
8/4/2026 at 6:16:53 PM
> Why do you think that?From (highly upvoted) parent:
> LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence > LLM-based “AI” ....
Hardly neutral statements, hardly accurate statements, yet highly upvoted. LLMs are AI, there is nothing to gain by pretending it isn't because of some secondary motive or opinion someone has.
> LLM has solved enough problems that almost nobody has the view you ridicule here
I think I have just shown you the parent literally claims LLMs are not AI, and do not possess intelligence.
> but there are still many who think LLM are thinking just like humans and that you can trust them just like humans.
I don't think many people think that, especially on HN, for two reasons:
1. Most high profile AI tools have immediately visible disclaimers saying, more or less, "AI makes mistakes". 2. People don't trust humans either, if anything people trust the AI more because it is not human.
People may be incorrectly worshipping LLMs, but they are doing it precisely because it is not human. If you put chatgpt behind a believable chatbox so people would actually think it was human, they would be much more skeptical.
by vouwfietsman
8/3/2026 at 5:11:24 PM
Much of an LLM's capability comes from the structure encoded in its learned representations. The probabilistic outputs are primarily a way of expressing uncertainty and generating fluent text, while compression during training is what forces the model to discover that underlying structure.by diydsp
8/3/2026 at 5:16:34 PM
> Much of an LLM's capability comes from the structure encoded in its learned representationsAnd thats encoded as a set of next token predictions. So the way to see how reliably it solves a problem is to look at the chain of predictions, and see where it is unreliable at finding the next spot, or where it always fails and you need to add that link to the dataset to train it.
This isn't magic, today we understand pretty well how to add new skills to LLM, and the better this is understood the faster progress will be.
This also means that if a context doesn't have any good predictions, it will produce a dumb prediction for that context. This results in these bad outcomes, because currently LLM doesn't have a map for where predictions are good or bad.
by Jensson
8/3/2026 at 4:56:49 PM
[dead]by cindyllm
8/3/2026 at 4:30:04 PM
Would be nice to get high karma commenter votes count only ..by apples_oranges
8/3/2026 at 4:57:17 PM
Not a dichotomy actually. Highly depends on the task.by germandiago
8/4/2026 at 11:00:58 AM
I agree. To me it seems blindingly obvious that a huge swath of the HN audience is gripped by fear and a loss of identity as a result of what LLMs have demonstrated over the last couple years, and they're lashing out as a result. Shaking their fist at the sky because they don't like the weather. Very understandable, but it's getting old to read month after month.by senordevnyc
8/3/2026 at 5:44:46 PM
Opinions differ. This is not news.by chrisjj
8/3/2026 at 4:50:56 PM
Not sure what your point is? Those things can both be true.Or should the discourse in a diverse community like HN only reflect the positions you personally hold?
by unknownfuture
8/3/2026 at 5:01:15 PM
Tell me how a 'nExT toKeN prEdIcTor' can make breakthroughs in math or play a game of chess. These activities aren't pure symbol manipulation, they require actual understanding at some level.by Yajirobe
8/3/2026 at 5:42:24 PM
By predicting next tokensby inigyou
8/3/2026 at 5:04:08 PM
Eh, I'm not going to litigate your claims.My point is it's silly to whine that HN is a place where multiple points of view on the topic are aired out and discussed.
If you want a personal echo chamber where only your own beliefs are affirmed and anything else is flagged off or downvoted, I'm sure you can go find one or, worst case, vibe code one into existence.
by unknownfuture
8/3/2026 at 5:17:51 PM
Fair so let me be clear. I’m whining because the “next token predictor” reductionist point of view has been wrong and is only growing more wrong with time. Clearly these things can do things that actually matter. Do you disagree?by gbnwl
8/3/2026 at 5:24:39 PM
Even now you're engaging in this discussion as though I'm trying to litigate your point and that somehow forcing me to concede is, what, winning? I don't know.I get the impression you want me to concede that the particular points of view you disagree with aren't worthy of representation here on HN.
I'm not going to do that.
by unknownfuture
8/3/2026 at 5:40:32 PM
I just prefer HN comments to be better reflections of reality. There is an unspoken expectation here that people here know what they’re talking about especially when it comes to technical matters. The rise of LLMs has given way to a HN branded populism that willingly denies reality as well. “Next token predictor” truthism is just so dumb and completely ignores the reality of what these tool are able to do. Smash that upvote button every time it feels good if you want but it’s just a meaningless take at this point. It won’t help you predict anything that’s coming.Since we disagree on the present let’s informally do a “remind me 2 years” to this discussion and see what’s happened then.
by gbnwl
8/3/2026 at 5:52:59 PM
> I just prefer HN comments to be better reflections of reality.You mean your particular version of it.
It's interesting to see you consistently missing this point.
You've decided LLMs are clearly more than just complex but mindless statistical models.
You've decided that based on, it seems, the very impressive things these tools are capable of.
Therefore if anyone claims they're just mindless stastical models--with or without any attached judgement as to their actual utility or usefulness--then they are ipso facto wrong.
(And yes I just used endashes, damnit!)
That's on you.
It is in fact possible to simultaneously believe that LLMs are mindless token predictors and that they're enormously powerful.
These are entirely orthogonal beliefs.
Heck you could equally believe that LLMs represent true emerging AGI and that they still remain deeply flawed and are only an incremental step along the path of automation.
Or somewhere in between.
And discussing that space of possibilities is, I'd hope, precisely what HN is for.
by unknownfuture
8/3/2026 at 6:24:44 PM
It’s just a boring and unhelpful complaint that afaict largely serves to soothe the commenters ego rather than point at anything insightful that’s useful or predictive. Point me to your favorite “next token predictor” comment that was actually insightful or predictive. You have years of material to draw from.by gbnwl
8/3/2026 at 7:27:16 PM
https://news.ycombinator.com/item?id=49155075"These models are probabilistic, you shouldn't blindly trust them in spaces where accuracy is really important" seems like pretty sound advice to me.
by unknownfuture
8/3/2026 at 11:00:58 PM
More that the “discourse” around anything “AI” is just another culture war at this point - dominated by facile taking points that are shallow, ill-considered, uninformed-and-uninformative, and re-re-recycled into worthless pulp by their respective tribal bubble.And if someone’s first response to this comment is to try to peg which “side” I’m calling an idiot as if that’s the most important bit of info to dictate their response - that’s exactly what I’m talking about.
by blargey
8/3/2026 at 5:04:37 PM
It's almost as if there were many people using this site, and there is no clear consensus on LLMs, so people from various camps upvote interesting stores to support their cause. And people who are still somewhat undecided upvote both, if they present good evidence.I mean even perennially contentious topics will get this behavior.... some thing about emacs makes the front page, within a day or two there will be a vim post up there. Same with Rust is (good|bad), or if systemd creates an even more awesome tool, the haters will come along and recycle stories about bugs from over a decade ago.
There's a lot of people here. Not all of them read it every hour, and discussions like this among large groups often take a very long time with lots of repetition. Human group dynamics (aka politics) is slow.
> So tired of this discourse and this site.
You're welcome to leave if you don't like it. The site was like this long before you joined, and will like it long after you leave I'm sure.
It's also worth noting, that an awful lot of math discoveries are perfectly in line with dumb next token generators - they are finding a way to formally construct an argument and being surprised when it doesn't work, or surprised at the outcome of the grind. Not all of them are made by brilliant leaps of intuition.
by sophacles
8/3/2026 at 5:14:41 PM
This is a conflation of issues, predicated on false understanding of what LLMs are.This line of critique is pernicious because it is both technically correct, as description, and profoundly misleading.
Saying that outputs are a product of inputs is not interesting and to the point it is not explanatory.
What is interesting, is how they do what they do. What is the "statistically likely* next token? To answer that you can do exactly one thing, run the LLM. That's because what they are doing is interesting and not reducible.
What is more interesting is that in order to do what they do, given the architectures we apply and the training strategies we use and the harnesses we situate them in, LLM are recapitulating in their deep layers strategies observed in the animal brain. This is still suggestive, interpretibility is nascent: but it is also more than a little interesting. In some respects, for cognitive scientists interested in the manner in which mind merges from computational substrates, it is profoundly interesting.
One can incorporate this, and, still be viciously critical of bother the success and failure of LLM in the applications we have put them to, and of how we (as individuals and as institutions such as corporations) are integrating them into our work.
There is a lot to criticize!
But criticism can be taken more seriously when it is not obscured by misunderstanding or misrepresentation (intentional, or not) of what LLM are and why they are not remotely "parrots" in the pejorative sense.
The technology, as technology, at the scale we are architecting it, is doing things we did not imagine would be witnessed in our lifetime, if ever. Dismissing that and denying it because of the career, industry, society, and civilization challenges that technology brings are existential, is bad argumentation or bad faith.
Both can be true at once.
by aaroninsf
8/3/2026 at 5:26:21 PM
> We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are.I see no credible corroboration. More likely its folks having no more care for what they are doing than the bots themselves.
> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
People said the same about email spam ... until they engaged spam filters. CVE report slop is simply spam. Complaints are better directed at the filters, not the filtered.
by chrisjj
8/3/2026 at 5:18:55 PM
[flagged]by a3mrouter1
8/3/2026 at 4:22:49 PM
[flagged]by budsniffer952
8/3/2026 at 4:41:04 PM
> We are not going back, period.I didn't get this at all from the parent. They're simply stating that LLMs aren't entirely trustworthy, and that the responsibility is ultimately ours, not the LLM's.
by sedawkgrep
8/3/2026 at 4:46:15 PM
Okay.by budsniffer952
8/3/2026 at 3:55:11 PM
This is the right conclusion for completely wrong reasons haha. I love hn. (aside from being entirely wrong as most models are now mixed modal so they are token prediction engines)Each layer of attention can more through feature space “lit up weight clusters” in a way no other previous AI can. It can from that decode some rudimentary logic and world modeling and make deductions. Certainly better than any previous AI. Only a goof here would believe this wasn’t a serious advancement.
So don’t over sell it. But don’t sell it short with this “grrr in an engineer don’t threaten me with new tech” attitude.
This take is akin to teenage angsty takes and doesn’t really belong here.
by tsunamifury
8/3/2026 at 5:09:19 PM
> A human must verify everything an LLM presents as fact. Everything.I've thought about this for quite some time now.
No. A human doesn't need to verify everything. And the argument is really simple: stochastic.
Think of self-driving cars: We can show today - based on evidence and real data - that self-driving cars are safer than human drivers. That's a fact and the consequences are clear, more self-driving cars, less human-driven cars, less accidents, less hurt people, less dead people.
Are the cars 100% safe and NEVER make a mistake? No. But they don't need to. Nothing is ever 100% (in the real world).
Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.
Yes, there will be another layer in the system, another level of abstraction that will/must end at the human boundary.
by adjfasn47573
8/3/2026 at 7:02:32 PM
> Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.Let me know once the majority of software engineering organizations start only checking in markdown files and let code be generated non-deterministically from these specs in CI. If this is not happening now, there’s clearly a sufficiently high level of distrust in blind LLM output (both code itself and reviews).
And even then, are you suggesting humans shouldn’t at least review the markdown specs? Why not have LLMs review the specs then? Is there, perhaps, some fundamental quality to human review process that is desirable here?
by batshit_beaver
8/4/2026 at 5:11:27 AM
This is weird I didn't imply or say any of these things that you're arguing against right now.by adjfasn47573
8/3/2026 at 5:37:45 PM
This reduction of everything to stochasticity is silly. Or, to put it differently: Do you accept a value with some error appearing in your bank account on salary day?We have plenty of systems where complete accuracy is the only acceptable thing. Computers are great for such things. Until we all get caught up in a way of delusion and start writing those systems as natural prose passed through an improperly understood stochastic machine.
by gspr
8/4/2026 at 5:18:37 AM
I've never said or implied what you're arguing against right now.An LLM building software benefits from using a formal language, formal tests or having a formal specification in a similar way we do. Everything you can formalize into something, so you have certainty about stuff, is always a benefit.
I wasn't comparing exact science with LLMs. I was comparing current messy review processes of code performed by humans with future messy review processes of code likely performed by LLMs.
If you find a way to convert code reviews into a fully formalized process, then this is clearly the winner. If the LLMs find a way to do that, same.
If you find a way to formalize the process of driving through any street in any situation, then this is clearly the winner. Until then, driving stays messy and stochastic.
by adjfasn47573
8/3/2026 at 4:35:18 PM
We can chalk this up as another example of over-exhuberance by what folks believe humans can accomplish vs. what they actually are.Flesh-based “brain” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your flesh-based brain gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
Humans are prediction engines. They are not Pure Intelligence, and shouldn’t not be treated in any form or fashion as if they possess pure intelligence. What bothers me about this entire situation is that presumably the folks that have relied on the flesh-based “brains” to generate these vulnerabilities knew (or should have known) enough about their "tool" to know this would happen, but did not: To err is to be human.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this over reliance on fallible “brains".
A human must verify everything another human presents as fact. Everything. If you don’t, we all pay the price. Using a human does not remove the onus of responsibility on the human being in charge, if anything they amplify it because humans work for peanuts in some countries, and can generate lots more output more quickly that needs to be verified by the humans in charge.
by SubiculumCode
8/3/2026 at 4:21:05 PM
You are right with the analysis, but wrong with the conclusions. Yes, LLM „thinking process“ is kinda non-deterministic in a sense that it does not follow logical reasoning and will not produce logically correct results in 100% cases. It has an error margin.However, error margins are in the center of any engineering discipline. We cannot produce things measured with 100% accuracy. This is accepted fact. The focus is always not on eliminating errors, but on reducing them to acceptable minimum. With LLMs we should not expect an ideal logical thinker, but a process that may error sometimes, and we must design quality controls instead that push LLM outputs within acceptable margins. And it can work.
by ivan_gammel
8/3/2026 at 4:27:36 PM
Yes but the key here is doing proper risk assessment. "What is the consequence if the LLM gets this wrong?" "How do we verify the output?" "What are the legal ramifications for using the LLM in this way?" "Who is responsible when the LLM fails?" "Whats the expected accuracy here?" etc.In the current AI mania, there's a lot of due diligence simply being ignored. Plenty of "Well humans make mistakes too!" going on here on HN too.
by kentm
8/3/2026 at 5:38:19 PM
The due diligence not being done is people putting cats in microwaves. It‘s not the dangerous part. The real danger is risk assessments coming to wrong conclusions, because it is still terra incognita. Talented engineers were in this situation before, doing mistakes with cars, airplanes, buildings etc.by ivan_gammel
8/3/2026 at 7:37:16 PM
No, I'm sorry but I think thats a cop out. The fact that LLM are stochastic and can give incorrect answers is not particularly difficult to comprehend, and the risks that fall out of that are reasonably understandable. The issue is entirely down to bad choices by the people driving LLMs, because they are engaging with what they wish LLMs do instead of what they actually do.by kentm
8/3/2026 at 9:22:14 PM
So you say that it is easy to understand what LLMs are, yet the issue is that people don‘t understand it. Either you put too many people in the bucket labelled „idiots“ or you are blowing that problem out of proportion, at the same time seeing only the tip of the iceberg.Risks are not binary. Most people are not idiots. They do understand that LLMs aren‘t thinking humanly or 100% logically. You are talking about some sort of faith, I think this is more about trust, which is built on observations. Statistically, the outcomes of LLM work may hit the goal quite a lot. More of that in the beginning of the journey than in the end. For many reasonable people that becomes a trap, where minor, acceptable deviations accumulate into a fireball. Still, saying that LLMs are failure with inevitable fireball at the end and giving up is a trivial and stupid solution. Staying within razor-thin distance from that fireball and managing to deliver a working solution is what everyone tries at the moment. This is how it always happened and always worked. It‘s the art of engineering.
by ivan_gammel