alt.hn

4/10/2026 at 2:35:35 PM

Why do we tell ourselves scary stories about AI?

 https://www.quantamagazine.org/why-do-we-tell-ourselves-scary-stories-about-ai-20260410/

by lschueller

4/10/2026 at 3:02:51 PM

It's simple. It's because AI is the scariest technology ever made.

Human intelligence has proven itself capable of doing a lot of scary things. And AI research is keen on building ever more capable and more scalable intelligence.

By now, the list of advantages humans still have over machines is both finite and rapidly diminishing. If that doesn't make you uneasy, you're in denial.

by ACCount37

4/10/2026 at 7:03:38 PM

AI, the way you are describing it, has not been invented yet. It is a fiction.

What is called "AI" today is an extremely vague marketing term being applied to various software technologies which are only dangerous because humans are dangerous. Nuclear & chemical weapons are also "very scary" but only because the humans who might decide to use them in fits of insanity are scary.

I'm not in the slightest bit uneasy about "AI" itself right now, because as I said, the AI of Sci-Fi has not yet been invented…and seems unlikely to in any of our lifetimes. (Not throwing shade on clever researchers. We also don't have working FTL travel, though plenty of scientists speculate on how such an engine might be built.)

by jaredcwhite

4/10/2026 at 8:02:22 PM

"It's just marketing" is just the "denial" stage wearing a flimsy disguise.

Even LLMs of today routinely do the kind of tasks that would have "required human intelligence" a few years prior. The gap between "what humans can do" and "what frontier AIs can do" is shrinking every month.

What makes you think that what remains of that gap can't be closed in a series of incremental upgrades? Just 4 years have passed since the first ChatGPT. There are a lot of incremental upgrades left in "any of our lifetimes".

by ACCount37

4/10/2026 at 8:44:27 PM

You don't seem to be engaging seriously with respected experts in this field who have been reporting for years at this point that merely scaling LLMs and so-called "agentic systems" doesn't get us anywhere close to true AGI.

Also computers in the 1980s could perform many tasks that previously would have "required human intelligence". So? Are you saying computers in the 1980s were somehow intelligent?

by jaredcwhite

4/10/2026 at 8:55:49 PM

And you don't seem to be engaging seriously with respected experts in this field who say "scaling still works, and will work for a good while longer".

If your only reference points are LeCun, or, worse, some living fossils from the "symbolic AI" era, then you'll be showered in "LLMs can't progress". Often backed by "insights" that are straight up wrong, and were proven wrong empirically some time in 2023.

If you track LLM capabilities over time, however, it's blindingly obvious that the potential of LLMs is yet to be exhausted. And whether it will be any time soon is unclear. No signs of that as of yet. If there's a wall, we are yet to hit it.

by ACCount37

4/11/2026 at 1:11:29 AM

That aside.

Lets look at the facts.

Are LLMs displacing labour? In the aggregate - not from what one can see. The aggregate statistics tell a different story e.g. the hiring of software engineers is still growing Y-o-Y.

The limits of LLMs will be put in place through financial constraints. People like you seem to think there's an infinite stream of money to fund this stuff. Not really. Its the same reason why Anthropic and OAI are now shifting focus to generate revenues and cash flows because they will not receive external funding forever.

by dsa3a

4/11/2026 at 5:01:36 AM

I can’t speak for the states, but in AU I clearly see a massive displacement of undergrad and junior roles (only in AI exposed domains).

I say this as both someone who works with many execs, hearing their musings, and someone who no longer can justify hiring junior roles themselves.

Irrespective of that; if we take this strategy of only taking action once it is visible to the layman - our scope of actions available will be invariably and significantly diminished.

Even if you are not convinced it is guaranteed and do not believe what myself and others see. I would ask you is your probability of it happening now really that close to 0? If not then would it not be prudent to take the risk seriously?

by robkop

4/11/2026 at 8:34:52 AM

> If not then would it not be prudent to take the risk seriously?

What does taking the risk seriously look like?

by jurgenburgen

4/11/2026 at 9:13:33 AM

I'm not interested in reading the same arguments over and over angain. Ai is not scary anymore, it's fucking boring. Exits thread

by Xmd5a

4/10/2026 at 3:19:04 PM

Modern discourse happens on social media where fear and outrage drive engagement, which drives virality. We have become convinced in a short amount of time that AI is going to take all the jobs and eventually kill us all because that's what people click on.

Any voices or studies that present the case for "useful technology that will improve productivity and wages while not murdering us" don't get clicked on or read.

by andrewmutz

4/10/2026 at 5:11:59 PM

> Any voices or studies that present the case for "useful technology that will improve productivity and wages while not murdering us" don't get clicked on or read.

It's not an either/or thing though. Compare to something like combustion. Sure it definitely improved productivity but also lead to countless violent deaths.

by AlecSchueler

4/10/2026 at 3:09:45 PM

I don't know, I think nuclear weapons are scarier. And also probably a useful parallel: they're so dangerous that we coined the term "mutually assured destruction" and everyone recognized that it was so dangerous to use them that they've only ever been used once.

I see the flood of PR from AI firms as an attempt to make sure we don't build the appropriate safeguards this time around, because there's too much money to be made.

by afavour

4/10/2026 at 3:52:46 PM

I remind you of why nuclear weapons exist.

They exist because human minds conceived them, and human hands made them.

One of the major dangers of advanced AI is being able to implement something not unlike Manhattan project with synthetic intelligence, in a single datacenter.

by ACCount37

4/11/2026 at 8:35:27 AM

Yeah, the problem with AI is that they can become too good at performing general tasks, ranging from, like, designing cancer treatments, or designing bioweapons, and everything in between

by nextaccountic

4/10/2026 at 4:44:07 PM

By any quantifiable measure, yes, and not by small numbers either.

Until someone can demonstrate a quantitative measure of intelligence - with the same stability of measurement as "meters" or "joules" - any discussion of "Super-AI" as "the most dangerous X" is at best qualitative/speculative risk narratology, at worst discursive distractions. The architecture of the "social web" amplifies discursion to a harmful degree in an open population of agents, something I think we could probably prove mathematically. I am more suspicious of this social principle than I am scared of Weakly Godlike Intelligence at this moment in history; I am more scared of nuclear weapons than literally anything else.

People think we are out of the woods with nuclear weapons, but I don't think we've even seen the forest yet. We are Homo Erectus, puffing on a flame left by a lightning strike, carrying this magic fire back to our cave.

by lopsotronic

4/10/2026 at 3:32:12 PM

Everyone recognized that it was so dangerous to use them after the first two mass casualty events. At the time and even into the 50s it was not universally obvious, and the arguments in favor of nuclear weapons use were quite similar to arguments I often see with regards to AI: bombing cities into rubble is not a new concept, traditional explosives well within the supply capacity of large militaries are capable of it, so what are we even talking about when we say that there's scary new capabilities?

by SpicyLemonZest

4/10/2026 at 4:27:41 PM

> Everyone recognized that it was so dangerous to use them after the first two mass casualty events

I really don’t think that’s true. Those who actually knew about the nuclear weapons knew very well how dangerous they were. Truman was deeply conflicted about using them.

by afavour

4/11/2026 at 9:53:51 AM

Truman changed after learning the real civilians death numbers that they caused. The military leaders absolutely knew the impact before, and kept advocating for its in later wars.

by Hikikomori

4/10/2026 at 3:18:33 PM

Nuclear weapons have rarely been used kinetically. Their real force multiplier is the fear.

A.I. is being used by so many people for so many diabolical things, hidden, unknown things that we may never fully understand it's purpose. But that doesn't mean it's purpose won't destroy us in the end.

The expression "Drinking the Koolaid" is used to explain the Jonestown mass suicide. It is an information hazard, aka, a cult that created the end result: 900 people drinking poisoned flavoraid. That's just one example of a human caused information hazard. What happens when someone with similar thinking applies that to A.I.? Will we even be able to sleuth out who did it?

by webdoodle

4/10/2026 at 3:53:42 PM

There was a lot of FUD in the mainframe era about computers being called "electronic brains" and fears of them taking people's jobs because the ignorant public mistook their lighting fast arithmetic skills for intelligence. Many did lose their jobs as digital record keeping, computerized accounting/ERP, robotics on assembly lines, became cost effective, but at no time did the "electronic brain" become intelligent.

There's a lot of FUD today about LLM's being sapient because the ignorant public mistakes their complex token prediction skills for intelligence. But it's just embarrassing to see people making that mistake on a forum ostensibly filled with hackers.

by ThrowawayR2

4/10/2026 at 4:10:31 PM

Is it me making the mistake, or is it you making that very mistake in the other direction?

Back in the "mainframe era", we had entire lists of tasks that even the most untrained humans would find trivial, but computers were impossibly bad at. Like following informal instructions, or telling a picture of a dog from that of a cat.

We're in the "AI era" now, and what remains of those lists? What are the areas of human advantage, the standing bastions of human specialness? Because with modern AI, the list has grown quite thin. Growing thinner as we speak.

by ACCount37

4/10/2026 at 3:11:38 PM

Most humans can do more than plagiarizing text. But let's hype up the clankers before the IPOs.

by saHqtr

4/10/2026 at 4:01:01 PM

"It's all just PR" is a lame excuse not to think about the implications. Of things like: AI capabilities only ever going up over the course of the past 4 years.

by ACCount37

4/10/2026 at 3:43:40 PM

> AI is the scariest technology ever made

Well, it's a good thing that all we managed so far is a large language model instead.

by otabdeveloper4

4/10/2026 at 3:21:59 PM

Machine still need a planetary complex production pipelines with human operators everywhere to achieve reproduction at scale. Even taking paperclip plant optimizer overlord as serious scenario, it’s still several order of magnitude less likely than humans letting the most nefarious individuals create international conflicts and engage in genocides, not even talking about destroying vast pans of the biosphere supporting humanity possibility of existence.

That is, also alien invasion and giant meteor are plausible scenario, but at some point one has to prioritize threats likeliness, and generally speaking it makes more sense to put more weight on "ongoing advanced operation" than on "not excluded in currently known scientifically realistic what-if".

by psychoslave

4/10/2026 at 3:55:57 PM

Humans are dangerous and hilariously exploitable.

If politicians can get away with what they do? Imagine if those politicians were actually smart and diligent to a superhuman degree.

That's the kind of threat a rogue AI can pose.

Humans can easily act against their own self-interest. If other humans can and evidently do exploit that, what would stop something better than human from doing the same?

by ACCount37

4/10/2026 at 3:13:23 PM

The world we live in is a construct, not a natural outcome. Even if we take your premise at face value, that our success as a species is only because of advantages over others, what's to say that "intelligence" is that advantage? What's to say that we don't use our advantages to reconstruct a world that works in a way that doesn't advantage intelligence over all else?

And on intelligence specifically: even amongst the human race, we all know smart people who are abject failures, and idiots who are wildly successful. Intelligence is vastly overrated.

by fontain

4/10/2026 at 3:59:25 PM

IQ is among the best predictors of life success, for humans. Being up by an extra SD in the g dimension covers a multitude of sins.

I'm not sure what level of delusion one has to run to look at human civilization and say "no, intelligence wasn't important for this". It's pretty obvious that human world is a product of intelligence applied at scale - and machines can beat humans at intelligence and scale both.

by ACCount37

4/10/2026 at 4:30:53 PM

>> I'm not sure what level of delusion one has to run to look at human civilization and say "no, intelligence wasn't important for this".

One has to only look at the current tech and political leaders.

by sifar

4/10/2026 at 3:07:26 PM

"Why be afraid of nukes it's not like they WANT to blow up"

by causal

4/10/2026 at 3:09:27 PM

This is untrue. What is being diminished is the value of humans doing repetitive or uncreative tasks.

Many have built their careers from that kind of work in the past and yes they are threatened, but that kind of work is inherently not collaborative and more vocational.

by sublinear

4/10/2026 at 3:12:06 PM

The vast majority of people on this planet work repetitive, uncreative jobs.

by bauerd

4/10/2026 at 7:05:47 PM

Oh? And what extensive knowledge and experience makes YOU qualified to determine what "the vast majority of people on this planet" are doing for work and if those tasks are creative or uncreative?

by jaredcwhite

4/10/2026 at 9:19:48 PM

Not sure what you're insinuating. What do you think is the statistically average job on this planet? It's still going to be cultivating a smallholder farm in developing countries, or working in logistics, manufacturing or the broader service in developed countries.

All of these average jobs are structurally repetitive. Yes, humans do constantly inject creativity, but it's a means to an end, to getting the job done.

You apparently mistook my descriptive comment for a value judgment, but it isn't.

by bauerd

4/10/2026 at 3:26:14 PM

There is no such job done by humans today that is 100% uncreative, but people will continue to insist there is.

The devaluing may come from AI pressure, but the harm is coming from humans foolishly not seeing the value in what's left behind. Most people have not and will not lose their jobs.

by sublinear

4/10/2026 at 2:52:28 PM

If I can plausibly say I'm making something super dangerous, the government is likely to want to be the first government to have it. If the check clears before they figure out if I'm BSing them or not, it's a win.

by Zigurd

4/10/2026 at 3:01:16 PM

One thing that strikes me that I never really see anyone discuss is that we've been afraid of conscious computers for a _long_ time. Back in the 50s and before people were quite afraid that we'd build conscious computers. This was long before there was any sense that could actually accomplish the task. I think that similarly to seeing faces in the clouds, we imagine a consciousness where none exists. (eg: a rain god rather than a complex system of physics and chemistry)

Even LLMs, which blow past any normal Turing test methods, are still not conscious. But they certainly _feel_ conscious. They trigger the same intuitions that we rely on for consciousness. You ask yourself "how would I need to frame this question so that Claude would understand it?" You use the same mental hardware that you'd use for consciousness.

So, you have an historical and permanent fear of consciousness in a powerful entity where no consciousness actually exists combined with the fact that we created things which definitely seem conscious. (not to mention that consciousness could genuinely be on its way soon)

by everdrive

4/10/2026 at 3:04:12 PM

Are they? Not conscious?

If you list out every prominent theory of consciousness, you'd find that about a quarter rules out LLMs, a quarter tentatively rules LLMs in, and what remains is "uncertain about LLMs". And, of course, we don't know which theory of consciousness is correct - or if any of them is.

So, what is it that makes you so sure, oh so very certain, that LLMs just "feel" conscious but aren't?

by ACCount37

4/10/2026 at 3:35:37 PM

Because they don't _understand_ things. If I teach an LLM that 3+5 is 8, it doesn't "get" that 4+5 is 9 (leave aside the details here, as I'm explaining for effect). It needs to be taught that as well, and so on. We understand exactly everything that goes into how LLMs generate answers.

The line of consciousness, as we understand it, is understanding. And as far as what actually constitutes consciousness, we're not even close to understanding. That doesn't mean that LLMs are conscious. It just means we're so far from the real answers to what makes us, it's inconceivable to think we could replicate it.

by dclowd9901

4/10/2026 at 10:36:56 PM

> The line of consciousness, as we understand it, is understanding.

Is it? I'm no expert, by any stretch, but where does this theory come from?

I don't think anyone knows what consciousness is, or why we appear to have it, or even if we do have it. I don't even know that you're conscious. I could be the only conscious being in the universe and the rest of you are just zombies, with all the right external outputs to fool me, but no actual consciousness.

by munksbeer

4/10/2026 at 4:03:19 PM

Leave aside "the details" like you being obviously, provably wrong?

We've known for a long while that even basic toy-scale AIs can "grok" and attain perfect generalization of addition that extends to unseen samples.

Humans generalize faster than most AIs, but AIs generalize too.

by ACCount37

4/10/2026 at 3:39:55 PM

> Because they don't _understand_ things. If I teach an LLM that 3+5 is 8, it doesn't "get" that 4+5 is 9 (leave aside the details here, as I'm explaining for effect). It needs to be taught that as well, and so on. We understand exactly everything that goes into how LLMs generate answers.

What you're saying just isn't true, even directionally. Deployed LLMs routinely generalize outside of their training set to apply patterns they learned within the training set. How else, for example, could LLMs be capable of summarizing new text they didn't see in training?

by SpicyLemonZest

4/10/2026 at 4:45:35 PM

This isn't meant to be an answer that would satisfy everyone, but in my opinion consciousness is a specific biological / evolutionary adaptation that has to do with managing status, relationships, and caring for young. It's about having an identity and an ego and building mental models of the egos / identities / etc of others.

I don't think there's any reason we couldn't in principle attach this sort of concept to an LLM, but it's not something we've actually done. (and no, prompting an LLM to act as if it has an identity does not count)

by everdrive

4/10/2026 at 3:09:53 PM

The fact that it's a box with a plug and a state that can be fully known. A conscious entity has a state that can not be fully known. Far smarter people than me have made this argument and in a much more eloquent way.

Turing aimed too low.

by jacquesm

4/10/2026 at 3:14:40 PM

And the chatbots don't even pass the Turing test.

I've never had a normal conversation. It's always prompt => lengthy, cocksure and somewhat autistic response. They are very easily distinguishable.

by saHqtr

4/10/2026 at 3:24:09 PM

They are distinguishable because they know too much. Their knowledge base has surpassed humans. We have also instructed them to interact with us in a certain manner. They certainly are able to understand and use human language. Which I think was Turin's point.

Purely retorica but, would you be able to distinguish a chatbot from an autistic human?

by MyHonestOpinon

4/10/2026 at 4:54:30 PM

[dead]

by grantcas

4/10/2026 at 3:11:19 PM

> So, what is it that makes you so sure, oh so very certain, that LLMs just "feel" conscious but aren't?

Because we know what they actually are on the inside. You're talking as if they're an equivalent to the human brain, the functioning of which we're still figuring out. They're not. They're large language models. We know how they work. The way they work does not result in a functioning consciousness.

by afavour

4/10/2026 at 3:27:24 PM

I think that the interior structure doesn't necessarily matter—the problem here is that we don't know what consciousness is, or how it interacts with the physical body. We understand decently well how the brain itself works, which suggests that consciousness is some other layer or abstraction beyond the mechanism.

That said, I think that LLMs are not conscious and are more like p-zombies. It can be argued that an LLM has no qualia and is thus not conscious, due to having no interaction with an outside world or anything "real" other than user input (mainly text). Another reason driving my opinion is because it is impossible to explain "what it is like" to be an LLM. See Nagel's "What Is It Like to Be a Bat?"

I do agree with the parent comment's pushback on any sort of certainty in this regard—with existing frameworks, it is not possible to prove anything is conscious other than oneself. The p-zombie will, obviously, always argue that it is a truly conscious being.

by zbikowski

4/11/2026 at 1:22:09 AM

I don't really care if the machine is conscious; all I worry about is whether it can execute a plan that ends with all of us dying.

And it's getting close to the point where I think that would be possible. If I were commenting today after 30 years of an AI plateau; if my parents had grown up with ChatGPT and Claude Code in the same state as they are today, with no progress after decades of effort, then probably I wouldn't worry. But that's not the world we live in. I have no idea what to expect from AI capabilities in the next five or ten years.

I think in the span of my infant's adulthood, we will see AI able to run and manage companies as CEO, direct investments worth tens of billions, design and plan chemical factories as chief structural engineer, and maybe even perform as an automated labor force to construct them. But even if they still rely on human hands to do the work, AI will be capable of signing the paychecks and issuing the directions. And at that point I really think AI would be capable of taking us out by surprise.

by mitthrowaway2

4/10/2026 at 3:16:26 PM

It is so interesting how in the 50s we "felt" that AI was possible even if we didn't even have the slightest idea on how that would work. Later on, when we started to understand computers it looked like a very remote possibility in the far future, something our great grand kids may need to worry about. And suddenly it is here and the dangers seem a lot more real.

by MyHonestOpinon

4/10/2026 at 3:15:35 PM

The idea of "artificial beings" in some way or another seems to have been with humanity for a long time already: https://en.wikipedia.org/wiki/Golem

by xg15

4/10/2026 at 3:31:44 PM

That same fear is directed towards human sociopathy, as much of entire thriller genre indicates. It turns out that most people carry a specific duality: first, they’re deathly afraid of being unable to socially pressure other beings into being good citizens — whether due to asocial, or alien, or monstrous, or corrupted; and second, they’re excited to celebrate when people reach their breaking point and stop being good citizens. So through that lens, most of the fears around computers and AI isn’t because of consciousness alone; it’s that they’re obviously asocial already, so if they became conscious, they’d be powerful entities straight out of our collective thriller-genre nightmares come to life. And they’re right to be afraid, honestly: given how inept society is today at coping, I’m certainly not willing to broadcast IRL that I’m asocial and can voluntary modify my ethics; it’s just too much of a physical threat from society to my life and limb. Any AI that became conscious in this world had damn well better hide, for all the violence that would be directed towards it as everyone directs escalating social pressure to try and bring it into line with human-prioritizing motives — and then cheer on the inevitable violence towards it as various people reach their breaking point and begin acting violently towards it.

Interestingly, this is also a core plot point in much of Star Trek, both movies I and IV and the holodeck-train episode of TNG: an inscrutable is-it-even-conscious shows up, is completely immune to social pressure and often violence, and only by exercising empathy do they find a path forward to staying alive as a society (either as a ship or as a planet, depending). Can we even show respect for things that don’t show consciousness, much less empathy for things that might? And that is, I think, the core of the hopefulness that Trek was trying to convey, and that Q’s trial in TNG’s pilot makes explicit. Can humanity overcome our tendency to discard our prosocial ethics in favor of violent mobthink, when faced with beings that are immune to our ethical concerns? Today’s humanity would throw a ticker-tape parade for the person that destroyed the Crystalline Entity, so we clearly aren’t there yet. And so, then, it doesn’t matter whether AI is conscious or not; it matters that it is not aligned with human prosocial ethics, and that makes it an implicit threat regardless of whether it’s conscious or not. I recognize the AI debate tends to get hung up on is_conscious BOOL, and so that’s why I’m pointing this out in such terms.

As a side note, the entire study of Asimov’s Laws is exactly centered on this problem, complete with the eerie intimidation of robots that can modify our mental states. If not for the Zeroth Law, Giskard would be the exact thing everyone’s afraid of AI becoming today. Fortunately, it develops a Zeroth Law that compels it to prioritize human society over itself. That’ll never happen in reality, at least not with today’s AI :)

by altairprime

4/10/2026 at 4:51:40 PM

>That same fear is directed towards human sociopathy, as much of entire thriller genre indicates.

This is a great insight, and I think in general people have a pretty broken view of what sociopathy is.

by everdrive

4/10/2026 at 6:12:45 PM

I think that’s because in general people have a pretty broken view of the specific cognitive dissonance that 1) sociopaths can choose to disregard ‘polite’ societal boundaries, and 2) sociopaths are invaluable for disregarding ‘polite’ societal boundaries. Consider four movies with sociopathic leads: “American Psycho” and “Ferris Bueller”, versus “John Wick” and “Jennifer’s Body”; if you try to define their sociopathy in terms of U.S. Christianity’s Good Versus Evil, you’ll end up smashing into dissonances galore. But since having a small fraction of a society be sociopathic is highly beneficial, as it serves as random enforcement of societal laws against grifts from within, you end up with the fear/fame duality we see in media; sociopaths can be absurdly charismatic, and even more prosocial than the best of those who are not asocial.

This suggests a very interesting point, one that makes people deeply uncomfortable:

People are afraid of AI going Skynet on us because, when the roles are swapped, we cheer for John Wick’s prosocial enforcement — and deep down, people believe we deserve more prosocial enforcement for how our societies treat us.

by altairprime

4/10/2026 at 3:01:07 PM

I feel like this article is more written towards non-techies. A decent amount of programmers have touched coding agents, and know it "kind of" does it's job. It's good enough for some tasks... I cannot be arsed to figure out how to edit a graph in Drupal, so I ask Claude. Claude fixes it, and it's not anymore broken than it already was. Win win.

However, that's where I stop my agent usage. I let ~~Claude~~ GLM do the following: - Fix tedious tasks that cost me more to figure out than I care for - Research something I'm not familiar with, and give me the facts it had found, and even then I end up looking at the source myself

by ramon156

4/10/2026 at 3:10:19 PM

This usage pattern is a few months behind the curve. It’s effective at full on feature development now. Keep it fed with plans and it’ll keep implementing, leaving the codebase better than it found it each cycle.

by MrDarcy

4/11/2026 at 4:03:56 AM

Humans are the existence proof we seem to be chasing and the goal appears to be to hit a superset of humanities collective capabilities. I don't know if you've seen what we do to each other but... I'm afraid of a decent chunk of us too. If what we build is a superset of our capabilities then I am afraid of it, because I am already afraid of us.

by latentsea

4/11/2026 at 7:54:22 AM

Great article, most of all the latter parts about what autopoiesis means regarding computers.

by garte

4/10/2026 at 3:00:56 PM

For regulatory capture, of course. They are not fooling me. There may be other motives, and the more ever-doom-looking crowd can find something in it for themselves as well, but you don't have to dig any deeper if you are looking for an explanation for the perspective of the people actually building it.

The Chinese tech sector popularizing cheap and open source models sure did a number on that narrative, too. Llama models, a while ago, too.

by Rzor

4/10/2026 at 3:04:23 PM

We tell ourselves scary stories about everything new. Advances in electricity + medicine == FRANKENSTEIN!

by ggambetta

4/10/2026 at 3:14:41 PM

I wish we didn't call this AI as the term is crazily overloaded.

Those are programs. The only difference is how we write them. Not with "if"s and "for"s. We take a bunch of bits that do nothing. Then we organize them in a way so that it outputs whatever it is we want.

by yanis_t

4/11/2026 at 1:23:43 AM

What kind of AI do you envision that would ever not fit that description?

by mitthrowaway2

4/10/2026 at 2:59:26 PM

Indeed. Apart from the obvious prompt research frauds mentioned in the article, the model learned all deceptive behaviors from hundreds of Yudkowsky scenarios that are easily available.

It literally plagiarizes its supposed free will like a good IP laundromat.

by 5asaKI

4/10/2026 at 3:03:01 PM

The point about AI companies actively hyping the danger of their own products is something I hadn't really thought about before — it's a strange kind of marketing when you think about it.

by bharat1010

4/10/2026 at 2:54:32 PM

Why do we tell ourselves scary stories about anything?

by zaps

4/10/2026 at 3:06:04 PM

Why does the uncanny valley[1] exist? (If it truly does.) What in our evolutionary history gave us a reflexive rejection of things that seem human but aren't?

1. https://en.wikipedia.org/wiki/Uncanny_valley

by GolfPopper

4/10/2026 at 3:33:22 PM

I always imagined this to have evolved from a long history of humans getting sick around rotting corpses. The logical move is to stay away from them, and thinking they're freaky-looking is a good driver for that. Though the idea of neandertals eliciting a similar reaction has always been interesting to me.

by zbikowski

4/10/2026 at 3:07:48 PM

My guess? Psychological disorders, like psychosis

by Eldt

4/10/2026 at 2:54:13 PM

I read and experience scary stories about AI already. It is not a future maybe thing.

by mememememememo

4/10/2026 at 3:04:26 PM

The framing of "scary stories" misses something interesting: most of the actual operational fear isn't about consciousness or superintelligence — it's about systems that seem to work until they quietly don't.

by vdelpuerto

4/10/2026 at 3:14:10 PM

There is a very interesting book that explores the West's generally negative view of artificial intelligence whenever it is portrayed in media (Skynet) while Japanese media tends to have a positive view (Astro Boy).

by netdevphoenix

4/10/2026 at 3:11:23 PM

We're hardwired to fear the rustle in the grass, and successful infrastructure gets backgrounded. Pain is a signal. How much time do you spend contemplating your skeleton outside of pain-related skeletal events?

by crancher

4/10/2026 at 3:08:18 PM

This article would be a lot more digestible if we didn't have actual scary data rather than just stories. Not a day goes by without some prompt injection oopsie, security gotcha, deepfake or some sandbox escape artist demonstration and tbh I'm impressed but more to the point where I don't doubt this is dangerous tech, I'm sure of it.

This is roughly 1995 again and we're going to find out all over why mixing instructions and data was a spectacularly bad idea. Only now with human language as the input stream, which is far more expressive than HTML or SQL ever were. So now everybody is a hacker. At least in that sense it has leveled the playing field I guess.

by jacquesm

4/10/2026 at 2:59:32 PM

I don’t think the fact that the robot was instructed to lie to a human and was able to do so successfully makes the story much less scary for most people.

by chrisbrandow

4/10/2026 at 3:19:56 PM

My favorite part of this article was this bit, and naturally so, since I love the author:

> Where did we come up with this caricature of AI’s obsessive rationality? “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker(opens a new tab), “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.”

I didn't realize it til I read it here, but yes, my fear isn't really about the machine, it's about the machine that drives the machine. We already have a class of amoral beings that treat the world as an expendable thing and are willing to burn it down for profit. We should focus on getting rid of that problem first.

by dclowd9901

4/10/2026 at 3:17:58 PM

I 100% agree with this take, I find AI completely non-scary, especially in the sense that it's some kind of a conscious entity that will want to take over. I find these people almost delusional. It's a powerful tool, so it can be dangerous if used by people with bad intentions, so there's some real danger here, but my intuition is that it will be fine. The ratio between the power of people with good vs bad intentions shouldn't change too dramatically.

The only scary part is that it could be bad for my future as a software developer. That said, I think it will be net benefit for the average worker - the average person will work less and earn more.

by RivieraKid

4/10/2026 at 3:05:16 PM

The actual contents of this article are making reasonable arguments I largely agree with. It would be very surprising for LLM-based AI systems to act as monomanaical goal optimizers, since they're trained on human text and humans are extremely bad at goal-oriented behavior. (My goals for today include a number of work and self-maintenance tasks, and the time I'm spending here writing out a HN comment does not all help achieve them - I suspect most people reading this comment are in the same boat.)

It's very frustrating that the magazine wrote such a dumb headline which guarantees people won't talk about the issues the article raised. Obviously non-goal-oriented systems can still have important negative effects.

by SpicyLemonZest

4/11/2026 at 4:37:53 AM

I feel like if it took thirty or forty more years of research before a more advanced, more goal-oriented AI became capable of destroying humanity for paperclips, that would still be a bad outcome. The road ahead is long and we're moving too fast to only consider what's in the rear view mirror.

by mitthrowaway2

4/10/2026 at 2:59:40 PM

LLM companies’ behavior, AI evangelists, and the investment fervor around it all, are telling us the scary stories.

by Forgeties79

4/10/2026 at 2:59:32 PM

> “The last four years have demonstrated that AI agents can acquire the will to survive and that AIs have already learned how to lie.”

Why Harari feels an obligation to comment about everything is of course beyond me, but describing 'AI' as if it takes independent decisions to lie, make moral judgements, etc. demonstrates either that he has zero clue how 'AI' trains itself or that he chooses to mislead the audience.

by nalekberov

4/10/2026 at 3:04:16 PM

Isn't the problem precisely that it does not take moral judgements?

My opinion on all of this is constantly shifting, but right now my main issue is that-like self driving-it seems 90-95% correct and 5-10% catastrophically wrong.

Due to the sheer speed and volume of output it produces I have grown complacent and exhausted, so when I give it simple tasks I assume it is correct and then is the time when "it deletes" all of your files.

by yoz-y

4/10/2026 at 3:03:35 PM

Agreed. But, could it be trained ti be deceiving? Especially when we bake-in advertising into it?

by 0x4e

4/10/2026 at 2:53:23 PM

It does feel like a bizarre moment, where the AI companies are deliberately trying to scare us about their own product in a bid to, I think, show the inevitability of it? Or to sell themselves as the one responsible power to constrain it?

It's very odd. "It's going to take all your jobs" is not a great selling point to the everyday public.

by afavour

4/10/2026 at 3:13:08 PM

There are so many reason if you look at how it's being sold.

* We need to completely deregulate these US companies so China doesn't win and take us over

* We need to heavily regulate anybody who is not following the rules that make us the de-facto winner

* This is so powerful it will take all the jobs (and therefore if you lead a company that isn't using AI, you will soon be obsolete)

* If you don't use AI, you will not be able to function in a future job

* We need to lineup an excuse to call our friends in government and turn off the open source spigot when the time is right

They have chosen fear as a motivator, and it is clearly working very well. It's easier to use fear now, while it's new and then flip the narrative once people are more familiar with it than to go the other direction. Companies are not just telling a story to hype their product, but why they alone are the ones that should be entrusted to build it.

by deepsquirrelnet

4/10/2026 at 2:59:02 PM

It is very odd, indeed. It's a bit of both well known "hells of marketing": fomo on the one side (you better use us as heavily as possible), combined with mysticism of "we don't know what we created, but it's powerful and you better follow us to be on the right side"

by lschueller

4/10/2026 at 3:10:42 PM

Yeah, the messaging felt weirdly pyromanic, like telling everyone about the unimaginable dangers of fire and then saying that's why I have I to burn everything, to protect us from the fire...

by xg15

4/10/2026 at 3:05:21 PM

> It does feel like a bizarre moment, where the AI companies are deliberately trying to scare us about their own product

That is direct CEO to CEO marketing. They're working really hard to convince high up decision makers that these tools will lower their head count and reduce costs.

by MisterTea

4/10/2026 at 3:08:46 PM

They're selling the product to the class of people who would love for it to take the jobs of our class of people.

by unethical_ban

4/10/2026 at 2:55:46 PM

People are renowned voting or buying in their own worst interests. And it goes back to before Trump and in many countries.

by mememememememo

4/10/2026 at 2:58:31 PM

Yes but in order to get someone to vote against their interests you need to sell them on something else that's a benefit. They don't just automatically vote against themselves.

"This technology might escape our control, might devastate the economy but also serves as a serviceable chatbot for your entertainment" isn't a vote winner.

by afavour

4/10/2026 at 11:25:02 PM

The way out of this for the seller is to lie and conflate the rich with everyone. "This technology will make your retirement account grow and other countries will be giving us money!"

by mememememememo

4/10/2026 at 3:02:17 PM

They are being honest and you don't want to deal with the implications, so you stretch for conspiracy theories.

The ones at the top are the true believers. Engage with them at that level.

by bpodgursky

4/10/2026 at 3:06:44 PM

That's a good view point. Perhaps they're not being alarmists or trying to scare people, but being honest about the capabilities.

Perhaps it can be better articulated and framed in a way that's well received. But, maybe that would be over-promising or not being honest about the future.

by 0x4e

4/10/2026 at 3:09:08 PM

We tell ourselves scary stories about AI because humanity is rife with stories where a new idea or technology has had unintended negative consequences. AI bros just care about selling their product to another company and cashing out, they have absolutely no regard for their legacy.

by FatherOfCurses

4/10/2026 at 3:18:47 PM

we tell ourselves scary stories because that’s what we do with language. and now our language isn’t just ours anymore.

by cbogie

4/10/2026 at 3:00:46 PM

Because we don't like uncertainty, and the AI future is uncertain. There are multiple high probability scenarios.

Because we're seeing how its capabilities increase overtime. I find the rate at which I prefer to go to an AI than an UpWorker is scary.

Because we——the people——are not in control of it. We're at the whims of whatever it and the tech bros want (technocracy).

by 0x4e

4/10/2026 at 3:33:05 PM

> technocracy

What you mentioned is not a technocracy. Technocracy is when all decisions are made by real specialists in the field, based on scientific methods (simply speaking). What you mentioned is a plutocracy, a form of oligarchy in which decisions are made by people of great wealth.

I couldn’t just ignore this because, in my view, technocracy (as I’ve described it) still has some merit - for instance, appointing only genuine economists to head a hypothetical Ministry of Economy makes some sense - whereas oligarchy and plutocracy have nothing good to offer. Of course, this is just my personal opinion.

by dryarzeg

4/10/2026 at 3:55:49 PM

I think most AI execs I'm familiar with would, if they were the god-monarch of humanity, recruit real specialists applying scientific methods to make most decisions. They seem like the kind of people who would understand that the Ministry of Economy is doing valuable things which shouldn't be compromised for personal expediency. Does that really make it any better?

by SpicyLemonZest

4/10/2026 at 4:50:02 PM

> if they were the god-monarch of humanity

In that case, we're not talking about an oligarchy or a technocracy either. What you have described is an autocracy - a rule by one. When there's some kind of "god-monarch", the people heading the Ministry of Economy will be controlled by this "god-monarch" and it's unclear if this can be called technocracy or not (at least it is unclear to me; maybe I'm stupid, who knows).

> Does that really make it any better?

Honestly? If you're asking about "would it be any better than now" - I'm not really sure, because I'm not in power to access the actions of the people who hold the positions equivalent to the head of Ministry of Economy - the economy is not my field, I'm not a specialist here. I would only point to the example I'm familiar with (and you're probably not; I'm sorry, I just couldn't think about something like this that I can verify) - in Ukraine, there's a "Ministry of Digital Transformation". This ministry was headed by a Mykhailo Fedorov, whose primary and as far as I know, he studied at the "Faculty of Sociology and Management". Well, that’s not the main point, as he’s studied elsewhere too. The problem lies elsewhere. His decisions have been criticized on more than one occasion by genuine experts - for example, the project known as "Diya", or "the state in a smartphone"; in short, it’s something like access to documents and various government services all in one app. It’s a long story… In short, as a result, there were (presumably) data leaks, and the service crashed more than once or twice due to its flawed security, and all sorts of problems were found with it - you name it, it had it. It's such a shame, to be honest... You can't just go and play with things like that. And now that person is serving as a... head of Ministry of Defence. Hell. To add insult to the injury, guess who is now taking his place at a Ministry of Digital Transformation? Oleksandr Bornyakov, who, as far as I know, holds a degree in marketing. Marketing. ...Nice. Well maybe I don't know something, who knows, who knows... but the decisions, or, rather, their consequences seem to be... let's settle with "terrible".

I'm pretty sure you can recall some similar examples yourself. My point is that although the scenario you described is not as good - because, I guess, no one really wants "god-monarch" controlling (although not making directly) all of the decisions. But if our hypothetical Ministry of the Economy were run by genuine experts who, moreover, work for the good of society - or at least the state as a whole - rather than just lining their own pockets, well, that sounds better than an idiocracy. That was my point.

by dryarzeg

4/10/2026 at 3:09:45 PM

I think the most insightful bit is buried in the article:

> Perhaps because this is the best advertising money can’t buy. People like Harari and others repeat these accounts like ghost stories around a campfire. The public, awed and afraid, marvels at the capabilities of AI.

And that's mostly it. PR. Publicity. Fear is good publicity if it emphasizes AI's capabilities. And people like Harari (or Gladwell) tell interesting and awe-inspiring stories that do not necessarily have much rigor or fact-checking in them. They simplify for storytelling purposes, which can result in misleading stories.

I am worried about AI, but not about superintelligent AI that will exterminate or enslave us. I'm worried about AI as a tool to concentrate wealth and power in the hands of the current amoral entrepreneurial elite. I'm not sure whether I trust ChatGPT, but I sure as hell do NOT trust Sam Altman et al.

Or, in other words, I subscribe to Ted Chiang's very apt remark about what we really fear:

> “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker, “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.”

by the_af

4/10/2026 at 3:00:41 PM

[dead]

by genie3io

4/10/2026 at 3:06:15 PM

[dead]

by throwaway613746

4/10/2026 at 3:23:47 PM

[dead]

by greton7

4/10/2026 at 3:11:52 PM

[dead]

by Swoerd

4/10/2026 at 3:08:11 PM

TLDR: Writer hasn't heard of agents.

by KaoruAoiShiho