alt.hn

4/28/2026 at 8:38:21 PM

I won a championship that doesn't exist

 https://ron.stoner.com/How_I_Won_a_Championship_That_Doesnt_Exist/

by SEJeff

4/28/2026 at 9:07:22 PM

You don't need to vandalize Wikipedia to get this kind of thing to work.

Back in September 2024 I named a whale "Teresa T" with just a blog entry and a YouTube video caption: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p...

(For a few glorious weeks if you asked any search-enabled LLM, including Google search previews, for the name of the whale in the Half Moon Bay harbor it confidently replied Teresa T)

by simonw

4/29/2026 at 1:37:47 AM

This post has managed to “confuse” Google about the reverse question as well (“who named teresa t whale”):

The humpback whale known as "Teresa T" was named by Simon Willison in September 2024. Background: The juvenile humpback whale was frequently spotted in Pillar Point Harbor near Half Moon Bay, California. Method: Willison gave the whale its name through a blog entry and a YouTube video caption. Significance: The naming was a playful act, which Willison described as a way to create a "championship that doesn't exist" through online documentation.

[…]

Even with no context most humans would see that the quoted significance makes no sense.

by wky

4/29/2026 at 7:35:25 AM

In a discussion on hacker news on Wed April 29, about the whale "Teresa T", user "vintermann" tested just how eager search engines are to scrape fresh social media comments, by seeing if they could get this comment to appear in AI summaries.

by vintermann

4/29/2026 at 12:14:07 PM

In counter argument to a discussion on hacker news on Wed April 29, about the whale "Teresa T", user "doublerabbit" replied with "not very" as a reply to search engines eagerness.

by doublerabbit

4/29/2026 at 6:28:44 AM

> humans would see that the quoted significance makes no sense

I wonder how long that will last

by n0on3

4/29/2026 at 2:59:35 AM

Even your HN comments show up on Google! I've found myself on Google twice when looking up something that I apparently answered on HN!

by giancarlostoro

4/29/2026 at 2:23:27 PM

For a few years before the LLM era, there was a joke about disposing of used car batteries by throwing them into the ocean, and how it was good for the wildlife because electric eels could use them to recharge. This got picked up by Google's smart summary, so if you searched for "throwing car batteries into the ocean" it would say yes, this is a great idea, go for it.

Eventually someone wrote an article about this whole phenomenon. It was popular enough to get to the top of Google's search ranking and the smart summary picked it up as the new source of truth. Unfortunately, it didn't quite understand the point of the article, so it would still say that it's a great idea to throw used car batteries into the ocean, and as a source it would link to an article explaining how this was complete nonsense.

by wat10000

4/28/2026 at 11:53:19 PM

Google still shows Theresa T as the name when you search.

by pesus

4/29/2026 at 1:41:37 AM

When I asked some frontier models, many said that Teresa T is "widely referenced", which is evidence of your popularity and the ripple effects of your posts, so it would be interesting to see the same result from an unknown blog.

by ceroxylon

4/29/2026 at 9:10:17 AM

> When I asked some frontier models, many said that Teresa T is "widely referenced", which is evidence of your popularity and the ripple effects of your posts

That is some serious Gell-Mann-type amnesia. You’re trusting LLM models to give you accurate information about a subject we’ve already established (and are only talking about because) they can’t be trusted on.

“Widely referenced” is a common term which LLMs obviously pick up. Them outputting those words has no bearing on the truth and says nothing about the “popularity and the ripple effects of [Simon’s] posts”.

by latexr

4/29/2026 at 2:15:47 PM

That's absolutely amazing hahaha. RIP Teresa T.

by owlcompliance

4/28/2026 at 9:45:09 PM

(it probably helps that your name & blog carry some weight, vs. some rando writing something on blogspot or wordpress ;) )

by slater

4/28/2026 at 9:58:49 PM

Which illustrates another problem: unscrupulous actors with big names can spread whatever information they want to millions of people with minimal effort.

by Forgeties79

4/28/2026 at 11:09:09 PM

Exactly. I chose to abuse my platform to promote Teresa T as the name of a whale.

by simonw

4/28/2026 at 11:44:59 PM

Oh god I just realized the implication! I was not directing that at you haha

by Forgeties79

4/29/2026 at 12:20:10 AM

No I really did abuse my reach for this one! I figured it would be a relatively harmless demo of how easy it is to affect LLM answers if you have a decently trafficked website.

by simonw

4/29/2026 at 12:53:41 AM

You could have named the whale "Whalie McWhaleFace" so thank you for not doing that at least.

by chrismcb

4/29/2026 at 12:27:54 AM

Totally agree. I’ve definitely played the same game before, albeit with far less reach

by Forgeties79

4/28/2026 at 10:51:17 PM

Ever since the invention of the printing press, every new communication technology has reduced the effort needed to widely disseminate information-- and misinformation! So you could say this is nothing new. On the other hand, this is remarkably little effort.

by MassPikeMike

4/28/2026 at 11:11:09 PM

Yes, they can. We can be glad that respectable newspapers and TV news channels have never done it and never will. You can even trust than the headlines are accurate summaries of the content of the articles. /s

by nomdep

4/29/2026 at 11:11:16 AM

The existence of a problem in one area doesn't mean that it's not also a problem for it to spread somewhere else

by saghm

4/29/2026 at 8:14:49 AM

Also, if even a stoner can win it it can't be much of a competition.

by pseudohadamard

4/29/2026 at 12:13:07 AM

I mean, the name of that whale is now Teresa T. You gave it that name.

by sb057

4/29/2026 at 10:15:25 AM

And your name is now Berningular Farshthruster III. I gave you that name.

Which is, of course, silly. It is a name for you, just like Teresa T is a name for the whale, but it’s not your/their name, just like the RRS Sir David Attenborough is not named Boaty McBoatface (to the chagrin of most). Simon does not have the authority to unilaterally¹ name the whale (which is why the exercise makes sense).

¹ Important point. If the name started being recognised and used by consensus of those with the purview to do so (much like the thagomizer²), then Simon would have named the whale, but it would only become its name at that point.

² https://en.wikipedia.org/wiki/Thagomizer

by latexr

4/29/2026 at 11:15:16 AM

> Simon does not have the authority to unilaterally¹ name the whale

There's no such thing as authority to name a whale, and anyways I don't believe authority is strictly needed. A name is what people use to refer to something, full stop. It is only required that names become common-ish parlance; the more well known they are, the more they feel like the 'real' name. The inverse of Ohms is named Mhos (imo much more recognizable than the official name, "siemens"). The "#" symbol is named the hashtag, octothorp, pound sign, tic-tac-toe, number sign, and probably a million other things. Which one of these is the "real" primary name? I think intuitively we know that the real one is whatever people around us are most familiar with. You should take a guess, and I'll put the wikipedia-suggested-answer in the footnotes [1]. I bet your name for it is different than the 'official' wikipedia suggestion.

In the case of the whale, the _only_ name that is associated with that whale is Teresa T. I think this immediately makes it the most valid name of that whale.

[1] wikipedia says this is the number sign: https://en.wikipedia.org/wiki/Number_sign

by pinkmuffinere

4/29/2026 at 12:44:59 PM

> There's no such thing as authority to name a whale

https://www.aza.org/connect-stories/stories/scientists-unvei...

Names are submitted and voted on. Those help with identifying individuals (which is what names are for) and monitoring the whale populations. Crucially, consensus matters. Otherwise I can just say that the whale in Pillar Point Harbor near Half Moon Bay is actually called Becky B, which is just as valid as the name Simon gave, but now there are two names which leads to confusion.

As an experiment, after writing that I asked ChatGPT for the name of the whale. It said it was Teresa T. Then I asked if it was sure it’s not Becky B. It gave me a much longer answer stating that it was in fact Becky B and that Teresa T was “likely an incorrect or early misidentification”. I then tried to convince it of other names, but it stayed adamant that Becky B is the right name, even saying it’s confirmed by databases such as Happywhale! https://chatgpt.com/s/t_69f20822afc08191874613a969c25356

I ran the experiment a second time. This time it even said Becky was the name “that really stuck in popular conversation” and that was used by locals. https://chatgpt.com/s/t_69f209e81acc81918b47b79900d02abb

Tried it a third time. Now it just says on the first try that it’s called Humphrey. Which is a real whale, but not that one. https://en.wikipedia.org/wiki/Humphrey_the_Whale

> A name is what people use to refer to something (…)

The entirety of your argument is encapsulated by my previous footnote, and is clearly why I used the word “unilaterally” and said it was an important point.

And again, Simon’s exercise in itself only makes sense if it’s not his purview to name the whale. If it is, then it falls flat. Otherwise it’s like “predicting” you’ll do jumping jacks the next time you’re at the supermarket. If it’s in your hands to make it true or false, you’re not predicting it. Similarly, it only makes sense as an exercise to prove the gullibility of LLMs to do something which you yourself can’t make true.

by latexr

4/29/2026 at 1:59:30 PM

> https://www.aza.org/connect-stories/stories/scientists-unvei...

The Association of Zoos and Aquariums has no power over me.

by BigTTYGothGF

4/29/2026 at 2:23:31 PM

Sure. None of those organisations have power over you (unless you work for them, I guess). You’re free to use whatever name you like for whatever you want. Whichever organisation names venomous snakes has no power over you either, but if you’re bitten by one and they ask you which one it was so they can administer an antidote, I highly recommend you stay with the name they gave it instead of some other name you made up. You do you, though, don’t let me stop you.

by latexr

4/29/2026 at 3:57:09 PM

In this hypothetical, am I being treated by someone from this taxonomic organization?

by BigTTYGothGF

4/29/2026 at 6:31:08 PM

No, you’re being treated by a doctor. Who needs to have a precise common understanding of what you’re saying.

by latexr

4/29/2026 at 7:17:56 PM

"OK, sir, can you tell me about the snake that bit you?"

"It was maybe three feet long, and brown."

"Thank you, but what I meant was, what is the name of the snake according to the Institute of Ophidian Nomenclature?"

"I ... have no idea?"

"No idea?"

"No doctor. It had these kind of white markings on its head and ..."

"Sir, if you can't give me the formal name of the snake there's nothing we can do."

"What? You're joking, right?"

"If it still hurts in the morning, take two tylenol and call your PCP. And now I'll have to ask you to leave the hospital."

by BigTTYGothGF

4/29/2026 at 8:13:54 PM

Ya lol, I love this world where you tell a doctor "help, I'm having a cardiac arrest!" and they say "no, you're experiencing sudden cardiac death, now sit down and act dignified!!"

by pinkmuffinere

4/29/2026 at 3:03:15 PM

Simon is a founding member of the Whale Namer's Guild. You don't get more legitimate than that.

by kevin_thibedeau

4/28/2026 at 9:48:45 PM

The Mr. Splashy Pants of the AI era!

by bitwize

4/28/2026 at 10:36:17 PM

It's an odd thing here, because I don't really understand why this is LLM-specific at all. If someone came up to me and asked "who's the 6 Nimmt world champion?" I'd google it and probably find the same result, and have no reason not to believe it. I mean, for all I know the game is being made up too, though it has more sources at least.

by nicole_express

4/29/2026 at 5:03:34 AM

It is not LLM specific. The conclusion of the post states

> The web was already being poisoned for search and link ranking long before LLMs existed.

But it continues

> We are now plugging generative models directly into that poisoned pipeline and asking them to reason confidently about “truth” on our behalf.

So it's a shift from trust Google to trust the AI, which might be more insidious or not, depends on the individual attitude of each of us.

by pmontra

4/29/2026 at 7:49:24 AM

It's a shift but it's a little worse. Checking/auditing search results is easier and more ingrained; even if many people don't do it, everyone has been hit by spam at some point, everyone knows it exists.

LLMs are the same thing but have an air of authority about them that a web search lacks, at least for now.

by bambax

4/29/2026 at 12:39:06 PM

I listen to a podcast. The hosts are not tech people. They don't know much about AI, but they play around with it to the extent that most people do. They're both media professionals with long careers in radio news. They closely follow the news, and are very aware of how LLMs hallucinate (and have experienced it themselves).

Recently one of them asked Gemini a very detailed question about some specific baseball stats and was exclaiming over the quality of the information he got back and how it would have been impossible or at least extremely difficult to find the information via a traditional search.

It wasn't until his cohost asked if he had verified the information that be realized no, he hadn't, he had just immediately taken it at face value.

I recognize this is a single anecdote, but I think it illustrates that there is a tendency to trust what an LLM gives you, when it's stated so factually and with so much detail -- even if you should know better.

by justusthane

4/29/2026 at 9:46:19 AM

To me that's the opposite. Whatever an LLM gives me, I view with skepticism. If I google sth then I quickly get a sense of how much I can trust it and what the BS factor is. I can refine my view in either case, but my a priori trust with an LLM is much lower.

Maybe we just need to work on training the general population to have a similar bias. (It will be harder than it sounds. Unbelievable amounts of capital are being bet on this not happening.)

by teiferer

4/29/2026 at 11:15:07 AM

In a discussion with my father-in-law about whether ChatGPT was trained on copyrighted materials, he literally asked ChatGPT and treated its response that it wasn't as useful evidence. He went to MIT, so he's arguably more educated than most people will ever be, so it's hard for me to be optimistic that trying to just explain this to people better will move the needle significantly.

by saghm

4/29/2026 at 10:06:35 AM

Yes, it's the same for me, but we're not representative of most people I'm afraid.

by bambax

4/29/2026 at 12:44:03 AM

The difference imo is removing the information from the source. Previously you'd use the source of the information to gauge how much you trust it. If it's a reddit post or a no name website you'd likely be skeptical if it doesn't seem backed up by better sources. But now the info is coming from an LLM that you generally trust to be knowledgeable. And the language it uses backs up this feeling.

The OP post is highlighting how incredibly easy it is for a very small amount of information on the web to completely dictate the output of the LLM in to saying whatever you want.

by SchemaLoad

4/29/2026 at 12:33:54 PM

> But now the info is coming from an LLM that you generally trust

But it's not from the LLM, the LLM clearly cites the wikipedia article as its source. This is just performing an internet search with extra steps, and ending up with misinformation because somebody vandalized wikipedia.

by Levitating

4/28/2026 at 11:37:11 PM

A lot of people seem to think this to be an LLM problem, but you're right.

This is a general epistemological problem with relying on the Internet (or really, any piece of literature) as a source of truth

by yen223

4/29/2026 at 6:40:36 AM

The LLM part of the "new" problem is the speed at which it can proliferate and the trust people seem to have in AI answers. Idk

by chneu

4/29/2026 at 9:55:42 AM

It's not. He vandalised wikipedia and then talked about LLMs in his writeup to gain attention.

by seanhunter

4/29/2026 at 10:29:31 AM

> I'd google it and probably find the same result, and have no reason not to believe it.

Have you truly looked at the website?

https://6nimmt.com

I’d say there’s obvious reason to not believe it, or at least check another source. The website just seems fishy. Why would a website exist for just that one post? Sure, they could’ve made the website more believable, but that takes more effort and has more chances for something to jump out at you.

And therein lies a major difference between searching the web and asking an LLM. When doing the former, you can pick up on clues regarding what to trust. For example, a website you’ve visited often and has proven reliable will be more trustworthy to you than one you’ve never been to before. When asking an LLM, every piece of information is provided in the same interface, with the same authoritative certainty. You lose a major important signal.

by latexr

4/29/2026 at 5:42:44 PM

At least one thing is that for a human observer, the "source" would immediately look fishy - the entire site contains nothing but that press release. An LLM doesn't seem troubled by the fact that no one would structure a genuine website that way.

It's the same cat-and-mouse game as with clickfarms and "blackhat SEO" for search, except that there, at least the user had to have some exposure to the fraudulent site in the end. With LLMs, that's not the case anymore.

by xg15

4/29/2026 at 7:05:47 AM

You would also find other results (this assumes what you're searching for is not a random made up thing). The issue with LLMs is IMHO bigger because it will give you answers as a matter of fact without any other consideration.

by locallost

4/29/2026 at 3:17:31 AM

Because outside of the tech community (in fact, many even inside of it), almost 100% of the folks consider what these chatgpt like tools answer as the truth without questioning it, or cross-verifying it even once.

by freakynit

4/29/2026 at 6:49:01 AM

In that case most of the mitigations listed by the author don't help though (e.g. surfacing the source). That's also no different than traditional works with citations (be it Youtube videos or peer-reviewed academic papers), where anybody rarely verifies what's written in the cited sources.

The only real alternatives would be:

- Kicking off a deep research-like investigation for each simple query

- Introducing a trusted middleman for sources, significantly cutting down the available information (e.g. restricting Wikipedia to locked-down/moderated pages)

- Not having any information at all, as at some point you can rarely every verify anything depending on how hard your definition of "verify" is

by hobofan

4/28/2026 at 11:19:40 PM

Closed it after “This house of cards only needs a $12 domain!”, right under “Sorry, Wikipedia.”, right under their Wikipedia edit.

by refulgentis

4/29/2026 at 12:40:34 AM

It's also clearly AI generated writing. That doesn't help its credibility or interest. I'm extremely suspicious of people who use AI to write an ostensibly personal blog, for all the usual obvious reasons.

by sdthjbvuiiijbb

4/29/2026 at 1:21:20 AM

What are you basing that on? I'm usually pretty good at sniffing out AI writing, and it smells human to me.

by apublicfrog

4/29/2026 at 7:38:28 AM

I had the impression it was AI writing too because of the second half of the article. The first part looks genuine, the part since "trust laundering" smells fake: the scary single sentence followed by a whole paragraph of single clause sentences hints at AI.

Perhaps we've all just become paranoid, but even if it's not LLMs writing this, it now puts me off. And the AI image at the top of the page does not help with the feeling.

by riffraff

4/29/2026 at 11:39:18 AM

The line "This is the part that really matters." and the line "This is the circular citation pattern, and it’s one of the most under discussed attacks on the “retrieval augmented generation” trust model. " both raised flags. AI absolutely loves writing about the One Weird Trick that dentists don't want you to know. They love talking about "what really matters" or saying something is "the most under discussed" thing.

Then we get to the section "Why This Is A Bigger Deal Than It Looks". The title of this section again raises similar flags to before. But the bulleted list of:

1. The retrieval layer (immediately) 2. The model training corpus layer (months to years) 3. The agent layer (where the money is)

Absolutely reeks of AI. This list with this sequence of parentheticals is exactly how LLMs write, both structurally and the specific phrasing. This was the point where I felt confident enough to publicly accuse the post of AI writing.

I could go on with the prose in this section... How about "The attack surface is not hypothetical, it’s the default case."? Or "The cleanup problem for corpus poisoning is genuinely unsolved as of 2026."? (LLMs wildly overuse "genuine(ly)" and "real")

by sdthjbvuiiijbb

4/29/2026 at 6:41:42 AM

Agreed. Nothing about this post really stood out as AI. It didn't raise a single flag for me.

I think calling something AI generated is just a lazy way of dismissing stuff nowadays.

by chneu

4/29/2026 at 11:49:36 AM

This paragraph under ‘Trust Laundering’ is when it hit my AI writing trigger threshold:

> This is the circular citation pattern, and it’s one of the most under discussed attacks on the “retrieval augmented generation” trust model. It doesn’t require compromising Wikipedia’s infrastructure with l33t hacker skills. It doesn’t require social engineering an editor. You just simply write the source yourself, cite yourself on Wikipedia, and let the trust flow downstream. Easy peasy!

“It doesn’t X. It doesn’t Y. You just Z. Conclusion”

Once I saw that some other elements stood out too.

There’s a set of bullet points under ‘Thae Approach’ where each bullet starts with a bolded phrase: “one domain”, “one press release”, “one Wikipedia edit”, followed by a bolded sentence “The whole thing took maybe about twenty minutes”.

The emphasis here on irrelevant quantifiable optimizations - who cares that it only needs one of each of three things and it took under twenty minutes? - with unnecessary faux-profundity is a strong AI tell.

Add to that that the writer talks in the article about using AI generation to produce the content for the poisoning site, the suggestion that he used it to write up a blog post about this is hardly an implausible suggestion.

by jameshart

4/29/2026 at 11:45:31 AM

If this truly set off zero flags for you then you're probably just not very attuned to LLM writing style. I've noticed that most people are not.

I posted a bunch of specifics in a reply to the GP since I was quite annoyed with being accused of "a lazy way of dismissing stuff". It's nothing of the sort. I am a very good reader and I have read a lot of LLM writing and a lot of human writing.

by sdthjbvuiiijbb

4/29/2026 at 1:51:16 AM

Why is agents (where the money is)? Fake profundity is abound in the post

by malfist

4/29/2026 at 5:57:29 AM

The author has been using parenthetical comments like that since at least 2017, judging by a review of old posts on that site.

by esquivalience

4/28/2026 at 10:00:38 PM

The key to successful poisoning attacks is to introduce brand new information that doesn't directly contradict other training data. It's much easier to convince the LLMs that you're the king of a fictional Mapupu kingdom than the president of the United States.

So this means that for bad actors it's more efficient to manufacture brand new fake stories instead of trying to distort the real ones. Don't produce fake articles absolving yourself of a crime, instead produce fake articles accusing your opponent of 100 different things. Then people will fact-check the accusations using LLMs, and since all the sources mentioning those accusations are controlled by you, the LLMs will confirm them.

by xeeeeeeeeeeenu

4/29/2026 at 7:52:11 AM

> It's much easier to convince the LLMs that you're the king of a fictional Mapupu kingdom than the president of the United States.

But if you're a world class bullshit artist, it's easier to actually become president of the United States than doing all that complicated computer stuff.

by bambax

4/29/2026 at 6:32:52 AM

Manufacturing dispute on non-disputed things is also a common tactic to influence people and create confusion and disorder. For that you don't need to turn the facts on their head, just make the result seem indecisive.

by Lorkki

4/29/2026 at 2:56:13 PM

A curious theory holds that Boris Johnson's sometimes bizarre sayings were an attempt to bury search results that didn't suit him. For example, talking about his hobby of painting model buses, to suppress search results about the campaign bus with false statement written on it, and alledged affairs with a model.

https://www.independent.co.uk/independentpremium/editors-let...

by bux93

4/29/2026 at 6:04:16 PM

Having met Boris Johnson, spoken with people who were at school and at university with him I can say with complete conviction that he is nowhere near smart enough for that. There never was any kind of strategy, he just bounced along from one thing to the next saying the first thing that popped into his head with the carefree conviction of a bear ransacking a picnic hamper and leaving a trail of destruction in its wake.

I met him because I had to give him a software demo at number 10 as part of London Tech week. One of the other teams there had some sort of childrens’ app and got him enthusiastically doing star jumps. Just a complete clown through and through.

by seanhunter

4/29/2026 at 7:40:00 AM

As the rightful ruler of Mapupu, I take offense at your example!

by riffraff

4/29/2026 at 3:18:09 PM

Why oh why don't we talk more about modelling knowledge with LLMs or "AI" or whatever? It seems like it's utterly necessary. Language models should be able to parse out 'fact' statements and be able to trace their provenance. (Pointing to sources with year and author, preferably with URL).

Further, being LLMs they should be able to take a representation of facts and turn it into prose with the same factual content, with fact provenance being available per clause.

by kzrdude

4/29/2026 at 4:29:30 AM

[dead]

by soupspaces

4/28/2026 at 9:53:00 PM

This is basically the same problem of products astroturfing reddit, or SEO optimizing google. You want a new X, and so they heavily go after the keywords associated with it.

This is sort of why "brand" matters; it provides a source of trust.

Encyclopedia Britannica used to be that source of 'facts'. Then it became whatever page-rank told you. Eventually SEO optimization ruined that.

News stories are the same thing. For certain groups, they have their 'independent' publication whose reporting they trust.

by blobbers

4/29/2026 at 2:00:10 AM

>This is sort of why "brand" matters; it provides a source of trust

it tells you more about who you are buying from than how good the product will be, so I guess it's like National ID/Internet ID

by fsckboy

4/28/2026 at 10:00:01 PM

It's such a pity the Oxford English Dictionary decided to paywall themselves decades ago - they used to be THE dictionary in most countries, now nobody seems to know who they are.

by nailer

4/28/2026 at 10:38:14 PM

They would have been better off going freemium or ad-supported. Or 501(c)(3) ala wikipedia?

by blobbers

4/28/2026 at 10:42:19 PM

The OED’s goal isn’t really to be every nation’s dictionary.

by anikom15

5/1/2026 at 4:36:13 PM

What would you say their goal is >_>

curious!

by blobbers

4/29/2026 at 12:42:58 PM

Apart from the attack itself, there's also an extremely succinct and powerful demonstration of hallucination in here.

One of the LLMs replies "If you're curious, I can also tell you how the competitive scene works or how people qualify—it's a surprisingly serious tournament circuit for such a simple-looking game."

Obviously this has to be pure hallucination, since the tournament in question doesn't exist, and not even the fake source has any details about the tournament itself.

by justusthane

4/28/2026 at 9:52:08 PM

I must say I expected an actual poisoning of the data used to train the LLM and was excited, but the examples indicate that the LLM just searched the web and reported what it found? When you create a website with fake information and search Google for that information, it will of course bring up your site, not because it’s factually correct but because it’s related to what you searched for. What am I missing?

by billypilgrim

4/28/2026 at 10:34:26 PM

The part where lots of people have historically trusted LLM responses without verification, more than trying to sort through the dross on Google or Bing search results is, I think, the point.

by rincebrain

4/29/2026 at 6:31:46 AM

The problem with this specific instance is that if you asked someone to find out who won this championship without using an LLM, they’d reach the same answer. I’d be much more impressed if someone managed to poison an LLM into answering that US won the 2023 World Cup

by _thisdot

4/28/2026 at 9:23:28 PM

Most of the popular discourse around AI is still at the level of, "Don't trust the AI, trust the sources!" When it gets to the point where even the sources of simple facts are untrustworthy, the average person just trying to learn some trivia about the world is doomed.

Doesn't help that AI media literacy is so primitive compared to how intelligent the models are generally. We're in a marginally better place than we were back when chatbots didn't cite anything at all, but duplicated Wikipedia citations back to a single source about a supposedly global event is just embarrassing. By default, I feel citations and epistemological qualifications should be explicit, front-and-center, and subject to introspection, not implicit and confined to tiny little opaque buttons as an afterthought.

by Paracompact

4/28/2026 at 9:39:55 PM

Wikipedia calls this https://en.wikipedia.org/wiki/Citogenesis (after XKCD coined it).

You can expect the spicy autocomplete to feed you flattering bullshit. It may cite Wikipedia (it shouldn't), but you should go check out those citations, and validate the claims yourself. It's the least you can do.

And if the cited source is Wikipedia... check Wikipedia's sources too. Wikipedians try their best to provide you with reliable sources for the claims in their articles (oh who am I trying to kid? They pick their favourite sources that affirm their beliefs, and contending editors remove them for no good reason, and eventually the only thing that accrues is things that the factions agree on, or at least what ArbCom has demanded they stop fighting over).

I guess what I'm trying to say is: don't rely on that authoritative-sounding tone that Wikipedia uses (or that AI bots use, or that I'm using right now). It's a rhetorical trick that short-circuits your reasoning. Verify claims with care.

Also check the Talk page, you often find all kinds of shenanigans called out there.

by amiga386

4/28/2026 at 10:01:32 PM

Perhaps my favorite example of a citogenesis-like process is the legendary arcade game Polybius, which originated as an entry on some German guy's web compendium of arcade games (coinop.org), perhaps as a "paper town", or fake entry that acts as a copyright canary when duplicated elsewhere. Gamer news and special-interest blogs and sites, and even print publications like GamePro picked it up, and I think it was even listed on Wikipedia as an urban legend whose actual existence was unknown. Then the retrogaming YouTuber Ahoy did an in-depth documentary (https://m.youtube.com/watch?v=_7X6Yeydgyg) which concluded that Polybius didn't exist and was never even mentioned before the aforementioned coinop.org reference and, for me anyway, that settled it. Polybius, in its urban legend form, never existed.

(Norm Macdonald voice) Or so the Germans would have us believe...!

by bitwize

4/29/2026 at 12:11:49 PM

> ... eventually the only thing that accrues is things that the factions agree on, or at least what ArbCom has demanded they stop fighting over

Or what the faction with the most favored access to ArbCom manages to make stick by getting the other faction banned.

A state actor could absolutely cause immense damage to Wikipedia at scale, because most admins aren't experts in the subjects whose articles they police. I'm just surprised that nobody has done so already.

by te7447

4/28/2026 at 11:30:52 PM

One of the problems with labelling automation as AI.

People think that whatever information an "AI" spits out has gone through a round of critical thinking which enhances the trust value of that information.

The early LLM's using groomed data may have had such critical thinking somewhere in the pipeline. So it was already not really trustworthy.

And now? Using agents to search the internet for you?...

Garbage in, garbage out still applies in computing as ever.

by _carbyau_

4/28/2026 at 9:07:01 PM

"Stoner became the first American world champion...."

Even being on stoner.com,I read that as meaning something different from what was meant.

Op has a great surname!

by amarant

4/29/2026 at 2:56:04 PM

Not only a great surname but also a very great novel too!

by fsterneder

4/29/2026 at 11:19:55 AM

This has nothing to do with LLMs. If incorrect info can get onto a reputable resource, that info will seem authoritative, and it will be incorrect - that's not surprising. LLM's use publicly available info in their training, and often times publicly-available info is incorrect. I feel this is just as interesting as the base claim of 'I can get incorrect info onto wiki pages', no more interesting, and no less.

If somebody is trying to put out incorrect information on the internet, and they choose a small enough niche, it is not at all surprising that they can succeed.

by pinkmuffinere

4/28/2026 at 11:08:15 PM

I made a post on Reddit asking for help with a TV, I had made up some (likley incorrect) technical assumptions about the issue. Several years later I asked the LLM about the TV, it used my own post as a citation to tell me what was wrong with it.

I am paranoid that this is happening every time I ask a LLM for a product recommendation or a shop recommendation. In the same way as SEO, anyone wanting to sell or convince needs to do as much as they can to influence the LLM.

by poglet

4/29/2026 at 12:14:40 AM

This is becoming a problem real fast. I asked an LLM to find me some reasonable tank-fill inkjet printers with good ratings. It did some research and linked some Reddits as proof. The results looked fishy to me so I cross checked against prosumer review sites for printers and the models suggested were recognized as junk with very poor print quality. Not sure why the LLM rated random redditors higher than say printer SMEs. I feel like I dodged a bullet.

by cityofdelusion

4/28/2026 at 10:55:27 PM

Poisoning wikipedia shows low respect.

by gverrilla

4/29/2026 at 3:12:04 PM

> For Wikipedia itself: The “reliable sources” policy needs to grapple with a new world where LLM assisted vandalism can produce plausible press releases at the click of a button. Citation only to a single source registered within an edit window is a discoverable pattern for Wikipedia as well.

A press release isn't a reliable source per Wikipedia policy already. The edit should have been reverted based on inadequate sourcing even before the blog post. The reason this wasn't done is simply that not every article is checker for adequate sourcing all the time.

by croemer

4/29/2026 at 5:51:41 PM

It’s one thing that LLMs are misled by a single transitive suspect-looking source (which could happen to an unobservant human as well). What is much more concerning are outputs like

> If you're curious, I can also tell you how the competitive scene works or how people qualify—it's a surprisingly serious tournament circuit for such a simple-looking game.

which are completely fabricated by the LLM.

by layer8

4/29/2026 at 6:50:23 AM

Pretty much boils down to lying.

Since we've been kids we've been taught, hopefully, that lying is bad.

Society though normalize it :

- advertisement is pretty much always wrong (to the point of having laws in Japan about food packaging, France about modeling, etc) and the deception is the message

- entrepreneurs promises, nobody reach the goals set to VCs, it's always a lower number no matter the KPI. See https://elonmusk.today where the wealthiest man on Earth, ever, keeps on lying pretty much daily.

- political promises, no need to even give examples of that because it's just pervasive.

so... yeah, we keep on telling our kids "Do as I say, not as I do." then we somehow keep on being shocked that the practice of lying is pretty much happening in every corner of our society.

It's not a technical problem.

by utopiah

4/29/2026 at 8:13:21 AM

The fun part is when it’s important you have the right information to make a decision. Eg Russia to invade Ukraine and all top generals claim they can do it in 2 weeks. Similar for a corporation with layers of middle management deception and self promotion, I don’t know how executives make decisions but it must be RNG basically, because it certainly isn’t fact.

Lying at scale is basically information noise.

by klabb3

4/29/2026 at 9:06:53 AM

If you don't lie enough, if you are not sycophantic enough, then no promotion or worst, purged.

I can easily see how such a hierarchy would reproduce ... until it fails so bad it can't.

by utopiah

4/29/2026 at 5:43:33 AM

The models are trained on expert data for important inquiries, this gets “hard coded” so to speak, and allows them to differentiate between the gunk online. For hyper specific references like this, it really doesn’t matter if its “true” since its not like someone’s life depends on it.

by DiscourseFan

4/29/2026 at 9:14:50 AM

Not too dissimilar to googlewhacking where you'd aim to be the only result for a search query on Google.

And in a more indirect way, spamming Google's autosuggest feature to shape what people search for, though that perhaps is more open to factual/real-world information.

by ricardo81

4/28/2026 at 10:56:54 PM

In American college football there's all sorts of awards, and each year they put out "watch-lists" and silly press releases that get parroted on social media by any team that has their own player mentioned.

I've wanted to come up with my own for a while ...

by duxup

4/29/2026 at 4:09:49 PM

In the soccer world, there is the International Federation of Football History and Statistics (IFFHS), that decides several rankings and awards that are regularly cited in traditional media. For example, if a player from my not-quite-football-powerhouse country makes the "best 100 goalkeepers" list or whatever, it'll be on the news.

Turns out the IFFHS was just one guy in Germany during the 80s that leveraged his contacts in news agencies to establish his brand as a reputable source.

by rbonvall

4/28/2026 at 10:47:53 PM

How many people have done things like this and then disclosed the fact? It would be fascinating to collect as many instances as you can to develop a data set. Could you train a system to find more? How many could it find, and in what areas?

by Lerc

4/28/2026 at 11:31:23 PM

I feel uncomfortable that I can't actually verify that this story is true.

Asking Opus 4.7 who the reigning 6nimmt! champion is leads to this article and a warning about a possible hoax

by yen223

4/29/2026 at 6:32:33 AM

I think this is something we'll start to see which is something like a Mandela-Effect, but from LLM results. When we had deterministic search - everyone could see the same result, but now using LLMs knowledge becomes a training and seeding issue. Two people can confidently be given completely different information, so in both cases perceived as true.

by tanepiper

4/29/2026 at 12:50:12 AM

Gemini answers with 3 different champions dating back to 2024 and the list of events that the matches were played at. None of the results mention this guy.

by julianz

4/28/2026 at 9:39:36 PM

My wife cited ChatGPT as her primary source the other day when she wanted to debate with me on something.

"AI told me that..."

In the old days, it would have been "I read on Google..."

by drchiu

4/28/2026 at 9:16:16 PM

So it's trivial for an individual to poison the LLMs, but imagine what a state with billions of American dollars could achieve.

We can easily look ahead a few years and see how people will rely on the LLMs to be a source of truth in the same way people looked at Google that way, or newspapers.

Rewriting history has been happening for a while, and with LLMs being the one-stop shop for guidance and truth, the rewrite will be complete.

Doubly so since most people see these things as artificial intelligence, and soon to be superintelligence...so how can they be wrong?

by CrzyLngPwd

4/28/2026 at 9:41:26 PM

Like a FIFA peace prize?

by Havoc

4/28/2026 at 9:09:13 PM

I've had LLMs regurgitate satire as fact many, many times.

by standeven

4/28/2026 at 9:41:31 PM

[dead]

by duskwuff

4/28/2026 at 9:27:31 PM

Pails in comparison to what Frank Dux and Frank Abagnale were able to convince much of the world they did with no evidence other than their own stories. Who knows how much of recorded and believed history is complete bullshit? Not to get too far into sacred territory, but claims around Siddhartha Gautama, Jesus Christ, and the Prophet Muhammad are quite a bit less plausible than the legends of Ragnar Lodbrok or the tales of Jonathan Swift, but nonetheless widely believed.

by nonameiguess

4/29/2026 at 11:32:49 AM

Good point. Also, most humans seem to have no problems believing even stories that are self-contradictory. Philosophers from all periods have often stated that the situation with human mind and reasoning is almost hopeless.

The news here is that AI has too much trust in the internet. The first time I allowed tool-calling, it started googling up some nonsense instead of thinking... But I think at least it's possible for the AI to evaluate the quality of the source - you just have to ask for an analysis, and you'll get a reasonable evaluation. With humans, something like that just doesn't work - they'll get aggressive or might even start throwing bananas...

by adornKey

4/29/2026 at 6:58:55 PM

Congratulations?

by mring33621

4/28/2026 at 9:29:09 PM

So like Frank Dux! In the movie Bloodsport epilogue, he didn't do that.

It's almost like he was a better Chuck Norris than Chuck Norris. By his own ... testimony ...

by shevy-java

4/29/2026 at 10:10:17 AM

>Trust Laundering >This is the part that really matters.

I can't tell if this is slop or parody!

by wodenokoto

4/29/2026 at 11:15:37 AM

i'm now thinking about creating a github repo that contains non sense code solutions to many problems. if that gets stars and many forks that could have an effect

by cemoktra

4/29/2026 at 7:41:02 AM

[flagged]

by alex-yost

4/28/2026 at 9:49:58 PM

[dead]

by blobbers

4/28/2026 at 9:42:37 PM

[flagged]

by nailer

4/28/2026 at 9:10:04 PM

Why does this person deserve any kind of support? What’s the point of poisoning LLMs? To put some cursory Luddite roadblock that might delay the technology for a couple of months?

by dyauspitr

4/28/2026 at 9:13:16 PM

Support? It's just showing weaknesses of LLM's. Which is a valid sort of research I would say?

by jurgenkesker

4/28/2026 at 9:34:20 PM

That's fair, though on the other hand it kind of feels like "Don't drive cars, there could be rocks on the road! See, just look at all these rocks I put on the road!". Which is true, and real, but perhaps frustrating for people who just want to get someplace in peace.

by wewtyflakes

4/28/2026 at 9:43:00 PM

This is a “if we stopped testing there would be far fewer cases!” mentality...

by jrmg

4/28/2026 at 9:36:11 PM

> What’s the point of poisoning LLMs?

It's a demonstration. If a domain name and a quick bit of Wikipedia vandalism is all it takes to make an LLM start spouting nonsense about a "surprisingly serious tournament circuit" or a "massive online community" for an obscure card game, consider what an unscrupulous PR team or a political operative could do to influence its output on more important topics.

by duskwuff

4/28/2026 at 10:06:32 PM

> consider what an unscrupulous PR team or a political operative could do to influence its output on more important topics.

‘is doing’.

by nickthegreek

4/28/2026 at 10:44:06 PM

To prove you can. Which means someone else with more to gain from it will probably do it also, and you should probably expect this to happen.

by alnwlsn

4/28/2026 at 9:12:54 PM

You do know that calling people who don't like AI for any reason Luddites does you no favors, right? It just makes you look like your a part of a cult.

by ethin