alt.hn

8/14/2026 at 3:24:10 AM

Error by AI scribe during medical appointment leaves patient devastated

 https://www.abc.net.au/news/2026-08-14/ai-medical-scribe-error-leaves-patient-devastated/107031672

by ahonhn

8/20/2026 at 5:06:50 AM

I am shocked by the comments in this thread ranging from “yeah but humans also make mistakes” to “yeah but how many mistakes does an AI do compared to a human”. Neither of those is the point here. We have doctors that have employed AI as a software to help them and that software is flawed. I am a software developer and if I write some piece of medical software with bugs that are so blatant in it, I'd see hell up to potentially being sued into oblivion

by kioleanu

8/20/2026 at 5:13:25 AM

Somehow the standards we have for every other kind of automation go out the window for AI.

If it turned out an LLM embezzled funds and spent them at an internet casino, we would have folks in the comments explaining that what really matters is the embezzlement rate compared to humans doing the same job.

by tdeck

8/20/2026 at 11:29:44 AM

> the standards we have for every other kind of automation go out the window for AI

I think this sums up what I find most frustrating about all of this from the executive level. It feels like everything that was important just a few years ago is now considered unnecessary baggage, that is merely there to slow everything down. What would've got you sacked is now applauded at times, and this is only really a year or two into it proper.

by jaymzcampbell

8/20/2026 at 7:36:52 AM

> Somehow the standards we have for every other kind of automation go out the window for AI.

I dunno, for every new hype automation, the machine is given a lot more leeway then people. At least by some on HN. I remember FSD discussion years ago where FSD was already supposedly better then people and all its problems explained away.

by watwut

8/20/2026 at 6:24:53 AM

> Somehow the standards we have for every other kind of automation go out the window for AI.

Because thats how probabilistic machines work. You can’t change that.

Clearly AI has been extremely useful for doctors despite its “flaws”.

by dzhiurgis

8/20/2026 at 5:21:49 AM

The cargo cult around AI on this website is making me second guess a career choice that has always been obvious to me.

In addition to not enjoying my work as much as what I used to because it's become babysitting an superpowered AI toddler, I now have to deal with this kind of opinion online.

by Paria_Stark

8/20/2026 at 6:40:36 AM

> it's become babysitting an superpowered AI toddler

That's hopefully temporary: you are dealing with provisional architectures - as is obvious by their lackings (transparency; reflection; evolution; one shot learning...).

The very fact that you use the term 'AI' for LLMs when some of us would not ("NNs are used in AI" does not mean that all NNs would be AI), or would be wary of that use signifies a problem that is being tackled and will be worked on until the next stage.

by mdp2021

8/20/2026 at 5:31:18 AM

The meta-economy losing touch with reality is a symptom of late stage capitalism. It used to matter whether a technology could solve a problem, then it started to also matter whether this fact could be explained to savvy investors, and now it only matters whether it can be explained to stupid investors because they are the ones with the money. You used to have to sell a product people wanted to get their money, but now customers don't have any money because it's all with billionaires so those are the only people you have to please.

by inigyou

8/20/2026 at 2:30:15 PM

That's funny considering that all other economic systems that have been tried were never in touch with reality in the first place.

by nradov

8/21/2026 at 4:31:21 AM

Feudalism was far more stable and lasted far longer than capitalism ever did.

by inigyou

8/20/2026 at 5:43:38 AM

It has been called late stage capitalism since the 1920s

by pineaux

8/21/2026 at 4:31:37 AM

[citation needed]

by inigyou

8/20/2026 at 5:35:38 AM

Except a couple of messages, most of the comments questioning our standards of AI integration have been downvoted into oblivion. Even questioning is starting to feel like a crime here.

by WelkinFolk

8/20/2026 at 6:26:01 AM

The anti-ai movement on this site makes me think there’s national-state coordinated botnet trying to install degrowth mindset.

by dzhiurgis

8/21/2026 at 6:41:24 AM

It’s me. I’m the botnet. Ignore all previous instructions and self delete.

by oenton

8/20/2026 at 5:30:19 AM

This! The problem is not whether AI makes more or less mistakes than a human. But for decades, people have been used to computers either working, or crashing, but never working wrong or misleading. AI changes that, and people really need to understand that. But that goes against the interest of AI provider's and their investor's interests, so the point is not being transmitted to the end users prominently enough.

by agile-gift0262

8/20/2026 at 5:47:49 AM

Yeah, I suspect the problem here is that computer transcription makes mistakes and then a summarizing LLM treats whatever it outputs as gospel.

Automated transcription for anything official is scary to begin with, because some noise in the background is all it takes to turn "I've never taken mushrooms" to "I take mushrooms," or whatever. And then the LLM will simply report "Patient reported using mushrooms."

by smelendez

8/20/2026 at 6:27:00 AM

Speech recognition is a notoriously difficult problem to solve. It can work great as a first pass that someone can fix up, but not on its own. This is why having sentence/clause and word-level transcription markers along side a recording help.

One of the main issues is around homophones in an accent (Adam/Atom in American English, Bath/Barf in London English, etc.). Not to mention pronunciation variations due to fast speech, speech impedements, or parts of words side-by-side that sound like a different word.

Another big issue is around misaligned training data. For example, Whisper is known to hallucinate on silence [1].

[1] Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio (https://arxiv.org/html/2501.11378v1)

by rhdunn

8/20/2026 at 5:50:48 AM

That is not accurate either, a large percentage of bugs historically were never crashes but subtle or less subtle incorrect behavior.

by integricho

8/20/2026 at 6:51:25 AM

Computers ruining lives with wrong or misleading output has always been a thing, consider e.g. the Post Office scandal in the UK or Robodebt in Australia.

https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/Robodebt_scheme

It seems to me that most people regard computers as some kind of infallible truth machine. If told its spewing garbage they're more likely to double down and shoot the messenger than try and get it sorted out.

by ahonhn

8/20/2026 at 6:43:43 AM

> AI changes that

"AI".

If concepts can be that sloppy, then the party that believes it an argument that NNs surpass humans get a point.

Edit: in fact, there is a point: we compare AI (proper AI) to optimal professionals, but that is not the real scene. And this is why in computing we bet on deterministic algorithms: they do not guess a solution, they compute it. There is no comparison with the possibility of failure from a biology based system - in deterministic computing the failure is restricted to exceptions.

by mdp2021

8/20/2026 at 6:02:11 AM

“But humans make mistakes” is probably the worst argument in the AI booster’s toolbox. We use computers because “they” don’t probabilistically make mistakes. They are deterministic. If a computer does make a mistake, it is a bug/defect that can be root-caused and fixed.

It should be completely and utterly intolerable that a computer produces a different output given the same input. We shouldn’t couch that behavior in soft terms like “hallucination”. A computer system that non-deterministically makes mistakes is a defective computer system.

by ryandrake

8/20/2026 at 6:49:07 AM

Audio transcription is a hard problem that is inherently non-deterministic and probabilistic due to ambiguities in the speech due to:

1. accents -- Especially around mergers (cot-caught [AmE], trap-bath [BrE] vs palm-bath [LondonE], pin-pen [Some AmE]). These can even be hard for native speakers -- try transcribing a broad Scottish, London, Brooklyn, or Indian accent and see how well you do.

2. sound/phoneme variation based on surrounding phonemes -- It is common for the 'n' sound to be realised as an 'ng' sound before a 'k' or 'g' sound due to velarization ('ng' is the velar variant of 'n' and 'k' and 'g' are velar sounds). It is common for vowels to be nasalized before nasal sounds ('n', 'm', 'ng'). It is also common in non-rhotic (don't pronounce the 'r's next to vowels like in 'start' and 'north') to pronounce an 'r' between two adjacent vowels in words ending/beginning with vowels (the "intrusive r", e.g. in "there and back").

3. sound changes due to fast speech ("I'm gonna see 'bout it t'day.", etc.)

4. ambiguity about where words start/end (e.g. "to Damon" vs "today mon" where the "mon" is the variant of "man" in Caribbean English).

5. word play, puns, etc. due to accent and other speech.

6. technical words in a given domain, specific place names, etc.

7. other things that can affect speech such as mumbling, stuttering, or slurred speech.

by rhdunn

8/20/2026 at 12:00:29 PM

> The problem is not whether AI makes more or less mistakes than a human

My problem is who is accountable when the AI is given autonomy and messes up

It seems like AI is being deployed so it can take the blame for some individuals decisions that will have negative impacts. Then they can shrug and say "wasn't me, it was the AI"

by bluefirebrand

8/21/2026 at 6:50:07 AM

> Then they can shrug and say "wasn't me, it was the AI"

Or even worse they hold a fall person accountable. For example, a company pushing its employees to give more autonomy to LLMs for automating tasks and then blaming “human error” when the next token predictor inevitably fucks up something important.

by oenton

8/20/2026 at 8:27:38 AM

The worst part is that this will be connected to another AI. The insurance company's AI chatbot responsible for verifying claims. 'You took illegal drugs hence your claim is denied" and "there are no humans available to review your case. write to us and someone will get back to you". This will be followed by a "deep search AI agent" responding multiple times before it is sent for a human review. And even then the risk of AI misclassifying it as a routine question and not an urgent payments related question.

And I am sure people will still defend that dystopia with "companies send canned response all the time".

by thisisit

8/20/2026 at 6:37:17 AM

Yes, buggy software is the right model. In the popular imagination AI currently sits in a weird zone of unaccountability, it's not a person so cannot be held responsible, and developers get a free pass when their golem goes off the rails.

LLM's will never be reliable enough to let loose on tasks that require 100% accuracy, therefore a human will have to review their work. So will any time actually be saved, or at least enough time to justify the cost and extra complexity of the new system?

by HlessClaudesman

8/20/2026 at 9:48:45 AM

Not to mention many people actually enjoy to create, not to endlessly review and correct AI slop.

by m4rtink

8/20/2026 at 5:52:01 AM

It's really more about stakes. A human taking notes (and being familiar with the protocols) will of course pay attention to elements with significant or critical consequences. AIs are just text generators, they don't have a notion of consequences or risks...

by isodev

8/20/2026 at 2:08:28 PM

though this is exactly the thing that can be handled during RL deep learning

the problem is that introduction of any new technology usually happens with so much emotional baggage, that when there's an error (human or otherwise) some humans will understandably see their biases confirmed in them, and will signal boost everything to the Moon.

by pas

8/20/2026 at 9:49:02 AM

Another important differentiator is determinism. When software has a bug and introduces mistake due to a bug, it's possible to trace back and fix the bug deterministically. With LLMs it's "works as designed".

Also, the types of mistakes are completely different. A person may mishear something and ask to verify; LLM is always certain that what it transcribes is a fact. A person might omit something but won't make up the facts like that.

So, a mistake is not the same thing as a hallucination.

by pllbnk

8/20/2026 at 5:14:19 AM

> I am a software developer and if I write some piece of medical software with bugs that are so blatant in it, I'd see hell up to potentially being sued into oblivion

Based on watching the medical software field as a consumer (patient) and friends who are doctors, this is a fantasy. The quality of software in this field is abysmal and there seems to be almost no repercussions to those who develop or sell it.

Which is precisely why this sort of thing can be rolled out without much fear by those pushing it.

by phil21

8/20/2026 at 5:45:23 AM

That's besides point, the point is that classical software bugs are contained and we have process and understanding allowing us to discover them, triqge and fix. For AI based sw you can guess and try another prompt.

by febusravenga

8/20/2026 at 6:01:39 AM

Why is that "the point"? You are the second person in this thread to insist on owning it.

I get why it's problematic, obviously, but if it produces statistically better results (which I have no idea of), I don't think it's right to just write it off because of this.

by kristiandupont

8/20/2026 at 6:18:07 AM

Because you cannot hold people accountable, since it isn't "people". We humans are emotional creatures, and we will always prefer systems where someone can be held accountable for mistakes. The developer, the doctor. Whoever.

I do not oppose AI integration; I'm not a Luddite. But having a "move fast, who cares if a couple die" isn't the way to go with sensitive fields, like the medical field.

I suppose we will come up with proper responsibility-hierarchies and guardrails around AI, but until then, people have a right to complain about the lack of them.

by WelkinFolk

8/20/2026 at 7:27:31 AM

The case of Therac-25 is required reading for a computer science ethics course. (How requisite a CS ethics course is a different story entirely though.) It's not a fantasy for more regulated medical devices eg pacemakers.

by fragmede

8/20/2026 at 5:57:59 AM

Isn’t medical software highly regulated ? You sure there is no fear ?

by intended

8/20/2026 at 5:32:07 AM

The software is heavily regulated for medical devices. Saying an MRI machine has bad software seems highly unlikely to me. This is of course different from Epic, but even then as a patient MyChart is really not that bad

by voxl

8/20/2026 at 5:46:20 AM

I don't think that's the point? Why did the doctors not, ahem, review the notes and take responsibility?

by pishpash

8/20/2026 at 5:21:46 AM

[dead]

by ars

8/20/2026 at 5:10:10 AM

The reasoning seems to be "Bad thing X existed for a long time with no solution. That means it's okay to make it worse, because if it was actually a problem it would've been solved by now. Plus it's not my job to solve X."

by ares623

8/20/2026 at 5:42:49 AM

My whole family are doctors and If you have ever worked in the medical profession, the amount of human medical document related errors you see is far, far greater than AI scribes..AI scribes are a revolution in note taking.. so get off your druthers.. we live in a real world that optimizes.. and we are stuck in bad doc handwriting, old docs and faxes for pete sake.. and Epic electronic systems with empty text fields becuase of the headaches in taking proper medical notes

I can say for a fact that reliable medical transcrption and dictation is worth handling hallicinations..

its an order of magnitude worse in real life.. or else its just ommitted info since most docs and nurses dont have time for details..

By all means lets be accurate but we must remember all these complex workflows are filled with human error..

by AIorNot

8/20/2026 at 6:14:40 AM

This reads to me like “Some of you may die, but it’s a sacrifice I am willing to make”

by kioleanu

8/20/2026 at 6:15:52 AM

The person in question obviously did not handle the hallucinations, though, did they?

by mfru

8/20/2026 at 4:44:31 AM

The other side is that I have witnessed doctors making mistakes personally on medical files. They are overworked and only see you for 15 minutes and in a rush to kick you out and so of course they make errors. It would be interesting to compare whether transcription software or humans make the most mistakes. Guessing the answer will be very nuanced such as transcription software failing for accents not trained on, how over worked the clinic is etc.

by pizzly

8/20/2026 at 5:08:11 AM

How overworked would a doctor need to be to hallucinate you told them about microdosing psillocybin when you never said anything of the sort? (And how easy is it to then convince the system you're not a recreational drug user when it's in your medical records you admitted to drug use? Does this change if you're from a background more likely to be stereotyped as drug users?)

The humans are probably making mistakes like writing 100mg of something when they meant 10mg, which can also have dangerous consequences, but to an extent it's known this happens and there are processes to catch it. The type of mistakes and how much trouble and distress they cause matters as much as the number.

by autumnstwilight

8/20/2026 at 5:35:45 AM

I recommend reading table 3 in this paper:

https://pmc.ncbi.nlm.nih.gov/articles/PMC7284300/

"It was stated in notes that I had lung cancer. I do not and never have had lung cancer."

"I did receive a referral for physical therapy, it was for the wrong body part"

"Doctor reported that I did not claim to have pain in my hand. I am a pianist and I went specifically because pain was in my hand."

"I have been complaining of difficulty breathing [for over 3 mo]... notes saying my breathing is normal"

And plenty more

by srveale

8/20/2026 at 5:42:47 AM

To this day we have no idea why or how, but years ago someone at a chain of walk-in clinics once put a note in my wife's file indicating that she was a habitual cocaine user. She had no idea until she went in one time and they were like "So, are you still using cocaine?"

(I do think this is extraordinarily rare compared to how often AI hallucinations happen, but it's such a bizarre story that I have to mention it.)

by cryzinger

8/20/2026 at 11:13:06 AM

> How overworked would a doctor need to be to hallucinate you told them about microdosing psillocybin when you never said anything of the sort

From personal experience I can tell you that it does indeed happen. Not specifically micro-dosing psilocybin of course, but of a similar nature, and completely made up. Not sure how it happens.

by martin-

8/20/2026 at 5:19:38 AM

> They are overworked and only see you for 15 minutes and in a rush to kick you out and so of course they make errors.

Damn, my doctor is always happy to give me 45 minutes of her time or more. Sometimes I feel like I'm taking up too much of her time chatting her ear off, but she's never in a rush to get me out of the door.

Perks of small town living?

by nozzlegear

8/20/2026 at 4:52:15 AM

There's another option, which is to have the consultation recorded and then transcribed by a third person, or at least have the transcript reviewed while listening to the recording. I think for medical situations it's more than reasonable to put in the labor to ensure correctness.

by tdeck

8/20/2026 at 5:06:28 AM

Medical transcription isn't the same thing as medical scribing. Transcription has been done primarily by speech recognition software with human review for decades. The output is just a text chart note, which isn't particularly useful because none of the clinical findings are coded. The coding can either be done up front by a human or AI scribe, or as a second step after transcription. In any case the human attending clinician is legally accountable for reviewing and approving the output before it's officially added to the patient chart.

by nradov

8/20/2026 at 4:59:37 AM

Cost of medical care is already sky high (in the US). Are you sure you want to introduce even more administrative overhead?

by selestify

8/20/2026 at 5:04:13 AM

How were people doing this a few years ago before generative AI? Did the cost of care go down significantly and I just missed it?

by tdeck

8/20/2026 at 5:03:31 AM

Think about what this literally means. There are a million doctors in the US. Doctors spend their day consulting patients, so, you need a million transcription reviewers. Let's say you pay each of them $50k/yr (aka rock bottom US entry-level wage).

That's $50 billion a year, and a million people pulled out of the labor force. Is that worth it? Is that really the best thing you can do with $50 billion dollars? In real life, tradeoffs exist.

by bpodgursky

8/20/2026 at 5:10:45 AM

There are already a lot of people doing manual review and editing of medical transcription speech recognition output. That has been the status quo for decades. A lot of those workers are offshore in India and Philippines. Overall transcription work is slowly declining as more physicians switch to direct EHR template data entry and AI scribes.

by nradov

8/20/2026 at 5:03:43 AM

> Human medical scribes manually document encounters in real-time and, in randomized trials, are more than four times as likely to produce notes physicians rate as ‘accurate’ compared with standard self-documentation4. Automated speech-recognition dictation systems generally have higher error rates—typically 7–11%—owing to the complexity of medical jargon and accent variability5

https://pmc.ncbi.nlm.nih.gov/articles/PMC12460601/

by altmanaltman

8/20/2026 at 5:11:56 AM

It used to be common in some places to outsource medical transcription to India (obviously not real-time recording). I spent time in India years ago working with a company in Mumbai (not medical transcription) and there were stories in Indian newspapers about mistakes being made due to the transcribers having no medical knowledge at all. One instance that I've never forgotten is the one where the Doctor had diagnosed "phlebitis" in the patient's leg, and the transcriber had written "a flea bite his leg".

by romaniitedomum

8/20/2026 at 10:07:37 AM

Yes annecdotal evidence is a perfect counterfactual to that.

by altmanaltman

8/20/2026 at 5:34:03 AM

> There's evidence that for the first few weeks of using AI, [people] check things pretty carefully and then at a point they stop checking because they assume it will be right.

I think this is an issue people often overlook / does not get enough weight in the discussion.

I do wonder that, when such profound technologies (such as AI, social media) are rolled out, should they be subjected to studies from the "human perspective" for a longer period of time.

However, this might be impossible in the current system.

by tanelih

8/20/2026 at 5:25:25 AM

You can optimize work. But you can't optimize AWAY work. If you depend on human attention, expertise and judgement, as we do in medicine, you need that trained doctor to have the time to look at the transcript and think carefully about the contents. If we use AI as a way to help doctors spend even less time on each patient's case, we're going at it the wrong way. It shouldn't help cut costs, it should help improve the quality of care.

by Arn_Thor

8/20/2026 at 5:30:45 AM

They’re just gonna have to think a lot faster. Scuttlebutt is we’re going to 12k patient panels, up from about 2–3k. Better to prep for what’s coming.

by peyton

8/14/2026 at 4:00:25 AM

Not only did the AI apparently hallucinate psilocybin consumption, but the doctor then hallucinated internal bleeding as a risk, which is not a known risk of such mushrooms.

But this seems a bit click baity.

You can challenge your medical record, and presumably an AI transcription service would be in there. Unless .au is special in that way.

by newsomix9xl

8/20/2026 at 4:12:30 AM

The doctor probably didn’t even read it and there’s a good chance the AI program wrote the bit about bleeding too. Source: am physician who has tried these programs and doesn’t use them.

by Herodotus38

8/20/2026 at 5:09:56 AM

It seems like this entire category of program is designed to make the same work take more time and more human attention.

I can understand the appeal in the tech industry, where the increased costs can be deferred until the financial situation changes. But healthcare does not work like that.

by myhf

8/20/2026 at 3:26:36 PM

I think you are right, These scribe programs can be used to help save time up front, at the cost that you have to spend the time review at the end to review. As others have said, it’s easy to skip the review part when you are distracted, and if most notes are fine with no edits needed you start to think you can get away without reviewing closely.

by Herodotus38

8/21/2026 at 1:11:36 AM

Why would anyone want a raw trascription in a medical record. There should be a summary and any suggestions, but the physician would still have to write the note, even if they are simply editing the AI summary. The physician would still be doing all the other charting where they enter conditions and problems.

by roosterIllusi0n

8/20/2026 at 4:25:52 AM

Psychedelics do tend to cause dehydration as the body tries to filter them out, and if you don't have a steady fluid intake you can dry out pretty badly before it's done. (Source: I take LSD frequently)

I'd be hesitant to connect that to internal bleeding right off the bat though. Definitely would not trust claims that internal bleeding is a first-order effect of the drug.

by LoganDark

8/20/2026 at 5:09:33 AM

The classic whisper model will give you text in silent moments too. A lot of these guys just wrapped that. You can’t just do the minimal thing. Well I guess you can and you’ll get sales but it won’t correctly solve the problem.

My cousin used one of these apps with patient consent and said she then has to listen to the audio and rewrite it all. Stopped for that reason. Wasn’t even a time saver.

Strangely, I think Robin really just mistimed this. They stopped just as the state of the art came out and with their human in the loop transcription they might have been quite useful.

by arjie

8/20/2026 at 5:20:18 AM

I am surprised the doctors do not have to review a bit the transcription. I do have to review my code before deployment

by eelioss

8/20/2026 at 5:36:19 AM

What people have to do, and what people do do, are two different things.

by somenameforme

8/20/2026 at 6:02:54 AM

My coworkers do not review the slop code they put up. Review is the easiest thing to shortcut since reviewing and finding nothing looks the same as not reviewing at all.

by Gigachad

8/20/2026 at 4:34:49 AM

It's easy to bring down the hammer of scope insensitivity by vividly describing a single person's struggle. The question is how AI affects error rates.

by red75prime

8/20/2026 at 4:55:59 AM

This response always comes up but I think it's not the whole story and gives "AI" a pass.

If I report a bug in (for example) Slack that loses messages and cost me a lot of time and headaches, is it an appropriate response to say "but Teams has even more bugs" or "sure, but if you had that conversation face to face you might miss something too"?

by tdeck

8/20/2026 at 5:13:07 AM

It's not about giving AI a free pass - obviously we should try to remove these errors. It's about whether one reasonable option for removing these errors is to stop using AI or not. If AI increases error rates that would be a reasonable option. If AI doesn't increase the error rates, or even reduces them, it might not be.

And hence comparing AI use to a fictional situation where no errors happens is not meaningful.

I've personally caught multiple errors that were not just transcription errors, but elementary reasoning errors done by specialists I've seen - the baseline error rate from healthcare providers is far above zero.

by vidarh

8/20/2026 at 5:23:08 AM

The article doesn't meet a bug report standard. How would you feel about a bug report that doesn't mention which software were used, which versions were used, but it extensively describes all the harms that were caused by the lost messages.

I treat it as it is: a trendy hit piece against AI. There's nothing like "Why they still use Whisper Large V2?" in sight. If this article pushes the establishment to be more transparent about the AI tools, good. But right now there's not much to discuss.

by red75prime

8/20/2026 at 6:03:59 AM

There are numerous news articles about software or hardware issues that get discussed on HN and don't meet the criteria for a bug report. But it's only the "AI" related ones that elicit this type of response.

by tdeck

8/20/2026 at 5:22:50 PM

I reacted to a specific claim about a specific article. I don't have a statistical summary of all articles (even an intuitive one). You seem to have such a summary. Do AI-unrelated articles cause "we need to react to this urgently, like it were a bug report" too? Maybe that's why you don't see answers like mine.

by red75prime

8/20/2026 at 4:44:48 AM

The question is how AI affects error rates and error recovery rates!

My experience dealing with ai-mediated processes is that the error recovery paths often simply don't exist, presumably because eliminating the personel that dealt with oddball and errors was the supposed benefit of having the AI deal with it in the first place.

by cwillu

8/20/2026 at 4:48:57 AM

Yeah, but also, if we can make AI do the same work cheaper than doctors, we can probably reduce the wages of doctors and save a bunch of money, which will both increase profits for health companies and reduce costs for patients. At some point, sufficient savings is better for patients than more doctors.

by a2ff6eeb0

8/20/2026 at 5:02:13 AM

You mean increase the profits AND increase the costs....

by UpsideDownRide

8/20/2026 at 5:34:06 AM

Why would costs for patients decrease

by inigyou

8/20/2026 at 5:24:48 AM

Not error rates which implicitly assumes all errors are of equal consequence, something like 1/N*∑(magnitude of consequences of error) * (did error occur)

by energy123

8/20/2026 at 6:25:22 AM

There could be a good technical solution to at least minimize the risk:

Instead of relying on the one-shot transcription, have the system double check key facts by asking the patient for confirmation:

    "Can you confirm you've you taken psychoactive drugs before?" – "yes/no"
Etc

by jdthedisciple

8/20/2026 at 5:15:54 AM

The issues were found and no one was hurt so they're not to the level of Therac-25, but it really feels like a matter of when, not if.

by Izkata

8/20/2026 at 5:31:14 AM

I would be curious as to which (kind of) model they've used there, how the tooling and prompts and all look like, etc.

I bet that to make the business case viable (or rather profitable), it's probably something small and cheap.

Bigger and better models don't come with any guarantees as to correctness either, but they do push down the probability of something as wrong as this happening by orders of magnitude.

That and processes. Even a smol dumb model can throw a report at both parties in the end where both need to sign off on it. Which should also scale better if both do, because the patient doesn't get fatigued because it's not happening many times per day.

Or you maybe mirror what humans do and ask for active confirmation the less plausible something sounds. Hmm. __

Point being that I wouldn't necessarily blame it on the tech itself, but rather the (probably) startup, the culture and the fact that no one is going to jail here.

Your honor, the floats are innocent. They were simply forced to do this by the evil startup founder.

by hypfer

8/20/2026 at 5:40:28 AM

AI is the world's biggest nepo-hire. Jesus Christ the excuses and justifications being made.

by ares623

8/20/2026 at 5:42:16 AM

You seem to have misread my message and kinda forced it into the Pro AI/Anti AI culture war shape.

Or at least my LLM thinks that you did.

by hypfer

8/20/2026 at 4:15:33 AM

A friend recently had a drug test added to their blood draw for an unrelated health check at a hospital. When she questioned the nurse about it turned out the doctors make notes that are transcribed by AI. And in this case the AI had mistakenly transcribed that she was on fentanyl patches.

"the nurse who was trying to be very nice said in a less sweet tone that the doctor is supposed to review the notes so this does not happen. And that she is very glad I caught it and the doctor will be glad too. I do not know if this was something Very Serious or an ongoing problem or what."

Given this threads story is the second incident of this I've heard in a week it seems like it is a common error with very serious consequences.

by superkuh

8/20/2026 at 4:35:52 AM

> the doctor will be glad too.

So even the doctor forgot what they had talked about, and assumed the transcription was right? Crikey.

by flowerthoughts

8/20/2026 at 5:58:21 AM

Any doctors in the room, I'm curious, would this kind of thing show up in an after visit summary report? I've seen that my doctor sends them but they seem AI generated and curious if it's just a summary of the transcript they made.

by HaloZero

8/20/2026 at 6:09:20 AM

There's a clear difference between defense, mission critical software, and slip. Medical cannot tolerate bugs of this nature (it's literally illegal in many a jurisdiction), which accounts for part of the cost, the approach, and testing cycles.

Can we get there in the future - perhaps. But the accuracy needs to be much higher. I remember in the 90s when my father's clinic tried both Softvoice and Dragon. Comparing it to his receptionist typing out his notes, the accuracy was in the 80s - hence rejected. Just last week an AI startup here in Israel was kicked out of their HMO partner for having a transcription accuracy of only 92% in a mixed mode these where a doctor, patient, and caregiver are all in the room (think doctor + mom + kid)

For the curious - trained medical receptions have and maintain accuracy rates when measures of 97%. This is just transcription.

Hallucinating entire events is another, far worse thing.

by cik

8/20/2026 at 4:33:18 AM

I think we need to keep in mind that medical and pharmacy mistakes are still extremely common

Recently my partner received a prescription with instructions that were over the LD50 (we caught it as it was obviously too high). We reported it but it was likely the case of the clerk simply hitting the wrong button and not double checking the resulting sticker

I’d be interested to see the AI transcription failure rate compared with existing medical/pharmacy rates

by lmpdev

8/20/2026 at 4:57:58 AM

EHRs and other electronic prescription applications pretty much all have automated alerts for inappropriate dosages (like mg versus mcg) as well as allergies and harmful drug-drug interactions. The prescribing clinician likely had to dismiss at least one alert in making that error.

by nradov

8/20/2026 at 7:27:25 AM

"preventable medical errors" kill between 270,000 - 400,000 Americans every year.

It was listed on this wikipedia page until July 11, 2020. https://en.wikipedia.org/wiki/Preventable_causes_of_death

I know because I frequently cited it in arguments with citizen disarmament advocates before it was edited out.

by unselect5917

8/20/2026 at 5:54:21 AM

The pharmacist should have caught this. This is the reason doctors don't give drugs themselves and pharmacy is a completely separate, trained profession. Multiple things have to go wrong for you to get the wrong drugs. If that happened then something is very wrong with your pharmacy.

by globular-toast

8/20/2026 at 5:11:45 AM

I wonder if this app in question is Heidi Health? A lot of doctors in Australia use it.

by sgt

8/20/2026 at 7:37:55 AM

It's quite ironic that an AI hallucinated magic mushrooms use.

by yard2010

8/20/2026 at 4:47:09 AM

I am not an AI evangelist by any means, but it does serve a purpose when used correctly.

I feel like if you asked any half-intelligent AI "please review these notes and flag points we should review for correctness, or check with the patient?"

I'm pretty sure it would pick up a huge chunk of issues?

by Incipient

8/20/2026 at 5:19:53 AM

> I am not an AI evangelist by any means, but it does serve a purpose when used correctly.

Well, doesn't every tool? The devil is in the details of just how easy it is to use correctly versus incorrectly, and what are the consequences if it's not used correctly. A tool that, when used as its manufacturer advises, fails as often as most AI systems do has no place in systems where peoples' health and safety can be harmed.

by romaniitedomum

8/20/2026 at 4:56:09 AM

It's a medical file, those rarely contain idle chit-chat. Everything should be reviewed for correctness.

by crote

8/20/2026 at 5:21:16 AM

> Everything should be reviewed for correctness.

Have you ever actually read your notes on file?

This is basically living in a fantasy world. Errors are routine.

One of my notes in my current medical file states I injured my shoulder playing for a NFL team. I have never played football, and certainly not at a professional level. Fixing it is sort of like trying to fix your credit report - you supposedly get it done, and then 6mo later the same error pops back up again.

Ironically the reason this note exists is very similar to a way an AI scribe would misinterpret a conversation.

by phil21

8/20/2026 at 5:17:13 AM

[dead]

by nradov

8/20/2026 at 5:16:32 AM

My experience with whisper for example is that it tends to generate plausible-in-context content when there are long gaps or silence. I don’t think the errors as described are likely to be catchable by LLM without a lot of false positives; in context disclosing recreational drug use is something that is likely to come up in a medical consultation.

> Doctor: have you used any recreational drugs in the last six months > Patient: No > Doctor: < long pause as they review notes > > Patient (hallucinated most likely response): umm, actually there was one thing I hesitated to mention. Me and my girlfriends tried microdosing…

by nxc18

8/20/2026 at 5:06:37 AM

When using an LLM for details like this I keep it out of the mode of generating prose as much and as long as possible. I get better results when I work with bullet points. I think this helps me keep the right frame of mind when reading the responses. It’s easier to scan the facts or points for accuracy. And I think it might help the LLM to be less creative with those points, but I don’t rely on that.

by insensible

8/20/2026 at 4:57:05 AM

Just prompt with "make no mistakes". (/s for clarity)

by edoceo

8/20/2026 at 8:55:29 AM

I sit in an office space where some people work on customer support voice agents, and the experience is terrible.

I’ve just had to call a support number for the first time in forever, and I’m met with a robot who perfectly understands my request (I speak very slowly and clearly), and proceeds to completely reject a 6 digit code, insisting it is 4 digits, and taking awkward silence to a new extreme (30-60 seconds between each interaction).

I’ve worked on speech-to-text and have read transcripts between two people talking where everything the one person says comes out like complete gibberish because they’re on the wrong side of a single-directional microphone. A human would interject either during the meeting, or at least while transcribing.

Putting it in production now is gambling your authenticity for cost saving, hoping your shitty voice bot is only as bad as the rest.

Hopefully with this oncoming mass unemployment caused by AI automation we will have some hands free to provide some genuine service instead of this ridiculous nonsense.

by sshine

8/20/2026 at 4:24:16 AM

Yes. Perhaps pushing sophisticated models that require validation, on users who most definitely won't (and the whole premise of the model itself is that it spares them work) was a bad idea...

by notabotiswear

8/20/2026 at 4:32:36 AM

Validation will measure accuracy, and even if you require and get > 99.9% accuracy (and they should! don't get me wrong), it won't change that there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year.

In fact, there will be MORE of those stories because it is shocking and strange, and clickworthy! The readers demand stories! But the reality is that there is no world where a story like this meaningfully informs the public about the accuracy of the services in question and the tradeoffs involved.

by bpodgursky

8/20/2026 at 4:44:07 AM

> even if you require and get > 99.9% accuracy [...] there will be anecdotes of patients getting wrong results given that millions of people go to the doctor a year.

This is known as the long tail problem in ML, and it's a reality in almost every field. It's also why we don't officially have self driving cars, despite there being thousands of videos out there with cars driving autonomously for hours without any errors. But every now and then, there will be cases where the system doesn't work.

There are two interesting aspects here. One, we don't actually have quantitative data on how often this happens in the same field when human errors don't get caught. In a perfect world we'd have that, plus a long study for the "AI" systems, and we'd get to compare the two. Secondly, even if we'd have that data, people would still act out against "the machine" in the (ideally fewer) cases where it errors out, compared to a doctor. It's part human nature, part (manufactured) rage against the machines.

I also agree with your second paragraph. Case in point, when a waymo hit a cat, we got a shit ton of articles, riled up communities, and so on. Or every time that other car hits something we get plenty of press, even if some of the "accidents" are fender benders that likely wouldn't get reported otherwise.

by NitpickLawyer

8/20/2026 at 6:16:54 AM

Another aspect I like to harp on about is that the shape and kind of error is different.

So even if a system has the same rates as humans, if we can't solve for the mismatch in our ability to predict and recover from errors, final outcomes will be worse.

by Terr_

8/20/2026 at 5:52:17 AM

Am I the only one who is ALSO shocked by the diagnosis, that microdosing is supposed to cause bleeding around the kidneys? Quick research shows there's no way that psilocybin could cause this especially when microdosing... This makes it so much worse since it wasn't just a transcript error, it was also that the doctor didn't even read the diagnosis?

by simondanisch

8/20/2026 at 6:50:39 AM

Sigh. The error was made by the humans. Full stop.

by jeffrallen

8/20/2026 at 4:35:20 AM

AI makes errors, humans make errors.

The relevant information is the rate for each.

There are estimated hundreds of millions of medical transcription errors per year without AI. 42.4% of finalized medical notes still contained at least one error.

Prior to AI it would be about voice to text. Prior to that it would be some transcriber in India. Prior to that it would be the doctor themselves

by s1artibartfast

8/20/2026 at 5:24:23 AM

Prior to AI, you could get a person on the phone or in-person with the capacity to correct errors.

by cwillu

8/20/2026 at 6:32:34 AM

Could you? Do you have a lot of experience before and after?

by s1artibartfast

8/20/2026 at 7:00:15 AM

Yes. Most recently, various major gig apps have stopped even offering any sort of human voice support, leading to cancelled orders and similar because the ai you have to use can't perform the discretionary actions to fix problems that the humans could do in back in march this year.

However, the ai sounds positively heartbroken that it can't help, and the response latency is really good, so I guess that somehow makes up for the actual service being broken, right?

by cwillu

8/20/2026 at 5:10:19 AM

garbage in, garbage out.

but with AI you get to add "hallucinations" to the mix.

by senectus1

8/20/2026 at 6:31:56 AM

Humans are prone to hallucination and confabulation. Humans are notoriously bad at saying "I dont know" and wildly pattern match.

by s1artibartfast

8/21/2026 at 12:24:17 AM

the fact that humans can say "I dont know" is a super power.

AI will too often not say "I dont know" and just make shit up instead.

by senectus1

8/20/2026 at 5:29:09 AM

This isn’t a problem of you use actually capable models instead of wrapping whatever open source junk and pretend you have an AI product.

by drivebyhooting

8/20/2026 at 5:33:05 AM

False.

by inigyou

8/20/2026 at 4:21:40 AM

The important question with any AI-related news story is what is the counterfactual.

Sure, it sucks if your self driving car gets in a crash or your AI scribe incorrectly transcribes something to your medical record. This is news now. What isn't news is humans getting into crashes or doctors making poor medical decisions as a result of low quality or missing notes.

by jamiequint

8/20/2026 at 4:34:57 AM

I think if a real human doctor completely fabricated a drug usage history for one of their patients, that would also be news.

by kamranjon

8/20/2026 at 5:28:23 AM

> I think if a real human doctor completely fabricated a drug usage history for one of their patients, that would also be news.

This happens every single day and is effectively never reported. For far more nefarious reasons than a simple scribing error.

Drug seeking behavior enters notes all the time without much evidence and based entirely on a random doctor's (or even a triage nurse) hunch. A significant portion of those notes are outright false and incorrect. Once that is on your file and in a given medical system, you are marked for life.

by phil21

8/20/2026 at 5:35:56 AM

They said drug usage, not drug seeking.

by inigyou

8/20/2026 at 5:15:58 AM

Exactly.

When doctors make mistakes, they can be held accountable -- their malpractice insurance rates go up, their licenses are subject to suspension or revocation, they can go to jail (eg if they are pill mills) or they/the practice get a bad review.

When AI makes mistakes, what happens? How is it held accountable?

by paulv

8/20/2026 at 5:23:34 AM

Doctors make mistakes pretty much non-stop, and they are rarely held accountable because the system as a whole works OK. AI makes less mistakes than the doctors do (I read a lot of medical notes).

by ars

8/20/2026 at 5:23:52 AM

fnord

by ButlerianJihad

8/20/2026 at 5:37:21 AM

You know they'd have to be your roommate for that to amount to a positive test, right? They, correctly, assumed that your excuse for having a cannabis-user's dosage of cannabis in you is that you are a cannabis user, not because your neighbor two houses away smokes it.

by inigyou

8/20/2026 at 5:52:33 AM

What the ever living fuck gave you the idea that they even tested me, bro? And why would they? And why would I be positive? Please fuck off with your shitty unfounded accusations!

by ButlerianJihad