alt.hn

7/28/2026 at 12:06:53 AM

Don't ask an LLM for a confidence score

 https://justinflick.com/2026/07/27/llm-confidence-scores.html

by pamplemeese

7/28/2026 at 12:45:44 PM

That's what I thought was interesting about the Watson Jeopardy Challenge. Watson had a sense of its confidence, and it seemed to be accurate. The answers it got wrong were the ones it knew it had difficulty understanding.

3 videos at https://archive.org/details/Jeopardy_2011-02-14_The_IBM_Chal...

by fhdkweig

7/28/2026 at 10:32:01 AM

There was a post earlier on HN where they trained a probe which could give a realistic confidence score on an LLM, they claim with 81% accuracy. They used it to interrupt and switch to a smarter model if a dumber one had low confidence: https://news.ycombinator.com/item?id=49010782

by foo12bar

7/28/2026 at 2:22:47 PM

I was designing a system that pulls structured data out of an unstructured transcript, and Opus initially suggested a confidence score for each value. When I asked it “do you think the model will generate a meaningful number there?” the answer was basically “ha, you caught me.”

What did work was asking the model to emit citations in the form of line numbers from the transcript. It dramatically dropped hallucination.

by bensyverson

7/28/2026 at 2:39:04 PM

> I asked it “do you think the model will generate a meaningful number there?”

This also isn't likely to give you good results, models don't have a good sense of their own capabilities.

This is also in the same camp of asking a model why it did something. It'll generate something plausible and likely totally incorrect.

by abound

7/28/2026 at 3:01:01 PM

> models don't have a good sense of their own capabilities.

It actually had some memorized knowledge about LLMs being bad at confidence intervals, then it did a cursory web search. But it didn't need to do any of that; I already knew they would be bad numbers.

by bensyverson

7/29/2026 at 2:36:24 PM

Using a 0-100 scale is a pretty bad idea in general. Neither humans not models know what to do with it. Split your metric into several concrete variables with discrete values well understood by the user (human or model). If you still need something more granular after that, calculate the aggregate score.

by orbital-decay

7/28/2026 at 1:06:40 PM

The statement near the top of the post > "The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless."

is definitely too strong of a claim and directly undercut by what is said near the end of the post: > "Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities that are better calibrated than the model’s own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. And Anthropic’s Language Models (Mostly) Know What They Know found encouraging results asking models to estimate the probability that their own proposed answer is true."

My own experience is that stated confidence is a helpful tool and of course you need a rubric and a proper prompt, but this is clearly less work than training a classifier (as advocated by the post) and requires less data.

by dmrivers

7/28/2026 at 1:47:58 PM

Fully agree. I found for most of my work with LLMs and Finance, ~90% of the time a high or low confidence score was accurate. There's the occasional ambiguous case, and that'll happen, but the engineering work that comes with building a classifier makes it not practical for my usecases.

by StellaSearch

7/28/2026 at 10:27:03 PM

I'm working on a universal one. My belief is that as agentic capabilities increase, behavioral attribution will be increasingly needed to maintain quality.

by inspectorSlap

7/28/2026 at 4:21:56 PM

Agreed. However my research is showing that saturation of the rubric occurs in unique configuration that drive the transfer curve to near binary. Effectively turning a 1-10 rating system into yes/no.

by inspectorSlap

7/28/2026 at 2:58:30 PM

[dead]

by smohare

7/28/2026 at 5:35:06 PM

In human memory, we often use a sense of familiarity to guide our memory decision, in the absence of explicit recollection of details. There is a whole memory literature about "recollection and familiarity" that dissociates the two cognitive processes, recollection which involves retrieval of specific details of an experience, and familiarity, which is a sense of memory strength, but absent of any qualitative detail. Familiarity is a faster process, and can often spur subsequent retrieval attempts that can lead to actual recollection..e.g. you see someone that seems familiar, but can't place where...and after a few moments, you remember who they were and where you had met them.

When measuring these processes, one approach has been to ask participants to provide confidence ratings. Recollection tends to lead to threshold-like, very high confident responses. Familiarity is more graded and continuous. Many then use a dual-process ROC model to identify the recollection and familiarity components, on average, of a person's memory of a memory test (see work by Andy Yonelinas).

This kind of work goes beyond memory, but applied to the general problem of how people judge their confidence in answers.

Its likely been applied to LLMs. A familiarity signal would probably be pretty easy to generate... The recollection kind of component might take some of those introspection type of approaches. For example, these papers, which I have not read, [1] https://arxiv.org/html/2603.17839v1 [2]https://arxiv.org/abs/2603.09250 might be getting at these ideas.

This might be relevant

by SubiculumCode

7/28/2026 at 12:22:41 PM

I find the recommendation in favour of discrete categories over numeric probability for confidence level intriguing, because it goes against the grain of all advice I've read for humans interested in forecasting.

The easiest argument in favour of numeric probability is that it's simple to evaluate. Someone claiming to be 90 % certain better be right about nine out of ten times -- no fewer, but also not too often! This is useful when you're learning to judge your own confidence, which most people are bad at. (Almost everyone can train themselves to be better at i but some people seem to be good at it with no training.)

The typical argument against categories is that people can mean very different things with the same word: https://i.ibb.co/kQT4Ymz/q.png

It seems like TFA gets around these problems by creating a translation table between a fixed set of categories and the probabilities implied by the model for those categories. That's an interesting approach!

by kqr

7/28/2026 at 12:20:09 PM

You don’t ask an LLM, but certain LLMs expose internal metrics you can tell how many token candidates where there what was the score which one and which one was selected.

So there are objective ways to control hallucinations as well as figuring out how “correct” the answer is to some extent.

by dogma1138

7/28/2026 at 1:00:51 PM

But those scores are only the likelihood of the given token being the next one in a plausible phrase, not that the underlying data is correct.

by bigfishrunning

7/29/2026 at 7:50:01 PM

Yes that is correct but it’s still better than nothing, there are some assumptions to be made such as that the model was not trained on junk data.

by dogma1138

7/28/2026 at 4:23:29 PM

Maybe confidence is a loaded term, but LLMs do have a sense of procedural uncertainty. Couldn’t you ask it to respond y/n in response to a question and look at the next token sampling probabilities?

by morelandjs

7/28/2026 at 11:57:55 AM

Who can read that text color and background combo? I had to turn on reader mode in Firefox.

by SubiculumCode

7/28/2026 at 12:41:30 PM

Firefox on desktop renders as black text on white background. The link colors are yellow, which is not great, but still okay for me to read.

I use the "Dark Background and Light Text" add-on. Sometimes the add-on makes a bad website worse (like IMDB), but it can be switched off with a click.

by fhdkweig

7/28/2026 at 12:05:27 PM

renders fine here on firefox android, black text on slightly yellow tinted light background

by ed_mercer

7/28/2026 at 5:17:11 PM

very strange. for me, on Firefox Android, it was light gray text on white background....

by SubiculumCode

7/28/2026 at 7:02:08 AM

I agree with the author if we are trying to use this as some sort of absolute scale of confidence. It only develops meaning when we control for many other variables. Looking at confidence scores across two different models or prompts is probably not a good idea.

by bob1029

7/28/2026 at 12:06:31 PM

So I agree with the general geist of this, a few counter-examples though:

If you have the raw log-probs, I show how to use conformal inference to set false-positive or recall rates, https://crimede-coder.com/blogposts/2026/ConfClassification

Some of the peer reviewed papers with the older models did show calibration was bad (not close to monotonic). This post with newer models (for one example, classifiying injuries) is not that bad, https://gmcirco.github.io/blog/posts/ai-calibration/calibrat....

I mean it just depends on the application, what level of error you can consider. But the second post shows how to recalibrate the scores as well if you need calibrated probabilities.

by apwheele

7/28/2026 at 12:18:08 PM

This resonates with what I'm seeing in healthcare right now and the bad taste it's leaving.

There are a whole host of new EMRs popping up that aim to help clinicians make judgement calls about how to answer certain questions and even when particular procedures are relevant. The last one we demoed, each piece of information it retrieved from the LLM has a confidence score attached to it. Our nurses have to fill out 200+ question forms when taking on a new patient that all HAVE to be completed in a single-go meaning that you can't split it up into multiple forms and it's gotta be one cohesive unit.

Imagine being a nurse with little technical skill and almost no idea how these tools work trying to make sense of what the difference between a 90% and 70% is across 200 different questions. "We're 60% sure the patient is allergic to nuts" means jack shit to them. Granted, sometimes the scores are complimented with actual references in the underlying documentation (history and physical, referring info) but sometimes it's not.

by mexicocitinluez

7/29/2026 at 10:58:50 AM

these can work as long as the evidence is attached (to your point). And in healthcare absolutely must (your EMR example is terrifying). The patient may be allergic to nuts (confidence .84) >> cite: 1) lab value X, PRO Y, ... >> otherwise I'm 62% confident I'm an elephant.

by inspectorSlap

7/28/2026 at 12:23:29 PM

Healthcare still uses fax. No surprise they arent using Frontier AI to answer these questions. (And if you still don't trust AI, have the report on the left side of the screen and the source pages on the right side of the screen)

by t3hTao

7/28/2026 at 12:41:31 PM

If you work in a field which matters, not trusting even “Frontier AI” is also known as “basic professional competence”. If you’re vibe-coding a marketing site nobody dies if you miss a mistake. The same is not true in healthcare or real engineering, and because AI systems can’t be held accountable you will be.

by acdha

7/28/2026 at 3:15:48 PM

Interesting read. Dear author, side note, please increase the body font weight, it's barely readable.

by arm32

7/28/2026 at 3:04:05 PM

It's obvious to me that writing "Give me a confidence score from 1-5" in your prompt will have disappointing results if you use that score directly. If you want self-reported confidence scores that are at all useful, you need to think about statistics and score calibration regardless of whether those scores are produced by an LLM or a human.

Take a corpus of problems you know the answers to, but the AI sometimes gets wrong [1]. Have the AI try to solve each problem and give you a 1-5 integer confidence score. The known proportion of correct answers in each bucket gives you a mapping from scores to probabilities [2].

[1] One possible corpus creation strategy might be math problems that you really need to run a program to solve. You can run the program to generate the correct answer key, but disallow tool calling for the AI.

[2] I'm pretty sure there's some stats wizardry that will let you put error bars on each bucket's probability based on the number of problems that end up in each bucket.

by csense

7/28/2026 at 10:56:44 AM

> the capability is highly unreliable and highly context-dependent.

Sounds like what's highly unreliable is the evidence, making the "capability" just another imagining.

by chrisjj

7/28/2026 at 11:56:44 AM

“That’s real, and it’s interesting.”

Stop with the slop.

by empthought

7/28/2026 at 3:46:17 PM

Dang, my bad. Thought I cleaned that up. To be clear, I use AI to help clean up scattered thoughts, and then I edit it further but sometimes I miss the AI-isms that it inserts.

by pamplemeese

7/28/2026 at 12:23:54 PM

Use AI to condense, not extend.

by t3hTao

7/28/2026 at 4:41:24 PM

Slop with the stop!

by pixel_popping

7/28/2026 at 12:06:50 PM

Unfortunately I think that we need to be much more judgmental than is tolerated on this forum if we want to get people to stop writing with AI assistance.

People are not going to stop spamming you with slop text unless the narrative is "Using AI for writing is a mark of poor literacy/low intelligence".

by dan_gee

7/29/2026 at 6:43:47 AM

[dead]

by Byte-Naut

7/28/2026 at 12:51:28 PM

[flagged]

by zenith605

7/28/2026 at 12:29:13 PM

[dead]

by cadeos

7/28/2026 at 10:47:30 AM

[dead]

by Keywordstat

7/28/2026 at 4:08:29 PM

[dead]

by inspectorSlap

7/28/2026 at 1:26:09 PM

[flagged]

by nexustoken

7/28/2026 at 8:54:43 AM

[dead]

by capestart