alt.hn

5/25/2026 at 1:30:14 PM

Confidence Scores for Exam Questions

https://nomagicpill.substack.com/p/confidence-scores-for-exam-questions

by surprisetalk

5/28/2026 at 8:47:43 PM

I seem to remember some medical related multiple choice tests in the UK use a mechanism of +1 for correct , 0 for unanswered , -1 for incorrect.

by vmilner

5/28/2026 at 10:37:57 PM

Typically you have -1/N for incorrect selection, where N is the number of choices. For N=4, you would grade incorrect answers as -0.25.

If you have a person taking an exam that is not confident in themselves or generally knows the subject area, you don't want to negatively impact educated guessing.

by bArray

5/29/2026 at 12:54:59 AM

> Typically you have -1/N for incorrect selection, where N is the number of choices. For N=4, you would grade incorrect answers as -0.25.

That is definitely not typical. -0.25 is the appropriate adjustment for N=5. For N=4 you want -0.33. -1/N makes no sense at all.

Note that doing this preserves the expected value of everyone's score, but artificially widens the variation, which you might not want. It does allow you to diagnose partial knowledge, which you probably do want.

by thaumasiotes

5/29/2026 at 11:05:51 AM

Maybe it is typical only for me. I did question it and was told that we don't want to completely remove the incentive for educated guessing. We also usually have a scale of question difficulty, so getting people to a pass is not too difficult if they know the subject at all, but getting towards 100% gets significantly harder.

I think the real reason is that our questions are usually N=4, negative marks of 0.25 allows for quick adding.

by bArray

5/29/2026 at 12:39:53 PM

> I did question it and was told that we don't want to completely remove the incentive for educated guessing.

So... you were told some unmotivated nonsense?

On an item with four answers, +1 for a correct answer and -0.25 for a wrong answer means that in expectation you will receive 0.0625 points for a completely uneducated guess. The only correct adjustment you can make is to dock 0.33(3...) for a wrong answer, in which case an uneducated guess is worth 0.0000 points and a minimally-educated guess, one in which you're capable of eliminating just one of the four answers, is worth... 0.0833(3...) points.

> I think the real reason is that our questions are usually N=4, negative marks of 0.25 allows for quick adding.

You think adding fourths is easy, but adding thirds is hard? If you really believe that, it'd be simple enough to add fifth choices to your questions.

Are you sure the real reason isn't just that nobody ever bothered to put any thought into what they were doing?

by thaumasiotes

5/28/2026 at 9:32:36 PM

A system like that seems especially appropriate for a practice where the foundational principle is "do no harm."

by CGMthrowaway

5/28/2026 at 10:12:06 PM

Would probably be applicable to engineers as well, or any other field where the practitioner has an obligation to be aware of the limits of their competency.

by bee_rider

5/28/2026 at 10:39:13 PM

and yet.

by DANmode

5/28/2026 at 9:29:18 PM

It would make more sense to just use IRT for grading the responses than trying to add more complexity to the answers themselves.

by clickety_clack

5/29/2026 at 1:33:47 AM

Terence tao had a blogpost about this for the case of true/false questions, with multiple choice left as an exercise for the reader.

https://terrytao.wordpress.com/2016/06/01/how-to-assign-part...

At least with the formulation Tao had, one unfortunate side-effect is that if you answer with 100% confidence and get it wrong, you basically fail the entire course.

by krackers