By Roman Aglaganov
Ask a chatbot something it does not know, and too often you will not get “I don’t know”. You will get a fluent, confident, completely invented answer. In 2023, two New York lawyers learned this the hard way, submitting a brief full of court cases ChatGPT had made up, and paying a $5,000 fine for it.
Trained to guess
A 2025 paper from OpenAI researchers, “Why Language Models Hallucinate”, makes an uncomfortable argument: we taught models to bluff. Most benchmarks score answers as simply right or wrong. Under that system, a model that guesses gets some answers right by luck, while a model that says “I don’t know” gets zero. Optimise for the leaderboard and you get a confident guesser, the AI equivalent of a student filling in every multiple-choice bubble.
Change the incentives
The fix the authors propose is simple in principle: penalise confident wrong answers more than honest uncertainty, and give partial credit for saying “I’m not sure”. Some exams for humans already do this. AI evaluations should too.
What users can do now
Ask for sources and check them, especially for anything legal, medical or financial.
Invite uncertainty: tell the model it is fine to say it does not know. Many will take you up on it.
Reward the right behaviour: when a tool admits its limits, that is a feature, not a failure.
We do not trust people who are never unsure. We should not trust machines that are never unsure either.
