# Opinion: Every AI should learn to say “I don’t know”

URL: https://ai-pulse.aglaganova.com/blog/opinion-every-ai-should-learn-to-say-i-dont-know
Published: 2026-10-09
Author: Roman Aglaganov, Full-stack developer
Category: Opinion
Region: Global
Tags: benchmarks, safety

> Chatbots hallucinate partly because they are trained and tested in ways that reward confident guessing. It is time to reward honesty instead.

![Typographic cover: “I don’t know.” Three words every AI should learn](https://ai-pulse.aglaganova.com/api/media/file/dont-know-1.png)

Ask a chatbot something it does not know, and too often you will not get “I don’t know”. You will get a fluent, confident, completely invented answer. In 2023, two New York lawyers learned this the hard way, submitting a brief full of court cases ChatGPT had made up, and paying a $5,000 fine for it.

## Trained to guess

A 2025 paper from OpenAI researchers, “Why Language Models Hallucinate”, makes an uncomfortable argument: we taught models to bluff. Most benchmarks score answers as simply right or wrong. Under that system, a model that guesses gets some answers right by luck, while a model that says “I don’t know” gets zero. Optimise for the leaderboard and you get a confident guesser, the AI equivalent of a student filling in every multiple-choice bubble.

## Change the incentives

The fix the authors propose is simple in principle: penalise confident wrong answers more than honest uncertainty, and give partial credit for saying “I’m not sure”. Some exams for humans already do this. AI evaluations should too.

## What users can do now

**Ask for sources** and check them, especially for anything legal, medical or financial.

**Invite uncertainty:** tell the model it is fine to say it does not know. Many will take you up on it.

**Reward the right behaviour:** when a tool admits its limits, that is a feature, not a failure.

We do not trust people who are never unsure. We should not trust machines that are never unsure either.

## Sources

- [Kalai et al., Why Language Models Hallucinate (arXiv:2509.04664)](https://arxiv.org/abs/2509.04664)