By Roman Aglaganov
Every AI launch now comes with the same slide: a table of benchmark scores, the new model’s numbers in bold, every rival a few points behind. It is the tech industry’s version of a car advert where the vehicle drives along an empty mountain road. Technically accurate, not much like your commute.
Problem one: the tests get used up
Benchmarks have a short life. MMLU, a test of knowledge across 57 subjects released in 2020, was once a serious challenge; top models now score around 90%, leaving little room to tell them apart. Researchers keep building harder ones. Humanity’s Last Exam, published in early 2025, collected about 2,500 expert-level questions precisely because the old tests had run out of headroom, and the best models at launch scored under 10%. Those scores have been climbing ever since.
Problem two: the answers leak
Models learn from enormous scrapes of the internet, and benchmark questions live on the internet. If test questions end up in the training data, a high score may measure memory rather than ability. It is the AI equivalent of finding the exam paper in the bin the night before.
Problem three: Goodhart’s law
“When a measure becomes a target, it ceases to be a good measure.”
The economist Charles Goodhart’s observation fits AI uncomfortably well. Once a benchmark becomes the number everyone competes on, it is tempting to tune models for that test specifically, and the score drifts away from the thing it was meant to capture.
What to do instead
Benchmarks are still useful as rough signals, especially new and carefully guarded ones such as the ARC-AGI puzzles, which are easy for people and stubbornly hard for AI. But the best test of a model is the job you actually need it for. Try your own tasks, with your own documents, and keep the questions somewhere a web crawler cannot find them.
In short: read the chart, then run your own exam. And be suspicious of any score that ends in “state of the art” and starts with a marketing team.
