By Roman Aglaganov
The story of modern AI is mostly a story of more: more data, more parameters, more chips. Microsoft’s Phi models are a small rebellion against that.
Textbooks are all you need
In June 2023, Microsoft researchers published a paper with a cheeky title, “Textbooks Are All You Need”, a nod to the famous “Attention Is All You Need”. Their model, phi-1, had just 1.3 billion parameters, a fraction of the big models of the day, and was trained for four days on eight GPUs.
The secret was the diet. Instead of feeding it everything on the internet, they trained it on about 6 billion tokens of carefully filtered “textbook quality” web data, plus a billion tokens of textbooks and exercises written by GPT-3.5. On a standard coding test, HumanEval, it scored 50.6%, competitive with models many times its size.
A model in your pocket
In April 2024 came Phi-3-mini, with 3.8 billion parameters. Microsoft reported that it rivalled GPT-3.5 and Mixtral 8x7B on standard benchmarks, scoring 69% on the MMLU knowledge test, while being small enough to run on a phone. The technical report’s subtitle said it plainly: a highly capable language model, locally on your phone.
Why small matters
Cost: small models are dramatically cheaper to run, which matters when you answer millions of questions a day.
Privacy and offline use: a model on your device does not need to send your data anywhere.
Speed: fewer parameters means faster answers, especially on ordinary hardware.
Phi models are released with open weights, so anyone can download and adapt them. The catch is breadth: a small model knows less about the world, so it can be sharp at reasoning and coding but shaky on trivia. The lesson of Phi, now copied across the industry, is that quality of data can stand in for a lot of quantity. You are what you eat, even if you are a neural network.
