By Roman Aglaganov
Why are tech companies spending hundreds of billions of dollars on data centres? A big part of the answer is a handful of graphs from 2020.
Bigger is better, predictably
In January 2020, researchers at OpenAI published “Scaling Laws for Neural Language Models”. They found that a model’s performance improves smoothly and predictably as you increase three things: the size of the model, the amount of training data and the amount of computing power. Plot it on the right kind of chart and you get a straight line, which is catnip for anyone planning a budget.
Chinchilla: the diet plan
In 2022, DeepMind refined the recipe. Its “Chinchilla” paper found that many large models were undertrained: too many parameters, not enough data. For a fixed computing budget, it is better to train a smaller model on much more text, roughly 20 tokens (word pieces) of training data per parameter. To prove it, DeepMind trained Chinchilla, with 70 billion parameters and 1.4 trillion tokens of text, and it outperformed the company’s own 280-billion-parameter Gopher.
The limits
Scaling laws are observations, not laws of physics. They predict a model’s loss, a technical measure, more reliably than the specific skills it will have. And they assume you can keep finding more high-quality data, which is getting harder: the internet is big, but not infinite.
That is one reason labs have started scaling in a new direction: letting models spend more computing power while they answer, the approach behind today’s reasoning models. When one curve flattens, the industry goes looking for another.
