By Roman Aglaganov
In June 2017, eight researchers at Google published a paper with a playful title, a nod to the Beatles’ “All You Need Is Love”. It proposed a new kind of neural network for translating languages. It became one of the most cited papers in modern computer science, and the “T” in GPT.
What the transformer changed
Before transformers, language models read text mostly one word at a time, in order, like someone reading through a keyhole. That was slow, and they struggled to connect words far apart in a long sentence.
The transformer uses attention: every word looks at every other word in the passage at once and decides which ones matter for its meaning. In “The animal didn’t cross the road because it was too tired”, attention helps the model work out that “it” means the animal, not the road.
Why it took over
Because a transformer processes all the words in parallel, it is a perfect match for GPUs, which are built to do many calculations at once. That made it practical to train on vastly more text than before, and it turned out that transformers kept getting better the bigger they got. Within a few years they had taken over translation, then text generation, then images, audio, protein folding and more.
Where the authors went
All eight authors eventually left Google (one later returned), several to found AI companies of their own, among them Cohere and Character.AI. Rarely has one research paper launched so many start-ups, and so much compute spending, from a single diagram.
