By Roman Aglaganov
Every language model has a context window: the amount of text it can look at in one go, measured in tokens, the word-pieces models read. For years these windows were small. Early ChatGPT could hold a few thousand tokens, roughly a long essay, and would forget the start of a conversation if you went on too long.
One million tokens
In February 2024, Google announced Gemini 1.5 Pro with a context window of up to one million tokens, up from 32,000 in Gemini 1.0. In Google’s words, that is about an hour of video, 11 hours of audio, a codebase of more than 30,000 lines or over 700,000 words, all at once. In research, they tested up to 10 million.
Finding the needle
A big window is useless if the model cannot find things in it. The standard test is called Needle in a Haystack: hide one specific fact somewhere in a huge block of text, then ask about it. Gemini 1.5 Pro found the needle 99% of the time. It is the AI equivalent of reading the whole phone book and remembering one number on page 4,000.
Learning Kalamang
The most striking result was in the technical report. Kalamang is a language with fewer than 200 speakers worldwide, so there is next to nothing about it in a model’s usual training data. Researchers gave the model a grammar manual for the language in its context window, and it learned to translate English into Kalamang at a level similar to a person who had studied the same material. It did not know the language before; it learned it from the book, on the spot.
Why it matters: long context changed how people use AI. Instead of chopping documents into pieces, you can now give a model the whole contract, the whole codebase or the whole film and ask questions. Today million-token windows are common across the big labs.
