How a language model actually works: one trick, taken very far

In an earlier post I walked the long road to modern AI and ended on a deliberately deflating note: strip away the scale, the hardware and the billions of parameters, and a large language model is doing the thing Claude Shannon asked a person to do back in 1951, looking at a stretch of text and guessing what comes next. That was the punchline of the history. This post is the part I skipped past: how that one trick, taken very far, turns into something you can hold a conversation with.

No math, but nothing hand-wavy either. I want to give you an honest picture of what is actually happening under the chat window, including the parts where even the people who build these systems don't fully agree.

One trick

Everything starts with a single objective, and it is almost insultingly simple. Take a stretch of text, hide the end, and have the model guess what comes next. Then tell it whether it was right, and let it adjust. Do that a staggering number of times over a staggering amount of text. The whole training game is predicting the next bit, over and over; nothing in it asks the model to understand language or answer questions.

Shannon saw the shape of this in 1951. He wanted to measure how much genuine information English carries, and his method was a parlour game: show someone a sentence with the rest hidden and ask them to guess the next letter, then the next, then the next. People are good at it, because language is full of pattern, and exactly how good they are was the quantity he was measuring. A language model is that game run at a scale Shannon could not have pictured, by a machine instead of a person, but it is mechanically the same move.

Here is the part that sounds like a trick of words but isn't. Predicting well and compressing well are, mathematically, the same skill. If you can reliably guess what comes next, you can store text in less space, because the predictable parts carry little information and can be written cheaply. A model that predicts text brilliantly is therefore a brilliant compressor, and a recent DeepMind paper made this concrete in a way I find genuinely startling: a model trained only on text turned out to compress images and audio better than the specialist tools built for exactly that job (Delétang et al., 2023). One honest caveat the authors themselves flag, and which the headlines usually drop: those figures don't count the enormous size of the model doing the compressing. Amortise that in and the picture is far less magical. But the underlying point stands. To predict text well, a model has to find the patterns that make text predictable, and that pattern-finding is most of what these systems end up doing.

What the model actually sees

Before any of that prediction can happen, the text has to be turned into something the machine can work with, and the units it uses are not words. They are tokens: fragments, often chunks of a few characters, sometimes a whole common word, sometimes a single letter. The standard way of carving text into these pieces, byte-pair encoding, started life as a data-compression algorithm and was later borrowed for language (Sennrich et al., 2016). It works by starting from individual characters and repeatedly gluing together whichever pair shows up most often, until you have a vocabulary of useful fragments. Common words end up as one token; rare ones get spelled out from smaller pieces.

This is not a footnote. It quietly explains a whole family of odd failures. When a model miscounts the letters in "strawberry," it isn't being stupid; it never saw the letters. It saw two or three tokens, and the individual r's were never really visible to it as separate things. The model reasons over fragments of text, not over words the way you and I picture them, and a surprising number of its blind spots live in that gap.

How it learns

So the model plays the guessing game across an enormous slice of the written internet. How does it actually get better? Inside it are billions of numbers, called parameters, that start out essentially random. You can think of them as billions of tiny dials. When the model guesses the next token and gets it wrong, a procedure works backwards through all those dials and nudges each one a little in the direction that would have made the right answer slightly more likely. One nudge changes almost nothing. Trillions of nudges, across trillions of words, slowly turn that pile of random dials into something that has absorbed the patterns of language.

Nobody writes rules in this process, and no one tells the model that Paris is in France or that a question usually wants an answer. Those regularities are simply the cheapest way to get good at the guessing game, so the training pressure finds them on its own. Training squeezes the model, relentlessly, toward whatever internal arrangement predicts text well, and a lot of what looks like knowledge is a side effect of that squeeze.

Even in hindsight, the bet that this would keep paying off as the models got bigger wasn't an obvious one. When OpenAI trained GPT-3 in 2020, its 175 billion parameters were about ten times larger than any comparable model before it (Brown et al., 2020). (That "ten times larger" is a 2020 snapshot, worth remembering, since far bigger models followed within a couple of years.) What made it notable wasn't only the size. It was that the model could be handed a task it was never trained on, described in plain language with a couple of examples, and simply do it, with no retraining and no adjustment to those billions of dials. The capability fell out of scale.

Attention, and why this design won

None of this would have been practical without a change in the shape of the network itself, and this is the one piece of real machinery worth understanding.

Before 2017, the leading language models read text the way you read a sentence out loud: in order, one word after another, each step depending on the one before it. That sequential habit was a bottleneck. Because every step waited on the previous one, you couldn't spread the work across a lot of hardware at once, and that put a ceiling on how big and how fast you could train.

The 2017 paper that changed this was cheekily titled "Attention Is All You Need," and its move was to drop the read-in-order assumption entirely (Vaswani et al., 2017). Instead, every position in the text looks at every other position at the same time and works out how much each one matters to it. In a sentence like "the trophy didn't fit in the suitcase because it was too big," the mechanism lets "it" weigh every other word and settle heavily on "trophy." That weighing-of-everything-against-everything is called attention, and the architecture built around it is the transformer. Its great practical virtue is that all that weighing happens in parallel rather than in sequence, so training scales across enormous machines in a way the old designs never could.

I want to be careful here, because this is where tidy explanations tend to overreach. The 2017 paper was about machine translation and said nothing about the scaling revolution that followed; the story that "parallelism is why LLMs took off" is a fair retrospective reading, but it's stitched together from later work on how these models improve with size, not a single claim from one paper. The honest version is: the transformer removed the bottleneck that would have made today's models impossible to train, and then the scaling happened.

Why it feels like understanding

Which brings us to the uncomfortable question, the one that hides behind every impressive demo. When a model produces a fluent, correct, apparently thoughtful answer, is anything in there actually understanding, or is it an uncannily good mimic reciting the statistical shape of things it has seen?

There is a serious case for the mimic view, made most influentially in the 2021 paper that gave us the phrase "stochastic parrots" (Bender et al., 2021). The argument is that a model trained purely on the form of language, with no contact with the world the language is about, can only ever stitch together plausible sequences. It has the statistics of meaning without the meaning. When it seems to understand, that's us reading understanding into fluent output, the way we read a face into a cloud.

The trouble is that a sufficiently strong prediction machine has a reason to build more than surface statistics. If the cheapest way to guess the next move in a game is to internally track the state of that game, then predicting well might quietly force a kind of model of the world into existence. And there is now direct evidence that this can happen, at least in a small, clean setting. Researchers trained a model on nothing but sequences of legal moves in Othello, the board game, with no picture of the board and no statement of the rules, just move after move after move (Li et al., 2023). When they looked inside, they found the model had built its own internal representation of the board. Not a memorised list of move patterns, an actual model of the eight-by-eight state, which they confirmed by reaching in and altering it: change the model's internal board and its predictions about legal moves change accordingly. A follow-up study sharpened the finding, showing the representation was simpler and cleaner than first reported once you looked at it from the model's own point of view (Nanda et al., 2023).

I try to hold that result at its true weight, which means resisting the temptation to inflate it. It is strong evidence that "just surface statistics" is too glib. A system predicting a sequence really can construct an internal model of what generates that sequence. But Othello is a closed toy world with fixed rules, and the leap from "builds a model of a board game" to "understands language, and the world language describes" is exactly the leap the evidence does not license.

What I take from it

So here is where I land, and it is not a tidy verdict.

A language model is one trick, next-token prediction, driven by a squeeze toward good compression, running on an architecture that let the trick scale, and scaled until it did things nobody explicitly built in. That is not nothing. The compression view says learning to predict is learning to find structure, and the Othello result says that structure can include real internal models, not just word-level reflexes. "It's just autocomplete" leaves out most of that picture.

But the opposite story, that fluent output means comprehension, reads more into the evidence than it can carry, and the truth is that we don't yet have a clean answer for the full natural-language case. The honest position sits in the uncomfortable middle and stays there, because the question is still open.

For everyday use, though, the practical upshot is clear enough, and it's the thread running through most of what I write about these tools. The thing in front of you is a predictor of plausible text: extremely good at sounding right, with nothing in the mechanism that checks whether it is right. Knowing that it works by guessing what comes next is most of what you need to tell where to trust it and where to check it.

Sources

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. https://dl.acm.org/doi/10.1145/3442188.3445922

Brown, T. B., et al. (2020). Language models are few-shot learners. arXiv. https://arxiv.org/abs/2005.14165

Delétang, G., et al. (2023). Language modeling is compression. arXiv. https://arxiv.org/abs/2309.10668

Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2023). Emergent world representations: Exploring a sequence model trained on a synthetic task. International Conference on Learning Representations. https://arxiv.org/abs/2210.13382

Nanda, N., Lee, A., & Wattenberg, M. (2023). Emergent linear representations in world models of self-supervised sequence models. arXiv. https://arxiv.org/abs/2309.00941

Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. https://arxiv.org/abs/1508.07909

Shannon, C. E. (1951). Prediction and entropy of printed English. Bell System Technical Journal, 30(1), 50-64. https://archive.org/details/bstj30-1-50

Vaswani, A., et al. (2017). Attention is all you need. arXiv. https://arxiv.org/abs/1706.03762