2.1 Tokens, not words

A language model never sees your text. It sees a sequence of integers, each an index into a fixed vocabulary, and the thing that produces those integers explains a surprising amount of model behaviour.

The difficulty is that language does not have a fixed vocabulary. New names, typos, compound words and rare technical terms arrive constantly, but a neural network needs a finite output layer. Subword segmentation makes translation tractable as an open-vocabulary problem, letting a fixed vocabulary represent rare and previously unseen words as sequences of subword units (Sennrich et al. 2016, sec. 1).

The standard construction is byte pair encoding. Byte pair encoding builds a subword vocabulary by starting from characters and repeatedly merging the most frequent adjacent symbol pair (Sennrich et al. 2016, sec. 3.2). Run it over a corpus and common words collapse into single tokens, while rare ones stay split into pieces. The number of merge operations is the only hyperparameter of byte pair encoding; the final vocabulary size is the initial character vocabulary plus that number of merges (Sennrich et al. 2016, sec. 3.2).

That is the whole mechanism. It is worth sitting with how crude it is: frequency statistics over character pairs, with no linguistic knowledge of morphemes, roots or word boundaries.

Several familiar behaviours fall directly out of this. A common word arrives as one opaque token, so its spelling is simply not visible to the model — which is why character-counting questions go wrong in ways that look absurd for a system that can write an essay. Rare names shatter into several tokens, spending more of the context window. And a language poorly represented in the merge corpus gets segmented far more aggressively, so the same sentence costs noticeably more tokens than its English equivalent.

None of these are bugs in the model. They are consequences of a decision made before the model sees anything at all.

2.2 What attention actually computes

Once text is a sequence of tokens, the architectural question is how a token at one position gets to influence a token at another.

The previous answer was recurrence: walk the sequence left to right, carrying a hidden state. It works, but it has a structural cost. The sequential nature of recurrent models precludes parallelization within a training example (Vaswani et al. 2017, sec. 1). Position 500 cannot be computed until position 499 has been (Vaswani et al. 2017, sec. 1), so the length of the sequence sets a floor on how long a training step takes, no matter how many GPUs are available.

The Transformer dispenses with recurrence and convolutions entirely, relying solely on attention mechanisms (Vaswani et al. 2017, abstract).

2.3 What the operation is

Every token position emits three vectors: a query, a key, and a value (Vaswani et al. 2017, sec. 3.2.1). Scaled dot-product attention computes softmax(QK^T/sqrt(d_k))V, dividing by the square root of the key dimension to stop large dot products pushing the softmax into regions of vanishingly small gradient (Vaswani et al. 2017, sec. 3.2.1).

Read that as a soft lookup. Each query is compared against every key by dot product, producing a relevance score for every position (Vaswani et al. 2017, sec. 3.2.1). The softmax turns those scores into weights summing to one, and the output is the weighted average of the value vectors. A token gathers information from wherever in the sequence it is relevant, with the weighting learned rather than fixed.

The sqrt(d_k) term is easy to skip past, but it is doing real work. Dot products of high-dimensional vectors grow with dimension; without the scaling, the softmax saturates and gradients vanish, which is a training failure rather than a quality tweak.

2.4 Why this was the unlock

The property that matters is distance. A self-attention layer connects all positions with a constant number of sequential operations, giving a maximum path length of O(1) between any two positions, whereas a recurrent layer requires O(n) sequential operations (Vaswani et al. 2017, sec. 4, table 1).

In a recurrent model, information from token 1 reaching token 500 must survive 499 sequential transformations (Vaswani et al. 2017, sec. 4, table 1). Under attention, they are one operation apart.

Both the vanishing-signal problem and the parallelization problem dissolve together, which is why the change mattered so much more than a typical architectural adjustment.

One refinement: rather than a single attention operation, the Transformer uses multi-head attention, with the base model employing 8 parallel attention heads of dimension 64 each (Vaswani et al. 2017, sec. 3.2.2). Splitting the representation lets different heads attend on different bases at the same time, instead of forcing one averaged notion of relevance.

3.1 What pretraining optimizes

The training objective is smaller than most people expect.

A language model assigns a probability distribution over the possible next tokens in its vocabulary, given the preceding tokens as context (Jurafsky and Martin 2026, ch. 7, intro; Jurafsky and Martin 2026, ch. 7.4.1). Training compares that distribution against the token that actually came next, and adjusts the weights. That is the entire pretraining task.

What makes it scale is where the supervision comes from. Pretraining is self-supervised: the training signal comes from the corpus itself, since the next word is already known at every position, requiring no human labels (Jurafsky and Martin 2026, ch. 7.5.1). Every position in every document is a free training example (Jurafsky and Martin 2026, ch. 7.5.1). This is the property that let training sets grow to trillions of tokens — no annotation budget stands in the way.

Worth being precise about what this does not include. Pretraining produces a model that continues text plausibly. It does not by itself produce a model that follows instructions or declines harmful requests; those come from later stages, trained on data that is human-labelled.

3.2 What scale bought

The empirical finding that drove the last several years is regular enough to be surprising. Test loss falls as a power law in each of model size, dataset size and compute, when not bottlenecked by the other two, across trends spanning more than six orders of magnitude (Kaplan et al. 2020, sec. 1.1).

Stranger still: within reasonable limits, performance depends far more strongly on scale than on architectural hyperparameters such as the ratio of depth to width (Kaplan et al. 2020, sec. 1.1). The details practitioners once agonised over mattered less than how much of everything you had.

3.3 The correction

The first influential compute-allocation recommendation turned out to be wrong, and the way it was corrected is instructive.

Hoffmann et al. found that model size and training tokens should be scaled in approximately equal proportions, revising Kaplan et al.’s earlier recommendation to spend most additional compute on parameters (Hoffmann et al. 2022, sec. 3.4; Kaplan et al. 2020, sec. 1.1). The demonstration was direct: Chinchilla, with 70 billion parameters trained on 1.4 trillion tokens, outperformed Gopher’s 280 billion parameters trained on 300 billion tokens at equivalent compute (Hoffmann et al. 2022, abstract, sec. 4).

A model a quarter the size beat a model four times larger, on the same compute budget, because the compute was divided differently. Note what survived and what did not: the power-law relationship held, while the practical advice built on top of it was overturned within two years.

And a caveat that is routinely dropped: these laws predict loss, the model’s error at predicting the next token. They do not predict which capabilities appear, when a model will reason reliably, or whether a given benchmark will be passed. Loss is what was measured, and loss is what was extrapolated.

3.4 One token at a time

A trained model does exactly one thing: given a context, produce a distribution over the next token. Everything that looks like sustained writing is that single step, run in a loop (Jurafsky and Martin 2026, ch. 7, fig. 7.2 caption).

Text is generated autoregressively: each generated token is appended to the context and used as the prefix for generating the next one (Jurafsky and Martin 2026, ch. 7, fig. 7.2 caption). There is no plan, no draft, and no lookahead. The model has committed to every token it has already emitted, and conditions on them exactly as it would on text you wrote.

This explains a class of behaviour that otherwise looks like carelessness. A model that opens with a wrong claim will continue coherently from that claim, because its own output is now indistinguishable from the prompt. There is no mechanism by which an earlier token gets revised.

3.5 Choosing the token

Having a distribution is not yet having a token, and the choice matters.

The obvious rule is to take the most likely one. Greedy decoding always emits the highest-probability token, which makes it deterministic and produces text that is generic and often repetitive (Jurafsky and Martin 2026, ch. 7.4.1). That determinism sounds desirable — the same prompt would give the same answer — but the output quality is poor enough that it is rarely used for open-ended generation.

So decoders sample instead, and shape the distribution before they do. Temperature sampling divides the logits by a temperature parameter before the softmax, concentrating probability on higher-probability tokens as temperature falls (Jurafsky and Martin 2026, ch. 7.4.3). Low temperature approaches greedy behaviour; higher temperature flattens the distribution and admits less likely tokens.

Two practical consequences follow. The same prompt yields different answers across runs because the token was sampled, not chosen — this is a decoding decision, not evidence of an unstable model. And “temperature 0” outputs are not a more truthful model; they are the same model with its most predictable path taken, which is exactly the setting shown to produce generic text.

That is the whole pipeline: text becomes subword tokens, attention lets every position see every other, pretraining fits next-token prediction over enormous corpora, and generation samples that prediction one token at a time. Nothing in the loop is more sophisticated than those four mechanisms — the capability comes from their scale, not their complexity.