Almost every model you have used this year descends from one paper published in June 2017, and that paper is fifteen pages long. The architecture inside it is much simpler than its reputation.
Strip the notation and the Transformer is a block you can hold in your head. Each word looks at every other word, decides which ones matter to it, and rewrites itself as a blend of them. Stack that block six times and you have the encoder. Everything else is there to keep it stable, or to give back something it lost.
Why recurrence had to go
Before 2017, sequence models read a sentence the way you do, one word after another, carrying a hidden state forward. Word 40 could not be computed until word 39 existed. The paper’s complaint is not that this is inaccurate, it is that it “precludes parallelization within training examples”. Put that model on a machine with eight GPUs and most of the silicon waits.
Attention itself was not new. By 2017 it was standard equipment for letting a decoder look back at the source sentence, and the paper says so: “In all but a few cases, however, such attention mechanisms are used in conjunction with a recurrent network.” Adding attention to a recurrent model improves quality and leaves the bottleneck exactly where it was.
Both lanes do the same arithmetic. Only one of them is allowed to do it at the same time.
press Run and count the clock steps each lane needs. Simulated, with canned data.
Recurrence costs you distance too. A verb that depends on a subject ten words back needs that signal to survive ten steps in each direction. Self-attention connects any two positions in one hop, which the paper lists as O(1) maximum path length against O(n) for recurrence. Shorter paths make long-range dependencies easier to learn.
What self-attention actually does to a sentence
The sentence arrives as tokens, and each token starts as a vector of 512 numbers, its embedding. Nothing in that vector knows about the rest of the sentence. The word “it” is just the generic embedding of “it”.
Self-attention fixes that. Every token turns its embedding into three new vectors. A query, describing what it is looking for. A key, advertising what it offers. A value, which is what it hands over if it gets picked. The paper’s own definition is worth reading slowly: attention maps “a query and a set of key-value pairs to an output, where the output is computed as a weighted sum of the values, and the weight assigned to each value is computed by a compatibility function of the query with the corresponding key”.
That compatibility function is a dot product. Two vectors pointing in a similar direction score high, and a softmax turns the scores into weights that add up to one.
The word it needs to know what it refers to. Watch how self-attention decides.
Four steps, and that is the entire attention mechanism.
press Next four times and watch where the weight ends up. Simulated, with canned scores.
Four moves: ask, score, normalise, blend. The token “it” leaves the layer carrying mostly “cat”, which is how a model resolves a reference with no grammar rule written down anywhere. It reached six positions back for the price of reaching its neighbour.
Each word rewrites itself as a blend of the words it decided were relevant. That one sentence is most of the architecture.
Nothing in that formula knows about word order
A weighted sum does not care what order it adds things in. Shuffle the input words and self-attention hands back the same output, reshuffled. The layer sees a bag of tokens, not a sentence.
Recurrence had given order away for free, because reading in sequence encodes it. Dropping recurrence means putting order back deliberately.
Self-attention is a weighted sum, and a sum does not care what order it adds things in.
with encoding off, swap dog and man. The layer sees no difference at all. Simulated, with canned data.
The fix is a positional encoding added straight into each token’s embedding before the first layer, built from sines and cosines at different frequencies, with wavelengths forming a geometric progression from 2 pi to 10000 times 2 pi. The same word at two positions now arrives as two different vectors.
The honest footnote is that the authors also tried learned positional embeddings and found “the two versions produced nearly identical results”. They kept the sinusoids on a hunch, that the pattern might let the model extrapolate to sequences longer than the ones it trained on.
One square root is what lets it train at any size
There is one more term in the formula, skipped by most summaries, and the model will not train without it. Before the softmax, the dot products are divided by the square root of d_k, the key dimension.
Longer vectors produce bigger dot products, and softmax answers big numbers by putting nearly all the weight on the winner. The authors: for large key dimensions “the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients”.
Eight keys, one query, the same eight raw similarity scores throughout. Only the dimension changes.
turn scaling off, then drag dk up to 512. Simulated, with canned scores.
Turn the scaling off, drag the dimension up, and you can watch a layer stop learning. The square root cancels that growth exactly, which is why the same design works at 64 dimensions and at 512. The parts of a paper that look like housekeeping are often the load bearing ones.
Eight small heads beat one big one
A single attention distribution has to carry every kind of relation at once, so it averages them. Which word is the subject, which is the object, which determiner belongs to which noun, all collapsed into one set of weights.
Multi-head attention splits the 512 dimensions into eight heads of 64, runs them at the same time and concatenates the results. Each head gets a small space to specialise in, and because they are narrow the total cost stays close to one full-width head.
Query position: voids. Everything it attends to is shaded.
Press Next to run the heads one at a time.
step through all eight, then switch to h = 1. Attention patterns here are illustrative, not measured from the paper.
The ablation cuts both ways. One head scores 24.9 BLEU on the English to German development set against 25.8 for eight. Push to 32 heads and it falls back to 25.4. There is a middle that works.
The rest of the block is plumbing, and it matters
Around the attention layer sits the machinery that makes a deep stack trainable. Each sub-layer is wrapped in a residual connection and layer normalisation, so what flows forward is LayerNorm of x plus Sublayer of x. After attention comes a feed-forward network applied to each position separately, widening from 512 to an inner size of 2048 and back.
Attention plus feed-forward is one layer. The encoder stacks six of them. The decoder stacks six of its own, with two differences. Its self-attention is masked so that a position cannot attend to positions after it, which is what stops the model reading the answer it is meant to predict, and it adds a third sub-layer attending over the encoder’s output.
Nothing has really replaced that structure since. The decoder stack, masked and scaled up, is what sits inside the models you use today.
The numbers people switched for
On WMT 2014 English to German the big model reaches 28.4 BLEU, which the authors note beats the previous best results including ensembles by over 2 BLEU. The strongest ensembles they compare against score 26.30 and 26.36. On English to French it sets a single-model state of the art at 41.8 BLEU.
The training cost is what changed behaviour. Everything ran on one machine with eight NVIDIA P100 GPUs. The base model trained for 100,000 steps in twelve hours and already reached 27.3 BLEU. The big model took 300,000 steps and three and a half days. That is 3.3 times ten to the eighteenth floating point operations against 2.3 times ten to the nineteenth, which the paper positions as a small fraction of what the competing systems cost.
Twelve hours on one machine to beat an ensemble changes what a team is willing to attempt in a week. The authors also applied the same architecture to English constituency parsing, with large and with limited training data, to show it was not a translation trick.
Why this one stuck, and what it cost
It is easy to read the paper backwards and call it the invention of attention. It was not. What it did was remove the last sequential dependency from the architecture, at a moment when the hardware underneath was getting wider every year. A design that trains in parallel rides that curve. One that does not, does not.
The bill came with it. Every position attending to every other position costs O(n squared times d) per layer, which is cheap when sentences are short relative to the model width and expensive when they are not. The authors saw it coming, proposing attention restricted to a neighbourhood of size r for very long sequences and noting that this pushes the path length back up to O(n/r). Nine years later much of the long-context literature is still working that same trade, including treating long context as a learning problem rather than a storage one.
So take the shape rather than the slogan. Attention is not magic, it is a weighted average with learned weights. What made it win is that the whole sentence goes through it at once.