In June 2017, eight researchers at Google published a paper with a cheeky title and a deceptively simple claim: the recurrent and convolutional machinery everyone used for sequence modeling wasn't necessary. You could throw it all out and replace it with one idea — attention — applied in parallel across the whole sequence at once.
That sounds like a minor architectural tweak. It was not. The Transformer is the substrate of essentially every large language model you've used.
Why recurrence was the bottleneck
Before 2017, the dominant sequence models were recurrent: LSTMs and GRUs that read a sentence one token at a time, carrying a hidden state forward.RNNs can model long-range dependencies in principle, but gradients have to survive a long chain of multiplications to get there — which is exactly why LSTMs added gating in the first place. That sequential dependency is a problem for two reasons. First, you can't parallelize across the time dimension — step t needs step t−1 — so training doesn't exploit modern GPU hardware well. Second, information from early tokens has to survive a long relay of state updates to influence late ones.
Attention sidesteps both. Every position attends directly to every other position in a single step, so the path length between any two tokens is constant, and the whole operation is one big matrix multiply — embarrassingly parallel.
The architecture, one step at a time
Before the full pipeline, the one idea everything rests on — self-attention. Hover a token to see what it looks at:
Scroll through the core pipeline. The figure on the right tracks where you are.
1. Tokens become vectors
The input text is split into tokens and each token is mapped to a learned embedding
vector of dimension d_model (512 in the base model). From here on, the model only ever
sees vectors — the words themselves are gone.
2. Self-attention: queries, keys, values
Each token's vector is projected into three vectors: a query, a key, and a value. To compute a token's new representation, you take the dot product of its query with every token's key — a similarity score — scale it, softmax it into weights, and use those weights to average the value vectors. Tokens that "match" contribute more.The √d_k scaling matters: for large key dimensions, raw dot products grow large in magnitude, pushing softmax into regions with vanishing gradients. Dividing by √d_k keeps the variance sane.
3. Many heads, many relations
One attention pattern isn't enough. The Transformer runs 8 heads in parallel, each
with its own learned Q/K/V projections into a lower-dimensional subspace. One head might
track subject-verb agreement, another nearby word order. Their outputs are concatenated
and projected back to d_model. This is multi-head attention.
4. Where order comes from
Attention is permutation-invariant — shuffle the tokens and you get the same set of outputs. So the model has no built-in sense of order. The fix: add positional encodings to the embeddings — fixed sinusoids of different frequencies, one per dimension. The model learns to read position out of these wave patterns.The paper uses fixed sinusoids, but notes learned positional embeddings work about as well. Sinusoids have the nice property of extrapolating to sequence lengths not seen in training.
5. The encoder–decoder stack
The full model stacks 6 encoder layers and 6 decoder layers. Each encoder layer is self-attention followed by a position-wise feed-forward network, with residual connections and layer normalization around both. The decoder adds a masked self-attention (so it can't peek at future tokens) plus a cross-attention layer that attends to the encoder's output — that's how the target sequence conditions on the source.
6. It worked — and it was cheap
On WMT 2014, the Transformer hit 28.4 BLEU on English-to-German and 41.8 BLEU on English-to-French, beating the previous best — including ensembles — while training in a fraction of the compute. The base model has roughly 65M parameters and trained on 8 NVIDIA P100 GPUs. Parallelism wasn't just elegant; it was the whole point.
RNN/LSTM vs Transformer, side by side
RNN / LSTM
Reads tokens sequentially — step t depends on t−1, so training can't parallelize over the sequence. Long-range dependencies must survive a long chain of state updates. Cheaper per token on short sequences; painful to scale.
Transformer
Every token attends to every other in one parallel step — constant path length between any two positions. Trains far better on GPUs/TPUs. The cost: attention is quadratic in sequence length, so very long contexts get expensive.
The authors put the thesis plainly in the abstract:
“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”
What it unleashed
The Transformer was published as a translation model. Within two years it had become the default architecture for nearly all of NLP — and then vision, audio, and protein folding. Two papers in particular took the two halves of the architecture and ran:
- BERT took the encoder stack and pre-trained it bidirectionally — read the whole sentence, mask out words, predict them.
- GPT-3 took the decoder stack, scaled it to 175 billion parameters, and showed that sheer scale plus next-token prediction yields few-shot, in-context learning.
Everything since — instruction tuning, RLHF, the model you're probably chatting with — is a descendant of that 2017 diagram.
Read the paper if you haven't. It's nine pages, unusually clear, and you'll recognize every idea your favorite model is built on.
References & further reading
- Attention Is All You NeedVaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin · NeurIPS · 2017
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingDevlin, Chang, Lee, Toutanova · NAACL · 2019
- Language Models are Few-Shot Learners (GPT-3)Brown, Mann, Ryder, Subbiah, et al. · NeurIPS · 2020