“Attention Is All You Need” (Vaswani et al., 2017) replaced recurrence with attention and unlocked modern AI. Without it, GPT-4, Claude, Gemini and every LLM you use wouldn’t exist. Let’s open the black box — no PhD required, but no oversimplification either.
⚡ Core Idea
Before 2017: RNNs read words sequentially, forgetting early words by page 2. After: Transformers let every token directly attend to every other token in parallel, capturing long dependencies in O(1) steps. Cost: O(n²) attention, fixed by FlashAttention.
Why RNNs Failed and Transformers Won
| Architecture | How it reads | Parallel? | Long context? | Today’s use |
|---|---|---|---|---|
| RNN / LSTM (1997) | Word-by-word, hidden state carries memory | No — must wait for word 1 to process word 2 | Forgets after ~50 tokens (vanishing gradient) | Legacy; still in tiny on-device models |
| Transformer (2017) | Whole sentence at once, attention weights connect all pairs | Yes — GPUs love matrix multiplies | Handles 128K–2M via attention | Every frontier LLM |
The breakthrough wasn’t accuracy alone — LSTMs could match transformers on small datasets. The win was trainability at scale: transformers parallelize across 10K GPUs, RNNs cannot. When you train on 15T tokens, parallelism = feasibility.
Big Picture: Encoder vs Decoder vs Encoder-Decoder
The original Transformer for translation had Encoder (reads German) → Decoder (writes English). Today:
- Encoder-only (BERT, 2018): Reads bidirectionally (can look ahead). Great for classification, search, embeddings. Not generative.
- Decoder-only (GPT-3/4, Claude, Llama, Gemini): Reads left-to-right with causal mask. Perfect for next-token generation. This is the LLM.
- Encoder-Decoder (T5, UL2): Best for sequence-to-sequence (translation, summarization) but heavier to scale. Fading for pure LLMs, still used in some multimodal models.
Our focus: Decoder-only stack — repeat the same block 32–80 times, each refining token representations.
Self-Attention Step-by-Step (Q, K, V)
Take sentence: “The cat sat” → 3 tokens. Each token starts as a vector (embedding) of size d = 4096.
Step 1 — Project to Q, K, V
Each embedding x is multiplied by three learned matrices:
Q = x · W_Q— “What am I looking for?”K = x · W_K— “What do I contain?”V = x · W_V— “What will I contribute if attended to?”
Dimensions: Q,K = d_head (e.g., 128), V = d_head. For multi-head, there are head-specific W matrices.
Step 2 — Scores: Q · K^T / sqrt(d_head)
For “cat” as query, compute dot product with every token’s K: score(cat→The), score(cat→cat), score(cat→sat). High dot = high relevance. Divide by sqrt(d_head) (≈11) to keep softmax stable (prevents extreme spikes).
Step 3 — Softmax → Attention Weights
Softmax over scores gives probabilities summing to 1. Example for “it” in “animal … it was tired”:
| Query “it” → Key | Score | Weight after softmax |
|---|---|---|
| animal | 8.2 | 0.68 |
| street | 3.1 | 0.10 |
| didn’t | 1.2 | 0.02 |
| it (itself) | 4.5 | 0.20 |
Interpretation: 68% of “it”’s new representation will come from “animal”’s V vector. No hand-coded rule — learned from data.
Step 4 — Weighted Sum of V
New vector for “it” = 0.68·V_animal + 0.10·V_street + 0.20·V_it + … This mixes context into every token, in parallel, in one matrix multiply — GPUs excel here.
Intuition: Library vs Spotlight
Q = your question (“find subject of ‘it’”), K = book titles on spine, V = book content inside. Attention = beam of a spotlight; softmax = brightness assigned to each book. Multi-head = 16 spotlights of different colors looking for different clues simultaneously.
Multi-Head Attention: 16 Lenses at Once
One attention head averages; 32 heads specialize. Empirically visualized heads do:
- Head 3: attends to previous token (local syntax)
- Head 7: attends to matching parentheses or quotes (code structure)
- Head 12: tracks coreference (“it” → “animal”)
- Head 19: attends to distant topic word (“Inflation” 200 tokens earlier)
Mathematically: MultiHead = Concat(head_1, …, head_h) · W_O. Each head has its own W_Q, W_K, W_V, W_O slices. Total compute same as one big head, but representational power far higher.
Positional Encoding: RoPE & Why Order Matters
Attention is permutation-invariant — without position, “dog bites man” = “man bites dog”. Early transformers added sinusoidal absolute positions: PE(pos, 2i) = sin(pos / 10000^(2i/d)). Modern LLMs use RoPE (Rotary Position Embedding).
RoPE in one paragraph
Rotate Q and K vectors by angle pos × θ_i where θ_i = 10000^(-2i/d). The dot product Q^T·K then depends on relative distance (pos_q - pos_k) via cosine, not absolute. Benefits: extrapolates to longer sequences than trained on (with scaling tricks like NTK-aware, YaRN), and handles 128K+ contexts without re-training positions.
Other variants: ALiBi (adds linear bias by distance), used in MPT and older models, but RoPE won for 2026 frontier models (Llama 3, Gemma 2, DeepSeek).
The Full Transformer Block
Each layer = Attention → Feed-Forward (MLP), each with Residual + LayerNorm:
- LayerNorm: normalize vectors (mean 0, var 1, learned scale) for stable training.
- Self-Attention: as above, mixing information across tokens.
- Add & Norm: output = LayerNorm(x + Attention(x)) — residual highway lets gradients flow through 80 layers.
- MLP: Two linear layers with activation (SwiGLU in Llama:
gate = SiLU(xW_gate) * (xW_up); out = gate W_down). MLP mixes information within each token, adding depth of reasoning. This holds 2/3 of parameters! - Add & Norm again: final output of block.
Stack 32–126 such blocks → final hidden states → linear head → logits over vocab → softmax → next-token probabilities.
| Component | Parameters | Role |
|---|---|---|
| Attention QKV + O | ~30% | Context mixing |
| MLP (gate/up/down) | ~65% | Memory & reasoning inside token |
| Embeddings + Norm + Head | ~5% | I/O |
Causal Masking: Why LLMs Are Decoder-Only
During training, the model sees entire sentence “The cat sat on the mat”, but must not peek ahead when predicting “sat” given “The cat”. A causal mask sets attention scores for future tokens to -∞ before softmax → weight 0. This enforces left-to-right.
Why decoder-only won:
- Training is simple: one forward pass predicts all next tokens in parallel with mask.
- Inference is natural: generate token-by-token, each new token can attend to all previous generated tokens (with KV-cache).
- Encoder-decoder needs separate loss and cross-attention — more code, less throughput.
Walkthrough: Generating “cat sat”
Prompt: “The”
Tokens: [The] → 32 layers → logits → P(next) = {cat: 0.31, dog: 0.18, sky: 0.05, …} → sample “cat”
Prompt: “The cat”
Tokens: [The, cat] → with causal mask, “cat” can attend to “The”, “The” cannot attend to “cat” → logits → P(next | The cat) = {sat: 0.42, is: 0.21, …} → sample “sat”
Prompt: “The cat sat” → … continues until EOS token sampled.
Each step recomputes attention over full growing sequence — that’s why KV-cache exists (Part 4).
FlashAttention & Efficiency Tricks
Naive attention materializes n × n matrix (for n=128K, that’s 16B floats → 64GB per head, impossible). FlashAttention (Dao et al.) tiles computation in SRAM without forming full matrix, cutting memory 10× and speed 2–4×. All 2026 LLM training and inference use FlashAttention-2/3.
- Grouped-Query Attention (GQA): Llama 2 70B uses GQA-8: 32 query heads share 8 key/value heads → smaller KV-cache, faster decode, minimal quality loss. Standard in 2026.
- Sliding Window (Mistral): Each token attends to nearest 4K tokens + sparse global — handles long context with less compute.
- MoE: As in Part 1, only 2 experts per token active.
See attention in action — on your PDFs
Upload a thesis, ask “Where does attention fail on page 47?” — observe how retrieval + transformer answers grounded in context.