How LLMs Generate Text: Inference, KV-Cache, Decoding & Sampling Explained (2026)

LLM inference looks instant — but under the hood it’s 500 sequential forward passes, each attending to all previous tokens, each cached, sampled and streamed. Understanding this path explains latency, cost, temperature quirks, and why long context is expensive.

Autoregressive Generation: One Token at a Time

Unlike BERT which outputs all at once, LLMs are autoregressive: token t depends on tokens 1…t-1. Pseudocode:

Every step re-runs the whole 80-layer network — just with one token input plus cached history. So 500 output tokens = 500 full forward passes + 1 prefill.

KV-Cache: The 10× Speedup

Without cache, step n would recompute K/V for tokens 1…n-1 via O(n·d) per layer → total O(n²) for sequence length n. With cache, we store:

Component	Size per token per layer	Total for 128K, 32 layers, d=4096
K	2 bytes (BF16) × 4096 ≈ 8KB	2 × 8KB × 32 = 512KB per token → 64GB for 128K!
V	8KB

That’s why long context needs tricks:

• GQA (Grouped-Query Attention): Share K/V heads — Llama 3 70B GQA-8 cuts cache 4× vs MHA (64GB → 16GB for 128K).
• Quantized KV (INT8/INT4): Store K/V in 8-bit → half memory, 1–2% quality loss. Standard for 128K+ serving.
• PagedAttention (vLLM): Store cache in non-contiguous blocks like OS virtual memory → eliminates fragmentation, enables 10× higher throughput via continuous batching.
• Sliding window/eviction: Keep only recent 4K + sink tokens — for cheap long-context chat where old history less critical.

For “Chat with PDF” on a 40-page doc (10K tokens), cache is ~5GB in BF16, manageable. For 1M context (Gemini), cache is 500GB — sharded across 8 GPUs.

Logits → Softmax → Probabilities

Final linear head maps last hidden state h (4096-dim) → logits vector size vocab (e.g., 128K). Each logit is raw score for that token. Then:

This distribution is what sampling strategies operate on. The top logit isn’t always picked — that randomness creates creativity and also hallucinations.

Sampling Strategies: Temperature, Top-p, Top-k

Param	What it does	When high	When low / 0	Typical 2026 default
Temperature	Scales logits before softmax	More random, creative, diverse	0 = greedy (argmax), deterministic	0.7 (chat), 0.2 (code)
Top-p (nucleus)	Keeps smallest set with cumulative prob ≥ p	p=1.0 keeps all (no filtering)	p=0.1 keeps only 1–2 tokens	0.9–0.95
Top-k	Keeps top k tokens only	k=100 diverse	k=1 = greedy	Disabled (top-p preferred)
Repetition penalty	Divides logits of already-generated tokens by penalty	1.0 off, 1.2 penalizes repeats	1.0 none	1.0–1.1

Visual example

Prompt: “The future of AI is” → raw logits: {bright:2.8, uncertain:1.2, here:1.0, ...}

• Greedy (T=0): always “bright” → “The future of AI is bright and will …” (safe, boring)
• T=1.0, top-p 0.9: samples among “bright/uncertain/here/exciting” → diverse each run
• T=0.3: sharpens “bright” to 0.85 → mostly “bright” but occasional variation

ToolsLead chat uses T≈0.7 + top-p 0.9 — balanced factuality and fluency. For AI Question Generator, we bump to 0.9 for variety.

Beam Search vs Sampling vs Greedy

Method	Search	Diversity	Speed	Use in LLMs?
Greedy	1 path, best token each step	None — deterministic	Fastest	Code, extraction
Beam (k=4)	Keep 4 best sequences, prune	Low	Slow (4×)	Rare — translation, but not chat (LLMs prefer sampling)
Sampling (T, top-p)	1 path sampled	High	Fast	Chat — almost universal
Best-of-N	Sample N sequences, pick best by reward model	Medium	Slow	Reasoning (e.g., AlphaCode)

Why not beam? Beam finds higher log-probability overall but produces generic, dull text (“The future of AI is bright and promising.”). Sampling yields more human-like burstiness and is cheaper.

Streaming, Stop Tokens & Repetition Penalty

• EOS token: Model learned to emit special <EOS> (end-of-sequence) to signal “I’m done.” Decoder stops when sampled. Max tokens is fallback.
• Streaming: Server pushes each token via SSE/WebSocket as soon as sampled; browser appends without waiting for full response. Without streaming, 12s wait feels broken.
• Repetition penalty: Without it, models may loop: “very very very very…”. Penalty lowers logits of already-generated tokens by 1.1×, breaking loops. For PDF Q&A, we keep penalty low (1.05) to allow quoted repetition when citing.
• Seed + system fingerprint: With T=0 and seed fixed, same prompt gives same output (deterministic) — but across deployments, hardware non-determinism may still vary.

Speculative & Efficient Decoding

At 40 tokens/sec, 2K response = 50 seconds — too slow for chat. Tricks:

• Speculative decoding (Google, OpenAI): Draft model (e.g., 1B) predicts 4 tokens quickly → large model verifies in one forward pass (parallel). If draft matches ~75% of the time, speed up 2–2.5× with no quality loss.
• Continuous batching (vLLM, TensorRT-LLM): Pack many users’ decode steps into one GPU batch — utilization 80%+ vs 20% without. ToolsLead’s PDF services batch summarization jobs like this.
• Quantized decode (AWQ, GPTQ): Store weights in INT4 → 4× less VRAM, 1.5–2× faster, <2% quality loss for 70B models. Essential for running Llama 70B on single 48GB GPU.

Latency & Cost of Long Context

Context	Prefill time (H100)	KV cache (GQA, BF16)	Cost per 1K tokens (input/output)
4K	80ms	2GB	$0.005 / $0.015 (GPT-4o-mini)
32K	500ms	16GB	$0.01 / $0.03
128K	2–3s	64GB (needs sharding)	$0.10 / $0.30 (higher tier)

This explains why “Chat with 500-page PDF” costs 10× more: prefill is quadratic, cache huge, and decode slows because each step attends over 128K. Practical design: don’t feed whole PDF as context — use RAG (Part 5) to retrieve top chunks, not whole doc. Full-context is for tasks where every paragraph matters (legal deep-read, thesis review).

FAQs