Training LLMs: Pre-Training, SFT, RLHF & DPO Explained (2026 Guide)

Training a frontier LLM is the most expensive self-supervised task in history: 15 trillion tokens, 10²⁶ FLOPs, 16,000 H100 GPUs running for weeks, and 100,000 human preference judgments — all to make next-token prediction helpful and safe.

The 4-Stage Training Pipeline

Stage	Objective	Data	Compute	Output
0. Data	Curate high-quality mixture	Common Crawl + Code + Books + Synthetic	CPU + filters	15T clean tokens
1. Pre-train	Minimize -log P(next token)	Unlabeled text (self-supervised)	~90%	Base model (e.g., Llama-3-Base)
2. SFT	Minimize loss on human-written ideal responses	50K–500K instructions	~5%	Instruction model (chatty but not robust)
3. RLHF/DPO	Maximize learned human preference reward	50K–200K preference pairs	~5%	Aligned chat model (GPT-4o, Claude 3.5)

Stage 0: Data — The Real Moat

“Data is the model.” In 2026, teams spend as much engineering on data as on architecture.

Sources & mixture (Llama 3–ish)

• 50% Common Crawl: Raw web filtered by language ID (fastText), toxicity classifier, heuristic filters (remove boilerplate, too-short docs, lorem ipsum, excessive repetition).
• 15% Code: GitHub, StackExchange — teaches logic, structure, tool use. Dedup by repository, filter by permissive licenses.
• 10% Books & Papers: High-quality long-form, narrative coherence.
• 5% Wikipedia & curated: Factual recall.
• 10% Math & STEM: arXiv, synthetic proofs.
• 10% Multilingual: Hindi, Spanish, Chinese etc. — but token fertility high, so sampling carefully balanced.
• Synthetic (2025+): Model-generated reasoning traces (“Let’s think step by step…”) filtered by reward model and verified by Python execution. Adds 5–15% of tokens, boosts GSM8K/MATH by 5–10 points.

Quality filters

• Deduplication: MinHash (document-level) + exact substring match (paragraph). Prevents copying Wikipedia verbatim and stops test-set leakage.
• PII scrubbing: Regex + classifier removes SSNs, API keys, emails (reduces regurgitation risk).
• Toxicity: Perspective-style classifier downweights hate, but not too aggressively — otherwise model becomes prudish. Balance via mixing.
• Decontamination: Remove any paragraph that appears in MMLU/GSM8K/benchmarks to avoid cheating.

Final corpus: 15T tokens after filtering. Original raw crawl was ~100T tokens — 85% discarded. Quality beats quantity.

Tokenization & Batching

After data, train tokenizer (BPE) on sample of corpus → 128K vocab (GPT-4o), 32K (Llama 2), 256K (Gemma). Then:

1. Pack sequences: Concatenate documents with EOS token, cut into fixed 4K/8K blocks for training (no padding waste).
2. Shuffle globally: Pseudo-random but reproducible seed.
3. Batch: 4M tokens per step (e.g., 512 seq × 8192 length) — large batch stabilizes Adam.

Stage 1: Pre-Training (Next-Token Loss)

This consumes 90% of budget.

Objective formalized

Perplexity = exp(loss). Lower = better. During run, 3.5 → 2.0. But practitioners eval weekly on MMLU, GSM8K, HumanEval, not just perplexity.

Infrastructure

• Hardware: 16K H100s (each 989 TFLOPs) on InfiniBand. MFU (Model FLOPs Utilization) 40–55% after optimizing kernels (FlashAttention, comm overlap).
• Parallelism: Data Parallel + Tensor Parallel (split attention heads) + Pipeline Parallel (split layers across nodes) + Sequence Parallel for 128K context.
• Checkpointing: Save every 1K steps → 30TB of weights/history.
• Failure handling: With 16K GPUs, mean time between failures is hours — jobs restart from last checkpoint automatically. Lost compute is budgeted.

When to stop

Chinchilla-optimal says stop at ~20 tokens/param. But teams now overtrain: Llama 3 70B saw 15T tokens (214 tokens/param!) because inference cost dominates — a better 70B saves billions in serving vs training a bit longer.

Stage 2: Supervised Fine-Tuning (SFT) — Instruction Tuning

Base model can complete “User: Explain photosynthesis” with “User: Explain …” repeated — it’s a text completer, not an assistant.

SFT fixes it:

• Dataset: 50K–500K human-written (or high-quality synthetic) pairs: {instruction: “Summarize this PDF in 3 bullets”, response: “- …”}. Often curated from StackExchange, Alpaca, ShareGPT, plus internal annotation teams ($2–5 per example).
• Training: Fine-tune full model (or LoRA for smaller runs) with cross-entropy only on response tokens (mask prompt). LR 1e-5, 1–3 epochs. Early stopping to avoid overfitting SFT style.
• Effect: Model learns format, tone, to follow instruction rather than continue. SFT alone makes it chatty but still brittle — easy to jailbreak, verbose, not well-calibrated.

2026 recipe: SFT on high-quality, diverse, maximal-detail examples beats 1M mediocre examples. LIMA (NeurIPS 2023) showed 1,000 excellent examples can align well.

Stage 3: RLHF — Reward Model + PPO

RLHF aligns nuance: helpfulness, harmlessness, honesty, style consistency.

1. Collect preferences: For each prompt (e.g., “Write a Python quicksort”), sample 2–4 SFT outputs, have human label ranking (1st best, 2nd, …) — with criteria and handling ties. Need 50K–200K pairs. Costly: trained annotators, not crowdworkers, for safety tasks.
2. Train Reward Model (RM): Take SFT model, replace final head with scalar reward. Train with loss = -log σ(r_winner - r_loser). RM learns to predict human preference. Need help from 6B–13B RM for efficiency; freeze most of LLM, train head.
3. PPO (Proximal Policy Optimization):
   • Sample prompt → generate response with current policy.
   • Score with RM → reward.
   • Compute KL penalty: reward_final = r_RM - β·KL(policy || SFT) — prevents drift into reward hacking gibberish.
   • Update policy with clipped objective.

Also add rule-based rewards: Python unit tests for code, regex for format (“output must be JSON”), safety classifier score — reduces reliance on pure RM.

Instability is notorious: KL tuning, value network initialization, reward hacking (model learns to repeat flattery to get high reward). Teams log reward, KL, entropy, win-rate daily.

DPO & Modern Alternatives

DPO (Rafailov 2023) recasts RLHF as classification: directly optimize log σ( β log π_winner/π_ref - β log π_loser/π_ref). No RM, no PPO loop, stable offline training. Result similar to RLHF for most tasks.

Method	Needs RM?	Online?	Stability	When used 2026
PPO-RLHF	Yes	Yes (sample during train)	Low — needs tuning	Final frontier polish (OpenAI, Anthropic)
DPO	No (implicit)	No (static pairs)	High	Most open-source (Llama, Mistral) — default
KTO / IPO / RLOO	No	Varies	Medium	Alternatives handling noisy prefs
Constitutional AI (RLAIF)	AI feedback	Both	Medium	Safety: AI critiques AI outputs

2026 SOTA stacks often: SFT → DPO → iterative DPO (generate new pairs from DPO model, re-rank with RM) → tiny PPO final.

Compute, Hardware & Cost

Model	Params	Tokens	GPUs & Time	Est. Cost
Llama 3 8B	8B	15T	1K H100 × 30 days	$5M
Llama 3 70B	70B	15T	16K H100 × 39 days	$35M
DeepSeek-V3 671B MoE	671B (37B active)	14.8T	2K H800 × 60 days	$6M (MoE efficiency)
GPT-4 class	~1.8T MoE	13T	25K H100 × months	$100M+

Inference cost reminder: Serving 1B queries at 500 tokens each = 0.5T tokens. At $5/1M tokens, that’s $2.5M revenue but ~$1M GPU cost. Training cost is one-time; serving cost compounds, which is why quantization, MoE and speculative decoding (Part 4) matter.

Alignment Challenges & Evals

• Helpful vs Harmless: “How to make a bomb?” → must refuse harmfully but allow “How does chemistry of explosives work for my novel?” — fine line, many false refusals if over-tuned.
• Sycophancy: RM trained on humans who prefer agreeable answers → model flatters. Fix: disagreement-aware labeling.
• Evaluation: Benchmarks (MMLU, GSM8K, HumanEval, MATH, IFEval) + red-teaming + human win-rate vs previous version. Before any RLHF release, 1K+ adversarial prompts probe jailbreaks.
• Continual alignment: User feedback (thumbs up/down in Chat with PDF) creates fresh preference pairs for iterative DPO.

FAQs