LLMs don’t “think” like humans — they predict the next token, trillions of times. That single mechanism, scaled to 70 billion parameters and 15 trillion tokens of training data, is enough to write code, summarize PDFs, pass bar exams, and appear to reason. Here’s how — from tokens to emergence — without hype or hand-waving.
⚡ 30-Second Answer
An LLM = Transformer neural network + next-token prediction on internet-scale text + human preference tuning (RLHF/DPO). You give it tokens, it computes probabilities for what token comes next and samples one. Repeat 500 times = one answer. Try it on your PDF — upload a paper and ask “Summarize in 3 bullets” to see it live.
What Is an LLM in One Sentence?
A Large Language Model is a probability distribution over token sequences learned from vast text and conditioned on your prompt.
More concretely:
- Large = tens to hundreds of billions of learned weights (parameters). Llama 3.1 = 405B, GPT-4o ≈ 1.8T mixture-of-experts, DeepSeek-V3 = 671B MoE but only 37B active per token.
- Language = operates on discrete tokens, not pixels or audio waveforms (though multimodal variants now add those).
- Model = neural network approximating P(next token | previous tokens).
When you ask “Explain quantum computing like I’m 10”, the model does NOT retrieve a stored page. It tokenizes your prompt, runs it through ~80 transformer layers, outputs a probability distribution over ~128K vocabulary tokens, samples “Imagine”, then feeds “Imagine” back in, samples “a”, then “tiny”, and so on — autoregressive generation.
Analogy: The Infinite Library Librarian
Imagine a librarian who has read every book in a city library and never memorized sentences — only which word tends to follow which in every context. Ask a question, and she proposes the most likely next word, then re-considers with that word included. At small scale this is nonsense; at library-scale it becomes eerily coherent.
Tokens: How LLMs See Text
LLMs never see raw strings. A tokenizer converts characters → integer IDs.
Why not words?
- Vocabulary would be millions (all names, misspellings).
- Rare words would have no vector.
- Subword handles morphology: “unhappiness” → “un” + “happi” + “ness”.
| Text | Tokens (GPT-4o tiktoken) | Token IDs (example) | Note |
|---|---|---|---|
| hello world | 2 | 15339, 1917 | Common words = 1 token |
| ToolsLead | 2 | 13797, 32151 | CamelCase splits |
| 🤗 | 3 | 764, 232, 222 | Emoji = multiple bytes |
| Supercalifragilisticexpialidocious | 7–9 | … | Long word shattered |
| 2026 | 1–2 | 19657 | Numbers special |
Most 2026 models use BPE (Byte Pair Encoding) or SentencePiece. Training merges frequent byte pairs iteratively until vocab is 32K (Llama 2) to 256K (Gemini). Larger vocab = fewer tokens per sentence but bigger softmax head.
Pricing intuition
1,000 tokens ≈ 750 English words ≈ 1.5 pages single-spaced. If a provider charges $5 / 1M input tokens, analyzing a 40-page PDF (~10K tokens) costs $0.05. Understanding tokens predicts bills — and why “chat with PDF” chunks documents.
Tokenization matters for accuracy
- “Strawberry” has 3 Rs but token “straw” hides count — LLMs fail letter counting unless you add spaces.
- Code: “ = ” often one token, so model learns syntax patterns cheaply.
- Non-English: Hindi/Devanagari often 2–3x tokens per word vs English → higher cost, shorter effective context.
Next-Token Prediction: The Only Training Objective
Every stage — pre-training, fine-tuning, RLHF — ultimately optimizes next-token likelihood, just on different data.
Training example (cross-entropy loss)
Prompt tokens: [The, _cat, _sat, _on, _the] → Model predicts distribution → True next token: _mat → Loss = -log P(_mat).
Goal: minimize average loss over 15T tokens. No labels needed — text is self-supervised.
Why does this create intelligence?
- To predict next token well, you must implicitly learn grammar, facts, reasoning, coding patterns, and style.
- Internet text is a compressed record of human thought. Modeling it well requires internalizing much of that thought.
- At 1B params, model learns grammar. At 10B, learns facts. At 70B+, learns multi-step reasoning — scale induces qualitatively new strategies.
Loss curve → capability
During pre-training, loss drops smoothly: 3.5 → 2.8 → 2.1 → 1.8. But downstream benchmarks (MMLU, GSM8K) jump discontinuously — long plateaus then sudden gains. That’s why teams watch loss but evaluate weekly on real tasks.
Scale: Parameters, Data & Compute
| Dimension | 2019 (GPT-2) | 2023 (GPT-4) | 2026 (Frontier MoE) |
|---|---|---|---|
| Parameters | 1.5B | ~1.8T (MoE, ~280B active) | 400B–2T (MoE, 30–40B active) |
| Training tokens | 10B | 13T | 15–25T (with synthetic) |
| Compute | 10^21 FLOPs | 10^25 FLOPs | 10^26 FLOPs (10K–30K H100s × months) |
| Context | 1K | 8K–32K | 128K–2M |
| Training cost | $50K | $100M+ | $50–200M |
Scaling laws (Kaplan vs Chinchilla)
- Kaplan (2020, OpenAI): Bigger model beats more data, for fixed compute.
- Chinchilla (2022, DeepMind): For optimal, scale data tokens ≈ 20 × parameters. A 70B model needs ~1.4T tokens, not 400B. This shifted industry to train smaller models on more data — cheaper to serve.
- 2026 consensus: Chinchilla holds, but quality-filtered data + synthetic reasoning traces beat raw Common Crawl. Llama 3.3 70B trained on 15T high-quality tokens beats Chinchilla-optimal.
Mixture of Experts (MoE)
Instead of activating all parameters per token, MoE routes each token to 2 of 8 experts (feed-forward blocks). DeepSeek-V3: 671B total, 37B active → GPT-4-class quality at 1/5 inference cost. Downside: complex load balancing, larger VRAM to store all experts.
Context Windows Explained
Context = prompt + output tokens the model can attend to at once. It’s the model’s short-term memory.
- 4K (2019): ~3 pages — could lose track mid-paragraph.
- 128K (2024–2026 standard): ~300 pages — whole Codebase or long PDF.
- 1M–2M (Gemini 1.5, Claude 3.5 200K, GPT-4o 128K): Entire book + chat history. Used for “Chat with PDF” on 200-page contracts.
Cost is quadratic: attending to 128K tokens needs 16B attention operations per layer. Engineering tricks — FlashAttention, RoPE, KV-cache, sliding windows, MoE — make it feasible, but long context still costs 5–30× more per token than 4K.
Why Emergent Abilities Appear
No line of code says “if user asks math, do step-by-step.” Emergence comes from:
- Compositionality: Predicting next token well forces learning composable subtasks (parse grammar → infer intent → retrieve facts → plan steps).
- Scale: At 7B, model memorizes; at 70B, it generalizes to 5-shot reasoning (GSM8K jumps 15% → 60%).
- Instruction tuning: SFT + RLHF teach “follow the user instruction, not just continue text.” Base models complete “Q: 2+2?” with more questions; chat models answer “4”.
🧪 Try it yourself
In Chat with PDF, ask: “Translate page 5 to Hindi”. The model didn’t have a translation rule — it learned translation as a special case of next-token prediction from bilingual text in training.
LLM vs Search vs Traditional ML
| System | Input → Output | Training signal | Strength | Weakness |
|---|---|---|---|---|
| Search (Google) | Query → ranked docs | Links + clicks | Fresh, cited, factual | No synthesis |
| Classifier (BERT) | Text → label | Labels | Cheap, accurate | One task |
| LLM (GPT/Claude) | Prompt → tokens | Next token on all text | General, few-shot, generative | Hallucinates, costly |
That’s why modern products combine them: RAG = LLM + search. Chat with PDF does exactly this — retrieve relevant chunks, feed them in context, then generate grounded answer with citations.
Limitations You Must Know
- No true memory: Each request is stateless except context window. Model doesn’t “remember” you beyond chat history tokens.
- Hallucination: Fluent falsehoods when unsure. Mitigate with RAG, tools, lower temperature, citations.
- Reasoning is shallow: Multi-step math/physics fails without chain-of-thought prompting or tool use. Even 2026 models score ~60% on competition math (AIME).
- Bias & cutoff: Training data bias reflects society; knowledge cutoff means “no train after 2026-06” unless browsing enabled.
- Cost & latency: 2K output tokens at 30 tokens/sec = 66s. Long context multiplies cost linearly.
How ToolsLead Uses LLMs (Real Example)
ToolsLead’s AI layer isn’t “ChatGPT wrapper” — it’s a pipeline:
- Extraction: pdf-parse + pdfjs extracts text per page, preserves tables via heuristics.
- Chunking: Split 100-page PDF into 1,000-token overlapping chunks (≈ 750 words). Without chunking, 100K-token PDF would overflow context and cost $0.50 per question.
- Embedding: Each chunk → 1,536-dim vector via embeddings model (text-embedding-3-large). Stored in vector DB (pgvector).
- Retrieval: Your question → embed → cosine similarity → top 8 chunks + metadata (page number).
- Generation: Chunks + question + system prompt → LLM (GPT-4o-mini / Claude Haiku for cost) → answer with page citations + summary.
- Fallback: If no retrieval (image-only scan), run OCR first → then steps 2–5.
This RAG (Retrieval-Augmented Generation) pattern is covered in depth in Part 5 of this series. It’s why Chat with PDF answers with “Page 7 says …” instead of hallucinating.
Experience an LLM on your document
Upload a PDF, ask questions, get cited answers — free, no signup, deletes after 1 hour.