Beyond the Model: RAG, Fine-Tuning, LoRA, Quantization & LLM Evaluation (2026)

After pre-training and alignment, 90% of real LLM products live in the “last mile”: retrieval, adapters, quantization and evaluation. Choose wrong and you waste $50K training what RAG could do for $5. Choose right and you ship a grounded, fast, cheap assistant in a week — like ToolsLead’s Chat with PDF.

RAG vs Fine-Tuning vs Prompting: Decision Matrix

Need	Best tool	Cost	Cites source?	Updates
Answer from your PDFs/docs	RAG	$0 train, ¢ per query	Yes — page-level	Instant (add docs)
Brand voice / JSON schema adherence	Fine-tune / LoRA	$200–2K train + eval	No	Re-train
One-off or few examples	Few-shot prompting	0	No	Edit prompt
Stable logic + fresh knowledge	RAG + LoRA	$500 train + RAG	Yes	Both
Tool use / API calls	Function calling + RAG	Prompt + RAG	Yes	Add tools

Rule of thumb for startups: Start with RAG + strong prompting (1–2 days). Move to LoRA only when you have ≥1K high-quality examples and prompt brittleness appears. Never full fine-tune a 70B on <500 examples — you’ll memorize, not generalize.

RAG Deep Dive: Chunk, Embed, Retrieve, Generate

RAG (Lewis et al., 2020, revived 2023) is the architecture behind Chat with PDF, Perplexity, NotebookLM and every “chat with your docs” product.

Chunking choices matter most

Strategy	When	Pros/Cons
Fixed 1K overlap 100	General PDFs	Default, robust
Semantic (split at headings)	Structured thesis/manual	Better coherence, needs parser
Late chunking (MaxP)	Tables/code PDFs	Keeps tables together
Small 256 + re-rank	High-precision legal QA	More granular but more noise

Hybrid retrieval (dense cosine + BM25 keyword) + re-ranker (Cross-encoder like Cohere rerank) improves Recall@8 by 10–15% — worth it for legal/medical PDFs where keyword match matters (“Section 42A”).

Vector Databases & Embeddings

• Embedding models (2026): OpenAI text-embedding-3-large, Cohere embed-v3, BGE-large, E5-mistral. Dim 768–3072. Higher dim = better recall, more storage (1M × 1536 × 4 bytes = 6GB).
• Vector DBs:
  • pgvector (Postgres): easiest — ToolsLead uses it for <10M chunks, HNSW index (M=16, ef_construction=128, ef_search=64). 20ms query on 1M vectors.
  • Qdrant/Milvus: dedicated, sharded, for 100M+ vectors.
  • Pinecone/Weaviate: managed, pay per vector.
• HNSW index: Graph where each vector links to nearest neighbors; search greedy traverses graph in O(log N), not brute force. Tradeoff: recall vs speed via ef_search.
• Metadata filtering: “Search only in Q1 invoices” = cosine + SQL WHERE doc_id in (...) — crucial for multi-tenant SaaS.

Fine-Tuning & LoRA/QLoRA

When RAG isn’t enough — e.g., you want model to always output valid JSON for invoice extraction, or speak in your brand’s voice without 2K-token prompt.

Full fine-tuning vs LoRA

Method	Params trained	VRAM for 7B	For 70B	Quality
Full FT	100% (7B / 70B)	60GB (with optimizer)	600GB — multi-node	Best if data >10K
LoRA (r=16)	0.5–2% (~35M for 7B)	24GB	48GB on 1 GPU	≈ Full for many tasks
QLoRA (4-bit base)	0.5–2%	8GB	32–48GB	≈ LoRA, slight drop

LoRA math (one paragraph)

Freeze W (e.g., 4096×4096). Learn A (4096×r) and B (r×4096) with r=16 → ΔW = A·B (rank r). Forward is hW + hAB·α/r. r small = fewer params, still captures task-specific direction. Apply to attention Q/V and MLP; r=16–32 is sweet spot.

Practical LoRA recipe

1. Dataset: 1K–10K examples, JSONL: {"prompt": "Extract invoice total…", "response": "{"total": 1200}"}. Quality > quantity — fix bad examples manually.
2. Hyperparams: lr 2e-4, rank 16, alpha 32, dropout 0.05, epochs 2–3, early stop on val loss.
3. Tools: HuggingFace PEFT + TRL (SFT Trainer), Axolotl, Unsloth (2× faster). Train on 1× A100 40GB for 7B, 48GB for 70B QLoRA.
4. Merge or adapter: Keep adapter separate for sharing; merge (W+AB) for fastest inference without extra latency.

Quantization: Run 70B on One GPU

Frontier 70B in FP16 needs 140GB VRAM — 4× A100. Quantization squeezes to consumer hardware:

Format	Bits / weight	70B VRAM	Method	MMLU loss
FP16/BF16	16	140GB	Baseline	0
INT8 (LLM.int8)	8	70GB	Per-channel scale	~0.3%
INT4 GPTQ	4	40GB	Group-wise 128, calibration set	1–2%
INT4 AWQ	4	40GB	Activation-aware scaling preserves salient weights	0.5–1.5%
GGUF Q4_K_M (llama.cpp)	~4.5	42GB	K-quants + super-blocks	~1%
FP8 (H100 native)	8	70GB	Native H100, fast	0.2%

For ToolsLead self-host: AWQ INT4 70B on single A100 80GB with vLLM gives 35 tokens/sec with ~1% MMLU drop — acceptable for summarization. For 7B, Q4_K_M runs on MacBook M3 16GB at 12 tokens/sec via llama.cpp — edge demo friendly.

Evaluation: MMLU, HumanEval, Perplexity & Beyond

Leaderboards ≠ product quality, but you need both.

Benchmark	Tests	What it measures	2026 SOTA
MMLU (57 subjects)	15K MCQ	Broad knowledge	90–91% (GPT-4o, Claude 3.5)
GSM8K / MATH	8K grade-school / 12K competition math	Reasoning (needs chain-of-thought)	92% GSM8K, 55% MATH
HumanEval / MBPP	164/500 coding tasks	Code generation	90% HumanEval
Needle-in-Haystack	Retrieve hidden fact at 128K	Long-context recall	99% (Gemini 1.5 Pro)
IFEval / AlpacaEval	Instructions adherence	Follows format	85–90% win-rate
LiveBench / Chatbot Arena	Live human votes	Real preference	Ongoing Elo

Your golden set > leaderboard

Create 500 Q/A pairs from your PDFs and tasks. Metric: hallucination rate (answer not in retrieved context but stated as fact), citation accuracy, user thumbs-up. Track weekly. This catches regressions that MMLU misses — like model citing wrong page number after KV-cache optimization.

Agents & Tool Use (Function Calling)

2026 LLMs don’t just answer — they act:

• Function calling: Model outputs JSON like {"tool": "merge_pdf", "args": {"files": [...]}}; your code executes tool and feeds result back. Enables “Merge Q1 invoices then compress to 1MB”.
• ReAct loop: Thought → Act (tool) → Observe → Thought → … until done. Max 5–10 tool calls, with guardrails to avoid loops.
• ToolLlama / GPT-4o tool use: Trained on synthetic tool traces; SFT on user → assistant (tool_call) → tool → assistant (answer) sequences.

ToolsLead’s agentic roadmap: “Summarize PDF and translate to Spanish” → 1) summarize chunks → 2) translate summary → 3) produce bilingual PDF via Translate PDF.

Picking Your Stack (Flowchart)

Host options: Managed API (OpenAI, Anthropic, Gemini) for speed to market; Self-host (vLLM + AWQ) for data privacy, cost at scale, and no vendor lock. ToolsLead uses hybrid: managed for chat, self-host for batch OCR/summarization where cost matters.

FAQs