RAG vs Fine-Tuning: When to Use Each and When Neither Helps

RAG and fine-tuning are not competing religions. They modify different parts of the stack. Using RAG when you need behavioral change wastes money on retrieval infra you will bypass. Fine-tuning when you need fresh facts bakes stale knowledge into weights. This note is a decision framework from three production deployments — two successes, one expensive wrong choice.

Related: quantization war stories, ML drift monitoring.

Answer first: the decision tree

Need the model to know NEW facts that change weekly+?
  YES → RAG (or tool/API calls), not fine-tune alone
  NO ↓

Need STYLE / FORMAT / TASK behavior change?
  YES → Fine-tune (or prompt + few-shot if shallow)
  NO ↓

Need LOW latency + HIGH volume on fixed knowledge?
  YES → Fine-tune or distill after RAG prototype validates
  NO ↓

Need grounded citations in answers?
  YES → RAG with source attribution
  NO → Start with prompt engineering; measure before spending

When neither helps: the task needs deterministic logic (calculator, rules engine), human workflow change, or your data is too dirty to retrieve or train on. Fix data first.

What each approach actually changes

Layer	RAG	Fine-tuning
Model weights	Frozen (usually)	Updated
Knowledge source	External index	Training corpus
Update cadence	Re-index documents	Retrain / adapter refresh
Failure mode	Retrieval miss, chunk boundary	Catastrophic forgetting, overfit
Latency add	Retrieval + rerank + longer context	Minimal at inference
Cost driver	Vector DB, embedding calls, context tokens	GPU training, eval cycles

Case 1: RAG was correct (internal support bot)

Problem: 4,000 Confluence pages, updates daily, answers need doc links.

Why not fine-tune: Knowledge churns faster than retrain cadence. Week-old weights = wrong policy answers.

Stack:

• Embeddings: text-embedding-3-small (OpenAI) — good enough, cheap
• Chunking: 512 tokens, 64 overlap, markdown-aware splitter
• Vector store: pgvector on existing Postgres — ops team already on-call
• LLM: GPT-4o-mini with retrieved top-6 chunks, cite page_id

Metrics after 8 weeks:

• Answer accuracy (human eval sample): 78% → 71% if retrieval disabled — RAG carrying the value
• p95 latency: 2.1 s (retrieval 180 ms, generation rest)

Lesson: Invest in chunk quality and metadata filters (space, product line) before bigger models.

Case 2: Fine-tune was correct (structured JSON extraction)

Problem: Extract invoice fields to JSON from messy PDF text. Format rules strict; domain stable; 50k labeled examples accumulated.

Why not RAG: No external corpus — examples are the spec. Retrieval adds noise; schema compliance matters more than novelty.

Stack:

• Base: Llama 3.1 8B
• Method: LoRA, rank 16, 3 epochs
• Eval: JSON schema validator + field F1

Results:

• GPT-4 zero-shot: 91% schema-valid
• Fine-tuned 8B: 96% schema-valid, 4× cheaper per doc at volume

Lesson: Fine-tune when you have labels and the task is stable. Run shadow eval against base model forever — drift happens.

Case 3: Wrong choice (fine-tune when RAG needed)

Problem: Legal team wanted model to "know" current regulations.

What we did: Fine-tuned on regulation PDF snapshot.

Failure: Regulation updated monthly. Model confidently cited obsolete clauses. Worse than generic base model because confidence increased.

Fix: RAG over regulation corpus with effective-date metadata filter. Deprecated fine-tuned weights except for formatting/style adapter (tone: formal, section headers).

Cost of wrong path: ~$40k training + 6 weeks vs ~$8k RAG infra build.

When neither RAG nor fine-tuning helps

Dirty data. If retrieval returns garbage chunks or training labels are inconsistent, upscale the model or add memory — you are polishing noise.

Tool-shaped problems. "What is our stock price right now?" needs an API tool, not embeddings of yesterday's crawl.

Human process gaps. Model suggests refund; policy requires manager approval. No ML fixes approval workflow.

Context length already sufficient. Small fixed prompt with 20 examples — few-shot may beat both.

Hybrid pattern that often wins

1. Prototype with RAG + strong prompt — validate task-market fit
2. Collect failure logs — becomes fine-tune dataset
3. Fine-tune small model for format/behavior — keep RAG for facts
4. Quantize for deployment — see quantization war stories

This is not mandatory — but it avoids baking volatile facts into weights.

Evaluation checklist (before committing)

Question	RAG path	Fine-tune path
Knowledge refresh SLA	< 1 day re-index	Retrain budget?
Labeled examples	N/A	> 500 minimum, > 5k better
Citation required	Yes	Hard without RAG
Regression test suite	Retrieval recall@k	Held-out schema/behavior
Cost at 1M requests/mo	Model + index + embed	Amortized train + infer

Operational debt nobody budgets

RAG: Index staleness monitoring, embedding model upgrades (re-embed all), chunk migration when docs restructure.

Fine-tune: Eval harness on every base model upgrade, adapter versioning, catastrophic forgetting checks on general capabilities product still needs.

Track both in drift monitoring.

What I would decide today for common asks

Ask	Choice
Chat over company wiki	RAG
Fix JSON output format	Fine-tune small or constrained decoding
Code assistant on private monorepo	RAG + long context; fine-tune only if style rules extreme
Moderation classifier	Fine-tune classical or small LM
"Be more concise"	System prompt first; fine-tune if prompt ignored at scale

Related

Quantization war stories after you pick model size. Drift monitoring for either path in production.