RAG vs Fine-Tuning: When to Use Each (With Real Numbers)

RAG vs Fine-Tuning: When to Use Each (With Real Numbers)

Most rag vs fine tuning advice skips the one experiment that measured both on the same task. Balaguer et al., in arXiv:2401.08406 (cited 162 times), pushed an agriculture QA set through both: fine-tuning bought over 6 accuracy points, and RAG stacked another 5 on top of that. Their Table 18 puts GPT-4 at 75% raw, 81% fine-tuned, 86% fine-tuned with retrieval. So why do we still tell most teams to start with RAG? Because freshness, citations, and the cost math below decide more projects than a 1-point accuracy gap does.

Key Takeaways

• RAG is the default when knowledge changes often or answers must cite sources; fine-tuning wins on consistent format and latency.
• Fine-tuning costs land up front (training); RAG costs land per query (embeddings plus extra input tokens).
• Published same-task evidence: fine-tuning added 6 accuracy points, RAG 5 more on top, and the hybrid beat both alone.
• Run the five checks (data freshness, labeled examples, latency, citations, team skill) before writing any training code.

When Should You Use RAG vs Fine-Tuning? (Quick Verdict)

Choose RAG if your knowledge changes often or your answers must carry citations. Choose fine-tuning if you need consistent output format and low latency, and you hold labeled examples by the hundred. Use both once a deployment matures. RAG edits the context the model reads; fine-tuning edits the model itself. Most teams need the first one, not the second.

One line to keep: RAG changes what the model reads; fine-tuning changes what the model is. Pick based on which one your task actually needs.

NVIDIA's RAG glossary defines the retrieval side cleanly if you want the textbook version. Definitions don't pick your architecture, though. Evidence does, so start there.

What Does the Evidence Show? One Task, Both Approaches, Measured

The only same-task measured comparison ranking in Google's top five for this query is Balaguer et al. 2024, a Microsoft Research study cited 162 times. The team ran one agriculture QA task through a RAG pipeline, a fine-tuned model, and a hybrid of the two, then had GPT-4 score the answers. On their setup fine-tuning alone edged out RAG alone, and stacking the two beat either one by a wider margin.

The agriculture case study (arXiv:2401.08406)

The study, submitted January 2024 by Angels Balaguer and 15 co-authors, asks what it takes to give farmers location-specific insights. Their pipeline extracts information from PDFs, generates question-answer pairs from them, and evaluates Llama2-13B, GPT-3.5, and GPT-4 with and without retrieval.

Balaguer et al. report an accuracy increase of over 6 percentage points from fine-tuning, cumulative with RAG, which added another 5 points on top. The hybrid pipeline beat either approach alone. Their Table 18 shows the ordering for GPT-4: 75% with no help, 80% with RAG, 81% fine-tuned, 86% fine-tuned plus RAG. Note how close 80% and 81% sit; the gap between RAG alone and fine-tuning alone is one point, while the hybrid is five clear of both. In one experiment, the fine-tuned model drew on knowledge from other geographies to answer region-specific questions, lifting answer similarity from 47% to 72%.

The economics evidence

Snorkel AI's published study (November 2022) covers the cost side. On a 100-way legal classification benchmark (LEDGAR, 80,000 contract provisions), a fine-tuned RoBERTa model matched a fine-tuned GPT-3 while being 1,400× smaller, using under 1% of the ground-truth labels, and running at 0.1% of the production inference cost of the fine-tuned GPT-3 model, roughly one-thousandth. Total build: $1,915 with programmatic labeling versus $7,418 for manual annotation plus GPT-3 fine-tuning. One caveat: that is classification, not generative QA, so treat the ratios as directional.

Our read

Our interpretation: their setup is the friendliest case fine-tuning ever gets, and it still only won by a point. Balaguer et al. trained on a fixed PDF corpus and evaluated against that same frozen corpus, so nothing the weights learned had a chance to go stale mid-experiment. Most production knowledge bases do not hold still like that. A support bot answering questions about last week's release re-earns the 6 points every retrain cycle, while the index that feeds RAG updates the same afternoon. That is why we read a 1-point accuracy edge as the weakest input to this decision, and freshness as the strongest. Where the evidence does not generalize: Snorkel's result is a classification benchmark, and neither study tests tone or format control, which remains fine-tuning's strongest case.

Fine-tuning is the right answer less often than teams think, most projects that say "fine-tune" actually mean "retrieve."

How Does RAG Work, and When Does It Win?

RAG (retrieval-augmented generation) answers from documents you control instead of whatever the model memorized during training. First proposed by Lewis et al. in 2020, it became the default for knowledge work because the knowledge lives outside the model: update the index, and every answer changes tomorrow without a retrain.

The pipeline is four steps:

1. Ingest. Parse your documents (PDFs, wikis, tickets) into a corpus.
2. Chunk and embed. Split into chunks of a few hundred tokens and convert each into a vector with an embedding model.
3. Retrieve. At query time, find the top-K most similar chunks, plus keyword matches for exact strings like SKUs and error codes.
4. Augment and generate. Stuff those chunks into the prompt and let the LLM answer with sources attached.

RAG wins on three axes: freshness (re-index instead of retrain), citations (every answer points at the chunk it came from), and data control (customer data never enters a training run). If you want the full build walkthrough, here's how to build a RAG application step by step.

One warning on retrieval quality: the pipeline is only as good as its embedding and retrieval mix. Anthropic's Contextual Retrieval measured a 5.7% top-20 retrieval failure rate on plain setups, dropping to 2.9% with contextual embeddings plus BM25, and to 1.9% once a reranker was added. If exact-match queries keep failing, hybrid search (BM25 vs vector) is the fix.

When Does Fine-Tuning Win? (And What Is PEFT?)

Fine-tuning wins when the problem is how the model answers, not what it knows: consistent output format, brand tone, or a hard latency budget with no retrieval round-trip. It is also the lever for small-model economics. Snorkel's GPT-3-quality-at-0.1%-of-the-cost result above only exists because someone fine-tuned a small model instead of serving a big one.

Full fine-tuning vs PEFT (LoRA / QLoRA)

Full fine-tuning updates every weight in the model. It is expensive, slow, and rare outside large labs. Nearly everyone ships PEFT (parameter-efficient fine-tuning) instead. LoRA (Hu et al. 2021) freezes the base weights and trains a small low-rank adapter, typically 0.1-1% of the parameter count. QLoRA adds 4-bit quantization on top, so a 13B model fits on one consumer GPU. One related term worth knowing: continuous pretraining, where a model keeps pretraining on a raw domain corpus (unsupervised) before supervised fine-tuning on labeled examples.

A minimal LoRA config, per the Hugging Face PEFT docs:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                          # low-rank dimension; 8-64 typical
    lora_alpha=32,                 # scaling factor, commonly 2x r
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: ~13M || all params: 6.7B || trainable%: 0.19

For dataset prep, epoch counts, and evaluation, see our step-by-step fine-tuning guide.

The risks are real: overfitting on small datasets (a few hundred examples can memorize instead of learn), staleness (weights freeze your knowledge at the training cutoff), and no source attribution (a fine-tuned model cannot show its receipts). If any of those three is a dealbreaker, you just talked yourself back to RAG.

RAG vs Fine-Tuning vs Prompt Engineering: Where Do the Others Fit?

The three are a ladder, not rivals. Prompt engineering changes the instructions, RAG changes the context the model reads, and fine-tuning changes the weights. OpenAI's own fine-tuning guide places fine-tuning last in the loop: evals first, prompts second, training only when prompting stops being enough. Two newer options round out the toolkit.

One confusion worth naming: MCP servers and agent frameworks are orchestration, not customization. They decide which tools and sources the model can reach; they do not change how the model answers. You can run a RAG pipeline inside an agent and fine-tune the model underneath it, and many production systems do both. The autocomplete wars ("vs mcp", "vs agents") are category errors.

Is RAG Cheaper Than Fine-Tuning? The Real Cost Model

Short answer: at realistic query volumes, yes. Fine-tuning's bill lands upfront (labeled data plus training), while RAG's bill arrives per query (embeddings plus extra input tokens). OpenAI's fine-tuning docs charge training by the token, but token fees are pocket change next to the human cost of labeled examples. Here is the math on public list prices.

The break-even question: how many queries until RAG's cumulative per-query tax equals the fine-tuning investment?

break_even = training_cost / per_query_retrieval_delta
           = $1,915 / $0.0003
           ≈ 6.4 million queries

At 50,000 queries a month that is more than ten years. For most products, the fine-tuning investment never pays back through token savings alone; you fine-tune for format and latency, not to beat RAG on cost. The math flips past millions of queries per month or with very large retrieved contexts. And note the asymmetry: the fine-tuning bill renews every time data drift forces a retrain, while RAG scales linearly with volume times chunk size. If per-query spend is the real worry, start with cutting per-query LLM costs first; if you do go the training route, compare fine-tuning tooling before writing the check.

One freshness note: as of July 2026, OpenAI's fine-tuning docs state the hosted platform is being wound down for new users, with existing users keeping training access for the coming months. It is one more reason teams lean toward open-model PEFT or plain RAG.

5 Checks Before You Choose

Run these five yes-or-no checks before writing any training code; the pattern of answers points at RAG, fine-tuning, or hybrid more reliably than any benchmark does. Answer honestly, then count.

1. Does the knowledge change faster than you could retrain? Yes → RAG. A retrain per doc update is not an ops plan.
2. Do you have a few hundred labeled examples? No → RAG or prompt engineering. Fine-tuning on 40 examples memorizes; it does not learn.
3. Is there a hard latency budget? Tight → fine-tuning leans. Skipping the retrieval round-trip saves 50-200ms.
4. Must answers carry citations or an audit trail? Yes → RAG. Fine-tuned models cannot point at the source chunk.
5. Does the team have ML skill plus GPU or API budget for training? No → RAG. An index you can rebuild beats weights you cannot retrain.

Mostly yes on 1, 4, 5 → RAG. Mostly yes on 2 and 3 with a stable domain → fine-tuning. Split answers, or a mature product with real traffic → hybrid (next section). The point of the checklist is deciding by evidence, not by whichever technique is trending on your feed this month.

Can You Use RAG and Fine-Tuning Together?

Yes, and for mature deployments the hybrid pattern is the norm, not the exception. Fine-tune for domain fluency and output format (the how), retrieve for facts at inference time (the what). Balaguer et al. report exactly this on their agriculture task: the hybrid pipeline beat either approach alone, with RAG's 5-point gain stacking on top of fine-tuning's 6.

The maturity path we recommend: start with prompt engineering, add RAG the moment answers need private or fresh data, and add fine-tuning only once format inconsistencies or latency start hurting in production. Skip straight to fine-tuning and you pay the training tax before you know whether retrieval already solved the problem.

The hybrid pattern is not a compromise; for mature deployments it is the default, fine-tune for format, retrieve for facts.

How Do You Evaluate Your Winner?

Pick the winner the way you would pick a database: measure on your workload, not on vibes. The recipe fits in one paragraph and covers the four numbers that actually decide.

• Held-out question set. 100-300 real user questions. Not synthetic ones, never anything seen during training or indexing.
• Faithfulness and answer correctness. Faithfulness asks whether the answer is grounded in the retrieved context; answer correctness asks whether it is actually right. The pair, popularized by RAGAS, catches both hallucination and retrieval misses.
• Latency at p95, not the average. Retrieval adds a round-trip; measure the tail.
• Cost per 1,000 queries, tokens plus infra, measured rather than guessed.
• Re-run on drift. New docs, new model snapshot, new quarter: re-run the set.

The full metrics breakdown, including tooling, lives in our LLM evaluation guide.

Frequently Asked Questions

Can you use RAG and fine-tuning together?

Yes. Fine-tune for output format and domain fluency, and keep retrieval for facts at inference time. Balaguer et al. measured this hybrid on an agriculture QA task and found it beat either approach alone, with the accuracy gains stacking. Most mature production systems end up here: weights for the how, retrieval for the what.

When not to use fine-tuning?

Skip fine-tuning when your knowledge changes faster than you can retrain, when you have fewer than a few hundred labeled examples, when answers must carry citations or audit trails, or when there is no budget to retrain as data drifts. Those four conditions describe most early-stage products, which is why RAG is usually the right first move.

Is fine-tuning debunked?

No, but its territory shrunk. Long context windows and cheap RAG absorbed use cases that demanded fine-tuning in 2023. What remains is real: strict output format, brand tone, latency budgets without a retrieval round-trip, and small-model economics. If your problem is how the model answers rather than what it knows, fine-tuning is still the tool.

When would you use RAG vs fine-tuning?

Use RAG when answers depend on private or frequently updated knowledge, or when you need citations. Use fine-tuning when you need consistent format, tone, or latency and you hold enough labeled examples. Use both once the product matures. If unsure, start with RAG: it is cheaper to undo than a training run.

Is RAG cheaper than fine-tuning?

Upfront, yes. RAG's cost is per query (embeddings plus extra input tokens), while fine-tuning charges once for training and labeled data, then renews on every retrain. On public list prices, break-even lands around 6.4 million queries in our worked example, so at typical volumes RAG stays cheaper for the life of the product.

Is RAG better than fine-tuning for hallucinations?

Usually, but not for free. RAG grounds answers in retrieved chunks, so you can cite sources and audit failures. Bad retrieval poisons the answer, though: Anthropic measured a 5.7% top-20 retrieval failure rate on plain setups, cut to 1.9% with contextual retrieval plus reranking. Fine-tuning, meanwhile, can bake errors into the weights with no way to trace them.

RAG vs fine-tuning vs prompt engineering: what's the difference?

Prompt engineering changes the instructions you send. RAG changes the context the model reads at query time. Fine-tuning changes the model's weights. Each is a bigger intervention than the last: try prompts first, add retrieval when knowledge is the bottleneck, and train only when format, tone, or latency still hurts.

How do you evaluate RAG vs fine-tuning performance?

Build a held-out set of 100-300 real user questions and score both approaches on it: faithfulness (is it grounded?), answer correctness (is it right?), p95 latency, and cost per 1,000 queries. Re-run the set whenever your docs or model snapshot changes. Synthetic questions flatter both systems; real ones separate them.

Fine-tuning vs RAG for multi-hop questions on novel knowledge?

RAG, with better retrieval. The mechanism decides this one: a fine-tuned model can only reason over what its weights absorbed, so knowledge it never saw is unreachable no matter how well it was trained. Retrieval hands it the missing pieces at query time. The catch is that one retrieval pass rarely collects every hop, so plan for query decomposition or iterative retrieval plus a reranker, not a single top-K lookup.

Conclusion

The recap, minus the hedging:

• RAG is the default for changing knowledge and cited answers. Fine-tuning is the specialist for format, tone, and latency.
• The same-task evidence (Balaguer et al.) gives fine-tuning +6 p.p. and RAG a further +5 p.p. on top, with the hybrid best of the three. Our start-with-RAG advice rests on freshness, citations, and cost, not on that scoreboard.
• The cost math lands in RAG's favor at realistic volumes: break-even sat around 6.4 million queries in our worked example.
• Decide with the five checks, not with habit.

Picked RAG? See our ranked RAG tooling list for the stack around the pipeline.