Fine-Tuning vs RAG: When Each Makes Sense

Retrieval-augmented generation (RAG) changes what the model sees at question time: you fetch relevant passages from your documents and put them into the prompt. Fine-tuning changes the model itself: you continue training on your own examples so that its weights shift. The short answer to “fine-tuning vs RAG” follows from that difference. If the problem is that the model does not know your facts, or the facts change, or you need citations and access control, use RAG. If the model knows enough but answers in the wrong form, style, language register or level of detail, or you need a small model to do one task cheaply and fast, fine-tune. Most production systems that need both start with RAG and add a targeted fine-tune only where evaluation shows a gap that prompting and retrieval cannot close.

What each approach actually changes

A language model answers from two places: its parameters (everything it absorbed in pre-training and any later training) and its context window (the prompt, system instructions, conversation history and any documents you pass in). The two approaches act on different places.

With RAG, the model’s weights stay exactly as the vendor or open-source project shipped them. At query time a retriever searches an index of your content (vector search, keyword search such as BM25, or both), selects a handful of passages, and the application inserts them into the prompt with an instruction such as “answer only from the sources below and cite them”. The term comes from Lewis et al. (2020), the Facebook AI Research paper presented at NeurIPS that combined a neural retriever with a sequence-to-sequence generator.

With fine-tuning, you run additional training steps on a pre-trained model using examples of inputs and desired outputs (supervised fine-tuning), or preference pairs, or raw domain text (continued pre-training). The result is a new set of weights, or a small adapter that sits on top of the original weights. Nothing is retrieved at run time; whatever the model learned is baked in.

Three practical consequences follow:

• Updating knowledge in a RAG system means updating the index, which can happen minutes after a document changes. Updating knowledge in a fine-tuned model means training again.
• A RAG answer can point to the passage it used. A fine-tuned model cannot tell you which training example produced a sentence.
• A fine-tuned model needs no retrieval call, no long prompt and no index infrastructure at run time, so it can be faster and cheaper per request, especially if fine-tuning lets you switch to a smaller model.

Knowledge versus behaviour

The most useful distinction in this decision is between knowledge and behaviour.

Knowledge is facts: the warranty period for product line X, the clause numbering in your framework agreement, the current version of the travel policy, what the internal function ledger.reconcile() does. Behaviour is everything about how the model responds: output format (valid JSON against a schema), tone and register, length, which questions to refuse, how to structure a summary, domain vocabulary used correctly, the habit of asking for a missing order number before answering.

RAG is the right tool for knowledge. Fine-tuning is the right tool for behaviour. Most confusion in projects comes from trying to use fine-tuning to teach facts, which is exactly what the research below warns against.

There is a grey zone. Domain jargon is partly knowledge (what a term means) and partly behaviour (using it naturally and consistently). A model may answer correctly from retrieved passages yet phrase things in a way an underwriter finds odd; RAG supplies the facts and a modest fine-tune can fix the language.

What the research says

A handful of papers are worth knowing before you choose, because they explain results practitioners see in projects.

Ovadia, Brief, Mishaeli and Elisha (Microsoft), in “Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs”, first posted in 2023 and published at EMNLP 2024, compared unsupervised fine-tuning (continued training on the raw text) with RAG on knowledge-intensive question answering. RAG consistently came out ahead, both for knowledge the model had partly seen during pre-training and for entirely new facts. Fine-tuning gave some improvement, but models struggled to absorb new facts that way. One finding is useful for anyone who still wants to fine-tune on knowledge: exposing the model to many paraphrased variations of the same fact helped. That tells you how much data preparation fact-learning by fine-tuning really needs.

Gekhman and colleagues (Google Research and the Technion), in “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?” (EMNLP 2024), ran a controlled closed-book QA experiment and varied the share of fine-tuning examples that contained facts the model did not already know. Those examples were learned much more slowly than examples consistent with the model’s existing knowledge. As they were eventually learned, the model’s tendency to hallucinate rose roughly linearly with their number. The authors read this as support for the view that models acquire factual knowledge mainly in pre-training, while fine-tuning teaches them to use that knowledge more efficiently. In practical terms: a fine-tuning set full of new facts is a hallucination risk, and the usual remedy in the paper’s setting was early stopping or filtering out examples the model does not know.

Zhou et al. (Meta AI), in “LIMA: Less Is More for Alignment” (2023), fine-tuned a 65-billion-parameter LLaMA model on only 1,000 carefully curated prompt and response pairs and got answers that human raters often preferred to those of models trained on far larger instruction sets. The authors called this the “superficial alignment hypothesis”: knowledge comes almost entirely from pre-training, and alignment teaches the model which format and style to use. The evaluation, based on preference judgements over a few hundred prompts, has been criticised, but the direction agrees with the other papers: fine-tuning shapes behaviour, and data quality matters more than volume.

Zhang, Patil and colleagues at UC Berkeley, in “RAFT: Adapting Language Model to Domain Specific RAG” (2024), show that the two approaches are not exclusive. RAFT fine-tunes a model on questions paired with a mix of relevant documents and irrelevant “distractor” documents, with target answers that reason step by step and quote the relevant passage verbatim. The model learns to use retrieved context well and to ignore what does not help. On PubMed, HotpotQA and the Gorilla API datasets, RAFT improved over both plain fine-tuning and a general model with RAG. The paper’s analogy is studying for an open-book exam: you still bring the book, but you practise finding the right page.

Two papers matter for the cost and feasibility of fine-tuning. Hu et al. (Microsoft), “LoRA: Low-Rank Adaptation of Large Language Models” (2021), freeze the pre-trained weights and train small low-rank matrices added to each transformer layer. For GPT-3 175B they reported roughly 10,000 times fewer trainable parameters and a threefold reduction in GPU memory compared with full fine-tuning, with model quality on par with or better than full fine-tuning on the models they tested, and no extra inference latency once the adapter is merged. Dettmers et al. (University of Washington), “QLoRA: Efficient Finetuning of Quantized LLMs” (NeurIPS 2023), quantised the frozen base model to 4 bits and trained LoRA adapters on top, which made fine-tuning a 65B model possible on a single 48 GB GPU while preserving 16-bit fine-tuning performance on their benchmarks.

LoRA has a known trade-off. Biderman et al. (Databricks Mosaic AI and Columbia University), “LoRA Learns Less and Forgets Less” (TMLR 2024), compared LoRA with full fine-tuning on code and maths. In standard low-rank settings, LoRA learned the target domain less well than full fine-tuning, especially in continued pre-training, but it kept more of the base model’s general abilities. For behaviour-focused fine-tuning that is often the trade you want: change the format and style, do not damage everything else.

Two more results inform the baseline and the combination options below. Liu et al. (Stanford and others), “Lost in the Middle: How Language Models Use Long Contexts” (TACL 2024), found that models used information at the start or end of a long context much better than information in the middle, even models built for long contexts. And Hsieh et al. (Google Research and the University of Washington), “Distilling Step-by-Step!” (Findings of ACL 2023), trained small T5 models on labels plus rationales generated by a large model; on one benchmark a 770-million-parameter model beat few-shot prompted 540-billion-parameter PaLM while using 80% of the available training data.

Put together, the evidence supports a clear rule: put facts into context, put behaviour into weights, and expect the best results on hard domain tasks when the fine-tune teaches the model to work with retrieved context.

Start with prompting and long context

Before choosing either approach, build the cheapest baseline that could work. In many projects it is good enough, and if it is not, it gives you the reference point every later step must beat.

A careful system prompt with explicit rules, the output schema and a few worked examples fixes a surprising share of format and style problems without any training. Structured output features in current model APIs and open-source inference servers (JSON schema or grammar-constrained decoding) remove most broken-JSON issues outright. And if the relevant knowledge is small, skip retrieval and put it in the prompt. A handbook of a few hundred pages fits into the context window of several current models, and prompt caching reduces the cost of resending the same prefix. For a small, stable corpus that is a legitimate design, and simpler than RAG.

It stops working when the corpus outgrows the window or the token volume gets too slow at your traffic, when quality drops as the context grows (the effect Liu et al. measured), or when two employees may see different documents, because a monolithic prompt cannot apply per-user access rights.

Decision table by requirement

The table below maps common requirements to the approach that handles them. “Both” means the requirement is best met by a combination, usually RAG for the content and a fine-tune for the behaviour.

Two rows deserve more explanation.

Latency and cost per request are where fine-tuning wins most clearly, but the gain usually comes from model size, not from dropping retrieval. A fine-tuned 8-billion-parameter model that handles a narrow classification or extraction task can replace calls to a much larger general model and run on a single GPU in your own data centre. The extra retrieval step is acceptable in a chat assistant; in a pipeline that classifies a million documents overnight, it may not be.

The cost of change is often underestimated. In a RAG system you can edit the prompt, change the chunking or swap the embedding model, run the evaluation set and see the effect the same day. A fine-tuned model ties you to a training pipeline: every change in requirements means new or relabelled data, a training run, evaluation and a model release. That overhead is worth paying only for a benefit you can measure.

The GDPR angle: deleting data from an index vs from weights

Under Article 17 of the GDPR, a data subject can ask a controller to erase personal data concerning them, and the controller has to act on valid requests without undue delay. For a system built on emails, tickets or CRM notes, that is an operational requirement.

In a RAG system the path is clear. You delete the source record, the ingestion pipeline removes the affected chunks and embeddings from the index (by stable document ID, so design for it from day one), and caches and logs are purged under your retention rules. You can test it: search for the person’s name after deletion and confirm nothing comes back.

With fine-tuning, the personal data has influenced the weights. There is no reliable operation that removes one person’s data from a trained model. Machine unlearning is an active research field, but as of October 2026 it is not a mature, verifiable method you can point a regulator to. The practical remedy is to remove the record from the training set and retrain, which is slow and expensive if requests are frequent. The European Data Protection Board’s Opinion 28/2024 on AI models (December 2024) adds weight here: it says a model trained on personal data cannot automatically be considered anonymous, and that anonymity requires the likelihood of extracting personal data, directly or through queries, to be insignificant, assessed case by case.

The design consequence is simple. Keep personal data out of fine-tuning sets wherever possible: pseudonymise or remove names, customer numbers and free-text fields with personal details before training, and fine-tune on behaviour (formats, styles, task structure) rather than on records about people. Put personal and frequently changing data behind retrieval, where deletion and access control work. For employee-facing tools in Germany, the works council will usually want to know exactly this: what is stored where, who can see it, and how it is deleted.

Combining the two

Production systems in this space are rarely pure RAG or pure fine-tuning. Three combinations come up repeatedly.

Fine-tuned retriever or embedding model

Retrieval quality limits answer quality: if the right passage is not retrieved, no generator can use it. General-purpose embedding models are trained on web text and often struggle with internal abbreviations, product codes and domain terms. Fine-tuning the embedding model, or training a cross-encoder reranker, on pairs of queries and relevant passages from your own data is often the highest-return fine-tune in a RAG project. It is also cheaper and lower-risk than fine-tuning the generator: the models are small, the training data can be partly generated (an LLM writes plausible questions for each chunk, a person reviews a sample), and the effect is easy to measure with retrieval metrics such as recall@k.

Fine-tuned generator trained on RAG inputs

This is the RAFT pattern. You build training examples that look like production: a question, the retrieved passages (including irrelevant ones), and a target answer in the desired format that cites the right passage and says so when the passages do not contain the answer. The model learns your answer structure, your citation style and, importantly, when to refuse. Do not train only on cases where the gold passage is present, or the model learns to trust whatever it receives.

Distillation of a large model into a small one

You use a large, capable model (with RAG, if the task needs knowledge) to produce high-quality outputs for many inputs, filter and review them, and fine-tune a small open-weight model on the result. The small model then runs on your own hardware. Hsieh et al. showed that including the large model’s rationales in training helps the small model more than labels alone. Two cautions: check the licence terms of the model whose outputs you use for training, since some providers restrict using outputs to train competing models; and evaluate the small model on its own, because it inherits the teacher’s errors as well as its strengths.

A fourth variant appears in agent systems: a small fine-tuned model classifies intents or picks the retrieval source, while a larger model writes the answers. See our article on AI agent development.

Data requirements for fine-tuning

Fine-tuning projects fail more often on data than on training. What you need depends on the goal:

• For format, style and task structure, a few hundred to a few thousand high-quality examples are typical starting points with LoRA on a modern base model. LIMA is the extreme demonstration that a small, carefully curated set can go a long way for behaviour.
• For classification or extraction tasks, you need enough examples per class or field to cover the variation, including the rare and awkward cases. A set that contains only clean, typical inputs produces a model that fails on the messy ones.
• For domain adaptation with continued pre-training on raw text, the volumes are much larger and the gains less predictable, as Biderman et al. found for LoRA in that setting. This is rarely the right first step for a business project.

Quality beats quantity. Each example should be one you would be happy to see the model produce. Inconsistent labels (two reviewers formatting the same kind of answer differently) teach inconsistency. In line with Gekhman et al., avoid training examples whose answers depend on facts the base model does not have, unless the same facts will be supplied in context at run time, as in RAFT.

Sources of data, in order of preference: historical outputs experts already approved (support replies, reviewed contract summaries), then outputs from a strong model corrected by experts, then synthetic examples, which need review because a model generating its own training data reproduces its own blind spots.

Labelling is the expensive part. Write a labelling guide with good and bad examples, measure agreement between two labellers on a sample, and fix the guide where they disagree. A clean, reviewed set of 500 examples is usually worth more than an unreviewed set of 20,000.

Hold out a test set before training and never train on it. Keep it stable across iterations so results stay comparable.

Evaluation design for both approaches

You cannot decide between RAG and fine-tuning, or justify adding a fine-tune to a RAG system, without an evaluation set that reflects your real questions. Build it in the first weeks, before any architecture discussion is settled.

A workable evaluation set has a few hundred real or realistic inputs, sampled to include the common cases, the hard ones and the cases where the right answer is “I don’t know” or “this is not in the documents”. For each, record a reference answer or the facts an answer must contain, and for RAG the documents that should be retrieved.

Measure separately for each layer:

Retrieval (RAG only)
  recall@k        share of questions where a relevant passage is in the top k
  MRR / nDCG      how high the relevant passage ranks
Generation
  correctness     answer matches reference facts (expert-graded, or LLM-graded and spot-checked)
  faithfulness    every claim is supported by the retrieved passages
  citation check  cited passage actually supports the sentence
  refusal         says "not found" when the answer is not in the sources
  format          output parses and passes schema validation
Behaviour / style
  pairwise preference by domain experts, blind to which system produced it
Operations
  latency p50 / p95, tokens per request, cost per 1,000 requests
Regression
  general capability checks after fine-tuning (forgetting)

Separating retrieval from generation tells you where the gap is. If recall@5 is low, fine-tuning the generator will not help; fix chunking, add hybrid search, add a reranker or fine-tune the embedding model. If retrieval is good but answers are unfaithful or in the wrong form, that is a generation problem, and a RAFT-style fine-tune is a candidate.

LLM-as-judge grading scales the evaluation but needs calibration: have experts grade a sample, compare with the judge, and adjust the rubric until agreement is acceptable. For fine-tuning, also check what the model has lost. Run a small set of general tasks before and after training; Biderman et al. are a reminder that forgetting is real, and that LoRA limits it but does not remove it.

Operations: what you have to run afterwards

The operational burden differs more than the build effort, and it is easy to ignore at the proof-of-concept stage.

A RAG system needs an ingestion pipeline that keeps the index in sync with the sources: detecting new, changed and deleted documents, re-chunking and re-embedding them, carrying permissions across, and monitoring for failures. If you change the embedding model, you re-embed the whole corpus and re-run the retrieval evaluation. Prompts and retrieval parameters should be versioned like code. Our sibling article on a RAG chatbot for internal documents goes through these components in detail.

A fine-tuned model needs model versioning: which base model, which data snapshot, which hyperparameters, which evaluation results, deployed where. Store the training data snapshot with the model version, or you will not be able to reproduce it later. Base models change: providers deprecate hosted versions on published schedules, and better open-weight releases appear. Adapters are tied to the exact base weights they were trained on, so each move means re-running the fine-tune and the evaluation. Budget for that as a recurring task. With hosted fine-tuning services, check whether you can export the weights and what happens when the base model reaches end of life.

LoRA helps here. Adapters are small files, several can be served on one base model, and swapping an adapter is a deployment of megabytes, not of a full model. Inference servers such as vLLM support serving multiple LoRA adapters on one base model.

A typical project path

The sequence below is a common, low-risk way to run such a project. Durations are indicative for a single, well-scoped use case with a clear owner.

1. Use case and evaluation set (2–4 weeks). Define the task, the users, what a good answer is and what must never happen. Collect a few hundred realistic inputs and reference answers with domain experts. Clarify data protection: which data may be processed, where, and under which legal basis.
2. Prompting baseline (1–2 weeks). A strong general model, a careful prompt, few-shot examples, structured output, and if the corpus is small, the documents in context. Measure it.
3. RAG (4–8 weeks to a production-ready first version). Ingestion with permissions, hybrid retrieval, reranking, citation, refusal behaviour. Measure retrieval and generation separately.
4. Targeted fine-tuning, only where evaluation shows a gap (3–6 weeks per fine-tune). Usually in this order of likelihood: embedding model or reranker for retrieval gaps; a small model for a high-volume narrow step such as classification or extraction; a RAFT-style generator fine-tune for answer format, style or faithfulness that prompting cannot fix.
5. Operations. Monitoring, feedback capture, re-indexing, scheduled re-evaluation, and a plan for base model upgrades.

Many projects stop after step 3, and that is a good outcome. Fine-tuning earns its place when there is a measured gap, a stable task definition and enough clean examples.

Worked examples

The four cases below are general illustrations of how the decision usually falls, not descriptions of specific client projects.

Customer support assistant

A manufacturer wants an assistant that drafts replies to support tickets using product manuals, known-issue notes and warranty rules. The facts change with every product release and known-issue update, agents need to see which manual section a reply is based on, and some documents are internal only. That is RAG territory: index the manuals and notes with metadata for product line and audience, retrieve by product, and require citations in the draft.

Fine-tuning comes in, if at all, for house style: a fixed structure (acknowledge, answer, next step, sign-off) and phrases the team avoids. If prompting with examples does not produce consistent drafts, a LoRA fine-tune on a few hundred approved historical replies, paired with the passages that would be retrieved for them, teaches the structure. Personal data in historical tickets has to be removed before training.

Contract clause extraction

A legal department wants to extract specific clauses (liability caps, termination notice periods, governing law, change-of-control provisions) from thousands of supplier contracts into a structured table. There is little “knowledge” to retrieve: the information is in the contract being processed. The challenge is behaviour: find the right clause despite varied wording, return it in a fixed schema, say “not present” reliably.

The baseline is a large model with a detailed prompt and schema-constrained output, run per contract. If that is accurate enough, stop. If volume, cost or an on-premises requirement argue against a large hosted model, this is a good case for distillation: let the large model process a sample, have lawyers correct a few hundred outputs, and fine-tune a smaller model on the corrected set.

Code assistant for an internal framework

An engineering organisation has an internal framework with its own APIs and conventions that no public model has seen. Developers want an assistant that writes code using it correctly.

The tempting idea is to fine-tune a code model on the framework’s source. The research above suggests caution: function signatures and deprecations are knowledge, they change with every release, and a model that half-learned them produces plausible but wrong calls. RAG over the current API reference, examples and migration guides stays current. Symbol and keyword search usually beats pure vector search here, because developers ask about exact names.

A fine-tune can still help with conventions: the house patterns for error handling, logging and tests. Train on reviewed internal code with retrieved API docs in the context, so the model learns to rely on the documentation it is given rather than on memory.

German-language domain style

A German insurer wants generated letters and summaries in correct German insurance language: the right technical terms, the formal register customers expect, consistent wording for standard situations. Current general models write good German, but domain register and house terminology are where they drift.

Facts about policies and claims belong in retrieval, with access control. The language is a behaviour problem and a good fine-tuning case: a set of a few hundred to a few thousand approved letters, pseudonymised, trains the register and terminology. Because personal data is everywhere in this domain, the data preparation and the data protection assessment are the main work, and the training itself is the smaller part. Running the fine-tuned open-weight model in a German or EU data centre answers the data residency question that a hosted API may not.

FAQ

Is RAG better than fine-tuning?

For adding or updating knowledge, yes. Ovadia et al. found RAG consistently ahead of unsupervised fine-tuning for both known and new facts, and RAG adds citations, access control and easy deletion. For behaviour, such as format, style or a narrow task on a small model, fine-tuning is the stronger tool.

Can fine-tuning teach a model new facts?

It can, but slowly and with side effects. Gekhman et al. showed that examples with new knowledge are learned more slowly and, once learned, increase the model’s tendency to hallucinate. If facts must come from your data, supply them in context.

Do I need a lot of data to fine-tune an LLM?

For behaviour and format, often a few hundred to a few thousand well-chosen, consistent examples are enough with LoRA; LIMA used 1,000. Quality and diversity matter more than volume. Continued pre-training on domain text needs far more data and gives less predictable results.

Can I combine RAG and fine-tuning?

Yes, and for demanding domain tasks the combination is usually the best option. Common patterns are a fine-tuned embedding model or reranker, a generator fine-tuned on RAG-style inputs with distractor passages (as in RAFT), and distillation of a large model’s outputs into a small self-hosted model.

How does GDPR affect the choice?

Data in an index can be deleted and access-controlled per user; data absorbed into model weights cannot be reliably removed short of retraining. Keep personal data out of fine-tuning sets, and put it behind retrieval where erasure requests under Article 17 can be fulfilled and tested.

What about long context windows instead of RAG?

For a small, stable set of documents and one group of users, putting the documents into the prompt is valid and simpler. It stops working when the corpus outgrows the window, when volume or latency becomes a problem, or when different users may see different documents.

Choosing for your use case

The choice is less “fine-tuning vs RAG” than which problem you have: missing knowledge, wrong behaviour, or both. An evaluation set built from real inputs answers that question faster than any architecture debate, and it is the same instrument you will need to justify each later step.

AI Superior is an AI engineering company in Darmstadt, founded in 2019 by AI scientists, that builds LLM applications, RAG systems and AI agents for B2B clients. If you are weighing the options for a specific use case, our LLM development and consulting services cover the evaluation, baseline and RAG stages, and our LLM fine-tuning services cover data preparation, LoRA and QLoRA training, distillation and model operations. For budgeting, see our article on the cost of fine-tuning an LLM.

Sources and further reading

• Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
• Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
• Ovadia, O., Brief, M., Mishaeli, M., Elisha, O. (2024). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. EMNLP 2024.
• Gekhman, Z. et al. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? EMNLP 2024.
• Zhang, T., Patil, S. G. et al. (2024). RAFT: Adapting Language Model to Domain Specific RAG.
• Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023.
• Zhou, C. et al. (2023). LIMA: Less Is More for Alignment.
• Biderman, D. et al. (2024). LoRA Learns Less and Forgets Less. TMLR.
• Liu, N. F. et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. TACL.
• Hsieh, C.-Y. et al. (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Findings of ACL 2023.
• Regulation (EU) 2016/679 (GDPR), Article 17: Right to erasure.
• European Data Protection Board (2024). Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models.