Fine-tuning vs RAG: when to fine-tune an LLM, and when not to

As of 12 November 2026. Each source below was read on that date. Models, documentation and services change; check the source before you rely on a detail.

The short version.

	A better prompt	Retrieval (RAG)	Ordinary code	Fine-tuning
What it changes	The instruction the model is given.	The information the model has when it answers.	Whether a model is involved at all.	The model itself.
Use it when	The model can do the task but misreads what you want.	The answer depends on facts that are private, large or changing.	You can write the rule down, or the output only needs to fit a schema.	The task is narrow, stable and repeated, and you have examples of correct output.
It will not fix	Cost and latency of a large model on every call.	A model that reasons badly about text it has been given.	Language that does not follow rules.	Missing or out-of-date knowledge.
What it needs	A test, and time to iterate.	An index, and a way to keep it current.	A rule that holds.	Labelled examples, a held-out test, and a training run.
Cost to change your mind	Lowest: edit the text.	Low: update the documents.	Low: change the code.	Highest: new data and a new run.

Work from left to right. Each option to the right costs more to set up and more to change, so it should be chosen because the ones before it were tried or ruled out, with measurements.

Before any of them: decide how you will know.

None of the four can be chosen sensibly without a test. The documentation of the companies that sell the models says the same. Claude's prompt engineering guide assumes you already have “A clear definition of the success criteria for your use case” and “Some ways to empirically test against those criteria”, and tells you to establish them first if you do not.^[1] OpenAI's guide suggests you “might even consider writing evals before you start writing prompts”, and its advice on testing is: “Run evals using test data that is representative of real world inputs.”^[2]

In practice that means a set of real inputs with the correct output for each, kept apart from anything you tune against, and a written statement of what score is good enough. Every option below is then a change you can measure.

A better prompt.

The cheapest change, and the first to try. If the model produces the right answer when the instruction is clearer, when it is shown a few examples, or when the output format is spelled out, then the problem was the instruction. OpenAI puts it plainly: “The prompt engineering process may be all you need in order to get great results for your use case.”^[2]

It is the right answer when the task is new or still changing, when volume is low, or when the failures are misunderstandings: the wrong format, the wrong tone, a category applied too broadly.

It runs out when the prompt works and the bill or the response time is the complaint. A longer prompt on a large model makes both worse. Claude's guide notes that not every failing test is best solved by prompting: “you can sometimes improve latency and cost more easily by choosing a different model.”^[1]

Retrieval (RAG).

Retrieval-augmented generation looks things up at the moment of the question and puts what it finds in front of the model. The paper that named it paired a pre-trained generator with a searchable index of documents, so that the model's answers could draw on text outside its own parameters.^[3] The model is unchanged. What changes is what it can see.

It is the right answer when the task depends on knowledge: your documentation, your policies, this customer's account, last week's prices. Knowledge like that changes, and an index can be updated the same day. It also lets an answer point to where it came from.

Fine-tuning is a poor substitute for it. One study that compared the two directly as ways of giving a model knowledge reported that retrieval “consistently outperforms” unsupervised fine-tuning, “both for existing knowledge encountered during training and entirely new knowledge”, and that models “struggle to learn new factual information through unsupervised fine-tuning”.^[4] Those are findings about knowledge, on the tasks and models that study used. They are a good reason not to fine-tune facts into a model.

It will not fix a model that has the right passage in front of it and still classifies, extracts or formats badly. That is a behaviour problem, and retrieval does not change behaviour.

Ordinary code.

Some model calls are doing work a function could do. If the rule can be written down (“an order number is two letters and eight digits”, “anything from this sender goes to that queue”), write it. Code is exact, costs almost nothing to run, and can be tested the way the rest of your software is. Keep the model for the cases the rule cannot place.

The same goes for output format. If the complaint is that the model's JSON sometimes fails to parse or omits a field, that is often solved at the interface, without touching the model. OpenAI's Structured Outputs, for example, “constrains completed, non-refusal model output to your supplied JSON Schema”.^[5] A schema guarantees the shape. It does not guarantee the values are right, which is a separate thing to measure.

It is the right answer whenever it is possible. It runs out where language stops following rules, which is why the model was there in the first place.

Fine-tuning.

Fine-tuning changes the model. In supervised fine-tuning you give it labelled examples and, in Amazon's description, “the model learns to associate what types of outputs should be generated for certain types of inputs”, with its parameters adjusted in the process.^[6] The result is a model that does the kind of task in the examples without being told how each time.

Its main use is the one this site is about. OpenAI's guide describes it: “You can train a smaller, cheaper, faster model to excel at a particular task where a larger model is not cost-effective.”^[2] The prompt gets shorter because the instruction is in the weights, and the model gets smaller because it only has to do one thing.

What makes a task a candidate.

Narrow. One job with a defined output: a label from a fixed set, a record with fixed fields, one of a known list of routes.
Stable. The categories, the schema and the rules are not changing month to month. A fine-tuned model knows what it was trained on.
Repeated. It runs often enough that the cost or the delay per call adds up to something worth removing.
Evidenced. You have examples of correct output that a person or a system of record stands behind, enough to train on and to hold some back for the test.
Already working. A prompt on a general model does the task acceptably. The complaint is price, speed or consistency, not that nobody knows how to do it.

What it costs you.

Data work. The examples have to be collected, checked for duplicates and sensitive content, and split. This is most of the effort.
A training run, and another each time the task changes.
Something to operate. A model you fine-tuned is a model you serve, or pay someone to serve.

Why LoRA and open weights come up.

Training every parameter of a model is expensive. LoRA “freezes the pre-trained model weights and injects trainable rank decomposition matrices” into the model's layers, which greatly reduces the number of parameters that have to be trained.^[7] The Hugging Face documentation adds that the trained adapter can be merged into the base weights, so that LoRA “does not add any inference latency”.^[8]

Open-weight models are ones whose weights you can download and run. Qwen2.5-7B-Instruct and Qwen3-8B, for example, are published on Hugging Face under an Apache-2.0 licence.^[9]^[10] A fine-tune of a model like that is a file you hold. A fine-tune inside a vendor's service depends on that service. When this page was written, OpenAI's guide carried the notice “OpenAI is winding down the fine-tuning platform”, with fine-tuned models to “remain available for inference until their base models are deprecated”.^[2]

Where the labels come from matters.

A common shortcut is to use the large model's own outputs as the training labels for the small one. This is distillation, and cloud services offer it as a product: Amazon Bedrock, for instance, generates responses from a teacher model and uses them to fine-tune a student.^[11] It works, and it has a catch. The student learns whatever the teacher did, including what the teacher got wrong, and a test set labelled by the same teacher will not show it. Keep a record of where every label came from, and judge the result on examples a person or a system of record has confirmed.

They combine.

These are not rival camps. A fine-tuned model can sit behind a retrieval step; a rule can handle the easy cases and pass the rest to a model. One case study on an agricultural dataset reported “an accuracy increase of over 6 p.p. when fine-tuning the model” and found the gain “cumulative with RAG, which increases accuracy by 5 p.p. further”.^[12] That is one dataset and one set of models, and the figures do not transfer to yours. The point that does transfer is that the two address different things, so using one does not rule out the other.

Six questions, in order.

Can you measure it? If you have no test set and no written bar, build those first. Nothing below can be judged without them.
Can the rule be written down? Then write code, and use a model only for what the code cannot place.
Does the answer depend on facts that change, or that the model was never given? Then retrieve them. Do not train them in.
Does the model misunderstand what you want? Fix the prompt, add examples, constrain the output format. Measure again.
Does it now work, but cost too much, respond too slowly, or vary too much? And is the task narrow, stable and repeated, with examples you trust? Then a smaller fine-tuned model is a candidate.
Does the candidate meet your bar on examples it has never seen? If yes, switch. If no, you have your answer for now, and it cost an experiment and not a migration.

Where Daidon fits.

Daidon is being built for questions five and six: to take a team's examples, measure the model they run today, fine-tune a smaller open-weight model, and report whether it meets the bar on held-out data. It is also being built to stop at questions two, three and four when the examples point there, and to say that retrieval, a prompt or code is the better tool. The five steps are set out on their own page.

Daidon is in development. There is nothing to install or run yet. Join the waiting list and we will email you once, when early access opens.

Questions

Is fine-tuning better than RAG?

They solve different problems. Retrieval gives a model facts it was not trained on, at the moment it answers. Fine-tuning changes how a model behaves on a kind of task. If the difficulty is missing or changing knowledge, use retrieval. If the difficulty is doing one stable task consistently, cheaply and quickly, fine-tuning is the candidate. They can be used together.

When should you fine-tune an LLM?

When the task is narrow and stable, runs at volume, has examples of correct output, and a prompt on a general model already works but costs too much, is too slow, or is not consistent enough. And when you can measure the result: fine-tune only against a held-out test and acceptance criteria you set beforehand.

When should you not fine-tune?

When the answer depends on facts that change, when the task itself is still changing, when you have no trustworthy examples, when a clearer prompt has not been tried, or when the rule can be written as code. In each of those cases another tool is cheaper and easier to keep right.

Can fine-tuning teach a model new facts?

Poorly, on the evidence of one study that compared the two directly: it found that models struggled to learn new factual information through unsupervised fine-tuning, and that retrieval did better for knowledge both seen and unseen in training. Use retrieval for facts.

How many examples do you need to fine-tune?

There is no figure that holds across tasks, and this page does not give one. It depends on how many categories or fields there are, how varied the inputs are, and how rare the hard cases are. The practical answer is to hold out a test set, train on what you have, and see whether the result meets your criteria.

What is LoRA?

Low-rank adaptation. The original model's weights are frozen and a small set of added matrices is trained instead, which greatly reduces the number of trainable parameters. The adapter can be merged into the base weights so that it adds no inference latency.

Does Daidon do this today?