Fine-tuning or context: which problem are you solving

Fine-tuning vs RAG: which failure are you fixing

Start from the broken output, not from the method. There are five ways a model output can be wrong, and each one has an owner. It does not know a fact, so the fix is retrieval. It knows the fact and says it in the wrong shape, so the fix is format. It says the right thing in the wrong voice or skips a step, so the fix is behaviour. It is right and slow, so the fix is latency. It is right and expensive, so the fix is cost.

Fine-tuning is good at two of those five: format and behaviour. It is weak at the first one, facts, and that is the one people buy it for. Feeding a model a thousand pages of your handbook as training data does not give it a reliable memory of the handbook. It gives it the sound of the handbook, which is worse than not knowing, because the answers now look right.

Retrieval is good at the first one and does nothing for the other two. Put the right page in the prompt and the model will use it. It will still write in a voice you did not ask for, and it will still skip the step you told it about once, three thousand tokens ago.

So the question is never which method is better. The question is which of the five failures you have, measured on real outputs, before anyone opens a training script.

• Wrong fact, right shape. Cause: the fact was not in the prompt. Fix: retrieval.
• Right fact, wrong shape. Cause: no schema, or a schema said once in prose. Fix: ask for structured output, then tune if it still slips.
• Right shape, wrong voice, or a step left out. Cause: behaviour. Fix: show it examples first, then tune if the examples get long.
• Right and slow. Cause: a long prompt, or too many hops. Fix: cache, cut, or move behaviour into weights.
• Right and dear. Cause: a big model doing a small job. Fix: a small model tuned on the big one's outputs.

What fine-tuning actually changes inside the model

Fine-tuning takes a set of example exchanges and nudges the weights so the model is more likely to produce that kind of output. The unit is a sample, not a document. A sample is an input and the output you wanted, in the format the model will see at run time.

That shape matters more than people expect. If your training file is full of documents rather than exchanges, you are teaching the model a style of text, not a job. Style is what it will learn. A small set of samples that match the real task beats a large pile of handbook pages, and the reason is not scale, it is that only one of the two sets contains the thing you want repeated.

The common cheap method is a low-rank adapter, from the LoRA paper in the sources below: a small set of extra weights trained on top of a frozen base. It cuts the cost and the storage and it lets you keep a few adapters around. The OpenAI fine-tuning guide in the sources shows the hosted version of the same idea, down to the file format. It does not change the shape of the problem. You still need samples, and the samples are still the hard part.

One cost that surprises people: a tuned model can get worse at things you did not train on. The field calls it catastrophic forgetting, and in practice it shows up as a model that now writes your ticket summaries perfectly and can no longer do the arithmetic it used to do. That is why an eval set covering the old behaviour, not just the new one, is part of the price of the method.

The file itself is plain. One line per exchange, in the same chat format the model answers in, which for a hosted service reads like the block below. If you cannot write two hundred lines of that by hand or harvest them from work you already approved, you are not ready to train.

• {"messages": [
• {"role": "system", "content": "You are a support triage agent."},
• {"role": "user", "content": "Card declined on renewal, third time."},
• {"role": "assistant", "content": "{\"queue\": \"billing\", \"severity\": 2}"}
• ]}

What retrieval changes, and where it breaks

Retrieval augmented generation, from the 2020 paper in the sources, means one thing: find the text that answers the question and put it in the prompt before the model answers. Nothing is learned. Nothing is stored in the model. Change the document at noon and the answer changes at noon, which is the whole reason to use it.

It breaks in four places, and all four look like the model being stupid. The chunk boundary cuts a table in half, so the model reads a column with no header. The ranker returns the right topic and the wrong version, so you get last year's policy stated with confidence. The question does not share words with the answer, so the search finds nothing and the model answers from memory anyway. Or the text is there and buried in the middle of a very long prompt, with no label saying what it is, so the model treats it as background.

Here is the quick way to see which half you are in. Open one bad answer. Look at the text the search put in the prompt, and ask if the fact was in it. If the fact was there and the answer is still wrong, the model is the problem. If it was not there, the search is the problem, and the model did what it could with what it had. That one check splits the work in two and it takes a minute.

None of those get better with fine-tuning. Some of them get worse, because a tuned model is more sure of itself. If your team is arguing about tuning and nobody has read the retrieved chunks for ten bad answers, stop the argument and go read the chunks. That single hour reorders most of these debates.

• Chunking: split on structure, not on a fixed character count. Keep the heading with the body.
• Ranking: filter by date and by version before you rank by how close the text looks. Right topic, wrong year is the classic bad answer.
• Recall: the question rarely uses the words the document uses. Run keyword and vector search together, and take the union.
• Placement: put the retrieved text where the model reads it, and say what it is, not just paste it.
• Scope: an agent that can see more than the user can is a bug, not a feature. Filter by who is asking.

Symptom to fix: a table you can run against your own logs

Take ten bad outputs from last week. Real ones, from real users, not the ones you made up to test with. Put each one in a row of this table and see where they land. If eight of them land in the retrieval column, your fine-tuning project is a way to avoid fixing the index.

Do it on paper. One row per bad output. What you saw. What went wrong. What the row says to fix. Ten rows take half an hour and they end most of the fight, because the pile is rarely even. It is eight of one kind and two of the other. The eight is the work, and the two can wait.

This is a diagnosis aid, not a law. Two rows can be true at once, and the right order then is to fix the cheap one first and re-measure, because the second problem is often smaller than it looked while the first one was still there.

Why teams reach for fine-tuning when the real problem is retrieval

The first reason is honest confusion. The symptom for a missing fact and the symptom for a missing behaviour are the same sentence: the model does not know our business. Said out loud, that sentence points at training, because training is what we do to people who do not know things. The machine does not work that way, and the phrase hides the difference.

The second reason is that tuning feels like ownership. A fine-tuned model is a file with your name on it. An index is plumbing, and plumbing does not feel like a moat. That feeling has cost more quarters than any technical mistake on this page, because it survives the evidence: teams keep a tuning project alive after the tests say retrieval was the problem, since cancelling it means admitting the asset was never an asset.

The third reason is that prompt work has a bad name. It sounds like fiddling. It is fiddling, for about a day, and then it is a versioned file with a test set behind it, which is the same discipline as a training run at a tenth of the cost. Teams skip the day of fiddling and buy the quarter of training.

The tell is the sentence people use to justify it. If the sentence is we need the model to know our data, you have a retrieval problem. If the sentence is we need the model to always answer in this exact shape, and the examples in the prompt are now longer than the prompt itself, you have a real case for tuning.

• The words to watch: we need it to know our data. That is a retrieval line, not a training one.
• Also watch: it has to come out in this exact shape, every time. That one can be a real tuning case.
• A tuned model is a file with your name on it. An index is a pipe. Both are work. Only one of them feels like an asset.
• Ask who will run the next training job in a year. If no one can name a person, do not run the first one.

Three cheap tests that tell you which one you need

Run these in an afternoon before you spend a quarter. Each one has a result you can read without any tooling, and each one rules something out.

Test one, the paste test, separates knowledge from everything else. Take a question the system got wrong. Find the document that holds the answer by hand. Paste it into the prompt and ask again. If the answer is now right, you have a retrieval problem, full stop. No amount of tuning will fix it, because the model already proved it can use the fact when the fact is present.

Test two, the repeat test, separates format from behaviour. Send the same input twenty times at your normal settings. If the content is right every time and the shape moves around, that is a format problem, and structured output or a schema fixes it in an hour. If the shape holds and the content drifts, that is behaviour, and it is where examples and then tuning belong.

Test three, the examples test, sizes the tuning case. Put three good examples in the prompt. Measure. Put ten. Measure again. If ten examples get you most of the way and the prompt is still a sane size, you do not need tuning yet, you need a prompt in a file with a test set. If the gain only arrives past thirty examples, and the prompt is now so long it costs real money on every call, that is the signal to move the behaviour into weights.

• Test one, paste the source doc into the prompt. Right answer means a retrieval problem.
• Test two, send the same input twenty times. If the shape moves and the content holds, it is format. If the shape holds and the content moves, it is behaviour.
• Test three, try three examples, then ten, then thirty. If the gain flattens early, keep the prompt and skip the training run.
• Write the result of each test down. The argument restarts in six weeks and the notes end it faster.

What each one costs to keep alive after launch

Both methods have an upkeep bill and neither vendor page leads with it. The shapes are different, and the difference decides more cases than the build cost does.

A fine-tune ages against the model. When the provider ships a better base model, your tuned adapter is stuck on the old one until you run the job again, and a tuned old model can end up behind an untuned new one. So the real cost is not the training run, it is the promise to redo the training run, keep the dataset alive, and re-score the same eval set every time the ground moves.

A retrieval system ages against your documents. It needs a freshness path, a delete path, and permissions that follow the user rather than the index, which is the part most teams add after the first incident. Change the embedding model and you re-embed everything. Add a reranker and you buy accuracy with latency, and the size of that trade is something to measure on your own data rather than take from anyone's table, this one included.

The honest summary: tuning costs you a standing commitment to retrain, retrieval costs you a standing commitment to keep an index true. If you cannot staff one of those commitments, pick the other one, whatever the accuracy numbers say.

• Fine-tune upkeep: keep the data set alive, run the job again each time the base model moves, score the old behaviour as well as the new one, version the adapters, and serve them.
• Retrieval upkeep: pull the docs in, keep them fresh, honour a delete, filter by who is asking, re-embed the lot when the model changes, and score the search step on its own.
• Shared: an eval set of real cases. Without it, neither method can tell you whether a change helped.
• Shared: a per-run trace with the retrieved text in it. Bad answers are unreadable without it.

When fine-tuning is the cheaper answer, not the fancier one

There are cases where tuning is plainly right, and they share a trait: you are paying for the same tokens over and over to teach the model something that never changes.

The first is a strict output format at volume. Classification, routing, extraction into a fixed schema. The job is narrow, the shape never moves, and a tuned small model can do it at a fraction of the price of a large one reading a long instruction each time.

The second is a voice or a policy that takes many examples to pin down. When the prompt carries fifteen examples on every call to hold the style, those examples are rent. Tuning converts the rent into a one-off payment plus upkeep, and the break-even depends on your call volume, which you can work out from your own bill rather than from anyone's table.

The third is latency inside a tight budget. A shorter prompt is a faster first token. If you have squeezed the prompt and cached what can be cached and the budget is still missed, moving stable behaviour into weights buys you room that no prompt trick will.

The tradeoff in all three: you gain a cheap, fast, consistent model and you lose the ability to change its behaviour in five minutes. A prompt edit ships in a commit. A behaviour change in a tuned model ships in a training run, a scoring pass, and a deploy. Teams that iterate weekly on what the agent should do usually find that trade bad, even when the cost model says otherwise.

Using both without paying for both twice

The two are not rivals. They sit at different points: retrieval decides what is in front of the model, tuning decides what the model does with it. A support agent that answers from today's policy, in your exact ticket format, needs both, and the order of operations matters.

Build retrieval first and measure it on its own. Score whether the right chunk was in the prompt, separately from whether the final answer was good. Teams that only score the final answer cannot tell a bad retriever from a bad prompt, and they end up tuning the model to paper over an index they never checked.

Then write the prompt with examples and keep it in a file under review. Only when the examples stop paying for themselves does the tuning question become live, and by then you have something valuable for it: a set of real inputs with the outputs you approved, which is exactly the training file you need and the artifact nobody has on day one.

One caution on the combination. Never train on examples whose answers came from the retrieved text, unless you keep the retrieved text in the training input too. Otherwise you are teaching the model to state facts it cannot see at run time, which produces a confident, fluent, wrong model, the most expensive kind.

• Score retrieval alone: was the right passage in the prompt, yes or no.
• Score the answer given a perfect passage. That splits retriever bugs from prompt bugs.
• Keep the prompt in git with the eval set beside it.
• Harvest approved outputs into the training file. That is your tuning dataset, earned rather than written.

The third option nobody names: fix the context, not the model

Between the two big methods there is a cheaper one that gets skipped because it has no vendor: change what goes into the window and how it is arranged. The glossary on this site has an entry for context engineering, and the short version is that the window is a budget and most teams spend it badly.

Common wins, in the order they usually pay: cut the instructions that repeat, label each block of context so the model knows what it is reading, move the task to the end where the model attends to it, drop the tool definitions the task cannot use, and cache the stable prefix so you stop paying for it on every turn. Prompt caching in particular changes the cost arithmetic of long system prompts, and it is a config change rather than a project.

The reason this works is unglamorous. A long prompt is not free attention. A fact can be in the window and still go unused, because it sits in an unlabelled wall of text a long way from the question, so an answer fails on layout rather than on retrieval. Rearranging a prompt is an afternoon. It should be the first thing tried and it is usually the last.

Where Agentik sits on this question

For the record, since this is our blog: Agentik {OS} does neither of these things to your model. We do not fine-tune and we do not run a vector database for you. The host you already pay, Claude, Claude Code, Cursor, ChatGPT, Codex, or Hermes, runs the model and pays the tokens. What we ship is the third option above, done for a line of work.

The mechanism is context selection. One MCP tool, talk, takes a normal conversation turn and returns a compact pack of the agents and rules that turn needs, about 2 KB on Growth OS against about 110 KB for the full roster of 48 agents. The point is not the number. The point is that most of what an agent system needs is a decision about what to put in the window, which is the cheapest of the three interventions on this page and the one with no training bill.

That is also the limit of it. If you need a small tuned classifier at high volume, or an index over ten years of your own documents with per-user permissions, that is a different job than the one we do, and you should build or buy it separately.

What this guide does not settle

The tests on this page tell you which failure you have. They do not tell you whether the result will be good enough, and no framework can. That number comes from an eval set of your own cases, scored by someone who knows what a right answer looks like in your business. If you do not have that set, build it before you pick a method, because without it both methods are guesses with different invoices.

The second limit is that model behaviour moves. Advice about what needs tuning today can expire with one release, and I would not bet on any particular gap staying open. That biases this piece toward reversible fixes: prompts, retrieval, context layout. I am stating the bias rather than hiding it.

The third limit is scope. This guide is about text models answering with facts and formats. If you are training on images, audio, or a task with no natural language target, the shape of the decision changes and the tests above do not transfer cleanly. Where I do not know, I would rather say so than write a rule that sounds complete.