Fine-tuning, prompting, and RAG: choosing the right lever
These three get compared as competing answers to one question: which technique should power my AI feature? They answer three different questions. What should the model do, what should it know, and how consistently should it do it at scale. A production system usually needs an answer to all three.
1. Three different problems wearing one debate
"Should we fine-tune or use RAG?" only makes sense if you've merged two unrelated things: giving a model access to knowledge it wasn't trained on, and shaping how it behaves (format, tone, edge-case handling). RAG answers the first. Fine-tuning can answer the second, and is a much weaker answer to the first than most people expect. Prompting sits under both: a system prompt does load-bearing work in nearly every system, whether or not RAG or fine-tuning is also involved. Most of choosing correctly is naming which of the three questions is open for a given feature.
2. Prompting: behavior and format, cheap to iterate
A system prompt (clear instructions, examples of the desired output format, guidance on edge cases) is the cheapest lever. Exhaust it before reaching for the other two. It's cheap because iteration is instant: a prompt change takes effect on the next call, with no training run, no data collection, and no new model artifact to deploy. For many behavior and format problems (tone, structure, handling an ambiguous request), prompting alone gets most of the way. Teams that jump to fine-tuning for a problem prompting could solve pay a much higher iteration cost for a result a good prompt would have reached faster and cheaper.
The limit is the context window. A prompt can carry only so many instructions and examples before it competes for space with everything else in the window, the budget problem context engineering describes. A prompt that has grown into a huge, ad hoc pile of edge-case patches shows the behavior has outgrown what static instructions can hold. That is one of the few signals that fine-tuning might be worth its cost.
3. RAG: knowledge that's large, dynamic, or needs to be attributable
Retrieval is for information the model needs that is too large to fit in a prompt, changes too often to bake into fixed instructions, or has to be traceable to a specific, verifiable source. Examples: a support system answering from a frequently updated knowledge base, a research tool that cites the document a claim came from, a system working over private data the model was never trained on. In each, the problem is what the model knows. No prompt or fine-tune substitutes for access to information that didn't exist, or wasn't available, at training time.
Claude and retrieval: long context vs. RAG covers the further choice between putting a bounded corpus directly in context and building a retrieval pipeline. Both answer the knowledge question. Neither is what fine-tuning is good at.
4. Fine-tuning: consistent behavior at scale, and its real cost
Fine-tuning earns its cost in fewer cases than its reputation suggests. The clearest one is a narrow, well-defined task at very high volume. The desired behavior moves into the weights, so every call needs a shorter prompt, and those savings can offset the upfront cost of curating data and running the fine-tune. It's also the right lever when a prompt can't guarantee enough consistency across a very large volume of varied inputs: a specific output format or style that has to hold with little deviation across millions of calls, where prompting leaves too much variance at the edges.
The costs are concrete. Curating a training dataset well is hard, and a poorly assembled one bakes in whatever biases and errors the data contained. Iteration is slow: testing a behavior change needs a new fine-tuning run. And a fine-tuned model is an artifact someone has to version, evaluate, and maintain, with its own regression risk every time it's retrained on updated data.
5. What fine-tuning doesn't fix
This mistake is common and expensive to discover late: fine-tuning is not a reliable way to teach a model new facts. A model fine-tuned on data containing a set of facts learns something closer to the style and pattern of that data than a dependable memory of each fact. Asked to recall a detail that appeared once in its fine-tuning data, it is much less reliable than a retrieval system that fetches the source document containing that detail on demand, verifiable every time. Teams that fine-tune to give a model knowledge of a proprietary dataset are using the wrong tool. Retrieval, or retrieval plus a lighter fine-tune for format and tone, gets a more reliable result for the knowledge half of the problem.
6. They compose
None of the three excludes the others. Most production systems that look like they picked one run a combination: a system prompt for tone and format, retrieval for the specific, current, or proprietary knowledge a response must be grounded in, and, where it earns its cost, a fine-tune for a narrow, high-volume task where consistency matters more than flexibility. Treating the choice as exclusive ("we're a RAG system", "we fine-tuned our model") over-invests in whichever lever got attention first and under-invests in the one that closes the gap it can't.
7. Worked example: a support bot needing all three
A support bot has three requirements:
- Answer in the company's tone and format (a behavior problem).
- Answer from a knowledge base updated daily with new articles and product changes (a knowledge problem).
- Classify every incoming ticket into one of a fixed set of categories at very high volume, with tight consistency for downstream routing (a consistency-at-scale problem).
Solving all three with one lever produces a worse system. Prompting shapes tone well but can't keep pace with a daily-updated knowledge base without a huge, constantly rewritten prompt. RAG solves the knowledge problem and does little for classification consistency, which is a narrow, high-volume, well-defined task suited to a lightweight fine-tune. Fine-tuning the bot's general conversational behavior, beyond the classification step, would be slow to iterate on and still wouldn't keep knowledge fresh, which retrieval does better and cheaper.
The working system layers all three, each at its own job: a system prompt for tone and general behavior, retrieval against the daily-updated knowledge base for current or proprietary information, and a small purpose-built classifier, possibly the one fine-tuned piece, handling ticket categorization at the volume and consistency routing needs.
This pairs with RAG failure modes for debugging retrieval once it's the right lever, and with eval-driven development for measuring whether a change to a prompt, a retrieval pipeline, or a fine-tune moved the system's performance or only felt like an improvement.