RAG or fine-tuning: choose by what needs to change

“Should we use RAG or fine-tuning?” is often asked before the failure is defined. A model that lacks current policy documents has a knowledge-access problem. A model that receives the right evidence but repeatedly uses the wrong format has a behavior problem. Those problems call for different controls.

Begin with four options, not two

Prompting, retrieval-augmented generation, fine-tuning, and a hybrid system are complementary. Use the least complex option that meets a measured requirement.

Fine-tuning should not be the default way to load a changing knowledge base. Training data changes model parameters; it does not create a dependable database, source link, deletion mechanism, or request-time permission check. A tuned model can still need RAG.

Use RAG when the answer depends on current evidence

A RAG system prepares a searchable knowledge source, retrieves candidate passages for the user’s request, and supplies selected evidence to the generator. The model answers from that context under application instructions. This can support citations and source-aware review, but only if retrieval and grounding are designed and evaluated.

RAG is a strong fit when

• Knowledge changes more frequently than a model training cycle.
• The content is private, tenant-specific, or filtered by user permissions.
• Users need references or the ability to inspect the supporting source.
• Records must be added, corrected, expired, or removed without retraining a model.
• The task needs a narrow subset of a corpus rather than a summary of everything.

RAG introduces its own production system. Data must be collected, parsed, segmented, enriched, embedded or indexed, secured, refreshed, retrieved, ranked, and assembled into context. Each step can fail independently.

RAG cannot repair missing or wrong evidence

A fluent answer can still be grounded in an irrelevant passage. When the system fails, inspect retrieval before changing the prompt or model. Determine whether the answer existed in the source, whether parsing preserved it, whether the query found it, whether ranking selected it, and whether the generator used it correctly.

Access control must happen in the retrieval and application layers, not through an instruction asking the model to ignore restricted content. Carry the authenticated identity and tenant boundary into filtering, and test that prohibited sources cannot enter the context.

Use fine-tuning when examples define the desired behavior

Fine-tuning adapts a base model using examples or preference signals. Depending on the supported method, it can improve consistent formatting, classification, extraction, style, tool selection, or a specialized task pattern. It requires a stable task, representative data, and an evaluation that can distinguish improvement from memorization or regression.

Fine-tuning is a stronger candidate when

• The desired behavior is difficult to express reliably through instructions alone but easy to demonstrate with high-quality examples.
• The task repeats at meaningful volume and prompt examples consume substantial context.
• A smaller adapted model can be tested against the required quality, latency, and cost target.
• Inputs and expected outputs form a stable contract, such as a classification, transformation, or constrained domain task.
• The team can maintain separate training, validation, and holdout data with clear provenance and permissions.

Start with an evaluation and the best practical prompt. Save the inputs where the base model fails and label the desired outputs. Review the dataset for duplicates, contradictions, leakage, sensitive data, narrow coverage, and low-quality synthetic examples. Keep a holdout set the training process never sees.

1. Define the behavior gap
   Name the failure that prompting, structured output, retrieval, tools, or deterministic code does not solve adequately.
2. Build a representative baseline
   Run the current model and prompt on real task categories, edge cases, and unacceptable outcomes.
3. Curate training examples
   Use consistent, authorized, high-quality input-output pairs that demonstrate the target behavior and relevant diversity.
4. Separate evaluation data
   Keep a holdout set for comparison and include categories likely to expose regressions or unsafe behavior.
5. Compare complete systems
   Evaluate the tuned model against the same prompt, context, tools, latency, cost, and operational constraints as the baseline.
6. Version and monitor
   Record the base model, dataset, training configuration, prompt, evaluation result, release, and rollback path.

Do not assume fine-tuning makes outputs factual. It can improve how the model performs a task while still generating unsupported claims. If factual correctness depends on external records, supply and verify those records at inference time.

Combine them when knowledge and behavior both need control

A support copilot may need RAG for current product documentation and account policy, while a fine-tuned model applies the organization’s response structure and escalation categories. A contract workflow may retrieve clauses and precedent, while an adapted model extracts a consistent risk schema. The retriever and tuned generator still need separate evaluation.

A hybrid architecture is not automatically more mature. It adds two data lifecycles, more versions, more failure paths, and more cost. Introduce it only when the evaluation shows distinct knowledge and behavior gaps that the combination resolves.

Evaluate the complete product path

Use the same representative task set to compare prompting, RAG, fine-tuning, and hybrid candidates. Measure the qualities that matter to the user and the operation rather than one generic score.

• Task result: correctness, completeness, format, action success, or user acceptance.
• Evidence: retrieval recall, context precision, groundedness, citation support, and permission correctness.
• Behavior: instruction following, consistency, calibration, refusal, and required escalation.
• Operation: latency, cost per accepted result, freshness, failure recovery, observability, and maintenance effort.
• Safety: sensitive-data handling, access boundaries, harmful outputs, prompt injection, and tool authority.

Evaluate changes by category. A higher overall average can hide a regression for a high-risk workflow or a tenant-specific permission path. Review failed examples and update the correct layer: data, parser, retriever, prompt, model, tool, or application rule.

Reference points

This guide uses current cloud and model-provider documentation as technical reference points. Specific platform capabilities can change and should be checked before implementation.

• Microsoft Foundry: retrieval-augmented generation — RAG for private or changing knowledge and fine-tuning for behavior, style, or task performance
• AWS Prescriptive Guidance: comparing RAG and fine-tuning — tradeoffs, changing documents, source references, and hybrid use
• OpenAI retrieval guide — semantic search, vector stores, ranking, attributes, and retrieval configuration
• OpenAI model optimization guide — evaluation, prompting, fine-tuning, and continuous comparison
• Anthropic contextual retrieval — preserving document context for retrieval and combining retrieval techniques