Chapter 2: When to Fine-Tune vs Use RAG

Making the Right Choice for Your Use Case

One of the most important decisions in LLM application development is choosing between fine-tuning, RAG (Retrieval-Augmented Generation), or simple prompt engineering. Each approach has distinct strengths.

---

The Three Approaches

1. Prompt Engineering

Carefully crafting input prompts to get desired outputs without any training.

2. RAG (Retrieval-Augmented Generation)

Retrieving relevant context from a database and including it in the prompt.

3. Fine-Tuning

Training the model on your data to internalize new knowledge or behaviors.

---

Decision Framework

Choose Fine-Tuning When:

Scenario	Why Fine-Tuning Works
Teaching a specific style or tone	Style is internalized, not just mimicked
Consistent output format	Model learns the structure deeply
Domain-specific terminology	Terminology becomes natural for the model
Custom task that's hard to prompt	Training provides more reliable behavior
Reducing prompt size/costs	Behaviors are built-in, not prompted

Choose RAG When:

Scenario	Why RAG Works
Knowledge changes frequently	Just update the database
Need citations/sources	Retrieval provides traceability
Large knowledge base	Can't fit in context or training
Factual accuracy critical	Retrieved facts vs learned facts
Privacy/access control	Control what context is retrieved

Choose Prompt Engineering When:

Scenario	Why Prompting Works
Quick experimentation	No training infrastructure needed
Simple format changes	Few-shot examples work well
Small number of use cases	Prompts can cover each case
Limited budget	No training or vector DB costs
Rapidly changing requirements	Change prompt vs retrain

---

Cost Comparison

Approach	Setup Cost	Per-Query Cost	Maintenance
Prompting	Low	High (long prompts)	Low
RAG	Medium	Medium	Medium (DB updates)
Fine-Tuning	High	Low (shorter prompts)	High (retraining)

Cost Crossover

Fine-tuning becomes cost-effective at scale. If you're making millions of API calls, the savings from shorter prompts often justify fine-tuning costs.

---

Hybrid Approaches

In practice, the best solutions often combine approaches:

Fine-Tuning + RAG

# Fine-tune for style and format
# Use RAG for knowledge retrieval

# 1. Retrieve relevant context
context = vector_db.search(query)

# 2. Use fine-tuned model with retrieved context
response = fine_tuned_model.generate(
    f"Context: {context}\nQuery: {query}"
)

Benefits:

• Model knows how to use retrieved information effectively
• Consistent style and format
• Up-to-date knowledge from retrieval

Fine-Tuning + Prompting

# Fine-tune for core behavior
# Prompt for variations

response = fine_tuned_model.generate(
    f"Respond in a formal tone: {query}"  # Prompt modifies fine-tuned behavior
)

---

Use Case Analysis

Case 1: Customer Support Bot

Requirement	Best Approach
Company voice and tone	Fine-Tuning
Product information	RAG
Response templates	Fine-Tuning
Recent announcements	RAG
Escalation rules	Fine-Tuning

Recommendation: Fine-tune for behavior + RAG for knowledge

---

Case 2: Legal Document Analysis

Requirement	Best Approach
Legal terminology	Fine-Tuning
Case law references	RAG
Citation format	Fine-Tuning
Recent rulings	RAG
Document structure	Fine-Tuning

Recommendation: Fine-tune for legal language + RAG for case database

---

Case 3: Code Assistant

Requirement	Best Approach
Coding style	Fine-Tuning
Company codebase context	RAG
Internal APIs	RAG + Fine-Tuning
Best practices	Fine-Tuning
Documentation	RAG

Recommendation: Hybrid approach for best results

---

Technical Comparison

Memory and Knowledge

Aspect	Fine-Tuning	RAG
Knowledge capacity	Limited by training	Limited by DB size
Knowledge update	Requires retraining	Update database
Latency	Fast (no retrieval)	Slower (retrieval step)
Accuracy	Can hallucinate	Citable sources
Scalability	Same cost at scale	DB costs grow

When Knowledge Matters

# RAG is better for factual queries
query = "What is our refund policy for Pro plan customers?"
# → Retrieve from policy database

# Fine-tuning is better for behavioral patterns
query = "Respond to an angry customer about billing"
# → Fine-tuned model knows the right tone and approach

---

Common Mistakes

❌ Fine-Tuning for Dynamic Knowledge

"We fine-tuned our model on company data, but it's already outdated."

Solution: Use RAG for knowledge that changes, fine-tune for behavior.

❌ RAG for Style/Format

"Our RAG system retrieves the right info but outputs are inconsistent."

Solution: Fine-tune for consistent style, RAG for content.

❌ Over-Engineering with Fine-Tuning

"We spent weeks fine-tuning when few-shot prompting would have worked."

Solution: Start with prompting, move to fine-tuning only when needed.

❌ Ignoring the Hybrid Option

"We had to choose between RAG and fine-tuning."

Solution: Combine approaches for best results.

---

Decision Tree

---

Practical Example: Building a Medical Assistant

Requirements:

1. Use proper medical terminology
2. Reference current treatment guidelines
3. Cite sources for recommendations
4. Follow HIPAA-compliant response formats

Solution:

# 1. Fine-tune for medical language and format
model = load_finetuned_model("medical-assistant-v1")

# 2. RAG for current guidelines and research
guidelines = medical_db.search(patient_query)

# 3. Combined generation
prompt = f"""
You are a medical assistant. Use the following current guidelines:
{guidelines}

Patient Query: {patient_query}

Respond using proper medical terminology and cite your sources.
"""

response = model.generate(prompt)

---

Key Takeaways

✅ Fine-tuning is best for style, format, and internalized behaviors

✅ RAG is best for dynamic knowledge and citable sources

✅ Prompting is best for quick iteration and simple customization

✅ Hybrid approaches often yield the best results

✅ Consider costs at both setup and per-query levels

✅ Match your approach to your specific requirements

---

Next Steps

Now let's explore the landscape of fine-tuning methods and tools in Chapter 3: Fine-Tuning Landscape.

---

Quick Check

Before moving on, make sure you can answer:

1. When would you choose RAG over fine-tuning?
2. What are the cost trade-offs between the three approaches?
3. How would you design a customer support system using both approaches?