Chapter 3: Catastrophic Forgetting

Preserving Knowledge During Fine-Tuning

Catastrophic forgetting is one of the biggest challenges in fine-tuning. Understanding it helps you design better training strategies.

---

What is Catastrophic Forgetting?

Catastrophic forgetting occurs when a neural network loses previously learned knowledge after being trained on new data.

Example Scenario

A pre-trained LLM knows:

• How to translate languages
• General world knowledge
• Mathematical reasoning
• Code generation

After fine-tuning on customer support data:

• ✅ Great at customer support
• ❌ May struggle with translation
• ❌ May lose some general knowledge
• ❌ Code generation quality may drop

---

Why Does It Happen?

Weight Interference

Neural networks store knowledge in their weights. When you train on new data:

Original weights (W) → encode general knowledge
New training → updates W to (W')
W' optimizes for new task → may break old task representations

Distribution Shift

The model adapts to your specific data distribution, potentially losing generalization.

---

Measuring Catastrophic Forgetting

Before/After Evaluation

from lm_eval import evaluator
from lm_eval.models.huggingface import HFLM

def evaluate_model(model_path, tasks=["hellaswag", "arc_easy", "mmlu"]):
    """Evaluate model on standard benchmarks."""
    model = HFLM(pretrained=model_path)
    results = evaluator.simple_evaluate(
        model=model,
        tasks=tasks,
        num_fewshot=0,
    )
    return results

# Before fine-tuning
baseline_results = evaluate_model("meta-llama/Llama-3.1-8B")
print("Baseline:", baseline_results)

# After fine-tuning
finetuned_results = evaluate_model("./my-finetuned-model")
print("Fine-tuned:", finetuned_results)

# Compare
for task in ["hellaswag", "arc_easy", "mmlu"]:
    baseline = baseline_results["results"][task]["acc"]
    finetuned = finetuned_results["results"][task]["acc"]
    diff = finetuned - baseline
    print(f"{task}: {diff:+.2%} ({'↑' if diff > 0 else '↓'})")

Task-Specific Testing

# Create a test suite for capabilities you want to preserve
test_cases = [
    {"prompt": "Translate to French: Hello, how are you?",
     "expected_contains": "Bonjour"},
    {"prompt": "What is 15 * 17?",
     "expected_contains": "255"},
    {"prompt": "Write a Python function to reverse a string",
     "expected_contains": "def"},
]

def test_capabilities(model, tokenizer, test_cases):
    """Test if model retains specific capabilities."""
    for test in test_cases:
        output = model.generate(
            tokenizer.encode(test["prompt"], return_tensors="pt"),
            max_new_tokens=100
        )
        response = tokenizer.decode(output[0], skip_special_tokens=True)
        passed = test["expected_contains"] in response
        print(f"{'✅' if passed else '❌'} {test['prompt'][:50]}...")

---

Mitigation Strategies

1. Use Lower Learning Rates

# Full fine-tuning: lower LR preserves more knowledge
training_args = TrainingArguments(
    learning_rate=1e-5,  # Lower than default
    # vs 2e-5 or higher for PEFT
)

2. Fewer Training Epochs

# Less training = less forgetting
training_args = TrainingArguments(
    num_train_epochs=1,  # Start with 1
    # Monitor validation performance
    # Only increase if underfitting
)

3. Regularization Techniques

Weight Decay

training_args = TrainingArguments(
    weight_decay=0.01,  # Penalize large weight changes
)

Elastic Weight Consolidation (EWC)

# EWC adds a penalty for changing important weights
# (Research technique, not directly in HF Trainer)
loss = cross_entropy_loss + lambda * sum(
    importance[i] * (theta[i] - theta_star[i])^2
    for i in parameters
)

4. Data Mixing / Replay

# Mix original pre-training style data with fine-tuning data
from datasets import concatenate_datasets

# Your specific data
finetuning_data = load_dataset("your/dataset")

# General instruction data to maintain capabilities
general_data = load_dataset("databricks/dolly-15k", split="train[:1000]")

# Mix them (e.g., 80% specific, 20% general)
mixed_dataset = concatenate_datasets([
    finetuning_data,
    general_data.shuffle().select(range(len(finetuning_data) // 4))
])

5. Use PEFT Instead

The most effective mitigation: don't update all weights

from peft import LoraConfig, get_peft_model

# Only update small adapters, preserving base model
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
)

model = get_peft_model(base_model, lora_config)
# Base model weights frozen → no catastrophic forgetting!

---

PEFT's Advantage

Why PEFT Prevents Forgetting

Empirical Evidence

Method	Task Performance	Benchmark Retention
Full FT	95%	70-85%
LoRA	93%	95-99%
QLoRA	92%	95-99%

PEFT methods typically retain 95%+ of baseline performance on unrelated tasks.

---

Evaluation Strategy

Comprehensive Evaluation Protocol

def comprehensive_evaluation(model_path, baseline_path):
    """Evaluate both task performance and capability retention."""
    
    # 1. Task-specific evaluation (your fine-tuning goal)
    task_score = evaluate_on_task(model_path, your_test_set)
    
    # 2. General capability benchmarks
    benchmarks = ["hellaswag", "arc_easy", "mmlu", "gsm8k"]
    baseline_scores = {}
    finetuned_scores = {}
    
    for bench in benchmarks:
        baseline_scores[bench] = evaluate_benchmark(baseline_path, bench)
        finetuned_scores[bench] = evaluate_benchmark(model_path, bench)
    
    # 3. Calculate retention
    retention = {}
    for bench in benchmarks:
        retention[bench] = finetuned_scores[bench] / baseline_scores[bench]
    
    avg_retention = sum(retention.values()) / len(retention)
    
    print(f"Task Performance: {task_score:.2%}")
    print(f"Average Capability Retention: {avg_retention:.2%}")
    
    # 4. Trade-off analysis
    if task_score > 0.90 and avg_retention > 0.95:
        print("✅ Good balance achieved")
    elif task_score > 0.90 and avg_retention < 0.90:
        print("⚠️ High task performance but significant forgetting")
    else:
        print("❌ Consider adjusting training")
    
    return {
        "task_score": task_score,
        "retention": retention,
        "avg_retention": avg_retention
    }

---

Best Practices Summary

To Minimize Catastrophic Forgetting:

Strategy	Difficulty	Effectiveness
Use PEFT (LoRA/QLoRA)	Easy	⭐⭐⭐⭐⭐
Lower learning rate	Easy	⭐⭐⭐
Fewer epochs	Easy	⭐⭐⭐
Data mixing	Medium	⭐⭐⭐⭐
Weight decay	Easy	⭐⭐
Early stopping	Easy	⭐⭐⭐

Recommended Approach

1. Start with PEFT (LoRA or QLoRA)
2. If PEFT is insufficient, try full fine-tuning with:
   • Low learning rate (1e-5)
   • Limited epochs (1-2)
   • Data mixing
   • Comprehensive evaluation

---

Key Takeaways

✅ Catastrophic forgetting is when models lose knowledge after fine-tuning

✅ It happens because weights are modified to fit new data distribution

✅ PEFT methods largely avoid forgetting by keeping base weights frozen

✅ For full fine-tuning: use low LR, few epochs, data mixing

✅ Always evaluate both task performance AND capability retention

✅ The trade-off is: task specialization vs. general capability

---

Next Steps

Test your knowledge with the Module 2 Quiz, then continue to Module 3: Parameter-Efficient Fine-Tuning.

---

Quick Check

Before moving on, make sure you can answer:

1. What is catastrophic forgetting and why does it occur?
2. How do PEFT methods help prevent catastrophic forgetting?
3. What are three strategies to minimize forgetting in full fine-tuning?