Chapter 3: Fine-Tuning Landscape
Methods, Tools, and Requirements Overview
The fine-tuning landscape has evolved rapidly, offering multiple approaches suited to different needs and resources. This chapter provides a comprehensive overview to help you navigate your options.
Fine-Tuning Methods Overview
Method Comparison
| Method | Parameters Trained | Memory Usage | Quality | Best For |
|---|---|---|---|---|
| Full Fine-Tuning | 100% | Very High | Highest | Maximum customization |
| LoRA | ~0.1-1% | Medium | High | General fine-tuning |
| QLoRA | ~0.1-1% | Low | High | Consumer GPUs |
| Prefix Tuning | ~0.1% | Low | Good | Task-specific |
| IA3 | ~0.01% | Very Low | Good | Minimal changes |
Full Fine-Tuning
What It Is
Updating all parameters in the model during training.
Pros
- Maximum flexibility and customization
- Can achieve the best results for complex tasks
- Full control over model behavior
Cons
- Requires significant GPU memory (7B model needs ~28GB+ for training)
- Higher risk of catastrophic forgetting
- Expensive in terms of compute
When to Use
- When you have ample compute resources
- For critical applications requiring maximum quality
- When PEFT methods don't achieve desired results
# Full fine-tuning example
from transformers import Trainer, TrainingArguments
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
fp16=True, # Mixed precision to save memory
)
trainer = Trainer(
model=model, # All parameters trainable
args=training_args,
train_dataset=dataset,
)
LoRA (Low-Rank Adaptation)
What It Is
Adding small trainable matrices to frozen model weights.
How It Works
Original: W (frozen)
LoRA: W + BA (where B and A are small trainable matrices)
Key Parameters
- Rank (r): Dimensionality of the low-rank matrices (e.g., 8, 16, 32)
- Alpha: Scaling factor for LoRA contributions
- Target Modules: Which layers to apply LoRA to (q_proj, v_proj, etc.)
Pros
- 90%+ memory reduction
- Fast training
- Adapter can be swapped or merged
- No quality loss for most tasks
Cons
- Slightly lower ceiling than full fine-tuning
- Hyperparameter tuning needed (rank, alpha)
from peft import LoraConfig, get_peft_model
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, lora_config)
QLoRA (Quantized LoRA)
What It Is
Combining 4-bit quantization with LoRA for extreme memory efficiency.
How It Works
- Load base model in 4-bit precision
- Add LoRA adapters in higher precision
- Train only the LoRA parameters
Memory Comparison
| Model Size | Full FT | LoRA | QLoRA |
|---|---|---|---|
| 7B | ~28GB | ~14GB | ~6GB |
| 13B | ~52GB | ~26GB | ~10GB |
| 70B | ~280GB | ~140GB | ~48GB |
Pros
- Train 70B models on consumer GPUs
- Minimal quality loss compared to LoRA
- Very cost-effective
Cons
- Slightly slower training
- Requires bitsandbytes library
- Some quantization artifacts possible
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"model_name",
quantization_config=bnb_config,
)
Alignment Methods
RLHF (Reinforcement Learning from Human Feedback)
- Trains a reward model from human preferences
- Uses RL (PPO) to optimize the language model
- Complex but powerful
DPO (Direct Preference Optimization)
- Simplifies RLHF by removing reward model
- Directly optimizes on preference pairs
- Much simpler to implement
ORPO (Odds Ratio Preference Optimization)
- Combines SFT and alignment in one step
- Even simpler than DPO
- New but promising results
# DPO example with TRL
from trl import DPOTrainer
trainer = DPOTrainer(
model=model,
ref_model=reference_model,
train_dataset=preference_dataset,
beta=0.1, # KL penalty weight
)
Tools Ecosystem
Hugging Face Stack
| Tool | Purpose |
|---|---|
| transformers | Model loading, tokenization, training |
| peft | Parameter-efficient fine-tuning |
| trl | SFT, DPO, RLHF trainers |
| datasets | Dataset loading and processing |
| accelerate | Distributed training |
| bitsandbytes | Quantization |
Alternative Frameworks
| Tool | Strength | Use Case |
|---|---|---|
| Axolotl | Easy configuration | Quick experiments |
| LLaMA-Factory | Unified interface | Multiple methods |
| Unsloth | 2x speed, less memory | Fast iteration |
Compute Requirements
GPU Memory Guidelines
Memory Needed ≈ (Model Parameters × Precision Bytes × Multiplier)
Where Multiplier:
- Inference: 1x
- LoRA Training: 2-3x
- Full Fine-Tuning: 4-6x
Recommended GPUs
| Budget | GPU | VRAM | Good For |
|---|---|---|---|
| Free | Colab T4 | 15GB | QLoRA on 7B |
| $ | RTX 3090/4090 | 24GB | QLoRA on 13B |
| $$ | A100-40GB | 40GB | LoRA on 70B |
| $$$ | A100-80GB | 80GB | Full FT on 13B |
| $$$$ | H100 | 80GB | Full FT on 70B |
Training Time Estimates
| Model | Method | GPU | Time for 1K Examples |
|---|---|---|---|
| 7B | QLoRA | A100-40GB | ~30 min |
| 7B | LoRA | A100-40GB | ~20 min |
| 7B | Full FT | A100-80GB | ~45 min |
| 70B | QLoRA | A100-80GB | ~3 hours |
Choosing Your Path
Decision Matrix
Recommendations by Scenario
| Scenario | Recommended Method |
|---|---|
| Learning/Experimenting | QLoRA on Colab |
| Production (cost-sensitive) | QLoRA on cloud GPU |
| Production (quality-critical) | LoRA or Full FT |
| Research | Full FT with checkpoints |
| Quick iteration | Unsloth + QLoRA |
The Course Ahead
In the following modules, we'll dive deep into each method:
| Module | Focus |
|---|---|
| Module 2 | Full Fine-Tuning details |
| Module 3 | PEFT concepts |
| Module 4 | LoRA deep dive |
| Module 5 | QLoRA and quantization |
| Module 6 | Data preparation |
| Module 7 | Hugging Face training |
| Module 8 | Advanced techniques (DPO, ORPO) |
| Module 9 | Evaluation and merging |
| Module 10 | Production deployment |
Key Takeaways
✅ Multiple fine-tuning methods exist for different resource levels
✅ PEFT methods (LoRA, QLoRA) make fine-tuning accessible on consumer hardware
✅ The Hugging Face ecosystem provides all necessary tools
✅ Method choice depends on compute budget and quality requirements
✅ Start with QLoRA for learning, scale up as needed
Next Steps
Test your knowledge with the Module 1 Quiz, then continue to Module 2: Full Fine-Tuning.
Quick Check
Before moving on, make sure you can answer:
- What is the main difference between LoRA and QLoRA?
- How much memory would you need to QLoRA a 7B model?
- What is the difference between DPO and RLHF?