Chapter 3: PEFT Methods Overview
All Major Parameter-Efficient Methods Explained
This chapter provides a comprehensive overview of all major PEFT methods, helping you choose the right one for your use case.
PEFT Methods Taxonomy
1. LoRA (Low-Rank Adaptation)
How It Works
LoRA adds trainable low-rank decomposition matrices alongside frozen weights.
Original: h = Wx
LoRA: h = Wx + BAx
Where:
- W is frozen (d × d)
- B is trainable (d × r)
- A is trainable (r × d)
- r is much smaller than d (low rank, e.g., 8-64)
Visual Representation
Configuration
from peft import LoraConfig
lora_config = LoraConfig(
r=16, # Rank (capacity)
lora_alpha=32, # Scaling factor
target_modules=[ # Where to apply
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
],
lora_dropout=0.05, # Regularization
bias="none", # Don't train biases
task_type="CAUSAL_LM",
)
Pros & Cons
| Pros | Cons |
|---|---|
| Very effective | Hyperparameter tuning needed |
| Well understood | Rank selection important |
| Widely supported | Small inference overhead |
| Can be merged | — |
2. QLoRA (Quantized LoRA)
How It Works
Combines 4-bit quantization with LoRA for extreme efficiency.
1. Load base model in 4-bit (NF4 format)
2. Add LoRA adapters in fp16/bf16
3. Train only the LoRA parameters
4. Gradients flow through dequantized weights
Configuration
from transformers import BitsAndBytesConfig
from peft import LoraConfig
# Quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, # Extra savings
)
# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
)
# Apply LoRA on top
lora_config = LoraConfig(r=16, lora_alpha=32, ...)
model = get_peft_model(model, lora_config)
Memory Comparison
| Model | LoRA | QLoRA | Savings |
|---|---|---|---|
| 7B | ~14GB | ~6GB | 57% |
| 13B | ~26GB | ~10GB | 62% |
| 70B | ~140GB | ~48GB | 66% |
3. Prefix Tuning
How It Works
Adds trainable "virtual tokens" as prefixes to the key and value states in attention layers.
Normal attention: Attention(Q, K, V)
Prefix tuning: Attention(Q, [P_K; K], [P_V; V])
Where P_K, P_V are learned prefix embeddings
Configuration
from peft import PrefixTuningConfig
prefix_config = PrefixTuningConfig(
task_type="CAUSAL_LM",
num_virtual_tokens=20, # Number of prefix tokens
prefix_projection=True, # Use MLP to project
)
Use Cases
- Task-specific conditioning
- When you can't modify model weights
- Continuous prompt optimization
4. Prompt Tuning
How It Works
Learns continuous embeddings prepended to the input sequence.
Normal: embed("What is AI?")
Prompt Tuning: [learned_prompts] + embed("What is AI?")
Configuration
from peft import PromptTuningConfig
prompt_config = PromptTuningConfig(
task_type="CAUSAL_LM",
num_virtual_tokens=8,
prompt_tuning_init="TEXT", # Initialize from text
prompt_tuning_init_text="Classify the following:",
)
Characteristics
- Fewest trainable parameters
- Works well for classification
- Less flexible than LoRA
- No modification to model internals
5. IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations)
How It Works
Learns to rescale activations rather than adding new weights.
Original: h = activation(Wx)
IA3: h = activation(W(l_v ⊙ x)) * l_k
Where l_v, l_k are learned scaling vectors
Configuration
from peft import IA3Config
ia3_config = IA3Config(
task_type="CAUSAL_LM",
target_modules=["k_proj", "v_proj", "down_proj"],
feedforward_modules=["down_proj"],
)
Characteristics
- Even fewer parameters than LoRA
- Minimal computational overhead
- Good for some tasks, less versatile
- Research-stage adoption
6. AdaLoRA (Adaptive LoRA)
How It Works
Dynamically allocates rank to different weight matrices based on importance.
Instead of: fixed rank r for all layers
AdaLoRA: rank_i based on importance score
Important layers → higher rank
Less important → lower rank
Configuration
from peft import AdaLoraConfig
adalora_config = AdaLoraConfig(
init_r=12, # Initial rank
target_r=4, # Target average rank
tinit=200, # Warmup steps
tfinal=1000, # When to stop adapting
deltaT=10, # Update frequency
beta1=0.85,
beta2=0.85,
)
When to Use
- When unsure about rank selection
- For optimal parameter allocation
- When willing to trade complexity for efficiency
Method Comparison
Parameters and Memory
| Method | Parameters | Memory | Quality |
|---|---|---|---|
| LoRA | ~0.1-1% | Low | ⭐⭐⭐⭐⭐ |
| QLoRA | ~0.1-1% | Very Low | ⭐⭐⭐⭐ |
| Prefix | ~0.1% | Very Low | ⭐⭐⭐ |
| Prompt | ~0.01% | Very Low | ⭐⭐⭐ |
| IA3 | ~0.01% | Very Low | ⭐⭐⭐ |
| AdaLoRA | Variable | Low | ⭐⭐⭐⭐ |
Use Case Recommendations
| Use Case | Recommended Method |
|---|---|
| General fine-tuning | LoRA |
| Limited memory | QLoRA |
| Multiple similar tasks | Prompt Tuning |
| Task conditioning | Prefix Tuning |
| Minimal parameters | IA3 |
| Optimal efficiency | AdaLoRA |
Combining Methods
LoRA + Quantization
# Most common combination
bnb_config = BitsAndBytesConfig(load_in_4bit=True, ...)
model = load_model(quantization_config=bnb_config)
model = get_peft_model(model, lora_config)
Multiple LoRA Adapters
# Train different adapters
model.add_adapter("task_a", lora_config_a)
model.add_adapter("task_b", lora_config_b)
# Switch between them
model.set_adapter("task_a")
# or
model.set_adapter("task_b")
Choosing the Right Method
Key Takeaways
✅ LoRA is the most versatile and widely used PEFT method
✅ QLoRA adds quantization for extreme memory efficiency
✅ Prompt/Prefix tuning work without modifying model weights
✅ IA3 offers minimal parameters but less flexibility
✅ Choose based on memory constraints and quality requirements
✅ Methods can be combined (e.g., QLoRA = Quantization + LoRA)
Next Steps
Test your knowledge with the Module 3 Quiz, then dive deep into LoRA in Module 4: LoRA Deep Dive.
Quick Check
Before moving on, make sure you can answer:
- What is the key difference between LoRA and QLoRA?
- How does Prefix Tuning work conceptually?
- Which PEFT method uses the fewest parameters?