Chapter 1: Understanding LoRA
The Intuition Behind Low-Rank Adaptation
LoRA (Low-Rank Adaptation) is an elegant technique that makes a simple but powerful observation: we don't need to update all parameters to adapt a model to new tasks.
The Core Insight
Key Observation
When fine-tuning pre-trained models, the weight updates often have a low "intrinsic rank"—meaning the changes can be captured by a much smaller set of parameters.
Analogy
Think of it like editing a document:
- Full Fine-Tuning: Rewrite the entire document
- LoRA: Add sticky notes with corrections
The original document (base model) stays intact, and the sticky notes (LoRA adapters) capture the changes.
How LoRA Works
Traditional Fine-Tuning
In a neural network layer:
h = Wx
Where W is a weight matrix of size (d × d).
Full fine-tuning updates W directly:
W' = W + ΔW
Where ΔW is the learned change to all (d × d) parameters.
LoRA's Approach
LoRA represents ΔW as a product of two smaller matrices:
ΔW = BA
Where:
- B has size (d × r)
- A has size (r × d)
- r is much smaller than d (r ≪ d)
The forward pass becomes:
h = Wx + BAx = (W + BA)x
Visual Representation
Parameter Savings
Example: A 4096 × 4096 Weight Matrix
# Original weight matrix
d = 4096
original_params = d * d # 16,777,216 parameters
# LoRA with rank 16
r = 16
lora_params = d * r + r * d # 131,072 parameters
# Savings
reduction = 1 - (lora_params / original_params)
print(f"Parameter reduction: {reduction:.2%}") # 99.22%
For a Full Model
| Model | Original | LoRA (r=16) | Reduction |
|---|---|---|---|
| 7B | 7,000,000,000 | ~4,000,000 | 99.94% |
| 13B | 13,000,000,000 | ~8,000,000 | 99.94% |
| 70B | 70,000,000,000 | ~40,000,000 | 99.94% |
Why Low Rank Works
Intrinsic Dimensionality
Research shows that fine-tuning updates tend to lie in a low-dimensional subspace:
- Pre-trained models already have rich representations
- Task adaptation requires "steering" rather than "rebuilding"
- Many dimensions of weight space are redundant for specific tasks
Empirical Evidence
From the original LoRA paper:
- r=4 often sufficient for many tasks
- r=8-16 recommended for complex adaptations
- r=64+ rarely needed, diminishing returns
Initialization
A Matrix: Random Gaussian
# A is initialized with small random values
A = torch.randn(r, d) * 0.01
B Matrix: Zero
# B is initialized to zero
B = torch.zeros(d, r)
Why This Initialization?
At training start: BA = 0 × A = 0
→ Output unchanged from base model
→ Stable training start
→ Gradual adaptation
Which Layers to Target
Transformer Architecture
Attention Block:
├── Query projection (q_proj) ← Common target
├── Key projection (k_proj) ← Sometimes target
├── Value projection (v_proj) ← Common target
└── Output projection (o_proj) ← Sometimes target
Feed-Forward Block:
├── Gate projection (gate_proj) ← Sometimes target
├── Up projection (up_proj) ← Sometimes target
└── Down projection (down_proj) ← Sometimes target
Common Configurations
| Configuration | Target Modules | Use Case |
|---|---|---|
| Minimal | q_proj, v_proj | Quick experiments |
| Standard | q_proj, k_proj, v_proj, o_proj | Balanced |
| Full | All attention + FF | Maximum capacity |
Merging LoRA
During Inference (With Adapter)
# Two paths: base + LoRA
output = W @ x + B @ A @ x
Slight overhead from additional computation.
Merged (No Adapter)
# Merge once
W_merged = W + B @ A
# Single path at inference
output = W_merged @ x
No overhead—identical to fine-tuned model.
Merge Example
from peft import PeftModel
# Load model with adapter
model = PeftModel.from_pretrained(base_model, "path/to/adapter")
# Merge adapter into base model
model = model.merge_and_unload()
# Now model is a regular model without adapter overhead
model.save_pretrained("merged-model")
LoRA Variants
Standard LoRA
The original formulation described above.
LoRA+
Applies different learning rates to A and B matrices.
DoRA (Weight-Decomposed Low-Rank Adaptation)
Separates magnitude and direction of weights.
GaLore (Gradient Low-Rank Projection)
Projects gradients into low-rank space.
Key Takeaways
✅ LoRA represents weight updates as low-rank matrix products
✅ Only trains ~0.01-0.1% of original parameters
✅ B initialized to zero ensures stable training start
✅ Common to target attention projections (q_proj, v_proj)
✅ Can be merged for zero inference overhead
✅ Low rank works because fine-tuning has low intrinsic dimensionality
Next Steps
Continue to Chapter 2: LoRA Mathematics for the mathematical foundations.
Quick Check
Before moving on, make sure you can answer:
- How does LoRA represent weight updates?
- Why is B initialized to zero?
- What does "low rank" mean in the context of LoRA?