Chapter 2: LoRA Mathematics

The Mathematical Foundations of Low-Rank Adaptation

Understanding the mathematics behind LoRA helps you make better decisions about hyperparameters and understand why it works so effectively.

---

Matrix Factorization

The Core Equation

For a pre-trained weight matrix W ∈ ℝ^(d×k):

W' = W + ΔW = W + BA

Where:

• W is the frozen pre-trained weight (d × k)
• B ∈ ℝ^(d×r) is the down-projection matrix
• A ∈ ℝ^(r×k) is the up-projection matrix
• r is the rank (much smaller than d and k)

Forward Pass

def forward(x, W, B, A, scaling):
    # Frozen path
    h_base = W @ x  # (d × k) @ (k × batch) = (d × batch)
    
    # LoRA path
    h_lora = B @ A @ x  # (d × r) @ (r × k) @ (k × batch) = (d × batch)
    
    # Combined
    return h_base + scaling * h_lora

---

Rank and Capacity

What is Rank?

The rank r determines the maximum "expressiveness" of the LoRA update.

Matrix ΔW = BA has rank at most r

If r is small, ΔW can only represent updates in a low-dimensional subspace.

Parameter Count

def lora_params(d, k, r):
    """Calculate LoRA parameters for a weight matrix."""
    return r * d + r * k  # B + A matrices

# Example: 4096 × 4096 matrix with r=16
params = lora_params(4096, 4096, 16)
print(f"LoRA params: {params:,}")  # 131,072

# Compare to full matrix
full = 4096 * 4096
print(f"Full params: {full:,}")  # 16,777,216
print(f"Ratio: {params/full:.4%}")  # 0.78%

Choosing Rank

Rank	Capacity	Use Case
4	Low	Simple tasks, minimal changes
8	Low-Medium	Standard fine-tuning
16	Medium	Default recommendation
32	Medium-High	Complex tasks
64	High	Maximum expressiveness
128+	Very High	Rarely needed

---

Alpha Scaling

The Scaling Factor

LoRA includes a scaling factor α (alpha):

ΔW = (α/r) × BA

Why Scale?

Without scaling, changing r changes the magnitude of updates:

• Higher r → more parameters → larger updates
• Lower r → fewer parameters → smaller updates

With α/r scaling, you can adjust r without changing the magnitude.

Common Configurations

# Common patterns
config1 = {"r": 16, "alpha": 16}   # Scale = 1
config2 = {"r": 16, "alpha": 32}   # Scale = 2
config3 = {"r": 32, "alpha": 64}   # Scale = 2

# Rule of thumb: alpha = 2 × r is common

Effect of Alpha

def effective_scaling(alpha, r):
    return alpha / r

# Examples
print(effective_scaling(16, 16))  # 1.0
print(effective_scaling(32, 16))  # 2.0
print(effective_scaling(64, 32))  # 2.0

Higher α/r → larger contribution from LoRA → faster adaptation but more risk of instability.

---

Initialization

A Matrix: Kaiming/He Initialization

# Small random values
A = torch.randn(r, k) * math.sqrt(2 / k)
# Or simpler:
A = torch.randn(r, k) * 0.01

B Matrix: Zero Initialization

B = torch.zeros(d, r)

Mathematical Reasoning

At initialization:

ΔW = BA = B × A = 0 × A = 0
h_total = Wx + (α/r) × BAx = Wx + 0 = Wx

The model starts exactly as the pre-trained model—no initial disruption.

---

Gradient Flow

Computing Gradients

For a loss L, we need gradients for B and A:

∂L/∂B = (α/r) × (∂L/∂h) × x^T × A^T
∂L/∂A = (α/r) × B^T × (∂L/∂h) × x^T

Key Property

Gradients only flow through the LoRA path during training:

• W is frozen (no gradients)
• B and A receive and update gradients

# In code
for name, param in model.named_parameters():
    if "lora" in name.lower():
        param.requires_grad = True
    else:
        param.requires_grad = False

---

Memory Analysis

Training Memory Components

For a weight matrix W ∈ ℝ^(d×k):

Component	Full FT	LoRA
Weights	dk × 2	dk × 2 (frozen)
Gradients	dk × 2	2rk × 2
Optimizer	dk × 8	2rk × 8
Total	12dk	2dk + 20rk

def memory_comparison(d, k, r, bytes_per_param=2):
    """Compare memory usage."""
    # Full FT: weights + grads + optimizer (2 states)
    full_ft = d * k * (2 + 2 + 8)  # 12 bytes per param
    
    # LoRA: frozen weights + LoRA training
    frozen = d * k * 2  # Just weights, no grads/optimizer
    lora = r * (d + k) * (2 + 2 + 8)  # Full training
    lora_total = frozen + lora
    
    print(f"Full FT: {full_ft / 1e9:.2f} GB")
    print(f"LoRA: {lora_total / 1e9:.2f} GB")
    print(f"Savings: {(1 - lora_total/full_ft) * 100:.1f}%")

memory_comparison(4096, 4096, 16)

---

Rank Selection Mathematics

Information Theoretic View

The rank determines how many independent "directions" the update can modify:

• Rank r allows r independent modifications
• Higher rank = more expressive but more parameters

Singular Value Decomposition

Any matrix can be decomposed:

M = UΣV^T

If M has low effective rank (most singular values are small), a low-rank approximation works well.

Practical Selection

def estimate_rank(task_complexity, model_size):
    """Heuristic for rank selection."""
    base_rank = 8  # Good default
    
    # Adjust for complexity
    if task_complexity == "simple":
        rank = max(4, base_rank // 2)
    elif task_complexity == "moderate":
        rank = base_rank
    elif task_complexity == "complex":
        rank = base_rank * 2
    else:
        rank = base_rank * 4
    
    return rank

# Examples
print(estimate_rank("simple", "7B"))    # 4
print(estimate_rank("moderate", "7B"))  # 8
print(estimate_rank("complex", "7B"))   # 16

---

Merging Mathematics

Merge Operation

At inference, we can merge LoRA into the base weights:

def merge_lora(W, B, A, alpha, r):
    """Merge LoRA into base weights."""
    scaling = alpha / r
    W_merged = W + scaling * (B @ A)
    return W_merged

After Merging

h = W_merged × x

Equivalent to:

h = (W + (α/r) × BA) × x

But computed with a single matrix multiplication—no overhead.

---

Summary of Key Equations

Concept	Equation
Forward pass	h = Wx + (α/r)BAx
Parameter count	(d + k) × r
Scaling factor	α/r
Merged weights	W' = W + (α/r)BA
At initialization	BA = 0 (B is zero)

---

Key Takeaways

✅ LoRA approximates weight updates as low-rank matrix products

✅ Rank r controls capacity; α controls magnitude

✅ Effective scaling = α/r; common value is 2

✅ Zero initialization of B ensures clean start

✅ Memory savings come from not storing gradients/optimizer for frozen weights

✅ Merging combines LoRA into base weights for zero overhead

---

Next Steps

Continue to Chapter 3: Implementing LoRA with PEFT for hands-on implementation.

---

Quick Check

Before moving on, make sure you can answer:

1. What is the effective scaling factor for r=16, alpha=32?
2. How many parameters does LoRA add for a 4096×4096 matrix with r=8?
3. Why is B initialized to zero instead of A?