Chapter 1: What is QLoRA?
Quantized Low-Rank Adaptation
QLoRA combines 4-bit quantization with LoRA to dramatically reduce memory requirements while maintaining quality.
The QLoRA Innovation
Memory Comparison
| Model | Full FT | LoRA | QLoRA |
|---|---|---|---|
| 7B | ~80GB | ~14GB | ~6GB |
| 13B | ~160GB | ~26GB | ~10GB |
| 70B | ~600GB | ~140GB | ~48GB |
How QLoRA Works
- Load base model in 4-bit (NF4 format)
- Add LoRA adapters in higher precision (bf16/fp16)
- Train only LoRA parameters
- Gradients flow through dequantized weights
from transformers import BitsAndBytesConfig, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model
# 4-bit configuration
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
quantization_config=bnb_config,
device_map="auto",
)
# Add LoRA on top
lora_config = LoraConfig(r=16, lora_alpha=32, ...)
model = get_peft_model(model, lora_config)
Key Components
NF4 (4-bit NormalFloat)
Quantization format designed for normally distributed weights:
- Equal number of values in each bin
- Better for neural network weights than uniform quantization
Double Quantization
Quantizes the quantization constants:
- Saves additional ~0.4 bits per parameter
- ~11% more memory savings
Paged Optimizers
Handles memory spikes during training:
- Pages optimizer states to CPU when needed
- Prevents OOM errors
Quality vs Memory Trade-off
| Method | Memory | Quality (relative) |
|---|---|---|
| Full FT | 100% | 100% |
| LoRA | 18% | 99% |
| QLoRA | 8% | 98% |
QLoRA achieves ~98% of full fine-tuning quality with ~8% of the memory.
When to Use QLoRA
✅ Use QLoRA when:
- GPU has < 24GB VRAM
- Training 7B+ models on consumer hardware
- Cost is a primary concern
- Experimenting and iterating quickly
❌ Consider LoRA/Full FT when:
- Maximum quality is critical
- Have abundant GPU memory
- Deploying merged model (merge overhead)
Key Takeaways
✅ QLoRA = 4-bit quantization + LoRA ✅ Reduces memory by ~60% compared to LoRA ✅ Minimal quality loss (~1-2%) ✅ Enables 70B model training on 48GB ✅ Uses NF4 format and double quantization