Quiz: LoRA Deep Dive
Test your knowledge about LoRA mathematics, implementation, and best practices.
30 questions - Click on each question to reveal the correct answer.
Question 1
What does LoRA stand for?
A) Low-Rank Adaptation
B) Linear Output Regularization Algorithm
C) Layer-wise Optimization for Representations
D) Large Output Reduction Approach
Show Answer
Correct Answer: A
LoRA stands for Low-Rank Adaptation, referring to the low-rank matrix decomposition used to represent weight updates.
Question 2
How does LoRA represent weight updates?
A) As a single matrix
B) As a product of two low-rank matrices (BA)
C) As a sparse matrix
D) As a vector
Show Answer
Correct Answer: B
LoRA represents weight updates ΔW as a product of two matrices: B (d×r) and A (r×k), where r is much smaller than d and k.
Question 3
Why is the B matrix initialized to zero?
A) To speed up training
B) So BA=0 at start, keeping initial output unchanged
C) To save memory
D) It's randomly initialized, not zero
Show Answer
Correct Answer: B
When B is initialized to zero, BA=0, meaning the model starts with exactly the same behavior as the base model, ensuring stable training start.
Question 4
What is the effective scaling factor for r=16 and alpha=32?
A) 0.5
B) 1.0
C) 2.0
D) 16
Show Answer
Correct Answer: C
The effective scaling is alpha/r = 32/16 = 2.0.
Question 5
For a 4096×4096 weight matrix with r=16, how many LoRA parameters are added?
A) 16,777,216
B) 131,072
C) 65,536
D) 4,194,304
Show Answer
Correct Answer: B
LoRA adds r×d + r×k = 16×4096 + 16×4096 = 131,072 parameters.
Question 6
Which layers are commonly targeted for LoRA in transformers?
A) Only embedding layers
B) Query, Key, Value projections (q_proj, k_proj, v_proj)
C) Only the output layer
D) Layer normalization
Show Answer
Correct Answer: B
The attention projection layers (q_proj, k_proj, v_proj, o_proj) are commonly targeted for LoRA.
Question 7
What is the purpose of the lora_alpha parameter?
A) Setting the learning rate
B) Scaling the contribution of LoRA updates
C) Determining batch size
D) Controlling dropout rate
Show Answer
Correct Answer: B
lora_alpha is a scaling factor that controls the magnitude of LoRA's contribution. The effective scaling is alpha/r.
Question 8
What function is used to apply LoRA to a model?
A) apply_lora()
B) get_peft_model()
C) create_lora()
D) add_adapter()
Show Answer
Correct Answer: B
get_peft_model(base_model, lora_config) is used to wrap a base model with LoRA configuration.
Question 9
What does model.merge_and_unload() do?
A) Saves the model to disk
B) Merges LoRA weights into base model and removes adapter
C) Loads a new adapter
D) Clears GPU memory
Show Answer
Correct Answer: B
merge_and_unload() merges the LoRA adapter weights into the base model weights, creating a standard model without adapter overhead.
Question 10
What is the typical learning rate for LoRA fine-tuning?
A) 1e-2
B) 1e-4 to 5e-4
C) 1e-6
D) 1.0
Show Answer
Correct Answer: B
LoRA fine-tuning typically uses learning rates in the 1e-4 to 5e-4 range, higher than full fine-tuning.
Question 11
What is the recommended starting rank for LoRA experiments?
A) r=1
B) r=16
C) r=256
D) r=1024
Show Answer
Correct Answer: B
r=16 is a common starting point, providing a good balance between capacity and efficiency.
Question 12
What does "low rank" mean in LoRA?
A) The model is ranked low in quality
B) The matrices have small dimensionality (r much smaller than d)
C) The learning rate is low
D) The model has few layers
Show Answer
Correct Answer: B
"Low rank" means the intermediate dimension r is much smaller than the original dimensions (r ≪ d), resulting in fewer parameters.
Question 13
After merging, what is the inference overhead of LoRA?
A) 10% slower
B) 50% slower
C) Zero (identical to base model)
D) 2x slower
Show Answer
Correct Answer: C
After merging, the LoRA weights become part of the base model weights, resulting in zero additional overhead at inference.
Question 14
What is the relationship between alpha and rank typically?
A) Alpha should be much smaller than rank
B) Alpha is typically 1-2x the rank
C) Alpha equals rank squared
D) They are unrelated
Show Answer
Correct Answer: B
Common practice is to set alpha to 1-2x the rank (e.g., r=16, alpha=32).
Question 15
How do you check trainable parameters in a PEFT model?
A) model.count_params()
B) model.print_trainable_parameters()
C) model.trainable()
D) model.params()
Show Answer
Correct Answer: B
model.print_trainable_parameters() displays the count and percentage of trainable parameters.
Question 16
What is lora_dropout used for?
A) Reducing model size
B) Regularization to prevent overfitting
C) Speeding up training
D) Memory optimization
Show Answer
Correct Answer: B
lora_dropout applies dropout regularization to the LoRA layers to help prevent overfitting.
Question 17
What is the bias parameter in LoraConfig?
A) Setting the learning rate bias
B) Whether to train bias terms in LoRA layers
C) Data sampling bias
D) Gradient bias correction
Show Answer
Correct Answer: B
The bias parameter controls whether to train bias terms: "none" (don't train), "all" (train all), or "lora_only" (train only LoRA biases).
Question 18
What is the forward pass equation in LoRA?
A) h = Wx
B) h = Wx + BAx
C) h = BA + Wx
D) h = W + BA
Show Answer
Correct Answer: B
The forward pass is h = Wx + (α/r)BAx, combining the frozen base path with the trainable LoRA path.
Question 19
What makes LoRA memory efficient during training?
A) Compressing the model
B) Only storing gradients/optimizer for small adapters, not frozen weights
C) Using int8 by default
D) Reducing batch size automatically
Show Answer
Correct Answer: B
Memory savings come from not storing gradients and optimizer states for the frozen base weights—only the small LoRA parameters need them.
Question 20
Can multiple LoRA adapters be used on one base model?
A) No, only one adapter allowed
B) Yes, using add_adapter and set_adapter
C) Only two adapters
D) Only during training
Show Answer
Correct Answer: B
Multiple adapters can be added with add_adapter() and switched using set_adapter().
Question 21
What is TaskType.CAUSAL_LM used for?
A) Classification tasks
B) Causal language modeling (text generation)
C) Image generation
D) Translation only
Show Answer
Correct Answer: B
TaskType.CAUSAL_LM indicates the model is used for causal language modeling, which is standard text generation.
Question 22
What should you do if training loss is NaN?
A) Increase learning rate
B) Lower learning rate, enable gradient clipping
C) Use larger batch size
D) Train longer
Show Answer
Correct Answer: B
NaN loss usually indicates numerical instability. Solutions include lowering learning rate, enabling gradient clipping, and checking for data issues.
Question 23
What is the typical LoRA adapter file size for a 7B model?
A) ~15GB
B) ~1GB
C) ~10-50MB
D) ~500MB
Show Answer
Correct Answer: C
LoRA adapter files are typically 10-50MB, compared to full models at ~15GB+ for 7B parameters.
Question 24
What is DoRA?
A) A dataset format
B) Weight-Decomposed Low-Rank Adaptation
C) A type of optimizer
D) A model architecture
Show Answer
Correct Answer: B
DoRA (Weight-Decomposed Low-Rank Adaptation) is a LoRA variant that separates weight magnitude and direction.
Question 25
How do you save only the LoRA adapter?
A) model.save_pretrained(path)
B) model.save_weights(path)
C) model.save_adapter(path)
D) Both A and C work
Show Answer
Correct Answer: A
model.save_pretrained(path) on a PEFT model saves only the adapter by default, not the base model.
Question 26
What is the purpose of gradient checkpointing with LoRA?
A) Saving training checkpoints
B) Trading compute for memory to reduce VRAM usage
C) Faster training
D) Better quality
Show Answer
Correct Answer: B
Gradient checkpointing recomputes activations during backward pass instead of storing them, reducing memory usage at the cost of additional computation.
Question 27
What parameter determines which layers get LoRA?
A) r
B) lora_alpha
C) target_modules
D) lora_dropout
Show Answer
Correct Answer: C
The target_modules parameter specifies which weight matrices receive LoRA adapters.
Question 28
What is the formula for parameter count in LoRA?
A) d × k
B) r × (d + k)
C) r × r
D) d + k
Show Answer
Correct Answer: B
LoRA adds r×d (for B) + r×k (for A) = r×(d+k) parameters per weight matrix.
Question 29
Why does LoRA work well for fine-tuning?
A) It uses more parameters
B) Fine-tuning updates have low intrinsic dimensionality
C) It trains faster always
D) It was designed by OpenAI
Show Answer
Correct Answer: B
Research shows that fine-tuning updates often lie in a low-dimensional subspace, meaning they can be effectively represented with low-rank matrices.
Question 30
What is the recommended approach for production deployment?
A) Keep adapter separate always
B) Merge adapter into base model for simpler deployment
C) Use full fine-tuning instead
D) Never use LoRA in production
Show Answer
Correct Answer: B
For production, merging the adapter into the base model simplifies deployment and eliminates adapter overhead at inference.
Results
| Score | Level |
|---|---|
| 27-30 | Excellent - LoRA master! |
| 21-26 | Good - Review implementation details |
| 15-20 | Average - Revisit all chapters |
| Below 15 | Needs comprehensive review |