Quiz: Module 3 - Training LLMs

Test your understanding of LLM training. 30 questions covering pre-training, data processing, and distributed training.

---

Question 1

What is the pre-training objective for GPT-style models?

1. Next token prediction (causal language modeling)
2. Masked language modeling
3. Sentence classification
4. Translation

Question 2

In masked language modeling, what percentage of tokens are typically masked?

1. 5%
2. 15%
3. 50%
4. 80%

Question 3

What is the primary advantage of causal LM over masked LM for LLMs?

1. Faster training
2. Natural fit for text generation
3. Better understanding
4. Less data required

Question 4

Why is deduplication important in training data?

1. To reduce storage costs
2. To prevent memorization and improve generalization
3. To speed up tokenization
4. To improve grammar

Question 5

What is Common Crawl?

1. A web browser
2. A large dataset of web pages commonly used for LLM training
3. A search engine
4. A quality filter

Question 6

What is data parallelism?

1. Each GPU processes different data with the same model
2. Each GPU has different model layers
3. Each GPU processes the same data
4. Splitting weight matrices across GPUs

Question 7

What is the limitation of pure data parallelism?

1. Slow training
2. Model must fit in a single GPU's memory
3. Poor accuracy
4. Requires special hardware

Question 8

In tensor parallelism, what is split across GPUs?

1. Training data
2. Different model layers
3. Individual weight matrices within layers
4. Attention heads only

Question 9

What does ZeRO stand for?

1. Zero Redundancy Optimizer
2. Zero Error Regularization Objective
3. Zoned Efficient Resource Optimization
4. Zero Entropy Reduction Operation

Question 10

What does ZeRO-3 partition?

1. Only optimizer states
2. Optimizer states and gradients
3. Optimizer states, gradients, and parameters
4. Only the training data

Question 11

What is the Chinchilla scaling recommendation for training tokens?

1. 1 token per parameter
2. 10 tokens per parameter
3. ~20 tokens per parameter
4. 100 tokens per parameter

Question 12

What precision is commonly used for LLM training to save memory?

1. FP64
2. FP32
3. BF16 (bfloat16)
4. INT8

Question 13

What is pipeline parallelism?

1. Processing data in batches
2. Placing different layers on different GPUs
3. Parallel tokenization
4. Concurrent data loading

Question 14

Which framework popularized ZeRO?

1. PyTorch
2. TensorFlow
3. DeepSpeed
4. JAX

Question 15

Why is quality filtering important for training data?

1. To reduce training time
2. To improve model quality by removing low-quality text
3. To reduce storage costs
4. To simplify tokenization

Question 16

How many training tokens did LLaMA 2 use?

1. 300 billion
2. 1 trillion
3. 2 trillion
4. 10 trillion

Question 17

What does gradient checkpointing do?

1. Saves gradients to disk
2. Trades compute for memory by recomputing activations
3. Checks gradients for NaN values
4. Compresses gradients

Question 18

What is the main challenge with pipeline parallelism?

1. Memory usage
2. Pipeline bubbles (idle time)
3. Communication overhead
4. Numerical instability

Question 19

What is the role of the learning rate warmup?

1. To cool down the GPUs
2. To prevent training instability at the start
3. To increase the learning rate
4. To reduce overfitting

Question 20

Which optimizer is commonly used for LLM training?

1. SGD
2. AdamW
3. RMSprop
4. Adagrad

Question 21

What does "context length" mean during training?

1. The length of the training run
2. The maximum sequence length the model can process
3. The size of the vocabulary
4. The embedding dimension

Question 22

What is perplexity in the context of LLM evaluation?

1. A measure of model size
2. Exponential of the average cross-entropy loss
3. The number of training steps
4. A complexity metric for code

Question 23

What is the typical batch size for LLM training?

1. 32 samples
2. 256 samples
3. Millions of tokens (using gradient accumulation)
4. 1 sample at a time

Question 24

What is loss spiking during training?

1. Normal behavior
2. Sudden increase in loss indicating training instability
3. A data loading issue
4. A sign of convergence

Question 25

What is the purpose of tokenizer training?

1. To learn an efficient vocabulary for the training corpus
2. To train the embedding layer
3. To filter the training data
4. To validate model outputs

Question 26

What is a common learning rate schedule for LLM training?

1. Constant learning rate
2. Linear increase
3. Warmup followed by cosine decay
4. Exponential increase

Question 27

What is the purpose of weight decay in training?

1. To speed up training
2. To regularize and prevent overfitting
3. To reduce memory usage
4. To increase model capacity

Question 28

What is FSDP?

1. Fully Sharded Data Parallel - PyTorch's native ZeRO implementation
2. Fast Synchronized Data Processing
3. Flexible Storage Data Partitioning
4. Full Scale Distributed Processing

Question 29

What is the benefit of using curriculum learning for LLMs?

1. Training on increasingly complex data can improve learning
2. It reduces training time
3. It eliminates the need for data filtering
4. It increases model size

Question 30

What is the typical duration to train a large LLM (100B+ parameters)?

1. Hours
2. Days
3. Weeks to months
4. Years

---

Quiz Complete!

Passing Score: 24/30 (80%)

← Chapter 3: Distributed Training

Module 4: Major LLM Families →