Quiz: Module 2 - Transformer Architecture

Test your understanding of the Transformer architecture. This quiz contains 30 questions covering attention mechanisms, multi-head attention, and positional encoding.

---

Chapter 1: Attention Mechanism

Question 1

In the attention formula, what does the Query (Q) represent?

1. What we're looking for or searching for
2. The content to retrieve
3. The index into memory
4. The output of attention

Question 2

Why is the attention score scaled by √d_k?

1. To make the computation faster
2. To prevent dot products from growing too large, which would push softmax into low-gradient regions
3. To normalize the output to unit length
4. To reduce memory usage

Question 3

What is the shape of the attention weights matrix for a sequence of length n?

1. n × d_model
2. d_k × d_v
3. n × n
4. 1 × n

Question 4

In causal (masked) attention, what does the mask prevent?

1. Looking at the same position
2. Looking at future positions
3. Looking at past positions
4. Looking at padding tokens only

Question 5

What is the time complexity of standard attention with respect to sequence length n?

1. O(n)
2. O(n log n)
3. O(n²)
4. O(n³)

Question 6

What value is used to mask out positions in attention?

1. 0
2. 1
3. -infinity (before softmax)
4. A random value

Question 7

What does the Value (V) matrix contain?

1. The search criteria
2. The relevance scores
3. The actual content to be retrieved based on attention weights
4. The position information

Question 8

What does softmax ensure about attention weights?

1. They sum to 1 for each query position
2. They are all equal
3. They are binary (0 or 1)
4. They are always positive integers

Question 9

Flash Attention primarily addresses which problem?

1. Accuracy of attention
2. Memory efficiency of attention
3. Training speed only
4. Model size

Question 10

How is the final attention output computed?

1. Sum of Q, K, and V
2. Product of Q and K
3. Weighted sum of V using attention weights
4. Maximum of attention weights

---

Chapter 2: Self-Attention and Multi-Head Attention

Question 11

What distinguishes self-attention from cross-attention?

1. In self-attention, Q, K, and V all come from the same input
2. Self-attention uses more parameters
3. Self-attention is faster
4. Self-attention only works with text

Question 12

If d_model = 512 and we have 8 attention heads, what is d_k per head?

1. 512
2. 256
3. 128
4. 64

Question 13

What is the main benefit of multi-head attention?

1. It reduces computation time
2. It allows the model to attend to different representation subspaces
3. It uses fewer parameters
4. It eliminates the need for positional encoding

Question 14

In multi-head attention, how are the outputs of different heads combined?

1. Averaged
2. Summed
3. Concatenated and then projected
4. Max pooled

Question 15

Where is cross-attention typically used?

1. Only in encoders
2. In encoder-decoder architectures, where decoder attends to encoder output
3. Only in GPT models
4. Never in Transformers

Question 16

Why can attention heads be computed in parallel?

1. Each head's computation is independent of other heads
2. They share weights
3. They process different sequences
4. They use different activation functions

Question 17

What might different attention heads learn?

1. The same patterns for redundancy
2. Different linguistic patterns (syntax, semantics, etc.)
3. Only positional information
4. Random noise

Question 18

What is the output dimension of multi-head attention with d_model = 512?

1. Depends on number of heads
2. 64 per position
3. 512 per position (same as d_model)
4. 1024 per position

Question 19

Why is self-attention permutation equivariant?

1. Because it uses RNNs internally
2. Because attention weights depend only on pairwise similarities, not positions
3. Because of the softmax function
4. It's not permutation equivariant

Question 20

What is the role of the output projection (W_O) in multi-head attention?

1. To compute attention weights
2. To project concatenated head outputs back to d_model
3. To add positional information
4. To compute the loss

---

Chapter 3: Positional Encoding and Embeddings

Question 21

Why do Transformers need positional encoding?

1. To reduce computation
2. Because self-attention alone is permutation invariant and loses position information
3. To increase model capacity
4. To handle variable-length sequences

Question 22

In sinusoidal positional encoding, what varies across different dimensions?

1. The amplitude
2. The frequency of oscillation
3. The phase shift
4. Nothing - all dimensions are identical

Question 23

What is the range of values for sinusoidal positional encoding?

1. 0 to 1
2. 0 to infinity
3. -1 to 1
4. Unbounded

Question 24

How are token embeddings combined with positional encodings?

1. Concatenated
2. Added together
3. Multiplied
4. Max pooled

Question 25

What is an advantage of learned positional embeddings over sinusoidal?

1. Better extrapolation to longer sequences
2. No parameters required
3. Can adapt to patterns specific to the training data
4. Works with any sequence length

Question 26

What is RoPE (Rotary Position Embedding)?

1. A type of attention mechanism
2. A method that encodes position by rotating embeddings
3. A training objective
4. A type of normalization

Question 27

Which models commonly use RoPE?

1. Original Transformer and BERT
2. LLaMA, Mistral, and many modern LLMs
3. Only GPT-2
4. Only encoder models

Question 28

Why is the token embedding scaled by √d_model in the original Transformer?

1. To reduce memory usage
2. To balance the magnitude of embeddings with positional encodings
3. To normalize gradients
4. To increase model capacity

Question 29

What is ALiBi's approach to position encoding?

1. Adding position vectors to embeddings
2. Rotating query and key vectors
3. Adding distance-based bias directly to attention scores
4. Learning separate position embeddings

Question 30

What is a limitation of learned positional embeddings?

1. They require too much computation
2. They have a fixed maximum sequence length
3. They can't capture relative positions
4. They don't work with multi-head attention

---

Quiz Complete!

Congratulations on completing the Module 2 Quiz! Review any questions you found challenging and revisit the relevant chapter sections.

Passing Score: 24/30 (80%)

---

← Chapter 3: Positional Encoding

Module 3: Training LLMs →