Quiz: Module 5 - Tokenization & Embeddings

Question 1

What is BPE (Byte Pair Encoding)?

1. A tokenization algorithm that iteratively merges frequent character pairs
2. A type of neural network
3. An embedding method
4. A compression algorithm for models

Question 2

Which tokenizer is used by BERT?

1. BPE
2. WordPiece
3. SentencePiece
4. Unigram

Question 3

What does SentencePiece treat text as?

1. Words
2. Characters
3. Raw bytes (language-agnostic)
4. Sentences

Question 4

What was GPT-3's context window?

1. 2K tokens
2. 4K tokens
3. 8K tokens
4. 32K tokens

Question 5

What is Claude 3's context window?

1. 32K tokens
2. 100K tokens
3. 200K tokens
4. 1M tokens

Question 6

Why use chunking with overlap?

1. To reduce processing time
2. To preserve context at chunk boundaries
3. To increase accuracy
4. To reduce token count

Question 7

What does RoPE stand for?

1. Rotary Position Embedding
2. Recursive Operator for Position Encoding
3. Random Orthogonal Position Encoding
4. Relative Order Position Embedding

Question 8

What do embeddings represent?

1. Token IDs
2. Dense vectors capturing semantic meaning
3. Probabilities
4. Attention weights

Question 9

How is semantic similarity typically measured?

1. Euclidean distance
2. Cosine similarity
3. Manhattan distance
4. Hamming distance

Question 10

What is a key application of embeddings?

1. Model training
2. Semantic search and RAG
3. Tokenization
4. Attention computation

Question 11

What's the typical vocabulary size for LLMs?

1. 1,000 tokens
2. 10,000 tokens
3. 30,000-50,000 tokens
4. 1,000,000 tokens

Question 12

What's the advantage of subword tokenization?

1. Handles unknown words by breaking them into known subwords
2. Faster processing
3. Smaller model size
4. Better grammar

Question 13

What does the ## prefix indicate in WordPiece?

1. Start of word
2. Continuation of a word (not start)
3. Special token
4. Unknown token

Question 14

What model has achieved 1M+ context?

1. GPT-4
2. Claude 3
3. Gemini 1.5
4. LLaMA 3

Question 15

What is the embedding dimension for GPT-3?

1. 768
2. 1024
3. 12288
4. 4096

Question 16-30

(Additional questions covering tokenization edge cases, embedding applications, context window management, and practical implementations)

Question 16

What happens when a word is not in the tokenizer's vocabulary?

1. It's ignored
2. It's broken into known subwords or characters
3. An error is raised
4. It's replaced with [UNK]

Question 17

What is vocabulary size's effect on model?

1. Larger vocab = more parameters in embedding layer
2. No effect
3. Only affects training speed
4. Only affects inference speed

Question 18

Which library is popular for sentence embeddings?

1. sentence-transformers
2. spaCy
3. NLTK
4. gensim

Question 19

What's the purpose of special tokens like [CLS] and [SEP]?

1. Mark sentence boundaries and enable special tasks
2. Reduce vocabulary size
3. Speed up tokenization
4. Improve accuracy

Question 20

What is token merging in tokenizers?

1. Combining frequent token pairs into single tokens
2. Removing duplicate tokens
3. Combining embeddings
4. Reducing vocabulary

Question 21

Why is byte-level tokenization beneficial?

1. Can represent any input without UNK tokens
2. Faster processing
3. Smaller vocabulary
4. Better accuracy

Question 22

What is the relationship between tokens and words?

1. 1 token = 1 word always
2. 1 word can be multiple tokens; 1 token can be partial word
3. 1 token = 1 character
4. 1 token = 1 sentence

Question 23

Approximately how many tokens equal one word in English?

1. 0.5 tokens
2. 1 token
3. ~1.3-1.5 tokens
4. 3 tokens

Question 24

What is a token's embedding learned from?

1. Manual labeling
2. Training data during model training
3. Dictionary definitions
4. User feedback

Question 25

What happens at context window limit?

1. Model cannot process additional tokens; truncation or chunking needed
2. Processing slows down
3. Model automatically summarizes
4. Quality improves

Question 26

What is the attention complexity issue with long contexts?

1. O(n²) memory and compute for sequence length n
2. O(n) memory usage
3. No complexity issue
4. Only affects training

Question 27

What is ALiBi used for?

1. Position encoding through attention biases
2. Tokenization
3. Embedding compression
4. Vocabulary building

Question 28

What are position embeddings added to?

1. Token embeddings
2. Attention weights
3. Output logits
4. Loss function

Question 29

What is embedding quantization?

1. Reducing precision of embedding values to save memory
2. Counting embeddings
3. Measuring embedding quality
4. Normalizing embeddings

Question 30

What is the benefit of using the same tokenizer for all LLaMA variants?

1. Consistent token-to-text mapping and easier model switching
2. Faster tokenization
3. Better accuracy
4. Smaller vocabulary

---

Passing Score: 24/30 (80%)

← Chapter 3Module 6 →