Quiz: Module 10 - Evaluation & Benchmarks

Question 1

What does MMLU measure?

1. Knowledge across 57 academic subjects
2. Code generation
3. Language translation
4. Image recognition

Question 2

What is HumanEval for?

1. Evaluating code generation ability
2. Human preference collection
3. Evaluating writing quality
4. Testing chatbots

Question 3

What is pass@k metric?

1. Probability of correct code in k attempts
2. Password attempts
3. Passing grade threshold
4. API rate limit

Question 4

What is red teaming?

1. Adversarial testing to find vulnerabilities
2. Team building exercise
3. Color-coded testing
4. Performance benchmarking

Question 5

What is data contamination in benchmarks?

1. Model trained on benchmark test data
2. Corrupted training data
3. Noisy evaluation data
4. Data privacy breach

Question 6

What does HellaSwag test?

1. Commonsense reasoning via sentence completion
2. Mathematical ability
3. Code generation
4. Translation quality

Question 7

Why use LLM as judge?

1. Scalable evaluation without human labelers
2. More accurate than humans
3. Required by regulations
4. Faster training

Question 8

What is TruthfulQA for?

1. Testing factual accuracy and avoiding false claims
2. Testing question answering speed
3. Testing coding ability
4. Testing translation

Question 9

What is prompt injection?

1. Hiding malicious instructions in input
2. SQL injection for LLMs
3. Adding prompts to training
4. Prompt engineering technique

Question 10

What is responsible disclosure?

1. Privately reporting vulnerabilities before public disclosure
2. Publishing all findings immediately
3. Never reporting issues
4. Disclosure to competitors

Questions 11-30

(Covering: GSM8K, WinoGrande, ARC, benchmark suites, evaluation frameworks, A/B testing, statistical significance, inter-rater reliability, calibration, perplexity, BLEU/ROUGE, human preference studies, safety benchmarks, multilingual evaluation, domain-specific evaluation, leaderboard gaming, evaluation costs, sampling strategies, evaluation pipelines, continuous evaluation)

Question 11

What does GSM8K test?

1. Grade school math word problems
2. Grammar
3. Spelling
4. Memorization

Question 12

What is perplexity?

1. Measure of how well model predicts text (lower is better)
2. Model confusion
3. Training difficulty
4. Dataset complexity

Question 13

What is BLEU used for?

1. Evaluating text similarity (often translation)
2. Color detection
3. Speed measurement
4. Safety testing

Question 14

What is inter-rater reliability?

1. Agreement between human evaluators
2. Model consistency
3. API reliability
4. Benchmark stability

Question 15

What is calibration in LLM evaluation?

1. Whether confidence matches actual accuracy
2. Hardware calibration
3. Weight initialization
4. Learning rate tuning

Question 16

Why is human evaluation gold standard?

1. Captures nuances automated metrics miss
2. Cheapest option
3. Fastest method
4. Most scalable

Question 17

What is benchmark saturation?

1. Models reaching near-perfect scores, limiting differentiation
2. Too many benchmarks
3. Data overflow
4. Memory limits

Question 18

What is WinoGrande?

1. Commonsense reasoning benchmark using pronoun resolution
2. Wine classification dataset
3. Image benchmark
4. Audio benchmark

Question 19

What makes evaluation expensive?

1. Compute costs and human annotation time
2. Data storage only
3. Benchmark licenses
4. Software costs

Question 20

What is continuous evaluation?

1. Ongoing assessment as model is updated
2. 24/7 monitoring
3. Infinite loops
4. Streaming evaluation

Question 21

What is A/B testing for LLMs?

1. Comparing two versions with real users
2. Testing alphabetically
3. Binary classification
4. Alphabet recognition

Question 22

What is statistical significance in evaluation?

1. Confidence that difference isn't due to chance
2. Statistical modeling
3. Significance level
4. P-value calculation

Question 23

What is evaluation pipeline?

1. Automated system for running evaluations
2. Data pipeline
3. Training pipeline
4. Deployment pipeline

Question 24

What is leaderboard gaming?

1. Optimizing specifically for benchmark scores
2. Playing games
3. Competitive gaming AI
4. Gamification

Question 25

What is ROUGE used for?

1. Evaluating summarization quality
2. Color detection
3. Error detection
4. Speed testing

Question 26

What is multilingual evaluation challenge?

1. Different benchmarks and quality standards across languages
2. Translation only
3. Character encoding
4. Font rendering

Question 27

What is domain-specific evaluation?

1. Testing on specialized fields (medical, legal, etc.)
2. Web domain testing
3. Network domain
4. Math domain

Question 28

What is sampling strategy in evaluation?

1. How to select test examples
2. Temperature sampling
3. Top-p sampling
4. Data collection

Question 29

What is the ARC benchmark?

1. AI2 Reasoning Challenge - science questions
2. Arc welding dataset
3. Architecture benchmark
4. Arithmetic test

Question 30

What is preference data collection challenge?

1. Subjectivity and annotator disagreement
2. Storage size
3. Format compatibility
4. Privacy only

---

Passing Score: 24/30 (80%)

← Chapter 3Module 11 →