Quiz: Module 8 - RLHF & Alignment

Question 1

What does RLHF stand for?

1. Reinforcement Learning from Human Feedback
2. Recursive Learning from Human Features
3. Refined Language from Human Feedback
4. Reinforced Learning for Human Functions

Question 2

What is the first step in the RLHF pipeline?

1. Pre-train base model
2. Train reward model
3. PPO training
4. Collect human preferences

Question 3

What does a reward model learn?

1. To predict which response humans would prefer
2. To generate better text
3. To classify topics
4. To translate languages

Question 4

What is PPO in RLHF?

1. Proximal Policy Optimization - the RL algorithm used
2. Pre-trained Policy Optimization
3. Parallel Processing Objective
4. Preference Prediction Optimization

Question 5

Why use KL divergence constraint in RLHF?

1. Prevent model from drifting too far from original behavior
2. Speed up training
3. Reduce memory usage
4. Improve accuracy

Question 6

What is Constitutional AI?

1. Training AI using a set of principles instead of case-by-case labeling
2. Government regulations for AI
3. Legal AI applications
4. Constitutional law research by AI

Question 7

What is RLAIF?

1. RL from AI Feedback - using AI to generate preference data
2. Reinforced Learning with AI Functions
3. Recursive Learning AI Framework
4. Real-time Learning AI Interface

Question 8

What is jailbreaking in the context of LLMs?

1. Attempting to bypass safety measures through clever prompts
2. Escaping from a virtual prison game
3. Unlocking model weights
4. Removing rate limits

Question 9

What is the "alignment tax"?

1. Potential reduction in helpfulness due to safety measures
2. Cost of alignment research
3. Government tax on AI
4. Training compute cost

Question 10

What is red teaming?

1. Systematic testing to find vulnerabilities
2. A type of model architecture
3. Training method
4. Color-coded error classification

Questions 11-30

(Covering: SFT vs RLHF, preference data collection, reward hacking, DPO, value alignment, interpretability, robustness, multi-stakeholder alignment, AI governance, emerging alignment techniques)

Question 11

What is SFT in RLHF pipeline?

1. Supervised Fine-Tuning on demonstration data
2. Sequential Feature Training
3. Safe Feedback Training
4. Synthetic Feedback Testing

Question 12

What is reward hacking?

1. Model finding ways to maximize reward without meeting intended goals
2. Attacking reward model
3. Increasing training rewards
4. Hacking techniques generated by AI

Question 13

What is DPO (Direct Preference Optimization)?

1. Alternative to RLHF that directly optimizes on preference data
2. Direct Policy Optimization
3. Data Processing Optimization
4. Distribution Parameter Optimization

Question 14

Why is interpretability important for alignment?

1. To understand why models make certain decisions
2. To make code readable
3. To speed up inference
4. To reduce model size

Question 15

What are the HHH criteria?

1. Helpful, Harmless, Honest
2. High, Heavy, Hard
3. Hybrid, Hierarchical, Holistic
4. Human, Hybrid, Helpful

Question 16

What is the reference model in RLHF?

1. Original SFT model used as baseline for KL penalty
2. The best performing model
3. Human reference
4. Documentation model

Question 17

What makes preference data collection challenging?

1. Subjectivity, expense, and potential biases
2. Technical difficulty
3. Data format
4. Storage requirements

Question 18

What is robustness in alignment context?

1. Maintaining safety under various inputs and distribution shifts
2. Model durability
3. Code stability
4. Hardware reliability

Question 19

What is value learning?

1. Inferring human preferences and values from behavior
2. Learning numerical values
3. Learning market values
4. Currency conversion

Question 20

What is scaling oversight?

1. Maintaining human supervision as AI capabilities grow
2. Monitoring model size
3. Scaling up human teams
4. Viewing scale metrics

Question 21

What made ChatGPT different from GPT-3?

1. RLHF alignment training
2. Larger size
3. Different architecture
4. More training data

Question 22

What is the goal of output filtering?

1. Check generated content for harmful outputs
2. Compress outputs
3. Format responses
4. Speed up generation

Question 23

Why train on adversarial examples?

1. To make model robust against attack attempts
2. To increase model size
3. To speed up training
4. To reduce compute costs

Question 24

What is specification in alignment?

1. Defining precisely what we want AI to do
2. Hardware requirements
3. API documentation
4. Model architecture

Question 25

What is assurance in alignment?

1. Verifying that model has desired properties
2. Insurance for AI
3. Performance guarantee
4. Customer support

Question 26

What is a critique-revision loop in Constitutional AI?

1. Model criticizing and revising its own outputs
2. Human review process
3. Code review
4. Training iteration

Question 27

What problem does RLHF solve that SFT doesn't?

1. Learning relative preferences between outputs
2. Generating text
3. Tokenization
4. Memory efficiency

Question 28

What is AI governance in alignment?

1. Deciding who determines alignment goals and safety standards
2. AI managing itself
3. Government AI
4. AI administration

Question 29

What is the reference model penalty in PPO?

1. β × D_KL(π_θ || π_ref) to prevent drift
2. Training loss
3. Memory cost
4. Compute cost

Question 30

What distinguishes a well-aligned model?

1. High helpfulness AND high safety
2. Maximum safety only
3. Maximum helpfulness only
4. Largest size

---

Passing Score: 24/30 (80%)

← Chapter 3Module 9 →