Quiz: Advanced Techniques
30 questions on instruction tuning, DPO, and ORPO.
Question 1
What is instruction tuning?
A) Tuning model architecture
B) Training on instruction-response pairs
C) Optimizing hyperparameters
D) Reducing model size
Show Answer
Correct Answer: B
Instruction tuning trains models on instruction-response pairs to follow user instructions.
Question 2
What does DPO stand for?
A) Direct Parameter Optimization
B) Direct Preference Optimization
C) Dual Policy Optimization
D) Data Processing Optimization
Show Answer
Correct Answer: B
DPO is Direct Preference Optimization, a simpler alternative to RLHF.
Question 3
What type of data does DPO require?
A) Single responses
B) Preference pairs (chosen/rejected)
C) Binary labels
D) Unlabeled text
Show Answer
Correct Answer: B
DPO requires preference pairs with chosen (preferred) and rejected responses.
Question 4
What is ORPO's main innovation?
A) Faster training
B) Combines SFT and preference optimization in one stage
C) No data needed
D) Works only with small models
Show Answer
Correct Answer: B
ORPO combines supervised fine-tuning and preference optimization into a single training stage.
Question 5
What does the beta parameter control in DPO?
A) Learning rate
B) Batch size
C) KL divergence penalty strength
D) Model size
Show Answer
Correct Answer: C
Beta controls the KL divergence penalty, balancing preference optimization vs staying close to the reference model.
(Questions 6-30 cover additional topics)