Chapter 3: ORPO and Other Methods
Cutting-Edge Preference Optimization
ORPO combines SFT and preference optimization in a single stage.
ORPO (Odds Ratio Preference Optimization)
Key Innovation
No separate SFT step needed—combines both objectives:
from trl import ORPOTrainer, ORPOConfig
orpo_config = ORPOConfig(
beta=0.1,
learning_rate=8e-6,
num_train_epochs=1,
bf16=True,
)
trainer = ORPOTrainer(
model=model,
args=orpo_config,
train_dataset=dataset,
tokenizer=tokenizer,
)
Method Comparison
| Method | SFT Step | Preference Step | Complexity |
|---|---|---|---|
| RLHF | Separate | PPO + Reward Model | High |
| DPO | Separate | Direct optimization | Medium |
| ORPO | Combined | Combined | Low |
Other Methods
KTO (Kahneman-Tversky Optimization)
Uses binary feedback (good/bad) instead of preferences.
IPO (Identity Policy Optimization)
Improves DPO with better theoretical grounding.
SimPO (Simple Preference Optimization)
Removes need for reference model.
Choosing a Method
| Scenario | Recommended |
|---|---|
| Have preference pairs | DPO or ORPO |
| Binary feedback only | KTO |
| Simple pipeline | ORPO |
| Maximum control | DPO |