Chapter 2: Direct Preference Optimization

Simpler Alternative to RLHF

DPO directly optimizes for human preferences without a separate reward model.

---

DPO vs RLHF

Aspect	RLHF	DPO
Reward Model	Required	Not needed
RL Training	PPO/similar	Direct optimization
Complexity	High	Lower
Stability	Can be unstable	More stable

---

Preference Data Format

{
  "prompt": "Explain quantum computing",
  "chosen": "Quantum computing uses qubits...",
  "rejected": "Quantum computers are just faster..."
}

---

DPO Implementation

from trl import DPOTrainer, DPOConfig

dpo_config = DPOConfig(
    beta=0.1,  # KL penalty weight
    learning_rate=5e-7,
    num_train_epochs=1,
    gradient_accumulation_steps=4,
    bf16=True,
)

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,  # Reference model for KL
    args=dpo_config,
    train_dataset=preference_dataset,
    tokenizer=tokenizer,
)

trainer.train()

---

Key Parameter: Beta

• Low beta (0.01-0.05): Stronger optimization
• High beta (0.5-1.0): Stay close to reference
• Default (0.1): Good balance

---

Continue to Chapter 3 →