Skip to main content

Chapter 1: Reinforcement Learning from Human Feedback

The RLHF Pipeline​

1. Pre-train base model
2. Supervised fine-tuning (SFT) on demonstrations
3. Train reward model on human preferences
4. Fine-tune with PPO using reward model

Reward Model Training​

Humans compare pairs of responses:

# Human preference data
{
  "prompt": "Explain quantum computing",
  "chosen": "Good explanation...",
  "rejected": "Poor explanation..."
}

# Reward model learns: reward(chosen) > reward(rejected)

PPO Training​

from trl import PPOTrainer, PPOConfig

ppo_config = PPOConfig(
    learning_rate=1.41e-5,
    batch_size=256,
    mini_batch_size=1,
)

ppo_trainer = PPOTrainer(
    model=model,
    ref_model=ref_model,
    tokenizer=tokenizer,
    config=ppo_config,
)

KL Divergence Constraint​

Prevent model from drifting too far from original:

L=−E[r(x,y)]+β⋅DKL(πθ∣∣πref)\mathcal{L} = -\mathbb{E}[r(x,y)] + \beta \cdot D_{KL}(\pi_\theta || \pi_{ref})