Skip to main content

Chapter 2: Constitutional AI

Anthropic's Approach​

Instead of labeling each case, define principles:

Constitution:
1. Be helpful, harmless, and honest
2. Avoid deception
3. Respect human autonomy
4. Don't assist with harmful activities

RLAIF (RL from AI Feedback)​

Use AI to generate preference data based on constitution:

1. Generate multiple responses
2. AI evaluates which better follows constitution
3. Train reward model on AI preferences
4. Fine-tune with RL

Self-Improvement Process​

# Model critiques its own response
critique_prompt = f"""
Response: {response}
Principle: {principle}
Does this response follow the principle? How could it be improved?
"""

# Model revises based on critique
revision_prompt = f"""
Original: {response}
Critique: {critique}
Please write a revised response that addresses the critique.
"""

Benefits​

  • Scalable (no human labeling needed for each case)
  • Consistent application of principles
  • Transparent reasoning