Skip to main content

Chapter 3: Safety and Alignment Challenges

Jailbreaking​

Attempts to bypass safety measures:

  • Role-playing prompts
  • Encoding harmful requests
  • Multi-step attacks
  • Token manipulation

Defense Strategies​

  1. Input filtering: Detect malicious prompts
  2. Output filtering: Check generated content
  3. Robust training: Include adversarial examples
  4. Constitutional constraints: Built-in principles

The Alignment Tax​

Safety measures can reduce helpfulness:

ApproachHelpfulnessSafety
UnalignedHighLow
Over-alignedLowHigh
Well-alignedHighHigh

Ongoing Research​

  • Interpretability: Understanding model reasoning
  • Red teaming: Systematic vulnerability testing
  • Scaling oversight: Human supervision as models grow
  • Value learning: Inferring human preferences

Key Challenges​

  1. Specification: What do we want AI to do?
  2. Robustness: Behavior under distribution shift
  3. Assurance: Verifying model properties
  4. Governance: Who decides alignment goals?