Chapter 1: Scaling Laws and Limits
Kaplan Scaling Laws (2020)
OpenAI found power-law relationships:
Where loss decreases predictably with model size.
Chinchilla Scaling (2022)
DeepMind's compute-optimal finding:
Previous wisdom: Train larger models on fixed data Chinchilla: Train smaller models on more data
| Approach | Params | Tokens | Compute |
|---|---|---|---|
| GPT-3 | 175B | 300B | High |
| Chinchilla | 70B | 1.4T | Same |
| Result | Chinchilla wins |
Rule of thumb: ~20 tokens per parameter
Diminishing Returns Debate
Are we hitting limits?
Arguments for limits:
- Benchmark saturation
- Emergent capabilities slowing
- Data availability constraints
Arguments against:
- New capabilities still emerging
- Algorithmic improvements
- Multimodal and reasoning advances
Efficiency Improvements
Beyond raw scaling:
- Better architectures (MoE)
- Improved training (curriculum)
- Data quality > quantity
- Algorithmic innovations