Chapter 1: GPT Series (OpenAI)
Evolution Overview
| Model | Year | Parameters | Key Innovation |
|---|---|---|---|
| GPT-1 | 2018 | 117M | Pre-training + fine-tuning |
| GPT-2 | 2019 | 1.5B | Zero-shot capabilities |
| GPT-3 | 2020 | 175B | In-context learning |
| GPT-3.5 | 2022 | ~175B | RLHF, ChatGPT |
| GPT-4 | 2023 | ~1.8T* | Multimodal, MoE* |
*Estimated
GPT-3: The Inflection Point
GPT-3 demonstrated that scale leads to emergent capabilities:
# Few-shot learning example
prompt = """
Q: What is the capital of France?
A: Paris
Q: What is the capital of Germany?
A: Berlin
Q: What is the capital of Italy?
A:"""
# Model completes: "Rome"
GPT-4 Capabilities
- Multimodal: Processes images and text
- 128K context: Handles book-length documents
- Improved reasoning: Better at complex tasks
- Safety improvements: More aligned with human values
Key Takeaways
Summary
- GPT evolution shows the power of scaling
- RLHF made GPT-3.5/4 more useful and aligned
- GPT-4 introduced multimodal capabilities