Chapter 2: From GPT to GPT-4
A Brief History of Modern LLMs
The journey from early language models to today's powerful AI assistants happened remarkably fast—just about five years transformed what AI can do with language.
What Does GPT Stand For?
GPT = Generative Pre-trained Transformer
- Generative: It generates new text
- Pre-trained: It's trained on lots of data before you use it
- Transformer: The architecture (type of neural network) it uses
The Transformer architecture, introduced in 2017, was the key breakthrough that made modern LLMs possible. We'll explore this more in Module 2.
The GPT Timeline
2018: GPT-1 - The Beginning
| Metric | Value |
|---|---|
| Parameters | 117 million |
| Training Data | ~5 GB of text |
| Capabilities | Basic text completion |
What it could do: Complete sentences, but often lost coherence over longer texts.
Significance: Proved that pre-training on lots of text, then fine-tuning for specific tasks, was powerful.
2019: GPT-2 - "Too Dangerous to Release"
| Metric | Value |
|---|---|
| Parameters | 1.5 billion |
| Training Data | ~40 GB of text |
| Capabilities | Coherent paragraphs |
What it could do: Write coherent paragraphs and short articles that often looked human-written.
The Controversy: OpenAI initially didn't release the full model, worried about misuse for fake news. This sparked debates about AI safety that continue today.
Famous Demo: Given the prompt about unicorns in the Andes, GPT-2 generated a convincing fake news article about scientists discovering unicorns.
2020: GPT-3 - The Game Changer
| Metric | Value |
|---|---|
| Parameters | 175 billion |
| Training Data | ~570 GB of text |
| Capabilities | Few-shot learning |
What it could do:
- Answer questions
- Write code
- Translate languages
- Summarize documents
- All without task-specific training!
The Breakthrough: "Few-shot learning"—you could just show GPT-3 a few examples in the prompt, and it would understand what to do.
Example of Few-Shot Learning:
Translate English to French:
English: Hello → French: Bonjour
English: Thank you → French: Merci
English: Good morning → French:
GPT-3 would correctly output "Bonjour" without any translation-specific training.
2022: ChatGPT - The Moment AI Went Mainstream
| Metric | Value |
|---|---|
| Base Model | GPT-3.5 |
| Key Innovation | Conversational fine-tuning |
| User Milestone | 100 million users in 2 months |
What changed: ChatGPT wasn't just a smarter model—it was trained to have conversations using a technique called RLHF (Reinforcement Learning from Human Feedback).
The ChatGPT Moment:
- Fastest-growing consumer application in history
- Reached 100 million users in 2 months (TikTok took 9 months, Instagram took 2.5 years)
- Made AI accessible to everyone
Why It Felt Different:
- Natural conversation flow
- Could refuse harmful requests
- Admitted when it didn't know something
- Felt like talking to a helpful assistant
2023: GPT-4 - Multimodal and More Capable
| Metric | Value |
|---|---|
| Parameters | Not disclosed (~1 trillion+) |
| Capabilities | Text + vision |
| Performance | Passes many professional exams |
New Capabilities:
- Vision: Can understand images, not just text
- Longer Context: Can process ~128,000 tokens (roughly 300 pages)
- Better Reasoning: Scores in top percentiles on many professional exams
Exam Performance:
| Exam | GPT-3.5 | GPT-4 |
|---|---|---|
| Bar Exam | Bottom 10% | Top 10% |
| SAT Math | 70th percentile | 89th percentile |
| GRE Verbal | 63rd percentile | 99th percentile |
What Improved With Each Version?
1. Coherence and Length
2. Reasoning Ability
- GPT-2: Could pattern match but not reason
- GPT-3: Basic logical reasoning
- GPT-4: Complex multi-step reasoning
3. Instruction Following
- GPT-2: Needed prompts to look like training data
- GPT-3: Could follow simple instructions
- ChatGPT: Designed for natural conversation
- GPT-4: Follows complex, nuanced instructions
4. Safety and Alignment
- Early GPT: Would generate any content
- ChatGPT: Trained to refuse harmful requests
- GPT-4: More nuanced understanding of safety
The Competitive Landscape
The success of ChatGPT triggered an AI race:
2023: The Year of LLM Competition
| Month | Event |
|---|---|
| February | Google announces Bard (now Gemini) |
| March | Anthropic releases Claude |
| March | GPT-4 launches |
| July | Meta releases Llama 2 open source |
| December | Google launches Gemini |
Why Competition Matters:
- Faster innovation
- Lower prices
- More options for users
- Open-source alternatives
Key Milestones Summary
| Year | Milestone | Significance |
|---|---|---|
| 2017 | Transformer architecture | The foundation of all modern LLMs |
| 2018 | GPT-1 | Proved pre-training works |
| 2019 | GPT-2 | Showed concerning capability |
| 2020 | GPT-3 | Demonstrated few-shot learning |
| 2022 | ChatGPT | Made AI accessible to everyone |
| 2023 | GPT-4 | Achieved human-level on many tasks |
| 2023 | Open source explosion | Democratized access to LLM technology |
Why GPT Became Famous
Several factors combined:
- Accessibility: ChatGPT was free and easy to use
- Capability: It could actually do useful things
- Conversational: It felt natural to interact with
- Timing: Remote work made digital tools more important
- Virality: Amazing outputs were shared widely on social media
Key Takeaways
✅ GPT evolved from 117M to 1T+ parameters in 5 years
✅ Each version brought qualitative improvements, not just quantitative
✅ ChatGPT's innovation was in usability, not just capability
✅ GPT-4 added vision and dramatically improved reasoning
✅ Competition is driving rapid innovation across the industry
Think About It
- Why do you think ChatGPT spread faster than any previous technology?
- What might GPT-5 be able to do that GPT-4 can't?
- How might open-source models change the AI landscape?
Next Steps
Now let's look at who else is building LLMs in Chapter 3: Major LLM Providers.