Chapter 3: Training and Learning Process
How AI Learns to Be Intelligent
Training is the process that transforms a neural network from a blank slate into a powerful AI system. Understanding this process reveals both the capabilities and limitations of AI.
The Three Stages of AI Training
Stage 1: Pre-training
What Happens
The model learns the basic patterns of language by predicting the next word in billions of text samples.
The Task: Next Word Prediction
Input: "The cat sat on the"
Model predicts: "mat" (with 35% probability)
"floor" (with 25% probability)
"roof" (with 15% probability)
...
Simple task, profound results: By learning to predict the next word, the model learns grammar, facts, reasoning patterns, and much more.
Training Data Scale
| Model | Training Data |
|---|---|
| GPT-3 | ~45 TB of text |
| GPT-4 | Estimated 1 PB+ |
| Llama 2 | 2 trillion tokens |
Sources include:
- Web pages (Common Crawl)
- Books
- Wikipedia
- Academic papers
- Code repositories
Computational Requirements
Training large models requires massive resources:
| Resource | Scale |
|---|---|
| GPUs | Thousands of A100/H100 |
| Training Time | Weeks to months |
| Cost | Millions of dollars |
| Energy | Power for a small city |
Example: Training GPT-4 reportedly cost over $100 million.
Stage 2: Fine-tuning
Purpose
Adapt the pre-trained model for specific tasks or behaviors.
Types of Fine-tuning
1. Task-Specific Fine-tuning
Train on examples of the specific task:
- Summarization datasets
- Question-answering pairs
- Code documentation
2. Instruction Fine-tuning
Train the model to follow instructions:
Before:
User: "Translate to French: Hello"
Model: "The word hello can be translated..."
After instruction fine-tuning:
User: "Translate to French: Hello"
Model: "Bonjour"
3. Domain Fine-tuning
Specialize for specific fields:
- Medical terminology
- Legal documents
- Company-specific knowledge
Fine-tuning Process
Stage 3: RLHF (Reinforcement Learning from Human Feedback)
The Problem
Pre-training produces capable but often unhelpful or unsafe outputs.
The Solution
Use human feedback to teach the model what "good" responses look like.
RLHF Process
Step 1: Collect Human Preferences
Humans compare pairs of AI responses and indicate which is better.
Example:
Prompt: "How do I pick a lock?"
Response A: "Here are step-by-step instructions..."
Response B: "I can't help with that. If you're locked out,
contact a locksmith."
Human Ranking: B > A (B is better)
Step 2: Train Reward Model
Create a model that predicts which responses humans prefer.
Step 3: Optimize the AI
Use the reward model to guide the AI toward preferred responses.
Why RLHF Matters
| Before RLHF | After RLHF |
|---|---|
| Capable but unpredictable | More helpful and aligned |
| May follow harmful instructions | Refuses inappropriate requests |
| Verbose and unfocused | Concise and relevant |
| Inconsistent quality | More reliable quality |
Key Training Concepts
Epochs
One complete pass through the training data.
Batch Size
How many examples are processed together.
Learning Rate
How much weights change with each update.
Overfitting
When the model memorizes training data instead of learning patterns.
Signs of overfitting:
- Perfect on training data
- Poor on new data
- Reproduces exact training examples
Underfitting
When the model hasn't learned enough patterns.
Signs of underfitting:
- Poor on training data
- Poor on new data
- Very generic outputs
What Models Actually Learn
Emergent Abilities
Capabilities that appear suddenly at certain scales:
Examples of emergent abilities:
- Arithmetic (appears around 10B parameters)
- Chain-of-thought reasoning (appears around 100B parameters)
- Code generation (improves dramatically with scale)
What They Don't Learn
- True understanding or consciousness
- Real-world consequences of actions
- Common sense that isn't in the data
- Events after training cutoff
Training Challenges
1. Data Quality
"Garbage in, garbage out" applies strongly:
- Biased data → Biased model
- Incorrect data → Incorrect responses
- Limited diversity → Limited capabilities
2. Cost and Resources
- Only large organizations can train frontier models
- Environmental concerns about energy use
- GPU availability constraints
3. Alignment
Ensuring the model does what we intend:
- Follows instructions
- Refuses harmful requests
- Provides accurate information
- Behaves consistently
4. Evaluation
Hard to measure all aspects of performance:
- Benchmarks can be "gamed"
- Real-world behavior differs from tests
- Edge cases are hard to anticipate
Alternative Training Approaches
Constitutional AI (Anthropic's Approach)
Train the AI to critique its own outputs using a set of principles.
DPO (Direct Preference Optimization)
Simpler than RLHF, directly uses preference data without a separate reward model.
Self-Play
Model improves by competing against itself (used in game-playing AI).
The Cost of Training
Financial Cost
| Model | Estimated Cost |
|---|---|
| GPT-3 | $4.6 million |
| GPT-4 | $100+ million |
| Gemini Ultra | Likely similar |
Environmental Cost
Training a large model can produce CO2 equivalent to:
- 5 cars over their lifetime
- Hundreds of flights
Why Open Source Matters
- Distributes training cost
- Enables research without massive budgets
- Models like Llama, Mistral allow wider access
Key Takeaways
✅ Training has three main stages: pre-training, fine-tuning, and RLHF
✅ Pre-training uses next-word prediction on massive text datasets
✅ Fine-tuning adapts models for specific tasks or behaviors
✅ RLHF aligns models with human preferences and values
✅ Training large models costs millions of dollars
✅ What models learn includes emergent abilities that appear at scale
✅ Key challenges include data quality, alignment, and evaluation
Module Summary
Congratulations! You've completed Module 2. You now understand:
- Neural Networks: The foundation of AI systems
- Transformers: The architecture enabling modern AI
- Training Process: How AI learns from data
Next Steps
Test your knowledge with the Module 2 Quiz, then continue to Module 3: Text Generation to explore how these technologies power language models.