Chapter 1: Understanding Language Models
From Autocomplete to Conversation
You use language models every day—probably without realizing it. When your phone suggests the next word as you type, that's a simple language model at work. Large Language Models are the same concept, but dramatically more powerful.
What Is a Language Model?
A language model is a computer program that predicts what text comes next. Give it a beginning, and it predicts a continuation.
The Autocomplete Analogy
Think of your phone's autocomplete:
- You type "Good mor..."
- Your phone suggests "morning"
- You type "How are..."
- Your phone suggests "you"
This is a language model in its simplest form. It's learned from millions of text messages that "Good mor..." is usually followed by "morning."
LLMs are the same idea—but instead of predicting single words, they can predict entire paragraphs, essays, or even books.
How Prediction Becomes Conversation
Here's the key insight: generating text is just predicting what comes next, one piece at a time.
When you ask ChatGPT "What is the capital of France?", here's what happens:
- The model sees your question
- It predicts the most likely next word: "The"
- Then the next: "capital"
- Then: "of"
- Then: "France"
- Then: "is"
- Then: "Paris"
- And so on...
Each word is a prediction based on everything that came before. The magic is that after training on billions of examples, these predictions become remarkably coherent and useful.
Why "Large" Matters
The word "large" in Large Language Model refers to two things:
1. Size of the Model (Parameters)
| Model | Parameters | Analogy |
|---|---|---|
| Simple autocomplete | Thousands | A pocket dictionary |
| GPT-2 (2019) | 1.5 billion | A small library |
| GPT-3 (2020) | 175 billion | A massive university library |
| GPT-4 (2023) | ~1 trillion+ | Every library in the world combined |
Parameters are like the model's memory cells. More parameters means the model can store more patterns and relationships.
2. Size of the Training Data
Modern LLMs are trained on:
- Billions of web pages
- Millions of books
- Scientific papers
- Code repositories
- Wikipedia (all languages)
- News articles
- And much more...
The Scale Changes Everything
Something remarkable happens when you make these models large enough: emergent abilities appear.
What Are Emergent Abilities?
These are capabilities that weren't explicitly programmed but "emerge" from scale:
- Reasoning: Working through logic problems step by step
- Code generation: Writing functional programs
- Translation: Converting between languages
- Summarization: Condensing long documents
- Creative writing: Generating stories, poems, and more
Small models can't do these things. Large models suddenly can—it's like a phase transition in physics.
From Calculator to Assistant
As models grew larger:
- 2018: Could complete sentences
- 2019: Could write coherent paragraphs
- 2020: Could answer questions and have conversations
- 2022: Could reason, code, and create in sophisticated ways
- 2023: Could process images, audio, and more
What LLMs Actually "Know"
Here's an important distinction:
LLMs Don't Store Facts Like a Database
They don't have a table saying "Capital of France = Paris." Instead, they've learned that when text starts with "The capital of France is...", it's almost always followed by "Paris."
They've Learned Patterns, Not Memorized Facts
This is why LLMs can:
- Generate creative content they've never seen
- Combine ideas in new ways
- Sometimes make mistakes about "facts"
Think of it like this: A well-read person doesn't memorize every sentence they've ever read. Instead, they develop an understanding of how language works and what concepts relate to each other.
Real-World Examples
Example 1: Email Drafting
You start typing: "I hope this email finds you..." The LLM predicts: "...well. I'm reaching out regarding our upcoming meeting..."
Example 2: Code Completion
You write: def calculate_average(numbers):
The LLM predicts the entire function based on the name and common patterns.
Example 3: Creative Writing
You prompt: "Write a haiku about programming" The LLM generates a poem that follows haiku structure because it's learned these patterns.
Key Takeaways
✅ Language models predict what text comes next
✅ "Large" means billions of parameters and vast training data
✅ Scale enables emergent abilities like reasoning and creation
✅ LLMs learn patterns, not facts—this has important implications
✅ Every response is a series of predictions, one token at a time
Think About It
Before moving on, consider:
- When you use autocomplete on your phone, what patterns has it learned from your typing?
- Why might predicting "the next word" lead to coherent paragraphs?
- What might go wrong when a model is just predicting patterns?
Next Steps
Now that you understand what language models are, let's explore how they evolved in Chapter 2: From GPT to GPT-4.