Chapter 2: Predicting the Next Word
The Core Mechanism of LLMs
Everything an LLM does—answering questions, writing stories, coding—comes down to one simple operation: predicting what word should come next.
The Basic Idea
An LLM generates text by repeatedly asking: "Given everything so far, what word is most likely to come next?"
A Simple Example
Prompt: "The capital of France is"
The model calculates probabilities:
- "Paris" → 95% likely
- "Lyon" → 2% likely
- "Berlin" → 0.1% likely
- "pizza" → 0.0001% likely
It picks "Paris" (or randomly samples weighted by probability).
How Prediction Creates Conversation
Let's trace how a question becomes an answer:
Your Input
"What color is the sky?"
The Model's Process
- Sees: "What color is the sky?"
- Predicts next word: "The" (likely start of an answer)
- Sees: "What color is the sky? The"
- Predicts: "sky" (makes sense in context)
- Sees: "What color is the sky? The sky"
- Predicts: "is" (grammatically correct)
- Predicts: "blue" (the answer pattern it's learned)
- Continues until: End of response detected
Output: "The sky is blue."
Why It Seems Intelligent
This simple mechanism produces intelligent-seeming behavior because:
1. Patterns Encode Knowledge
When trained on text like:
- "The sky is blue during the day"
- "The sky appears blue because of light scattering"
- "Blue sky is a sign of good weather"
The model learns that "blue" strongly follows "sky is" in question-answer contexts.
2. Context Shapes Prediction
The same word has different predictions based on context:
| Context | Likely Next Word |
|---|---|
| "The bank of the river..." | "was" (nature context) |
| "The bank where I deposit..." | "money" (financial context) |
| "Please bank the ball..." | "shot" (sports context) |
3. Scale Enables Reasoning
With enough parameters and training, the model can follow logical patterns:
"If A causes B, and B causes C, then A causes ___" → "C"
Probability and Randomness
Temperature: Controlling Creativity
LLMs have a "temperature" setting that affects how they choose the next word:
| Temperature | Behavior | Use Case |
|---|---|---|
| Low (0.1) | Always picks the most likely word | Factual answers, code |
| Medium (0.7) | Some variety, still coherent | General conversation |
| High (1.0+) | More random, creative, risky | Creative writing, brainstorming |
Example at Different Temperatures
Prompt: "The weather today is"
- Low temp: "sunny" (picked 10/10 times)
- Medium temp: "sunny" (7/10), "beautiful" (2/10), "nice" (1/10)
- High temp: "spectacular" (3/10), "reminiscent of spring" (2/10), "unexpectedly mild" (2/10)...
The Statistical Nature of LLMs
LLMs Are Probability Machines
This Explains Many Behaviors
Why LLMs can be "creative": They can choose less-likely-but-valid words.
Why outputs vary: Random sampling means different runs can give different answers.
Why they can be wrong: High probability ≠ correct. "Confident" patterns aren't always true.
Emergent Behaviors
When you combine:
- Billions of parameters
- Trillions of training examples
- Next-word prediction
You get capabilities that weren't explicitly programmed:
Emergent Ability Examples
| Ability | How It Emerges |
|---|---|
| Translation | Learned patterns from multilingual text |
| Code generation | Learned patterns from code repositories |
| Math | Learned patterns from math problems and solutions |
| Reasoning | Learned patterns from logical arguments |
| Summarization | Learned to condense based on summary examples |
Why This Works So Well
The Compression Hypothesis
To predict text well, a model must implicitly understand:
- Grammar and syntax
- Facts about the world
- Logic and reasoning
- Different writing styles
- Context and conversation flow
Prediction forces the model to learn all of this.
The Scaling Hypothesis
As models get larger:
- They learn more patterns
- They combine patterns more effectively
- New capabilities "emerge" at certain scales
- Performance keeps improving (so far)
Limitations of Prediction
1. Pattern Matching, Not Reasoning
The model finds patterns, not truth. If false information appears in patterns, it can be reproduced.
2. No Self-Awareness
The model doesn't "know" it's generating text. It's just predicting.
3. Hallucination Risk
When no strong pattern exists, the model generates plausible-sounding but potentially false text.
4. Prompt Sensitivity
Small changes in input can cause big changes in output.
The Autoregressive Loop
Autoregressive means the model's output feeds back as input for the next prediction.
Input: "Hello"
Predict: "there"
New input: "Hello there"
Predict: "!"
New input: "Hello there!"
Predict: "How"
New input: "Hello there! How"
...and so on
This is why:
- Long outputs are generated piece by piece
- Early errors can compound
- The model can "see" its own previous output
Key Takeaways
✅ Everything an LLM does is next-word prediction
✅ Probabilities are calculated for all possible next words
✅ Temperature controls randomness vs. predictability
✅ Scale and training enable emergent abilities
✅ Prediction ≠ understanding, leading to hallucinations
✅ Each word generated becomes input for the next prediction
Think About It
- If an LLM predicts "Paris" as 95% likely for "capital of France is ___", where does that 95% come from?
- Why might high temperature be good for creative writing but bad for factual questions?
- How does the autoregressive loop explain why early mistakes in a response can lead to more mistakes?
Next Steps
Now let's understand how LLMs "remember" conversation in Chapter 3: Context and Memory.