Chapter 3: Context and Memory
How LLMs "Remember" Your Conversation
When you chat with an LLM, it seems to remember what you said earlier. But how does this work? The answer involves understanding the context window—the model's short-term memory.
What Is the Context Window?
The context window is the maximum amount of text an LLM can "see" at once when generating a response.
Think of It Like a Window
Imagine you're reading a long document through a small window. You can only see a portion at a time. As you slide the window, you see new text, but you lose sight of old text.
That's exactly how context windows work for LLMs.
Context Window Sizes
Different models have different "window" sizes:
| Model | Context Window | Roughly Equivalent To |
|---|---|---|
| GPT-3.5 | 4K tokens | ~3,000 words (6 pages) |
| GPT-4 | 8K-128K tokens | 6-100 pages |
| Claude 3 | 200K tokens | ~150,000 words (a novel) |
| Gemini 1.5 | 1M tokens | Multiple books |
What Are Tokens?
Tokens are pieces of words—not exactly words, but close:
| Text | Approximate Tokens |
|---|---|
| "Hello" | 1 token |
| "Unbelievable" | 3 tokens (un-believ-able) |
| "ChatGPT is amazing" | ~4-5 tokens |
| 1 page of text | ~500-750 tokens |
How Context Creates "Memory"
The Illusion of Memory
LLMs don't actually have persistent memory. Instead:
- Your message + entire conversation history is sent each time
- The model sees everything in the context window
- It generates a response based on all of this
What Gets Sent to the Model
[System prompt: You are a helpful assistant...]
[User: Hello!]
[Assistant: Hi there! How can I help you?]
[User: I'm learning about AI]
[Assistant: Great! What would you like to know?]
[User: What is machine learning?] ← Current message
All of this goes to the model every time, not just your last message.
The Sliding Window Effect
When Context Gets Too Long
If your conversation exceeds the context window:
This is why:
- Very long conversations might "forget" early details
- The AI might ask about something you already told it
- Summarization can help preserve important context
Why LLMs "Forget"
1. Context Window Overflow
Oldest messages get pushed out when the window is full.
2. No Permanent Storage
Between sessions, nothing is retained (unless the app saves it).
3. Attention Dilution
Even within the window, the model pays more attention to recent and prominent content.
The System Prompt
A system prompt is hidden instructions that shape how the AI behaves throughout the conversation.
Example System Prompts
| Application | System Prompt Might Say |
|---|---|
| ChatGPT | "You are a helpful assistant..." |
| Customer Support Bot | "You are a support agent for XYZ company..." |
| Code Assistant | "You are a programming expert. Always explain your code..." |
| Role-play Game | "You are a medieval knight named Sir Galahad..." |
The system prompt takes up part of the context window but guides all responses.
Strategies for Working with Context Limits
1. Be Concise
Shorter messages = more room for conversation history.
2. Summarize When Needed
"Let me summarize what we've discussed so far..."
3. Reference Key Information
Repeat important details if the conversation is long: "As I mentioned earlier, my budget is $1000..."
4. Start Fresh for New Topics
Begin a new chat for unrelated topics to avoid confusion.
5. Use Explicit References
"Based on the code you showed in your second message..."
How Different Apps Handle Context
| Approach | How It Works | Trade-off |
|---|---|---|
| Simple Truncation | Oldest messages removed | Loses early context |
| Sliding Window | Keeps recent N messages | May lose important setup |
| Summarization | AI summarizes old messages | Loses detail, keeps gist |
| Hybrid | System prompt + summary + recent | Balances well |
Memory Features vs. Context
Some AI products add memory features beyond the context window:
ChatGPT Memory
- Stores facts you tell it explicitly
- Persists between conversations
- Uses separate storage, not context
Custom Knowledge
- Uploaded documents
- Retrieved when relevant
- A technique called RAG (Retrieval-Augmented Generation)
These are additions to the basic context mechanism, not replacements.
Practical Implications
Why the Same Question Gets Different Answers
- Different context (conversation history)
- Different random seed in generation
- Different version/time of day
Why Long Documents Are Tricky
- May not fit in context
- Model attention is limited
- Information at start/end remembered better than middle
Why Instructions Can Be "Forgotten"
- Pushed out of context in long conversations
- Newer instructions override older ones
- System prompts help but aren't perfect
Key Takeaways
✅ Context window is the LLM's "short-term memory"
✅ Entire conversation is re-sent with each message
✅ Different models have vastly different context limits
✅ Overflow causes loss of early conversation
✅ System prompts shape behavior but use context space
✅ Memory features are additions, not part of core LLM function
Think About It
- Why might a customer support chatbot "forget" your order number in a very long conversation?
- How does knowing about context windows help you get better results from AI assistants?
- Why is Claude's 200K context window valuable for analyzing long documents?
Next Steps
Test your understanding of how LLMs work in Module 2 Quiz, then continue to Module 3 to learn about using LLMs in practice.