Chapter 2: Token Limits and Context Windows
Context Length Evolution
| Model | Year | Context |
|---|---|---|
| GPT-3 | 2020 | 2K |
| GPT-3.5 | 2022 | 4K/16K |
| Claude 2 | 2023 | 100K |
| GPT-4 Turbo | 2023 | 128K |
| Claude 3 | 2024 | 200K |
| Gemini 1.5 | 2024 | 1M+ |
Handling Long Documents
Chunking Strategies
def chunk_text(text, max_tokens=4000, overlap=200):
tokens = tokenizer.encode(text)
chunks = []
for i in range(0, len(tokens), max_tokens - overlap):
chunk = tokens[i:i + max_tokens]
chunks.append(tokenizer.decode(chunk))
return chunks
RoPE for Long Context
Rotary Position Embedding enables length extrapolation:
- LLaMA uses RoPE
- Position info in rotation, not addition
Key Takeaways
tip
- Context windows have grown dramatically (2K → 1M+)
- Chunking with overlap preserves context
- RoPE enables better long-sequence handling