Chapter 1: Large Language Models (LLMs)
Understanding the AI Behind Text Generation
Large Language Models are the technology powering ChatGPT, Claude, and virtually every modern text generation system. Understanding what they are and how they work is fundamental to using them effectively.
What Makes a Language Model "Large"?
Parameters
The "large" in LLM refers primarily to the number of parameters—the adjustable values that store what the model has learned.
| Model | Parameters | Relative Size |
|---|---|---|
| BERT (2018) | 340 million | Small |
| GPT-2 (2019) | 1.5 billion | Medium |
| GPT-3 (2020) | 175 billion | Large |
| GPT-4 (2023) | ~1.7 trillion | Massive |
| Llama 3 (2024) | Up to 405B | Large |
Training Data
Large also refers to the training data—terabytes of text from:
- Books and publications
- Websites and articles
- Code repositories
- Conversations and forums
- Academic papers
Compute Resources
Training requires thousands of GPUs running for weeks or months.
How LLMs Generate Text
The Core Process: Next Token Prediction
At their heart, LLMs predict "what comes next" given the previous text:
The model doesn't retrieve information—it generates text based on learned patterns.
Token by Token
Generation happens one token at a time:
Step 1: "The weather today is" → "sunny"
Step 2: "The weather today is sunny" → "and"
Step 3: "The weather today is sunny and" → "warm"
...continues until complete
Key Capabilities of LLMs
1. Natural Language Understanding
LLMs can:
- Parse complex sentences
- Understand context and nuance
- Identify sentiment and tone
- Extract key information
2. Text Generation
LLMs can produce:
- Coherent paragraphs and essays
- Different styles and tones
- Technical and creative content
- Multiple languages
3. Following Instructions
Modern LLMs can:
- Respond to specific requests
- Follow multi-step instructions
- Adapt output format
- Ask clarifying questions
4. In-Context Learning
LLMs can:
- Learn from examples in the prompt
- Apply patterns from few examples
- Adapt without retraining
Important Parameters for Users
Temperature
Controls randomness/creativity:
| Temperature | Effect | Best For |
|---|---|---|
| 0.0 - 0.3 | Very predictable, focused | Factual content, code |
| 0.4 - 0.7 | Balanced | General writing |
| 0.8 - 1.0+ | Creative, varied | Brainstorming, creative writing |
Max Tokens
Sets the maximum length of the response.
Top-P (Nucleus Sampling)
Alternative to temperature—considers only the most probable tokens that sum to probability P.
Frequency/Presence Penalties
Reduces repetition by penalizing tokens that have already appeared.
Context Window: The Memory Limit
The context window determines how much text the model can "remember" in a conversation.
Context Window Sizes
| Model | Context Window | Approximate Words |
|---|---|---|
| GPT-3.5 | 16K tokens | ~12,000 words |
| GPT-4 | 128K tokens | ~100,000 words |
| Claude 3 | 200K tokens | ~150,000 words |
| Gemini 1.5 | 1M+ tokens | ~750,000 words |
Why Context Matters
- Longer conversations stay coherent
- Can analyze entire documents
- Reference information from earlier
- Better understanding of complex topics
Capabilities vs. Limitations
What LLMs Excel At ✅
-
Writing and editing
- Draft generation
- Style improvement
- Grammar correction
-
Summarization
- Long documents to key points
- Meeting notes
- Research papers
-
Translation
- Natural-sounding translations
- Cultural context awareness
- Multiple language pairs
-
Code assistance
- Writing code
- Debugging
- Explanation
-
Analysis
- Document review
- Sentiment analysis
- Pattern identification
What LLMs Struggle With ❌
-
Factual accuracy
- Can hallucinate facts
- No real-time information
- May confuse details
-
Mathematical reasoning
- Complex calculations unreliable
- Logic puzzles hit-or-miss
- Should be verified
-
Consistent memory
- Limited to context window
- No true long-term memory
- Can contradict itself
-
Current events
- Knowledge cutoff date
- No real-time data
- May be outdated
-
Physical world understanding
- Spatial reasoning weak
- Physical causality unclear
- Real-world consequences
The Importance of Prompts
The quality of your output depends heavily on your input.
Good Prompt Characteristics
❌ Bad: "Write something about AI"
✅ Good: "Write a 300-word blog post introduction
explaining artificial intelligence to small
business owners. Use simple language, include
one real-world example, and end with a
question to engage readers."
Prompt Components
- Task: What you want done
- Context: Background information
- Format: How the output should look
- Examples: Sample inputs/outputs
- Constraints: Limitations or requirements
Understanding "Hallucination"
LLMs can generate confident but incorrect information.
Why It Happens
- The model predicts plausible text, not verified facts
- Training data may contain errors
- The model has no way to check truth
How to Mitigate
- Verify important facts independently
- Ask for sources (though they may be invented)
- Use for drafts, not final authority
- Cross-reference with reliable sources
Key Takeaways
✅ LLMs are "large" due to parameters, training data, and compute
✅ They generate text by predicting the next token
✅ Temperature and other settings control output characteristics
✅ Context window limits how much the model can consider
✅ They excel at writing, summarizing, and analysis
✅ They struggle with facts, math, and real-time information
✅ Prompt quality significantly affects output quality
Next Steps
Now let's compare the major platforms in Chapter 2: ChatGPT, Claude, and Competitors.