Chapter 1: What are Large Language Models?

Introduction

Large Language Models (LLMs) represent one of the most significant advances in artificial intelligence. These massive neural networks have demonstrated remarkable abilities in understanding and generating human language, often exhibiting behaviors that weren't explicitly programmed.

Defining Large Language Models

The Scale Factor

A Large Language Model is a neural network trained on vast amounts of text data, characterized by:

1. Parameter Count: Billions to trillions of trainable parameters
2. Training Data: Terabytes of text from diverse sources
3. Compute Requirements: Thousands of GPU-hours for training

# Comparison of model sizes (approximate parameter counts)
model_sizes = {
    "BERT-base": "110M",
    "GPT-2": "1.5B", 
    "GPT-3": "175B",
    "GPT-4": "~1.8T (estimated)",
    "Llama-2-70B": "70B",
    "Claude-3": "Unknown (estimated >100B)"
}

What Makes Them "Large"?

The term "large" refers to multiple dimensions:

Dimension	Small Models	Large Language Models
Parameters	Millions	Billions to Trillions
Training Data	GBs	TBs to PBs
Context Window	512 tokens	4K-128K+ tokens
Training Compute	GPU-days	GPU-years

Core Capabilities

1. Text Generation

LLMs excel at generating coherent, contextually appropriate text:

# Example: Text completion
prompt = "The transformer architecture revolutionized NLP because"
# LLM completion: "it enabled parallel processing of sequences 
# and captured long-range dependencies more effectively than 
# recurrent approaches."

2. In-Context Learning

One of the most remarkable capabilities of LLMs is learning from examples provided in the prompt:

# Few-shot learning example
prompt = """
Translate English to French:
"Hello" -> "Bonjour"
"Thank you" -> "Merci"
"Good morning" -> """
# LLM output: "Bon matin" or "Bonjour"

3. Reasoning and Problem Solving

Modern LLMs demonstrate basic reasoning capabilities:

# Chain-of-thought reasoning
prompt = """
If a train travels at 60 mph for 2.5 hours, 
how far does it travel?

Let me think step by step:
1. Distance = Speed × Time
2. Speed = 60 mph
3. Time = 2.5 hours
4. Distance = 60 × 2.5 = 150 miles

Answer: 150 miles
"""

Emergent Capabilities

Emergent capabilities are abilities that appear only when models reach a certain scale, absent in smaller models.

Examples of Emergence

1. Multi-step Reasoning: Breaking complex problems into steps
2. Code Generation: Writing functional programs from descriptions
3. Translation: Translating between languages without explicit training
4. Instruction Following: Understanding and executing complex instructions

The Scaling Hypothesis

Research has shown predictable relationships between:

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}$$

Where:

• $L$ = Loss (lower is better)
• $N$ = Number of parameters
• $N_c, \alpha_N$ = Constants determined empirically

This suggests that simply scaling up models leads to predictable improvements in capability.

LLMs vs. Traditional Language Models

Statistical Language Models

Traditional models estimated probabilities using n-gram statistics:

$$P(w_n | w_1, ..., w_{n-1}) \approx P(w_n | w_{n-k}, ..., w_{n-1})$$

Limitations:

• Fixed context window
• Sparse data problems
• No semantic understanding

Neural Language Models (Pre-LLM)

Earlier neural approaches used RNNs and LSTMs:

# RNN-based language model (simplified)
class RNNLanguageModel:
    def __init__(self, vocab_size, hidden_size):
        self.embedding = Embedding(vocab_size, hidden_size)
        self.rnn = RNN(hidden_size, hidden_size)
        self.output = Linear(hidden_size, vocab_size)
    
    def forward(self, x, hidden):
        embedded = self.embedding(x)
        output, hidden = self.rnn(embedded, hidden)
        return self.output(output), hidden

Limitations:

• Sequential processing (slow training)
• Vanishing gradients
• Limited long-range dependencies

Large Language Models

LLMs overcome these limitations through:

1. Parallel Processing: Transformers process all tokens simultaneously
2. Attention Mechanism: Direct connections between any positions
3. Scale: Massive model capacity enables complex pattern learning

The Building Blocks of LLMs

1. Tokenization

Converting text to numerical representations:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
text = "Large Language Models are fascinating!"
tokens = tokenizer.encode(text)
# [16902, 15417, 32329, 389, 21120, 0]

2. Embeddings

Dense vector representations of tokens:

# Each token maps to a high-dimensional vector
embedding_dim = 768  # Common for base models
# "The" -> [0.23, -0.45, 0.12, ..., 0.89]  # 768 values

3. Transformer Layers

The core processing units:

Input → Embedding → [Transformer Block × N] → Output
                          ↓
                    Attention + FFN

4. Output Projection

Converting internal representations back to vocabulary probabilities:

$$P(w_i | context) = \text{softmax}(W_o \cdot h_i + b_o)$$

Practical Example: How LLMs Generate Text

def generate_text(model, tokenizer, prompt, max_tokens=50):
    """
    Simple autoregressive text generation
    """
    input_ids = tokenizer.encode(prompt, return_tensors="pt")
    
    for _ in range(max_tokens):
        # Get model predictions
        outputs = model(input_ids)
        next_token_logits = outputs.logits[:, -1, :]
        
        # Sample next token
        next_token = torch.argmax(next_token_logits, dim=-1)
        
        # Append to sequence
        input_ids = torch.cat([input_ids, next_token.unsqueeze(0)], dim=-1)
        
        # Stop if EOS token
        if next_token == tokenizer.eos_token_id:
            break
    
    return tokenizer.decode(input_ids[0])

Key Takeaways

Key Points

1. LLMs are defined by scale: billions of parameters, terabytes of training data
2. Emergent capabilities appear only at sufficient scale
3. The Transformer architecture enables efficient training and long-range dependencies
4. Autoregressive generation: LLMs predict one token at a time
5. In-context learning: LLMs can learn from examples in the prompt

Looking Ahead

In the next chapter, we'll trace the historical evolution of language models, from early recurrent neural networks to the revolutionary Transformer architecture that made modern LLMs possible.

---

← Module Overview

Chapter 2: History →