Chapter 2: History - From RNNs to Transformers

Introduction

The journey to modern Large Language Models spans decades of research in neural networks and natural language processing. Understanding this history helps us appreciate why the Transformer architecture was such a breakthrough.

The Pre-Neural Era (1950s-2000s)

Rule-Based Systems

Early NLP relied on hand-crafted rules:

IF input contains "weather" AND "tomorrow"
THEN search weather database for tomorrow
RETURN formatted weather response

Limitations:

• Couldn't handle language variability
• Required extensive manual engineering
• Poor generalization

Statistical Language Models

The statistical approach modeled language as probability distributions:

$$P(w_1, w_2, ..., w_n) = \prod_{i=1}^{n} P(w_i | w_1, ..., w_{i-1})$$

N-gram Models:

# Bigram model example
bigram_counts = {
    ("the", "cat"): 100,
    ("the", "dog"): 80,
    ("cat", "sat"): 50,
}

# P("cat" | "the") = count("the cat") / count("the")

Limitations:

• Data sparsity for longer contexts
• No semantic understanding
• Fixed vocabulary

The Neural Revolution (2000s-2017)

Feed-Forward Neural Language Models (2003)

Bengio et al. introduced neural language models:

[w_{t-3}] [w_{t-2}] [w_{t-1}]
    ↓         ↓         ↓
[Embedding] [Embedding] [Embedding]
         ↘     ↓     ↙
        [Hidden Layer]
              ↓
        [Softmax Output]
              ↓
           P(w_t)

Key Innovation: Learned distributed representations (embeddings)

Recurrent Neural Networks (RNNs)

RNNs introduced the concept of memory through recurrence:

class SimpleRNN:
    def forward(self, x, h_prev):
        """
        x: current input
        h_prev: previous hidden state
        """
        h_new = tanh(W_xh @ x + W_hh @ h_prev + b)
        return h_new

Architecture:

h_0 → [RNN] → h_1 → [RNN] → h_2 → [RNN] → h_3
        ↑            ↑            ↑
       x_1          x_2          x_3

Problems:

• Vanishing Gradients: Signals diminish over long sequences
• Sequential Processing: Can't parallelize training

Long Short-Term Memory (LSTM) - 1997

Hochreiter & Schmidhuber designed LSTMs to address vanishing gradients:

class LSTMCell:
    def forward(self, x, h_prev, c_prev):
        # Forget gate
        f = sigmoid(W_f @ [h_prev, x] + b_f)
        
        # Input gate
        i = sigmoid(W_i @ [h_prev, x] + b_i)
        
        # Candidate values
        c_tilde = tanh(W_c @ [h_prev, x] + b_c)
        
        # Update cell state
        c = f * c_prev + i * c_tilde
        
        # Output gate
        o = sigmoid(W_o @ [h_prev, x] + b_o)
        h = o * tanh(c)
        
        return h, c

Key Components:

Gate	Purpose
Forget Gate	What to remove from memory
Input Gate	What new information to store
Output Gate	What to output from memory

Sequence-to-Sequence (2014)

Sutskever et al. introduced encoder-decoder architectures:

Encoder:                    Decoder:
"Hello world" → [LSTM] → Context Vector → [LSTM] → "Bonjour monde"

Limitation: Single context vector bottleneck

Attention Mechanism (2014-2015)

Bahdanau et al. introduced attention to address the bottleneck:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

def attention(query, keys, values):
    """
    query: what we're looking for
    keys: what each position offers
    values: actual content at each position
    """
    scores = query @ keys.T / sqrt(d_k)
    weights = softmax(scores)
    output = weights @ values
    return output

Key Insight: Let the model learn which parts of the input are relevant for each output.

The Transformer Revolution (2017)

"Attention Is All You Need"

Vaswani et al. proposed removing recurrence entirely:

Key Paper

"Attention Is All You Need" (2017)

• Authors: Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin
• Introduced the Transformer architecture
• Enabled massive parallelization
• Foundation for all modern LLMs

Transformer Architecture Overview

Input Tokens
     ↓
[Embedding + Positional Encoding]
     ↓
┌─────────────────────────────┐
│   Multi-Head Attention      │
│         ↓                   │
│   Add & Normalize           │
│         ↓                   │
│   Feed-Forward Network      │
│         ↓                   │
│   Add & Normalize           │
└─────────────────────────────┘ × N layers
     ↓
Output Probabilities

Why Transformers Won

Feature	RNN/LSTM	Transformer
Parallelization	Sequential	Fully parallel
Long-range deps	Difficult	Direct attention
Training speed	Slow	Fast
Scalability	Limited	Excellent

The GPT Era (2018-Present)

GPT-1 (2018)

OpenAI's first "Generative Pre-trained Transformer":

• 117M parameters
• Pre-training + fine-tuning paradigm
• Decoder-only architecture

# GPT-1 architecture (simplified)
class GPT1:
    def __init__(self):
        self.layers = 12
        self.heads = 12
        self.d_model = 768
        self.vocab_size = 40478

BERT (2018)

Google's bidirectional approach:

• 340M parameters (BERT-Large)
• Encoder-only architecture
• Masked language modeling

# BERT pre-training objective
def masked_lm_loss(input_ids, model):
    # Randomly mask 15% of tokens
    masked_input = mask_tokens(input_ids, mask_prob=0.15)
    
    # Predict original tokens
    predictions = model(masked_input)
    
    # Loss only on masked positions
    return cross_entropy(predictions[masked_positions], 
                        input_ids[masked_positions])

GPT-2 (2019)

Scaling up and demonstrating emergent capabilities:

• 1.5B parameters
• Zero-shot task performance
• Initially withheld due to misuse concerns

GPT-3 (2020)

The scale that changed everything:

• 175B parameters
• In-context learning
• Few-shot prompting

# GPT-3 scale
gpt3_config = {
    "n_params": 175_000_000_000,
    "n_layers": 96,
    "d_model": 12288,
    "n_heads": 96,
    "context_length": 2048,
}

The Modern Era (2022-Present)

Model	Year	Parameters	Key Innovation
ChatGPT	2022	~175B	RLHF for alignment
GPT-4	2023	~1.8T*	Multimodal, MoE*
Claude 2/3	2023-24	Unknown	Constitutional AI
Llama 2	2023	7B-70B	Open weights
Mistral	2023	7B	Efficient architecture

*Estimated/unconfirmed

Timeline Visualization

Why History Matters

Understanding this evolution helps us:

1. Appreciate design decisions in modern architectures
2. Recognize patterns that lead to breakthroughs
3. Anticipate future directions based on trends
4. Debug and improve current systems

Key Takeaways

Summary

1. RNNs introduced sequence modeling but couldn't parallelize
2. LSTMs addressed vanishing gradients but remained sequential
3. Attention enabled focusing on relevant inputs
4. Transformers removed recurrence, enabling massive scale
5. Scale proved to be a key ingredient for capability emergence

Looking Ahead

In the next chapter, we'll survey the current landscape of Large Language Models, exploring the major players, their models, and the ongoing debate between open-source and proprietary approaches.

---

← Chapter 1: What are LLMs?

Chapter 3: LLM Landscape →