Skip to main content

Chapter 2: Transformers Architecture

The Architecture That Changed Everything​

The Transformer architecture, introduced in 2017, revolutionized AI. Every major language model (GPT, Claude, Gemini, Llama) is built on this foundation.

Before Transformers: The Problem​

Previous approaches to processing sequential data (like text) had major limitations:

Recurrent Neural Networks (RNNs)​

Problems:

  1. Sequential processing: One word at a time = slow
  2. Forgetting: Long-distance relationships get lost
  3. Limited parallelization: Can't take advantage of modern GPUs

Analogy: Like reading a book but only remembering the last few pages clearly.

The Transformer Solution​

The 2017 paper "Attention Is All You Need" proposed a radical idea: process all words simultaneously using an "attention" mechanism.

Key Innovation: Every word can "attend" to every other word, regardless of position.

Understanding Attention​

The Core Idea​

Attention allows the model to focus on relevant parts of the input when processing each element.

Example: "The animal didn't cross the street because it was too tired."

What does "it" refer to? A human knows "it" = "animal" because animals get tired.

The attention mechanism learns these relationships:

Self-Attention Explained​

For each word, the model asks: "How much should I pay attention to each other word?"

Step-by-step:

  1. Query (Q): "What am I looking for?"
  2. Key (K): "What do I contain?"
  3. Value (V): "What information do I provide?"

Analogy: Library Research​

Imagine you're researching a topic:

  • Query: Your research question
  • Keys: Book titles and summaries
  • Values: The actual content of the books
  • Attention: How relevant each book is to your question

Transformer Architecture​

Full Structure​

Key Components​

1. Embedding Layer​

Converts words into numerical vectors (lists of numbers) that capture meaning.

"cat" → [0.2, -0.5, 0.8, 0.1, ...]
"dog" → [0.3, -0.4, 0.7, 0.2, ...]

Similar words have similar vectors!

2. Positional Encoding​

Adds information about word position since attention processes everything simultaneously.

Without this, "Dog bites man" and "Man bites dog" would look the same.

3. Multi-Head Attention​

Multiple attention mechanisms running in parallel, each focusing on different aspects:

  • One head might focus on grammar
  • Another on meaning
  • Another on context

4. Feed-Forward Networks​

Standard neural network layers that process each position independently.

5. Layer Normalization​

Keeps values in a reasonable range, improving training stability.

Types of Transformer Models​

1. Encoder-Only (e.g., BERT)​

  • Processes input to create representations
  • Good for: Classification, understanding tasks
  • Can see all context (bidirectional)

2. Decoder-Only (e.g., GPT, Claude)​

  • Generates output word by word
  • Good for: Text generation
  • Only sees previous context (unidirectional)

3. Encoder-Decoder (e.g., T5, BART)​

  • Encoder processes input, decoder generates output
  • Good for: Translation, summarization

Why Transformers Excel​

1. Parallelization​

Can process all positions simultaneously → Much faster training

2. Long-Range Dependencies​

Attention connects any two positions directly → Better at understanding context

3. Scalability​

Architecture scales well with more data and parameters → Enables huge models

4. Transfer Learning​

Pre-trained models can be adapted to many tasks → More efficient development

Context Window​

The context window is the maximum amount of text a transformer can process at once.

ModelContext Window
GPT-34,096 tokens (~3,000 words)
GPT-4128,000 tokens (~100,000 words)
Claude 3200,000 tokens (~150,000 words)
Gemini 1.51,000,000+ tokens

Why It Matters:

  • Larger context = Better understanding of long documents
  • Can reference information from earlier in conversations
  • Enables processing entire books or codebases

Tokens: How Text is Processed​

Transformers don't process words—they process tokens.

What's a Token?​

A token is a piece of text, usually:

  • A common word: "the" = 1 token
  • A word part: "understanding" = "under" + "standing" = 2 tokens
  • A punctuation mark: "." = 1 token

Example​

Input: "ChatGPT is amazing!"

Tokens: ["Chat", "G", "PT", " is", " amazing", "!"]

Roughly: 1 token ≈ 4 characters or ¾ of a word in English

Limitations of Transformers​

1. Quadratic Complexity​

Attention compares every token to every other token:

  • 1,000 tokens → 1,000,000 comparisons
  • 10,000 tokens → 100,000,000 comparisons

This limits practical context sizes.

2. No True Reasoning​

Transformers recognize patterns; they don't truly understand or reason.

3. Static After Training​

The model's knowledge is frozen at training time (without retrieval augmentation).

4. Hallucination Risk​

The pattern-matching nature can produce confident but incorrect outputs.

Key Takeaways​

✅ Transformers process all input simultaneously using attention

✅ Attention allows every element to relate to every other element

✅ This architecture powers all major modern AI models

✅ Context window determines how much text can be processed

✅ Text is processed as tokens, not words

✅ Different transformer types (encoder, decoder) suit different tasks

Next Steps​

Now that you understand the architecture, let's explore how these models are trained in Chapter 3: Training and Learning.