Chapter 1: Neural Networks Basics
The Building Blocks of AI
Neural networks are the foundation of modern AI, including all generative AI systems. Understanding how they work provides crucial insight into what AI can and cannot do.
What is a Neural Network?
A neural network is a computing system loosely inspired by the human brain. It consists of interconnected nodes (called neurons) organized in layers that process information.
Neural networks are pattern recognizers. They learn to identify patterns in data and use those patterns to make predictions or generate new content.
The Brain Analogy
| Human Brain | Neural Network |
|---|---|
| Neurons | Artificial neurons (nodes) |
| Synapses | Weighted connections |
| Electrical signals | Numbers |
| Learning through experience | Learning through training data |
Basic Structure
Layers
A neural network has three types of layers:
- Input Layer: Receives the initial data
- Hidden Layers: Process the data (can be many layers)
- Output Layer: Produces the final result
Deep Learning
When a network has many hidden layers, it's called a deep neural network, and training it is called deep learning.
How Neurons Work
Each artificial neuron:
- Receives inputs from previous neurons
- Multiplies each input by a weight
- Adds all weighted inputs together
- Applies an activation function to produce output
Simple Example
Imagine a neuron deciding if an image shows a cat:
Input 1 (Has fur?): 1 × weight 0.3 = 0.3
Input 2 (Has whiskers?): 1 × weight 0.4 = 0.4
Input 3 (Barks?): 0 × weight 0.2 = 0.0
Sum = 0.7
If sum > 0.5 → Output: "Probably a cat"
Learning: Adjusting Weights
The magic of neural networks is in learning—the process of adjusting weights to improve accuracy.
Training Process
- Forward Pass: Data flows through the network
- Compare: Check output against the correct answer
- Calculate Error: How far off was the prediction?
- Backpropagation: Trace back which weights caused the error
- Adjust Weights: Make small corrections
- Repeat: Do this millions of times!
The Parking Analogy
Learning to park a car:
- First attempts: Hit curbs, crooked parking
- With practice: Small corrections each time
- Eventually: Parallel park perfectly
Neural networks learn the same way—through repeated practice and correction.
Types of Neural Networks
1. Feedforward Networks
The simplest type—data flows in one direction from input to output. Good for basic classification tasks.
2. Convolutional Neural Networks (CNNs)
Specialized for image processing. Use filters to detect features like edges, shapes, and patterns.
Used for: Image recognition, computer vision
3. Recurrent Neural Networks (RNNs)
Have loops that allow information to persist. Good for sequential data.
Used for: Early language models, time series
4. Transformers
The architecture powering modern AI. Process all inputs simultaneously using attention.
Used for: ChatGPT, Claude, DALL-E, and most modern AI
Parameters: The AI's Knowledge
What are Parameters?
Parameters are the adjustable numbers (weights and biases) in a neural network. They represent what the model has learned.
Scale Matters
| Model | Parameters |
|---|---|
| Small model | Millions |
| GPT-2 | 1.5 billion |
| GPT-3 | 175 billion |
| GPT-4 | ~1.7 trillion (estimated) |
| Llama 2 (largest) | 70 billion |
More parameters generally mean more capability, but also more computational cost.
Key Concepts Simplified
Activation Functions
Like a switch that determines if a neuron "fires" or not. Common ones include ReLU and Sigmoid.
Why it matters: Without activation functions, neural networks could only learn simple, linear patterns.
Loss Function
Measures how wrong the model's predictions are. Training tries to minimize this.
Analogy: Like a golf score—lower is better.
Gradient Descent
The method used to adjust weights. Move in the direction that reduces error.
Analogy: Like walking downhill in fog—take small steps in the direction that goes down.
Learning Rate
How big each adjustment step is. Too big = unstable. Too small = too slow.
What Neural Networks Can and Can't Do
Can Do ✅
- Recognize complex patterns
- Learn from examples
- Generalize to new, similar data
- Improve with more data and training
Can't Do ❌
- Reason like humans
- Truly understand meaning
- Apply common sense reliably
- Learn from very few examples (usually)
Real-World Example: Language Model
How a language model uses neural networks:
The network has learned patterns from billions of text examples, so it knows "mat" is more likely to follow "The cat sat on the" than "helicopter."
Key Takeaways
✅ Neural networks are pattern-recognition systems inspired by the brain
✅ They consist of layers of interconnected neurons
✅ Learning happens by adjusting weights through many examples
✅ More parameters generally mean more capability
✅ Different architectures excel at different tasks
✅ Transformers are the foundation of modern generative AI
Next Steps
Now that you understand the basics of neural networks, let's explore the revolutionary architecture that powers today's AI in Chapter 2: Transformers Architecture.