Chapter 2: Emerging Architectures
Beyond Transformers?
The Transformer has dominated, but alternatives are emerging.
State Space Models (SSMs)
Linear-time sequence modeling:
Transformers: O(n²) attention
SSMs: O(n) recurrence with parallelizable training
Mamba
Selective State Space Model:
# Mamba key innovation: input-dependent selection
def selective_scan(x, A, B, C, delta):
# delta (Δ) is input-dependent
# Enables dynamic, selective "attention-like" behavior
# With linear complexity
pass
Benefits:
- Linear scaling with sequence length
- Competitive with Transformers
- Better for very long sequences
RWKV
Combines RNN efficiency with Transformer quality:
- Linear complexity
- Parallelizable training
- Good performance
Comparison
| Architecture | Complexity | Long Context | Quality |
|---|---|---|---|
| Transformer | O(n²) | Limited* | Best |
| Mamba | O(n) | Excellent | Near-best |
| RWKV | O(n) | Good | Good |
*With optimizations like Flash Attention, Transformers remain competitive
Will Transformers Be Replaced?
Likely scenario: Hybrid architectures combining:
- Attention for key positions
- Linear layers for efficiency
- Best of both worlds