Module 3: Training LLMs
Module Overview
Training Large Language Models requires massive computational resources, carefully curated data, and sophisticated distributed training techniques. This module explores the complete training pipeline from data to trained model.
Learning Objectives
- ✅ Understand pre-training objectives (next token prediction, masked LM)
- ✅ Explain data collection and preprocessing pipelines
- ✅ Describe distributed training techniques (data, model, tensor parallelism)
- ✅ Understand the compute requirements for training LLMs
Module Contents
Chapter 1: Pre-training Objectives
Next token prediction, masked language modeling, causal vs bidirectional approaches.
Chapter 2: Data Collection and Preprocessing
Web scraping, data cleaning, deduplication, quality filtering, tokenization.
Chapter 3: Distributed Training Techniques
Data parallelism, model parallelism, tensor parallelism, pipeline parallelism, ZeRO.