Chapter 2: Data Collection and Preprocessing
Introduction
The quality and diversity of training data fundamentally determines model capabilities. Modern LLMs are trained on trillions of tokens from diverse sources.
Data Sources
Common Crawl
- Petabytes of web data
- Requires heavy filtering
- Most common starting point
Curated Sources
data_sources = {
"Wikipedia": "High quality, encyclopedic",
"Books": "Long-form, coherent text",
"ArXiv": "Scientific papers",
"GitHub": "Code and documentation",
"StackOverflow": "Q&A format",
}
Data Processing Pipeline
Raw Web Data → Deduplication → Quality Filtering → Tokenization → Training Data
1. Deduplication
from datasketch import MinHash, MinHashLSH
def create_minhash(text, num_perm=128):
"""Create MinHash signature for deduplication."""
m = MinHash(num_perm=num_perm)
for word in text.split():
m.update(word.encode('utf8'))
return m
2. Quality Filtering
def quality_filter(text):
"""Filter low-quality documents."""
# Length filter
if len(text.split()) < 50:
return False
# Repetition filter
lines = text.split('\n')
unique_lines = set(lines)
if len(unique_lines) / len(lines) < 0.5:
return False
# Language detection
if not is_english(text):
return False
return True
3. Tokenization
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")
def tokenize_dataset(texts, max_length=2048):
"""Tokenize and chunk texts for training."""
all_tokens = []
for text in texts:
tokens = tokenizer.encode(text)
all_tokens.extend(tokens)
# Create fixed-length chunks
chunks = [
all_tokens[i:i+max_length]
for i in range(0, len(all_tokens), max_length)
]
return chunks
Dataset Scale
| Model | Training Tokens | Data Size |
|---|---|---|
| GPT-3 | 300B | ~570 GB |
| LLaMA | 1.4T | ~1.4 TB |
| LLaMA 2 | 2T | ~2 TB |
Key Takeaways
Summary
- Data quality > data quantity
- Deduplication is critical for training efficiency
- Diverse sources improve model generalization