Chapter 2: Data Quality and Curation

Building High-Quality Training Datasets

Data quality determines model quality. This chapter covers filtering, cleaning, and curating datasets.

---

Quality Dimensions

Dimension	Description	Check
Accuracy	Factually correct	Human review
Relevance	On-topic for your use case	Domain alignment
Diversity	Covers different scenarios	Distribution analysis
Consistency	Uniform format and style	Automated checks
Completeness	Full, coherent responses	Length filtering

---

Data Filtering Pipeline

def filter_quality(example):
    """Filter low-quality examples."""
    output = example.get("output", "")
    
    # Length check
    if len(output) < 50:
        return False
    if len(output) > 4000:
        return False
    
    # Language check
    if not is_english(output):
        return False
    
    # Format check
    if contains_only_code(output) and not is_coding_task(example):
        return False
    
    return True

dataset = dataset.filter(filter_quality)

---

Deduplication

Remove duplicate or near-duplicate examples:

from datasets import load_dataset

# Exact deduplication
seen = set()
def deduplicate(example):
    key = hash(example["instruction"] + example["output"])
    if key in seen:
        return False
    seen.add(key)
    return True

dataset = dataset.filter(deduplicate)

---

Quality Scoring

Use a scoring model:

def score_quality(example):
    """Score example quality 1-5."""
    # Could use a classifier or LLM
    prompt = f"""Rate this response quality 1-5:
    Q: {example['instruction']}
    A: {example['output']}
    Score:"""
    
    score = scoring_model.generate(prompt)
    return {"quality_score": int(score)}

dataset = dataset.map(score_quality)
dataset = dataset.filter(lambda x: x["quality_score"] >= 4)

---

Dataset Size Guidelines

Use Case	Recommended Size
Style adaptation	100-500 examples
Task specialization	500-2,000 examples
Domain expertise	2,000-10,000 examples
General instruction	10,000+ examples

---

Key Takeaways

✅ Quality > Quantity always ✅ Filter aggressively for high quality ✅ Remove duplicates ✅ Ensure diversity in examples ✅ Human review for critical applications

---

Continue to Chapter 3 →