Chapter 3: Synthetic Data Generation
Using LLMs to Create Training Data
When real data is scarce, synthetic data generated by capable LLMs can be highly effective.
The Synthetic Data Pipeline
Generation Strategies
1. Self-Instruct
Generate new instructions and responses:
def generate_instruction():
prompt = """Generate a unique instruction for a helpful AI assistant.
Examples:
- Explain quantum computing to a 10-year-old
- Write a haiku about programming
- Create a meal plan for a vegetarian athlete
New instruction:"""
return llm.generate(prompt)
def generate_response(instruction):
prompt = f"""Provide a helpful response to this instruction:
Instruction: {instruction}
Response:"""
return llm.generate(prompt)
2. Evol-Instruct
Evolve simple instructions into complex ones:
def evolve_instruction(simple_instruction):
prompt = f"""Make this instruction more complex and detailed:
Simple: {simple_instruction}
Complex:"""
return llm.generate(prompt)
3. Domain-Specific Generation
def generate_domain_examples(domain, count=100):
prompt = f"""Generate {count} diverse instruction-response pairs
for a {domain} AI assistant.
Format each as:
Instruction: ...
Response: ...
---
"""
return parse_pairs(llm.generate(prompt))
Quality Control for Synthetic Data
def validate_synthetic(example, validator_model):
"""Validate synthetic example quality."""
prompt = f"""Is this a high-quality instruction-response pair?
Instruction: {example['instruction']}
Response: {example['response']}
Answer YES or NO with brief explanation:"""
result = validator_model.generate(prompt)
return "YES" in result.upper()
Best Practices
- Use strong models for generation (GPT-4, Claude)
- Validate with different model than generator
- Sample and review before full generation
- Mix with real data when available
- Check for contamination with eval sets
Key Takeaways
✅ Synthetic data can be highly effective ✅ Use strong models for generation ✅ Always filter and validate ✅ Mix with real data when possible ✅ Avoid eval set contamination