Chapter 3: Testing and Evaluation

Quality Metrics

Metric	What It Measures	How to Evaluate
Accuracy	Factual correctness	Manual verification
Relevance	On-topic responses	Rubric scoring
Completeness	All requirements met	Checklist
Consistency	Same quality across runs	Multiple tests
Format compliance	Matches requirements	Pattern matching

Creating Test Cases

Test Suite: Customer Support Prompt

Test Case 1: Simple inquiry
Input: "What are your business hours?"
Expected: Direct answer with hours

Test Case 2: Complaint
Input: "Your product broke after one week!"
Expected: Empathetic response + solution options

Test Case 3: Edge case - multiple issues
Input: "Wrong item sent AND it was damaged AND late"
Expected: Address all three issues systematically

Automated Evaluation

For scale, automate what you can:

Automated Checks:
- [ ] Response length within bounds
- [ ] Contains required sections
- [ ] No forbidden words
- [ ] JSON/format validity
- [ ] Sentiment matches expectation

Manual Review:
- Quality of reasoning
- Appropriateness of tone
- Accuracy of specific claims

Scoring Rubrics

Content Quality (1-5):
5 - Exceptional, exceeds requirements
4 - Good, meets all requirements
3 - Acceptable, meets most requirements
2 - Poor, significant gaps
1 - Unacceptable, doesn't address task

Format Compliance (Pass/Fail):
- Correct structure
- Within length limits
- Required elements present

Human Evaluation

For nuanced quality, use human reviewers:

Reviewer Guidelines:
1. Read the original prompt
2. Review the AI output
3. Score on: Accuracy, Relevance, Tone, Completeness
4. Note specific issues
5. Suggest improvements

Key Takeaways

Summary

1. Define metrics before testing
2. Create diverse test cases including edge cases
3. Automate format/structure checks
4. Use rubrics for consistency
5. Human review for nuanced quality

---

← Previous: Iterative Refinement | Take the Quiz →