Chapter 1: Common Benchmarks

Key Benchmarks

Benchmark	Measures	Tasks
MMLU	Knowledge	57 subjects
HellaSwag	Commonsense	Sentence completion
HumanEval	Code	Python functions
GSM8K	Math	Word problems
TruthfulQA	Truthfulness	Factual accuracy

MMLU (Massive Multitask Language Understanding)

57 subjects from elementary to professional level:

• STEM, humanities, social sciences
• Multiple choice questions
• Tests knowledge breadth

HumanEval

Code generation benchmark:

def check_solution(problem, solution):
    """
    Run test cases against generated code
    """
    test_results = run_tests(solution, problem.test_cases)
    return all(test_results)

# Metric: pass@k - probability of passing with k attempts

Benchmark Limitations

• Data contamination: Training on test data
• Gaming: Optimizing for benchmarks
• Narrow scope: May miss real-world performance

---

← OverviewChapter 2 →