Chapter 1: Evaluating Fine-Tuned Models

Comprehensive Model Assessment

Evaluation ensures your fine-tuned model meets quality requirements.

---

Evaluation Metrics

Metric	Use Case	Tool
Perplexity	Language modeling	Built-in
ROUGE	Summarization	evaluate library
BLEU	Translation	evaluate library
Win Rate	Preference	GPT-4 judge
Task Accuracy	Classification	Custom

---

Using lm-evaluation-harness

# Install
pip install lm-eval

# Evaluate
lm_eval --model hf \
    --model_args pretrained=./my-model \
    --tasks hellaswag,arc_easy,mmlu \
    --batch_size 4

---

Custom Evaluation

def evaluate_task_quality(model, tokenizer, test_cases):
    """Evaluate on custom test cases."""
    scores = []
    for test in test_cases:
        inputs = tokenizer(test["prompt"], return_tensors="pt")
        outputs = model.generate(**inputs, max_new_tokens=200)
        response = tokenizer.decode(outputs[0])
        
        # Score (could be rule-based or LLM-based)
        score = score_response(response, test["expected"])
        scores.append(score)
    
    return sum(scores) / len(scores)

---

Capability Retention

Always check if base model capabilities are preserved:

# Compare before/after on standard benchmarks
baseline = evaluate_benchmark(base_model)
finetuned = evaluate_benchmark(finetuned_model)

retention = finetuned / baseline
print(f"Capability retention: {retention:.1%}")

---

Continue to Chapter 2 →