Chapter 1: RAG Evaluation Metrics

Two Evaluation Areas

Area	What It Measures	Key Metrics
Retrieval	Quality of retrieved docs	Precision, Recall, MRR
Generation	Quality of LLM response	Faithfulness, Relevance

Retrieval Metrics

Precision@K

def precision_at_k(retrieved_ids, relevant_ids, k):
    """What % of retrieved docs are relevant?"""
    retrieved_k = set(retrieved_ids[:k])
    relevant = set(relevant_ids)
    return len(retrieved_k & relevant) / k

# Retrieved: [A, B, C, D, E], Relevant: [A, C, F]
# Precision@5 = 2/5 = 0.4

Recall@K

def recall_at_k(retrieved_ids, relevant_ids, k):
    """What % of relevant docs were retrieved?"""
    retrieved_k = set(retrieved_ids[:k])
    relevant = set(relevant_ids)
    return len(retrieved_k & relevant) / len(relevant)

# Retrieved: [A, B, C, D, E], Relevant: [A, C, F]
# Recall@5 = 2/3 = 0.67

MRR (Mean Reciprocal Rank)

def reciprocal_rank(retrieved_ids, relevant_ids):
    """How high is the first relevant doc?"""
    for i, doc_id in enumerate(retrieved_ids):
        if doc_id in relevant_ids:
            return 1 / (i + 1)
    return 0

# If first relevant at position 3: RR = 1/3 = 0.33

Generation Metrics

Faithfulness

Does the answer use ONLY information from context?

# High faithfulness:
# Context: "The policy allows 30 day returns"
# Answer: "You can return items within 30 days"

# Low faithfulness (hallucination):
# Context: "The policy allows 30 day returns"  
# Answer: "You can return items within 60 days for a full refund"

Answer Relevance

Does the answer actually address the question?

Context Relevance

Are the retrieved chunks relevant to the question?

Metrics Summary

Metric	Measures	Target
Precision@K	Retrieved relevance	>0.8
Recall@K	Coverage	>0.9
MRR	Ranking quality	>0.7
Faithfulness	No hallucinations	>0.9
Relevance	Answer quality	>0.8