Chapter 2: RAGAS Framework

What is RAGAS?

RAGAS (Retrieval Augmented Generation Assessment) is a framework for evaluating RAG pipelines automatically using LLMs.

Installation

pip install ragas

Basic Usage

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
)
from datasets import Dataset

# Prepare evaluation data
data = {
    "question": ["What is RAG?"],
    "answer": ["RAG combines retrieval with generation..."],
    "contexts": [["RAG is a technique that...", "It retrieves..."]],
    "ground_truth": ["RAG is Retrieval Augmented Generation..."]
}

dataset = Dataset.from_dict(data)

# Evaluate
results = evaluate(
    dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall
    ]
)

print(results)

RAGAS Metrics

Metric	Description	Range
faithfulness	Answer grounded in context	0-1
answer_relevancy	Answer addresses question	0-1
context_precision	Retrieved docs are relevant	0-1
context_recall	All needed info retrieved	0-1

Creating Test Dataset

def create_eval_dataset(qa_pairs, rag_system):
    """Generate evaluation dataset from Q&A pairs"""
    data = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
    
    for qa in qa_pairs:
        question = qa["question"]
        ground_truth = qa["expected_answer"]
        
        # Get RAG response
        response = rag_system.query(question)
        
        data["question"].append(question)
        data["answer"].append(response["answer"])
        data["contexts"].append(response["contexts"])
        data["ground_truth"].append(ground_truth)
    
    return Dataset.from_dict(data)

Interpreting Results

Score	Interpretation
>0.9	Excellent
0.7-0.9	Good, room for improvement
0.5-0.7	Needs work
<0.5	Significant issues

Continuous Evaluation

# Weekly evaluation pipeline
def weekly_eval(rag_system, test_questions):
    dataset = create_eval_dataset(test_questions, rag_system)
    results = evaluate(dataset, metrics=[...])
    
    # Store results
    save_metrics_to_db(results, timestamp=datetime.now())
    
    # Alert if degradation
    if results["faithfulness"] < 0.8:
        send_alert("Faithfulness dropped below threshold!")