Chapter 1: Automated LLM Testing

LLM Evaluation Metrics

Metric	Purpose	Tool
BLEU	Translation quality	sacreBLEU
ROUGE	Summarization	rouge-score
BERTScore	Semantic similarity	bert-score
Faithfulness	Grounded in context	RAGAS
Answer Relevancy	Answers the question	RAGAS

RAGAS Evaluation

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

# Prepare evaluation dataset
eval_data = {
    "question": ["What is MLOps?", "What are feature stores?"],
    "answer": ["MLOps is...", "Feature stores are..."],
    "contexts": [["Context 1", "Context 2"], ["Context 3"]],
    "ground_truths": [["MLOps stands for..."], ["Feature stores are centralized..."]]
}

# Run evaluation
results = evaluate(
    dataset=eval_data,
    metrics=[faithfulness, answer_relevancy, context_precision]
)

print(results)
# {'faithfulness': 0.85, 'answer_relevancy': 0.90, 'context_precision': 0.88}

Golden Dataset Testing

import json

def test_against_golden_dataset():
    with open("golden_dataset.json") as f:
        golden = json.load(f)
    
    results = []
    for item in golden:
        response = llm.generate(item["prompt"])
        
        # Check against expected patterns/keywords
        score = evaluate_response(
            response=response,
            expected_keywords=item["expected_keywords"],
            blocked_patterns=item["blocked_patterns"]
        )
        results.append(score)
    
    avg_score = sum(results) / len(results)
    assert avg_score > 0.85, f"Golden dataset score {avg_score} below threshold"

---

Next: Chapter 2: Human Evaluation