Quiz: Evaluation and Optimization
30 questions about RAG evaluation, RAGAS, and debugging.
Q1: What are the two main areas of RAG evaluation?
Answer: Retrieval quality and Generation quality.
Q2: What does Precision@K measure?
Answer: The percentage of retrieved documents (top K) that are relevant.
Q3: What does Recall@K measure?
Answer: The percentage of all relevant documents that were retrieved in top K.
Q4: What is MRR?
Answer: Mean Reciprocal Rank - measures how high the first relevant document appears.
Q5: What does faithfulness measure?
Answer: Whether the generated answer is grounded in (supported by) the retrieved context.
Q6: What is RAGAS?
Answer: Retrieval Augmented Generation Assessment - a framework for automated RAG evaluation.
Q7: What does context_precision measure in RAGAS?
Answer: Whether the retrieved documents are relevant to the question.
Q8: What does context_recall measure?
Answer: Whether all necessary information to answer was retrieved.
Q9: What's a good faithfulness score?
Answer: Above 0.9 is excellent; below 0.7 indicates hallucination issues.
Q10: What's the main cause of "wrong documents retrieved"?
Answer: Poor chunk size, wrong embedding model, or missing metadata filters.
Q11: What is LangSmith used for?
Answer: Tracing and debugging LangChain applications - view all calls and responses.
Q12: How do you know if the LLM is ignoring context?
Answer: The answer doesn't reflect the retrieved documents, or contradicts them.
Q13: What causes hallucinations in RAG?
Answer: Wrong docs retrieved, LLM extrapolating beyond context, or prompt issues.
Q14: How do you reduce hallucinations?
Answer: Better retrieval, explicit grounding prompts, lower temperature.
Q15: What data does RAGAS need?
Answer: Questions, answers, contexts (retrieved docs), and ground truth answers.
Q16: How often should you evaluate RAG systems?
Answer: Continuously/weekly, especially after changes to data or pipeline.
Q17: First debugging step for wrong answers?
Answer: Check what documents are being retrieved.
Q18: How to debug prompt issues?
Answer: Print the full prompt being sent to the LLM.
Q19: What reduces RAG latency?
Answer: Async operations, caching, smaller models, limiting context size.
Q20: What reduces RAG costs?
Answer: Caching, smaller embedding models, batching, limiting context.
Q21: What's answer_relevancy in RAGAS?
Answer: Measures if the answer actually addresses the question asked.
Q22: If context_recall is low, what's the problem?
Answer: Retrieval is missing relevant documents - need better retrieval.
Q23: If faithfulness is low, what's the problem?
Answer: LLM is making up information not in the context - hallucinating.
Q24: How to check if embeddings are working?
Answer: Test with known similar documents and verify they're close in vector space.
Q25: What's the ideal Recall@K?
Answer: Above 0.9 - want to capture most relevant documents.
Q26: Why track metrics over time?
Answer: Detect degradation as data changes or after pipeline updates.
Q27: What's A/B testing in RAG?
Answer: Comparing two RAG configurations to see which performs better.
Q28: What tool counts tokens in context?
Answer: tiktoken for OpenAI models, or model-specific tokenizers.
Q29: When to use automatic vs manual evaluation?
Answer: Automatic for scale/regression; manual for nuanced quality assessment.
Q30: What's ground truth in RAGAS?
Answer: The expected/correct answer that generated answers are compared against.
Results
| Score | Level |
|---|---|
| 27-30 | Excellent |
| 21-26 | Good |
| 15-20 | Average |
| Below 15 | Needs review |