Skip to main content

Chapter 3: Debugging RAG Pipelines

Common Failure Modes​

1. Wrong Documents Retrieved​

Symptoms: LLM gives wrong or irrelevant answers

Causes:

  • Poor chunk size
  • Wrong embedding model
  • Missing metadata filters

Solutions:

# Check what's being retrieved
docs = retriever.get_relevant_documents(query)
for doc in docs:
    print(f"Score: {doc.metadata.get('score')}")
    print(f"Content: {doc.page_content[:200]}...")
    print("---")

2. LLM Ignores Context​

Symptoms: Answer doesn't reflect retrieved docs

Causes:

  • Prompt template issues
  • Context too long
  • Context not formatted clearly

Solutions:

# Better prompt structure
prompt = """
CONTEXT (use this information to answer):
{context}

QUESTION: {question}

INSTRUCTIONS: Answer ONLY using the context above.
If the context doesn't contain the answer, say "I don't know."

ANSWER:"""

3. Hallucinations​

Symptoms: Confident but wrong answers

Causes:

  • Retrieved wrong docs
  • LLM extrapolating beyond context

Solutions:

  • Improve retrieval quality
  • Use explicit grounding prompts
  • Lower temperature

Tracing with LangSmith​

from langsmith import Client
import os

os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "your-key"

# Now all LangChain calls are traced
# View at smith.langchain.com

Debugging Checklist​

StepCheckTool
1Embeddings working?Embed test query
2Right docs retrieved?Print retrieved docs
3Context fits prompt?Check token count
4LLM sees context?Print full prompt
5Answer grounded?Compare to context

Performance Optimization​

Reduce Latency​

# Async retrieval
async def fast_rag(query):
    # Parallel embedding and retrieval
    embedding_task = embed_async(query)
    
    embedding = await embedding_task
    docs = await retrieve_async(embedding)
    
    return await generate_async(query, docs)

Reduce Costs​

  • Use smaller models where possible
  • Cache frequent queries
  • Batch embeddings
  • Limit context size