Chapter 3: RAG Architecture Overview

Introduction

A RAG system consists of three main phases: Indexing, Retrieval, and Generation. Understanding each phase is crucial for building effective RAG applications.

The Three Phases

---

Phase 1: Indexing (Offline)

The indexing phase prepares your documents for efficient retrieval. This happens before any user query.

Step 1: Document Loading

Load documents from various sources:

from langchain.document_loaders import (
    PyPDFLoader,
    TextLoader,
    CSVLoader,
    WebBaseLoader
)

# Load a PDF
pdf_loader = PyPDFLoader("company_handbook.pdf")
pdf_docs = pdf_loader.load()

# Load from web
web_loader = WebBaseLoader("https://example.com/docs")
web_docs = web_loader.load()

# Combine all documents
all_docs = pdf_docs + web_docs

Step 2: Text Splitting (Chunking)

Documents are split into smaller chunks for better retrieval:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,      # Characters per chunk
    chunk_overlap=200,    # Overlap between chunks
    separators=["\n\n", "\n", ".", " "]
)

chunks = splitter.split_documents(all_docs)
# Result: List of smaller text chunks

Why Chunk?

• Smaller text units = more precise retrieval
• Avoids retrieving irrelevant content
• Fits within LLM context windows

Step 3: Embedding Generation

Convert text chunks into numerical vectors:

from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Each chunk becomes a vector
# "The return policy states..." → [0.023, -0.891, 0.156, ...]

What are embeddings?

• Dense vector representations of text
• Capture semantic meaning
• Similar texts have similar vectors
• Typically 384-3072 dimensions

Step 4: Vector Storage

Store vectors in a specialized database:

from langchain.vectorstores import Chroma

# Create vector store
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# Now we have an indexed, searchable knowledge base

Indexing Pipeline Summary

Raw Documents
    ↓ (Load)
Document Objects
    ↓ (Split)
Text Chunks
    ↓ (Embed)
Vectors
    ↓ (Store)
Vector Database ✓

---

Phase 2: Retrieval (Online)

When a user asks a question, we retrieve relevant chunks from our indexed knowledge base.

Step 1: Query Embedding

The user query is converted to the same vector space:

user_query = "What is the return policy?"

# Convert to embedding using same model
query_vector = embeddings.embed_query(user_query)
# Result: [0.034, -0.782, 0.234, ...]

Step 2: Similarity Search

Find chunks with vectors similar to the query:

# Search for top 5 most similar chunks
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 5}
)

relevant_chunks = retriever.get_relevant_documents(user_query)

How similarity works:

Query Vector:  [0.034, -0.782, 0.234, ...]

Chunk 1:       [0.038, -0.791, 0.228, ...]  → 0.98 similarity ✓
Chunk 2:       [0.890, 0.123, -0.456, ...]  → 0.23 similarity ✗
Chunk 3:       [0.041, -0.768, 0.241, ...]  → 0.95 similarity ✓

Retrieval Strategies

Strategy	Description	Use Case
Similarity	Pure vector similarity	General purpose
MMR	Maximum Marginal Relevance (diversity)	Avoid redundancy
Hybrid	Combine keyword + semantic	Better recall

# MMR - balances relevance and diversity
retriever_mmr = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={
        "k": 5,
        "fetch_k": 20,
        "lambda_mult": 0.7
    }
)

---

Phase 3: Generation (Online)

Combine the user query with retrieved context to generate a response.

Step 1: Prompt Construction

Build a prompt that includes context:

def build_prompt(query, retrieved_chunks):
    context = "\n\n".join([
        chunk.page_content for chunk in retrieved_chunks
    ])
    
    prompt = f"""You are a helpful assistant. Use the following context 
to answer the question. If you don't know the answer based on the 
context, say "I don't have enough information."

Context:
{context}

Question: {query}

Answer:"""
    
    return prompt

Step 2: LLM Generation

Send the prompt to the LLM:

from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(model="gpt-4", temperature=0)

prompt = build_prompt(user_query, relevant_chunks)
response = llm.invoke(prompt)

print(response.content)

Step 3: Response Delivery

The final response is grounded in the retrieved documents:

User: "What is the return policy?"

Retrieved Context:
"Our return policy allows returns within 30 days of purchase. 
Items must be unused and in original packaging. Refunds are 
processed within 5-7 business days."

Generated Response:
"According to the company policy, you can return items within 
30 days of purchase, provided they are unused and in their 
original packaging. Refunds typically take 5-7 business days 
to process."

---

Complete RAG Pipeline Code

from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA

# Phase 1: Indexing
loader = PyPDFLoader("documents/handbook.pdf")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(chunks, embeddings)

# Phase 2 & 3: Retrieval + Generation
llm = ChatOpenAI(model="gpt-4")

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever()
)

# Query the system
response = qa_chain.invoke("What is the return policy?")
print(response["result"])

---

Architecture Diagram

---

Key Metrics

Phase	Metric	Target
Indexing	Throughput	100+ docs/min
Indexing	Chunk size	500-1500 chars
Retrieval	Latency	<100ms
Retrieval	Top-K	3-10 chunks
Generation	Latency	<3s
Generation	Context fit	<75% of limit

---

Summary

Phase	When	What	Output
Indexing	Offline (once)	Process & store documents	Vector database
Retrieval	Online (per query)	Find relevant chunks	Context documents
Generation	Online (per query)	Create response	Final answer

Key Takeaways

1. Three distinct phases with different timing and purposes
2. Indexing is offline - done once or periodically
3. Retrieval + Generation are online - happen per query
4. Chunk size matters - affects both retrieval quality and cost
5. The pipeline is modular - each component can be optimized separately

---

Next Steps

Test your knowledge with the Module 1 Quiz!