Chapter 1: Scaling RAG Applications

Scaling Challenges

Challenge	Impact	Solution
Millions of docs	Slow indexing	Batch processing
High QPS	Timeout	Load balancing
Large vectors	Memory limits	Sharding

Distributed Vector Databases

# Milvus cluster configuration
from pymilvus import connections, Collection

# Connect to distributed cluster
connections.connect(
    alias="default",
    host="milvus-cluster",
    port="19530"
)

# Data automatically distributed across nodes

Async Processing

import asyncio
from langchain.embeddings import OpenAIEmbeddings

async def batch_embed_async(texts, batch_size=100):
    """Embed texts in async batches"""
    embeddings = OpenAIEmbeddings()
    
    async def embed_batch(batch):
        return await asyncio.to_thread(
            embeddings.embed_documents, batch
        )
    
    tasks = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        tasks.append(embed_batch(batch))
    
    results = await asyncio.gather(*tasks)
    return [e for batch in results for e in batch]

Queue-Based Indexing

# Producer: Add to queue
import redis

r = redis.Redis()
r.lpush("index_queue", document_json)

# Consumer: Process queue
while True:
    doc = r.brpop("index_queue")
    process_and_index(doc)

Scale Recommendations

Documents	Architecture
<100K	Single instance
100K-1M	Replicated DB
1M-10M	Sharded cluster
>10M	Distributed system