Chapter 2: Text Splitting Strategies

Introduction

Text splitting (chunking) is crucial for RAG performance. The right chunk size and strategy significantly impacts retrieval quality and generation accuracy.

Why Chunking Matters

Problems with no chunking:

• Entire documents are too large
• Retrieval returns irrelevant content
• Wastes LLM context window

Key Parameters

Chunk Size

The number of characters/tokens per chunk:

Chunk Size	Pros	Cons
Small (256)	Precise retrieval	May lose context
Medium (512-1024)	Balanced	Good for most cases
Large (2000+)	More context	Less precise retrieval

Chunk Overlap

Shared text between consecutive chunks:

Chunk 1: [_______________]
Chunk 2:            [_______________]
              ^overlap^

Why overlap?

• Prevents cutting sentences mid-thought
• Preserves context across chunk boundaries
• Typical: 10-20% of chunk size

Splitting Strategies

1. Recursive Character Splitting

The most common approach - splits by hierarchy of separators:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_documents(documents)

Separator hierarchy:

1. Double newlines (paragraphs)
2. Single newlines
3. Periods (sentences)
4. Spaces (words)
5. Characters (last resort)

2. Token-Based Splitting

Uses actual token counts (important for LLM limits):

from langchain.text_splitter import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=500,      # tokens, not characters
    chunk_overlap=50,
    encoding_name="cl100k_base"  # GPT-4 tokenizer
)

3. Sentence Splitting

Preserves sentence boundaries:

from langchain.text_splitter import SentenceTransformersTokenTextSplitter

splitter = SentenceTransformersTokenTextSplitter(
    chunk_overlap=50,
    tokens_per_chunk=256
)

4. Semantic Splitting

Splits based on semantic similarity:

from langchain_experimental.text_splitter import SemanticChunker
from langchain.embeddings import OpenAIEmbeddings

splitter = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile"
)

# Chunks based on semantic similarity
semantic_chunks = splitter.split_documents(documents)

5. Markdown Header Splitting

Respects document structure:

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on
)

Choosing Chunk Size

def optimal_chunk_size(embedding_model, llm_context_window, top_k):
    """
    Calculate optimal chunk size
    """
    # Leave room for prompt template and response
    available_tokens = llm_context_window * 0.6
    
    # Divide by number of chunks to retrieve
    chunk_tokens = available_tokens / top_k
    
    # Convert to characters (roughly 4 chars per token)
    return int(chunk_tokens * 4)

# Example: GPT-4 with 8K context, retrieving 5 chunks
optimal = optimal_chunk_size(
    embedding_model="text-embedding-3-small",
    llm_context_window=8192,
    top_k=5
)
# Result: ~3900 characters per chunk

Comparison Table

Strategy	Best For	Chunk Quality
Recursive Character	General purpose	Good
Token-Based	LLM context management	Excellent
Sentence	Q&A systems	Good
Semantic	Varying content	Excellent
Markdown Headers	Structured docs	Excellent

Best Practices

1. Start with defaults - 1000 chars, 200 overlap
2. Test different sizes - Evaluate retrieval quality
3. Match to content - Technical docs may need larger chunks
4. Consider embeddings - Some models have max token limits
5. Preserve metadata - Track chunk source and position

# Production chunking pipeline
def chunk_documents(documents, chunk_size=1000, overlap=200):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=overlap,
        length_function=len,
        add_start_index=True  # Track position
    )
    
    chunks = splitter.split_documents(documents)
    
    # Add chunk index to metadata
    for i, chunk in enumerate(chunks):
        chunk.metadata["chunk_index"] = i
        chunk.metadata["total_chunks"] = len(chunks)
    
    return chunks

Summary

• Chunk size affects retrieval precision vs context
• Overlap prevents information loss at boundaries
• Strategy should match your document type
• Test and iterate to find optimal settings

---

Next Steps

Continue to Chapter 3: File Types to learn about handling complex documents.