Chapter 1: Document Loaders

Introduction

Document loaders are the entry point of any RAG pipeline. They extract text content from various file formats and sources, preparing it for further processing.

LangChain Document Loaders

LangChain provides 100+ document loaders for different sources:

PDF Loaders

from langchain.document_loaders import (
    PyPDFLoader,
    PDFPlumberLoader,
    PyMuPDFLoader,
    UnstructuredPDFLoader
)

# Basic PDF loading
loader = PyPDFLoader("document.pdf")
documents = loader.load()

# For better table extraction
loader = PDFPlumberLoader("document.pdf")
documents = loader.load()

# For scanned PDFs (with OCR)
loader = UnstructuredPDFLoader(
    "scanned.pdf",
    mode="elements",
    strategy="ocr_only"
)

Text and Markdown

from langchain.document_loaders import (
    TextLoader,
    UnstructuredMarkdownLoader
)

# Plain text
loader = TextLoader("readme.txt", encoding="utf-8")
docs = loader.load()

# Markdown with structure
loader = UnstructuredMarkdownLoader("docs.md")
docs = loader.load()

Web Content

from langchain.document_loaders import (
    WebBaseLoader,
    UnstructuredURLLoader,
    RecursiveUrlLoader
)

# Single page
loader = WebBaseLoader("https://example.com/docs")
docs = loader.load()

# Multiple URLs
loader = UnstructuredURLLoader(urls=[
    "https://example.com/page1",
    "https://example.com/page2"
])

# Crawl entire site
loader = RecursiveUrlLoader(
    url="https://docs.example.com",
    max_depth=3
)

Office Documents

from langchain.document_loaders import (
    Docx2txtLoader,
    UnstructuredWordDocumentLoader,
    UnstructuredPowerPointLoader,
    UnstructuredExcelLoader
)

# Word documents
loader = Docx2txtLoader("report.docx")
docs = loader.load()

# PowerPoint
loader = UnstructuredPowerPointLoader("presentation.pptx")
docs = loader.load()

# Excel
loader = UnstructuredExcelLoader("data.xlsx")
docs = loader.load()

LlamaIndex Readers

LlamaIndex offers SimpleDirectoryReader and specialized readers:

from llama_index import SimpleDirectoryReader
from llama_index.readers import PDFReader, DocxReader

# Load entire directory
reader = SimpleDirectoryReader(
    input_dir="./documents",
    recursive=True,
    exclude_hidden=True,
    required_exts=[".pdf", ".docx", ".txt"]
)
documents = reader.load_data()

# Specific reader
pdf_reader = PDFReader()
docs = pdf_reader.load_data(file_path="document.pdf")

Database Loaders

from langchain.document_loaders import (
    SQLDatabaseLoader,
    CSVLoader
)

# From SQL database
loader = SQLDatabaseLoader(
    query="SELECT content, metadata FROM documents",
    db=database_connection
)

# From CSV
loader = CSVLoader(
    file_path="data.csv",
    source_column="source",
    csv_args={"delimiter": ","}
)

Document Structure

All loaders return Document objects:

from langchain.schema import Document

# Document structure
doc = Document(
    page_content="The actual text content...",
    metadata={
        "source": "document.pdf",
        "page": 1,
        "author": "John Doe",
        "created_date": "2024-01-15"
    }
)

Best Practices

Practice	Description
Error Handling	Always catch file not found and parsing errors
Encoding	Specify UTF-8 encoding for text files
Metadata	Preserve source information for citations
Batch Loading	Load multiple files efficiently

# Robust loading with error handling
def load_documents_safely(file_paths):
    documents = []
    errors = []
    
    for path in file_paths:
        try:
            loader = get_loader_for_file(path)
            docs = loader.load()
            documents.extend(docs)
        except Exception as e:
            errors.append({"file": path, "error": str(e)})
    
    return documents, errors

Summary

• LangChain offers 100+ loaders for various formats
• LlamaIndex provides SimpleDirectoryReader for easy batch loading
• Always preserve metadata for source attribution
• Handle errors gracefully in production systems

---

Next Steps

Continue to Chapter 2: Text Splitting to learn about chunking strategies.