Chapter 3: Self-Query and Metadata Filtering

What is Self-Query?

The LLM extracts both the semantic query and structured filters from user input:

User: "Find HR policies from 2023"

Extracted:
- Semantic query: "HR policies"
- Filters: {department: "HR", year: 2023}

Implementation with LangChain

from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain.chains.query_constructor.base import AttributeInfo

# Define metadata attributes
metadata_field_info = [
    AttributeInfo(
        name="department",
        description="The department (HR, Engineering, Sales)",
        type="string"
    ),
    AttributeInfo(
        name="year",
        description="Publication year",
        type="integer"
    ),
    AttributeInfo(
        name="document_type",
        description="Type of document (policy, guide, memo)",
        type="string"
    )
]

# Create self-query retriever
retriever = SelfQueryRetriever.from_llm(
    llm=llm,
    vectorstore=vectorstore,
    document_contents="Company internal documents",
    metadata_field_info=metadata_field_info
)

# Query with natural language filters
results = retriever.get_relevant_documents(
    "Show me engineering guidelines from 2024"
)
# Automatically filters: department=Engineering, year=2024

Supported Comparisons

Operator	Example
$eq	year == 2023
$ne	status != "draft"
$gt	year > 2020
$gte	year >= 2020
$lt	year < 2024
$in	department in ["HR", "Legal"]

Best Practices

1. Clear metadata schema - Well-defined attributes
2. Good descriptions - Help LLM understand fields
3. Fallback logic - Handle extraction failures
4. Test thoroughly - Various query patterns

Combining Approaches

# Multi-query + Self-query
def advanced_retrieve(query, retriever):
    # Generate variations
    queries = generate_variations(query)
    
    # Self-query each variation
    all_results = []
    for q in queries:
        results = self_query_retriever.get_relevant_documents(q)
        all_results.extend(results)
    
    # Deduplicate and rank
    return rank_and_dedupe(all_results)