Chapter 1: Introduction to Vector Stores

What Makes Vector Databases Special?

Traditional databases are optimized for exact matches:

SELECT * FROM products WHERE id = 123;

Vector databases are optimized for similarity search:

# Find the 10 most similar vectors to query_vector
results = vector_db.search(query_vector, top_k=10)

Indexing Algorithms

HNSW (Hierarchical Navigable Small World)

The most popular algorithm for vector search:

# HNSW parameters
index = create_hnsw_index(
    vectors=embeddings,
    M=16,              # Number of connections per node
    ef_construction=200,  # Build-time quality
    ef_search=50       # Query-time quality
)

Characteristics:

• Excellent query speed
• High memory usage
• Best for most use cases

IVF (Inverted File Index)

Clusters vectors for faster search:

# IVF parameters
index = create_ivf_index(
    vectors=embeddings,
    nlist=100,         # Number of clusters
    nprobe=10          # Clusters to search
)

Characteristics:

• Lower memory than HNSW
• Requires training
• Good for large datasets

Flat Index

Brute-force exact search:

# No parameters - exact search
index = create_flat_index(vectors=embeddings)

Characteristics:

• Perfect accuracy
• Slowest search
• Good for small datasets

Algorithm Comparison

Algorithm	Speed	Memory	Accuracy	Best For
HNSW	Fast	High	~95%	Most cases
IVF	Medium	Medium	~90%	Large scale
Flat	Slow	Low	100%	Small datasets

Metadata Storage

Vector databases store metadata alongside vectors:

vectorstore.add(
    vectors=[embedding],
    documents=["Company policy document"],
    metadata=[{
        "source": "handbook.pdf",
        "page": 15,
        "department": "HR",
        "date": "2024-01-15"
    }]
)

# Filter by metadata during search
results = vectorstore.search(
    query_vector,
    filter={"department": "HR"},
    top_k=5
)

Key Features to Consider

Feature	Description
Filtering	Query by metadata
Updates	Modify existing vectors
Deletion	Remove vectors
Namespaces	Isolate data
Replication	High availability
Sharding	Scale horizontally

Summary

• Vector databases enable similarity search
• HNSW is the most common algorithm
• Metadata filtering is essential for production
• Choose based on scale and requirements

---

Next Steps

Continue to Chapter 2: Popular Options.