Chapter 3: Choosing the Right Embedding Model

Introduction

Selecting the right embedding model depends on your specific requirements: quality, cost, latency, language support, and deployment constraints.

Decision Framework

MTEB Benchmarks

The Massive Text Embedding Benchmark (MTEB) is the standard for comparing embedding models:

Leaderboard (Top Models)

Rank	Model	Average Score	Dimensions
1	text-embedding-3-large	64.59	3072
2	Cohere embed-v3	64.47	1024
3	BGE-large-en-v1.5	64.23	1024
4	E5-large-v2	62.25	1024
5	text-embedding-3-small	62.26	1536

Task-Specific Performance

Model	Retrieval	STS	Classification
text-embedding-3-large	59.2	65.4	71.2
Cohere embed-v3	58.9	64.8	70.1
BGE-large	57.8	63.9	69.5

Cost Analysis

Per Million Tokens

def calculate_monthly_cost(
    docs_per_month: int,
    avg_tokens_per_doc: int,
    model: str
):
    prices = {
        "text-embedding-3-small": 0.02,
        "text-embedding-3-large": 0.13,
        "cohere-embed-v3": 0.10,
        "self-hosted": 0.0  # Infrastructure costs separate
    }
    
    total_tokens = docs_per_month * avg_tokens_per_doc
    millions = total_tokens / 1_000_000
    
    return millions * prices[model]

# Example: 1M documents, 500 tokens each
cost = calculate_monthly_cost(1_000_000, 500, "text-embedding-3-small")
# Result: $10/month for embedding

Total Cost Comparison

Scale	OpenAI Small	OpenAI Large	Self-Hosted
100K docs	$1	$6.50	~$50/mo (infra)
1M docs	$10	$65	~$100/mo
10M docs	$100	$650	~$200/mo

Latency Considerations

API Latency

Model	Avg Latency	P99 Latency
text-embedding-3-small	50ms	150ms
text-embedding-3-large	80ms	200ms
Cohere embed-v3	60ms	180ms
Self-hosted (GPU)	10ms	30ms

Batching Strategy

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def batch_embed(texts, batch_size=100):
    """Efficient batching for large datasets"""
    embeddings = []
    
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        response = await client.embeddings.create(
            input=batch,
            model="text-embedding-3-small"
        )
        embeddings.extend([e.embedding for e in response.data])
    
    return embeddings

Language Support

Multilingual Models

Model	Languages	Quality
text-embedding-3	100+	Excellent
Cohere multilingual	100+	Excellent
BGE-m3	100+	Excellent
paraphrase-multilingual	50+	Good

Language-Specific

# For specific languages, dedicated models may be better
models_by_language = {
    "en": "BAAI/bge-large-en-v1.5",
    "zh": "BAAI/bge-large-zh-v1.5",
    "multilingual": "BAAI/bge-m3"
}

Deployment Options

Option 1: API (Recommended for most)

# Pros: No infrastructure, always updated
# Cons: Ongoing cost, latency, data leaves your network

from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()

Option 2: Self-Hosted

# Pros: No per-token cost, lower latency, data stays internal
# Cons: Infrastructure cost, maintenance

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/bge-large-en-v1.5')
# Load once, use many times

Option 3: Hybrid

# Use self-hosted for high volume, API for edge cases
def get_embeddings(texts, use_api=False):
    if use_api or len(texts) < 10:
        return api_embeddings.embed_documents(texts)
    else:
        return local_model.encode(texts)

Selection Checklist

Factor	Question	Recommendation
Budget	Monthly budget?	>$100: API, <$100: Self-host
Volume	Documents/month?	>10M: Consider self-host
Latency	Real-time needed?	<20ms: Self-host required
Languages	Multilingual?	Use multilingual models
Privacy	Data sensitivity?	Self-host if critical
Quality	Accuracy critical?	Pay for best model

Recommendation Matrix

Use Case	Recommended Model
Startup / MVP	text-embedding-3-small
Enterprise (quality)	text-embedding-3-large
Enterprise (cost)	BGE-large (self-hosted)
Multilingual	Cohere multilingual
Edge / Offline	all-MiniLM-L6-v2
Maximum quality	Cohere embed-v3

Summary

1. Start with API (OpenAI small) for prototyping
2. Evaluate MTEB scores for your task type
3. Calculate costs at your expected scale
4. Consider self-hosting at high volume
5. Test with your data before committing

---

Next Steps

Test your knowledge with the Module 3 Quiz!