Chapter 1: What are Embeddings?

Introduction

Embeddings are dense vector representations of text that capture semantic meaning. They enable computers to understand and compare the meaning of text, not just the words.

The Concept

Think of embeddings as coordinates in a "meaning space":

"happy" → [0.2, 0.8, -0.1, 0.5, ...]
"joyful" → [0.21, 0.79, -0.12, 0.48, ...]  // Similar!
"sad" → [-0.3, 0.1, 0.7, -0.4, ...]        // Different!

Similar meanings = Similar vectors = Close in space

How Embeddings Work

1. Text to Vector

from openai import OpenAI

client = OpenAI()

# Text becomes a vector
text = "Machine learning is fascinating"
response = client.embeddings.create(
    input=text,
    model="text-embedding-3-small"
)

embedding = response.data[0].embedding
# Result: [0.023, -0.891, 0.156, ..., 0.342]
# 1536 dimensions

2. Similarity Computation

import numpy as np

def cosine_similarity(vec1, vec2):
    """Calculate cosine similarity between two vectors"""
    dot_product = np.dot(vec1, vec2)
    norm1 = np.linalg.norm(vec1)
    norm2 = np.linalg.norm(vec2)
    return dot_product / (norm1 * norm2)

# Example
similarity = cosine_similarity(
    embed("What is RAG?"),
    embed("Explain Retrieval Augmented Generation")
)
# Result: ~0.92 (very similar!)

similarity = cosine_similarity(
    embed("What is RAG?"),
    embed("How to cook pasta")
)
# Result: ~0.15 (not similar)

Embedding Dimensions

Model	Dimensions	Vector Size
text-embedding-3-small	1536	6KB
text-embedding-3-large	3072	12KB
Cohere embed-v3	1024	4KB
BGE-small	384	1.5KB
BGE-large	1024	4KB

Why Dimensions Matter

• More dimensions = More nuanced representations
• Fewer dimensions = Faster search, less storage
• Trade-off between quality and efficiency

Similarity Metrics

Cosine Similarity

Most common for text embeddings:

# Range: -1 to 1
# 1 = identical direction
# 0 = orthogonal (unrelated)
# -1 = opposite direction

cosine_sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Euclidean Distance

Measures straight-line distance:

# Lower = more similar
euclidean_dist = np.linalg.norm(a - b)

Dot Product

When vectors are normalized:

# For normalized vectors, same as cosine similarity
dot_product = np.dot(a, b)

Visualizing Embeddings

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

# Reduce to 2D for visualization
texts = ["cat", "dog", "car", "truck", "happy", "sad"]
embeddings = [embed(t) for t in texts]

tsne = TSNE(n_components=2, random_state=42)
reduced = tsne.fit_transform(embeddings)

plt.scatter(reduced[:, 0], reduced[:, 1])
for i, text in enumerate(texts):
    plt.annotate(text, (reduced[i, 0], reduced[i, 1]))
plt.show()

Expected clusters:

• "cat", "dog" → Animals cluster
• "car", "truck" → Vehicles cluster
• "happy", "sad" → Emotions cluster

Key Properties of Good Embeddings

Property	Description
Semantic Similarity	Similar meanings → Close vectors
Compositionality	Phrases encoded meaningfully
Cross-lingual	Same concept similar across languages
Dense	All dimensions carry meaning

Embeddings in RAG

Summary

• Embeddings convert text to numerical vectors
• Similar meanings produce similar vectors
• Cosine similarity is the standard metric
• Dimensions affect quality vs efficiency
• Embeddings enable semantic search in RAG

---

Next Steps

Continue to Chapter 2: Embedding Models to explore available options.