Chapter 2: Multimodal LLMs

Vision-Language Models

Combining visual understanding with language:

Image → Vision Encoder → Image Tokens → LLM → Text Output

Key Models

Model	Company	Architecture
GPT-4V	OpenAI	Proprietary
Gemini	Google	Native multimodal
LLaVA	Open source	CLIP + LLaMA
Claude 3	Anthropic	Vision-enabled

LLaVA Architecture

# Simplified LLaVA structure
class LLaVA:
    def __init__(self):
        self.vision_encoder = CLIP()  # Image understanding
        self.projection = Linear()     # Align dimensions
        self.llm = LLaMA()            # Language model
    
    def forward(self, image, text):
        image_features = self.vision_encoder(image)
        image_tokens = self.projection(image_features)
        combined = concat(image_tokens, text_tokens)
        return self.llm(combined)

Capabilities

• Image description
• Visual Q&A
• Document understanding
• Chart/diagram analysis

Challenges

• Hallucination about image content
• Limited spatial understanding
• High compute requirements

---

← Chapter 1Chapter 3 →