Chapter 3: LLaMA, Mistral, and Open Source Models

Meta's LLaMA Series

Version	Sizes	License	Key Features
LLaMA 1	7-65B	Research	First competitive open model
LLaMA 2	7-70B	Commercial	Full open weights
LLaMA 3	8-70B+	Commercial	State-of-the-art open

Mistral AI

# Mistral's innovations
innovations = {
    "Sliding Window Attention": "Efficient long sequences",
    "Grouped Query Attention": "Memory efficient",
    "Mixtral (MoE)": "46B params, 12.9B active"
}

Using Open Models

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.2",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

Comparison

Aspect	Closed (GPT-4)	Open (LLaMA)
Capability	Higher	Competitive
Control	Limited	Full
Cost	Per-token	Infrastructure
Privacy	Data sent to API	Local

Key Takeaways

Summary

1. LLaMA democratized LLM access
2. Mistral innovates on efficiency
3. Open models enable customization and privacy

---

← Chapter 2: ClaudeModule 4 Quiz →