Chapter 3: Open Source Models

Why Open Source?

• Privacy: Run locally, data never leaves your system
• Cost: No per-token API fees
• Customization: Fine-tune for specific use cases
• Control: No external dependencies

Popular Open Source Models

Model	Creator	Sizes	Strengths
Llama 3	Meta	8B, 70B	General purpose, well-documented
Mistral	Mistral AI	7B	Efficient, strong performance
Mixtral	Mistral AI	8x7B	Mixture of experts, versatile
Phi-2/3	Microsoft	2.7B-3.8B	Small but capable

Prompting Differences

Open-source models may require:

• Specific prompt formats (e.g., [INST]...[/INST] for Llama)
• Adjusted expectations for smaller models
• More explicit instructions

Llama Format Example

[INST] You are a helpful assistant.

Write a haiku about programming. [/INST]

Deployment Options

• Ollama: Easy local deployment
• vLLM: Production-ready serving
• Hugging Face: Cloud or local with transformers
• llama.cpp: Efficient CPU inference

Key Takeaways

Summary

1. Open source = privacy, cost savings, control
2. Different models need different prompt formats
3. Smaller models need clearer instructions
4. Choose deployment method based on scale

---

← Previous: Claude | Take the Quiz →