Chapter 2: Local vs Cloud LLMs
Running AI on Your Own Machine
You don't have to use cloud APIs to access LLMs. With open-source models and the right hardware, you can run AI locally—completely private and offline.
What Is Local LLM Running?
Local LLM means running the AI model on your own computer or server, rather than sending requests to a cloud service.
The Key Difference
| Cloud LLM | Local LLM |
|---|---|
| Your data goes to OpenAI/Anthropic | Your data stays on your machine |
| Pay per use | Pay for hardware once |
| Always latest model | Update manually |
| Requires internet | Works offline |
Tools for Running Local LLMs
Ollama
The most user-friendly option for local LLMs.
Features:
- One-command installation
- Simple model management
- Works on Mac, Windows, Linux
- API compatible with OpenAI
Getting Started (example):
# Install Ollama, then:
ollama run llama3
LM Studio
Desktop app with a graphical interface.
Features:
- No command line needed
- Model browsing and download
- Chat interface built-in
- Works on Mac and Windows
llama.cpp
Technical but lightweight.
Features:
- Very efficient
- Works on minimal hardware
- Powers many other tools
- Full control over settings
Others
- Text Generation WebUI - Feature-rich interface
- GPT4All - Simple desktop app
- Jan - Modern, privacy-focused
Hardware Requirements
What You Need
Running LLMs locally requires decent hardware:
| Model Size | Minimum RAM | Recommended GPU |
|---|---|---|
| 7B parameters | 8GB | Optional (slower on CPU) |
| 13B parameters | 16GB | 8GB VRAM |
| 30B parameters | 32GB | 16GB VRAM |
| 70B parameters | 64GB+ | 24GB+ VRAM |
Understanding Model Sizes
| Model | Approximate Size | Quality Level |
|---|---|---|
| 7B | ~4-7 GB | Good for many tasks |
| 13B | ~7-13 GB | Better quality |
| 30B | ~15-30 GB | Approaching GPT-3.5 |
| 70B | ~35-70 GB | Near frontier quality |
Quantization: Making Models Smaller
Quantization reduces model precision to make it smaller and faster, with some quality trade-off.
Quantization Levels
| Level | Size Reduction | Quality Impact |
|---|---|---|
| FP16 | Baseline | Full quality |
| Q8 | ~50% smaller | Minimal loss |
| Q4 | ~75% smaller | Noticeable but usable |
| Q2 | ~87% smaller | Significant loss |
A 7B model can range from ~14GB (FP16) to ~3GB (Q4), making it runnable on laptops.
Privacy Benefits
Your Data Stays Private
When using cloud LLMs:
- Prompts are sent over internet
- Stored on provider servers
- May be used for training (check ToS)
- Subject to provider policies
When using local LLMs:
- Nothing leaves your machine
- No logging by third parties
- Full compliance control
- Works in air-gapped environments
Privacy-Critical Use Cases
| Industry | Why Local Matters |
|---|---|
| Healthcare | HIPAA compliance, patient data |
| Legal | Attorney-client privilege |
| Finance | Proprietary trading strategies |
| Government | Classified information |
| Enterprise | Trade secrets |
Performance Considerations
Latency
| Deployment | Typical Response Time |
|---|---|
| Cloud API (GPT-4) | 2-10 seconds |
| Local (M3 Max, 7B) | 1-3 seconds |
| Local (RTX 4090, 7B) | Less than 1 second |
| Local (CPU only, 7B) | 10-30 seconds |
Throughput
Cloud APIs often handle more parallel requests, while local deployment is limited by your hardware.
Cloud Advantages
Why Cloud Still Makes Sense
- No upfront investment - Don't need expensive hardware
- Always up-to-date - Latest models automatically
- Scalability - Handle traffic spikes
- Multi-modal - Image, audio, video capabilities
- Global availability - Access from anywhere
- Expert maintenance - Providers handle operations
Cost Comparison
Break-Even Analysis
| Scenario | Cloud Cost (Monthly) | Local Cost (Hardware) |
|---|---|---|
| Light use (1K queries) | ~$5-20 | $1,000+ GPU |
| Medium use (100K queries) | ~$500-2,000 | Same hardware |
| Heavy use (1M queries) | ~$5,000-20,000 | Same hardware |
Rule of thumb: Heavy users save money with local; light users save money with cloud.
Hybrid Approaches
Best of Both Worlds
Common Hybrid Patterns
| Pattern | How It Works |
|---|---|
| Sensitivity routing | Sensitive data → local; else → cloud |
| Complexity routing | Hard questions → better model |
| Fallback | Local first, cloud as backup |
| Local draft, cloud refine | Local generates, cloud polishes |
Getting Started with Local LLMs
Step 1: Check Your Hardware
- At least 8GB RAM for small models
- GPU recommended but not required
Step 2: Choose a Tool
- Beginners: LM Studio or Ollama
- Power users: llama.cpp or text-generation-webui
Step 3: Download a Model
- Start with Llama 3 8B or Mistral 7B
- Use Q4 quantization if limited RAM
Step 4: Experiment
- Try different prompts
- Compare to cloud models
- Find what works for your use case
Key Takeaways
✅ Local LLMs run on your own hardware, keeping data private
✅ Tools like Ollama and LM Studio make local running accessible
✅ Hardware requirements vary with model size (7B is most accessible)
✅ Quantization trades quality for smaller, faster models
✅ Privacy-critical applications often require local deployment
✅ Hybrid approaches combine local privacy with cloud capability
Think About It
- What personal or work data would you never want to send to a cloud LLM?
- How might edge devices (phones, laptops) change if they could run capable LLMs locally?
- When would you choose cloud even if local were technically possible?
Next Steps
Now let's look at where LLMs are heading in Chapter 3: Future Developments.