Chapter 1: GPU/TPU Management

GPU Selection Guide

GPU Type	VRAM	Use Case	Cost ($/hr)
NVIDIA T4	16GB	Inference, small training	$0.35
NVIDIA V100	32GB	Medium training	$2.50
NVIDIA A100	80GB	Large model training	$4.00
NVIDIA H100	80GB	LLM training	$8.00+

GPU Memory Optimization

import torch

# Enable gradient checkpointing
model.gradient_checkpointing_enable()

# Mixed precision training
from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()
with autocast():
    outputs = model(inputs)
    loss = criterion(outputs, targets)

scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

# Memory-efficient attention
from torch.nn.functional import scaled_dot_product_attention
# Uses Flash Attention when available

Key Takeaways

Summary

1. Match GPU to workload - don't over-provision
2. Use mixed precision for 2x speedup and memory savings
3. Monitor GPU utilization to identify bottlenecks
4. Consider spot instances for 60-90% cost savings

---

Next: Chapter 2: Distributed Training