Chapter 2: Inference Optimization

Quantization

Precision	Memory	Speed	Quality
FP32	100%	1x	Best
FP16	50%	2x	~Best
INT8	25%	3x	Good
INT4	12.5%	4x	Acceptable

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# 4-bit quantization
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    quantization_config=quantization_config,
    device_map="auto"
)

KV Cache Optimization

# vLLM handles KV cache automatically with PagedAttention
# This allows efficient memory usage for variable-length sequences

# Traditional: Pre-allocate max sequence length
# PagedAttention: Dynamically allocate in pages

llm = LLM(
    model="model-name",
    max_model_len=4096,
    block_size=16,  # Page size
    gpu_memory_utilization=0.9
)

---

Next: Chapter 3: Cost Management