Chapter 2: Optimization for Inference
Maximizing Inference Performance
Techniques for fast, efficient inference in production.
Inference Servers
| Server | Strength | Use Case |
|---|---|---|
| vLLM | High throughput | Production APIs |
| TGI | HF integration | Simple deployment |
| llama.cpp | CPU efficiency | Edge/local |
| Ollama | Ease of use | Local development |
vLLM Deployment
from vllm import LLM, SamplingParams
llm = LLM(model="./merged_model")
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)
Quantization for Inference
| Level | Memory | Speed | Quality |
|---|---|---|---|
| FP16 | 100% | 1x | 100% |
| INT8 | 50% | 1.5x | 99% |
| INT4 | 25% | 2x | 95-98% |
Optimization Checklist
□ Merge LoRA adapter
□ Apply inference quantization (INT8/INT4)
□ Enable KV cache
□ Use appropriate batch size
□ Enable Flash Attention
□ Consider speculative decoding
□ Set up proper monitoring