Chapter 3: Serving at Scale

vLLM

High-throughput serving with PagedAttention:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b-hf")
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)

outputs = llm.generate(["Hello, my name is"], sampling_params)

Text Generation Inference (TGI)

Hugging Face's production server:

docker run --gpus all -p 8080:80 \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id meta-llama/Llama-2-7b-hf

Comparison

Framework	Best For	Key Feature
vLLM	Throughput	PagedAttention
TGI	Production	HF integration
Triton	Multi-model	NVIDIA optimized

Load Balancing

# Example: Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3  # Multiple replicas for load balancing
  template:
    spec:
      containers:
      - name: llm-server
        resources:
          limits:
            nvidia.com/gpu: 1

---

← Chapter 2Quiz →