Chapter 2: Model Serving Frameworks

Framework Comparison

Framework	Best For	GPU Support	Multi-Model
TorchServe	PyTorch models	✅	✅
TF Serving	TensorFlow models	✅	✅
Triton	Multi-framework, high perf	✅	✅
BentoML	Rapid prototyping	✅	✅

TorchServe Example

# Create model archive
torch-model-archiver \
    --model-name churn_model \
    --version 1.0 \
    --model-file model.py \
    --serialized-file model.pth \
    --handler handler.py \
    --export-path model_store

# Start server
torchserve --start --model-store model_store \
    --models churn_model=churn_model.mar

Triton Inference Server

model_repository/
├── churn_model/
│   ├── config.pbtxt
│   └── 1/
│       └── model.onnx

# config.pbtxt
name: "churn_model"
platform: "onnxruntime_onnx"
max_batch_size: 64
input [
  { name: "features", data_type: TYPE_FP32, dims: [10] }
]
output [
  { name: "predictions", data_type: TYPE_INT64, dims: [1] }
]

---

Next: Chapter 3: Containerization with Docker