Chapter 1: Model Performance Monitoring

Key Metrics to Monitor

Category	Metrics
Model Quality	Accuracy, AUC, F1, precision, recall
Latency	P50, P95, P99 response times
Throughput	Requests per second
Errors	Error rate, error types
Business	Revenue impact, conversion rates

Prometheus Metrics

from prometheus_client import Counter, Histogram, Gauge

# Prediction counter
predictions = Counter(
    'model_predictions_total',
    'Total predictions',
    ['model_version', 'prediction_class']
)

# Latency histogram
latency = Histogram(
    'model_prediction_latency_seconds',
    'Prediction latency',
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0]
)

# Model accuracy gauge (updated periodically)
accuracy = Gauge(
    'model_accuracy_current',
    'Current model accuracy'
)

@app.post("/predict")
async def predict(request: PredictionRequest):
    with latency.time():
        result = model.predict(request.features)
        predictions.labels(model_version="2.1.0", prediction_class=str(result)).inc()
        return {"prediction": result}

---

Next: Chapter 2: Drift Detection