Chapter 3: Alerting and Incident Response

Alert Design Principles

1. Actionable: Every alert should have a clear response
2. Timely: Alert before users notice issues
3. Informative: Include context for diagnosis
4. Prioritized: Critical vs warning vs informational

Example Alert Rules

# prometheus-alerts.yml
groups:
  - name: ml-model-alerts
    rules:
      - alert: ModelLatencyHigh
        expr: histogram_quantile(0.99, rate(model_prediction_latency_seconds_bucket[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Model latency is high"
          description: "P99 latency is {{ $value }}s"
          runbook_url: "https://wiki/runbooks/model-latency"
          
      - alert: ModelAccuracyDegraded
        expr: model_accuracy_current < 0.8
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Model accuracy has dropped below threshold"
          description: "Current accuracy: {{ $value }}"

Runbook Template

# Runbook: Model Accuracy Degraded

## Summary
Model prediction accuracy has dropped below the acceptable threshold.

## Severity
Critical

## Impact
Incorrect predictions affecting user experience and revenue.

## Diagnosis Steps
1. Check recent data drift metrics
2. Review prediction distribution changes
3. Check for upstream data issues

## Resolution Steps
1. If data drift detected: Trigger retraining pipeline
2. If data quality issue: Alert data engineering
3. If bug: Rollback to previous model version

## Escalation
- L1: ML On-call Engineer
- L2: ML Platform Team Lead
- L3: Head of ML Engineering

---

Next: Module 9 Quiz