Capítulo 3: Alertas y Respuesta a Incidentes

Principios de Diseño de Alertas

1. Accionable: Cada alerta debe tener una respuesta clara
2. Oportuna: Alertar antes de que los usuarios noten problemas
3. Informativa: Incluir contexto para el diagnóstico
4. Priorizada: Crítica vs advertencia vs informativa

Ejemplo de Reglas de Alerta

# prometheus-alerts.yml
groups:
  - name: ml-model-alerts
    rules:
      - alert: ModelLatencyHigh
        expr: histogram_quantile(0.99, rate(model_prediction_latency_seconds_bucket[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Model latency is high"
          description: "P99 latency is {{ $value }}s"
          runbook_url: "https://wiki/runbooks/model-latency"
          
      - alert: ModelAccuracyDegraded
        expr: model_accuracy_current < 0.8
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Model accuracy has dropped below threshold"
          description: "Current accuracy: {{ $value }}"

Plantilla de Runbook

# Runbook: Model Accuracy Degraded

## Summary
Model prediction accuracy has dropped below the acceptable threshold.

## Severity
Critical

## Impact
Incorrect predictions affecting user experience and revenue.

## Diagnosis Steps
1. Check recent data drift metrics
2. Review prediction distribution changes
3. Check for upstream data issues

## Resolution Steps
1. If data drift detected: Trigger retraining pipeline
2. If data quality issue: Alert data engineering
3. If bug: Rollback to previous model version

## Escalation
- L1: ML On-call Engineer
- L2: ML Platform Team Lead
- L3: Head of ML Engineering

---

Siguiente: Cuestionario del Módulo 9