MODULE 13 - Observability and Monitoring
Overview
This module covers observability with Amazon CloudWatch. You will learn to monitor your infrastructures, create alarms, and develop incident response runbooks.
Learning Objectives
- Understand CloudWatch metrics
- Configure alarms and notifications
- Analyze centralized logs
- Create monitoring dashboards
- Develop a minimal incident runbook
Weekly Content
1. Amazon CloudWatch - Overview
┌─────────────────────────────────────────────────────────────┐
│ AMAZON CLOUDWATCH │
├─────────────────────────────────────────────────────────────┤
│ │
│ "The nervous system of your AWS infrastructure" │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ METRICS LOGS ALARMS │ │
│ │ ┌────────┐ ┌────────┐ ┌────────┐ │ │
│ │ │ CPU % │ │ App │ │ Alert │ │ │
│ │ │ Mem % │ │ System │ │ if >80%│ │ │
│ │ │ Network│ │ Lambda │ │ │ │ │
│ │ └────────┘ └────────┘ └────────┘ │ │
│ │ │ │ │ │ │
│ │ └──────────────┴──────────────┘ │ │
│ │ │ │ │
│ │ ┌──────┴──────┐ │ │
│ │ │ DASHBOARD │ │ │
│ │ │ Real-time │ │ │
│ │ │ Graphs │ │ │
│ │ └─────────────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
2. Important EC2 Metrics
| Metric | Description | Typical Threshold |
|---|---|---|
| CPUUtilization | CPU Utilization | Alarm > 80% |
| StatusCheckFailed | Instance Health | Alarm = 1 |
| NetworkIn/Out | Network Traffic | According to baseline |
| DiskReadOps | Disk Operations | According to baseline |
| CPUCreditBalance | T3 Credits | Alarm < 20 |
3. Alarm Configuration
┌─────────────────────────────────────────────────────────────┐
│ ALARM ANATOMY │
├─────────────────────────────────────────────────────────────┤
│ │
│ Alarm: "High-CPU-Alert" │
│ │
│ Metric : EC2 CPUUtilization │
│ Statistic : Average │
│ Period : 5 minutes │
│ Threshold : > 80% │
│ Evaluation : 3 out of 5 periods │
│ │
│ Actions: │
│ • ALARM → SNS Topic → Email/SMS │
│ • OK → SNS Topic → Resolution Notification │
│ │
│ States: │
│ ┌────────────────────────────────────────────┐ │
│ │ OK │◄─────────────►│ ALARM │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ │
│ │ └───────────────┴───────┘ │ │
│ │ ▲ │ │
│ │ │ │ │
│ │ INSUFFICIENT_DATA │ │
│ │ (not enough points) │ │
│ └────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
4. Runbook Structure
# RUNBOOK: High CPU on EC2 Instance
## Symptom
"High-CPU-Alert" alarm triggered
## Impact
- Degraded performance
- Possible timeout for users
## Diagnostic
1. Check the alarm in CloudWatch
2. Identify the affected instance
3. SSH into the instance: `top` or `htop`
4. Identify the consuming process
## Actions
### If legitimate application process
- [ ] Check if normal traffic spike
- [ ] Consider scale-out if ASG
- [ ] Increase instance size if necessary
### If suspicious process
- [ ] Identify the process (`ps aux`)
- [ ] Check application logs
- [ ] Kill if non-critical
- [ ] Escalate if malicious suspect
## Resolution
- [ ] Confirm CPU back to normal
- [ ] Document the cause
- [ ] Update the runbook if necessary
## Escalation
If unresolved in 30 minutes → Infra Team
5. Recommended Dashboard
┌─────────────────────────────────────────────────────────────┐
│ PRODUCTION DASHBOARD │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ CPU Utilization │ │ Memory Usage │ │
│ │ ▃▅▆▇█▇▅▃▂ │ │ ▂▃▄▅▆▇▇▆▅ │ │
│ │ Instance 1, 2, 3 │ │ Instance 1, 2, 3 │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Request Count │ │ Error Rate │ │
│ │ ▁▂▃▄▅▆▇███▇▅ │ │ ▁▁▁▁▁▂▁▁▁▁▁ │ │
│ │ ALB Requests/min │ │ 5xx Errors % │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────┐ │
│ │ Active Alarms │ │
│ │ 🟢 CPU OK 🟢 Disk OK 🔴 ErrorRate HIGH │ │
│ └─────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Exercise: Runbook Creation
Scenario
Create a runbook for the following scenario:
- Alarm: EC2 Instance StatusCheckFailed
- Impact: Website inaccessible
- Possible actions: Restart, replace, escalate
Template to Complete
- Symptom and impact
- Diagnostic steps
- Corrective actions (with decisions)
- Resolution criteria
- Escalation process
Deliverable
1-page Runbook + Alert Configuration:
- Structured runbook for a typical incident
- List of recommended alarms
- Thresholds and justifications
Navigation
- Observability Concepts
- Validation Quiz - 30 questions to validate your knowledge
- Lab: Monitoring
Next week: MODULE 14 - FinOps Governance