CloudWatch Alarms and Alerts
Introduction
CloudWatch alarms monitor your metrics and trigger automatic actions when thresholds are reached. It is an essential component of observability and operational responsiveness.
Alarm Architecture
┌─────────────────────────────────────────────────────────────────┐
│ ALARM FLOW │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────────────┐ │
│ │ Metric │────►│ Alarm │────►│ Actions │ │
│ │ │ │ │ │ │ │
│ │ CPU 85% │ │ Threshold>80%│ │ • SNS (email/SMS) │ │
│ └──────────┘ └──────────┘ │ • Auto Scaling │ │
│ │ • Lambda │ │
│ │ • EC2 (stop/reboot) │ │
│ └──────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
Alarm States
┌─────────────────────────────────────────────────────────────────┐
│ ALARM STATES │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ │
│ │ OK │ Metric within normal limits │
│ │ (Green) │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ Threshold exceeded │
│ ┌─────────────────┐ │
│ │ ALARM │ Threshold reached → Actions triggered │
│ │ (Red) │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ Missing data │
│ ┌─────────────────┐ │
│ │ INSUFFICIENT │ Not enough data to evaluate │
│ │ DATA │ │
│ │ (Gray) │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
Alarm Configuration
Configuration Elements
┌─────────────────────────────────────────────────────────────────┐
│ ALARM ANATOMY │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. METRIC │
│ └── Which metric to monitor? │
│ Ex: CPUUtilization, NetworkIn, Errors │
│ │
│ 2. STATISTIC │
│ └── How to aggregate the data? │
│ Average, Sum, Minimum, Maximum, SampleCount │
│ │
│ 3. PERIOD │
│ └── Time window for each point │
│ 60s, 300s (5min), 3600s (1h) │
│ │
│ 4. THRESHOLD │
│ └── What value triggers the alarm? │
│ > 80%, < 10, >= 1000 │
│ │
│ 5. EVALUATION PERIODS │
│ └── How many periods in violation? │
│ "3 out of 5 periods" (avoids false positives) │
│ │
│ 6. ACTIONS │
│ └── What to do when the alarm triggers? │
│ Notification, Auto Scaling, etc. │
│ │
└─────────────────────────────────────────────────────────────────┘
Example: EC2 CPU Alarm
# Pseudo-configuration CloudFormation
HighCPUAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: "EC2-HighCPU-Production"
AlarmDescription: "CPU greater than 80% for 5 minutes"
# Metric
Namespace: "AWS/EC2"
MetricName: "CPUUtilization"
Dimensions:
- Name: InstanceId
Value: "i-1234567890abcdef0"
# Evaluation
Statistic: "Average"
Period: 60 # 1 minute
EvaluationPeriods: 5 # 5 periods
DatapointsToAlarm: 3 # 3 out of 5 in violation
# Threshold
ComparisonOperator: "GreaterThanThreshold"
Threshold: 80
# Actions
AlarmActions:
- "arn:aws:sns:us-east-1:123456789:OpsAlert"
OKActions:
- "arn:aws:sns:us-east-1:123456789:OpsAlert"
Common Alarm Types
1. Performance Alarms
| Service | Metric | Typical Threshold | Reason |
|---|---|---|---|
| EC2 | CPUUtilization | > 80% | Overload |
| EC2 | StatusCheckFailed | > 0 | Failing instance |
| RDS | DatabaseConnections | > 80% max | Saturation |
| ELB | TargetResponseTime | > 2s | Latency |
| Lambda | Duration | > 80% timeout | Risk of timeout |
2. Error Alarms
| Service | Metric | Typical Threshold | Reason |
|---|---|---|---|
| Lambda | Errors | > 1% | Error rate |
| ALB | HTTPCode_Target_5XX | > 10 | Server errors |
| API Gateway | 4XXError | > 5% | Client errors |
| SQS | ApproximateNumberOfMessagesVisible | > 1000 | Queue saturated |
3. Cost Alarms
| Metric | Threshold | Action |
|---|---|---|
| EstimatedCharges | > Budget | Notification |
| EC2 Running Hours | > Planned | Investigation |
Composite Alarms
┌─────────────────────────────────────────────────────────────────┐
│ COMPOSITE ALARM │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Combine multiple alarms with AND/OR/NOT logic │
│ │
│ Example: Critical alert only if: │
│ - CPU > 80% AND │
│ - Memory > 90% AND │
│ - NOT in maintenance │
│ │
│ ┌────────────┐ │
│ │ CPU > 80% │───┐ │
│ └────────────┘ │ │
│ │ AND │
│ ┌────────────┐ ├────────► COMPOSITE ALARM │
│ │ Mem > 90% │───┤ │ │
│ └────────────┘ │ ▼ │
│ │ Critical actions │
│ ┌────────────┐ │ │
│ │NOT Maint. │───┘ │
│ └────────────┘ │
│ │
│ Advantage: Reduces false positives │
│ │
└─────────────────────────────────────────────────────────────────┘
Automatic Actions
SNS (Notifications)
┌─────────────────────────────────────────────────────────────────┐
│ SNS NOTIFICATIONS │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Alarm ────► SNS Topic ────► Subscribers │
│ │ │
│ ├──► Email (ops-team@company.com) │
│ ├──► SMS (+1-555-123-4567) │
│ ├──► Slack (webhook) │
│ ├──► PagerDuty (integration) │
│ └──► Lambda (automation) │
│ │
└─────────────────────────────────────────────────────────────────┘
Auto Scaling
Alarm CPU > 80% ────► Action: Scale Out (+2 instances)
Alarm CPU < 30% ────► Action: Scale In (-1 instance)
EC2 Actions
Alarm StatusCheck Failed ────► Action: Recover Instance
Alarm Instance Idle ────► Action: Stop Instance
Best Practices
1. Avoid False Positives
# BAD: Alarm too sensitive
Period: 60
EvaluationPeriods: 1
DatapointsToAlarm: 1
# → Triggers at the slightest spike
# GOOD: Alarm with tolerance
Period: 60
EvaluationPeriods: 5
DatapointsToAlarm: 3
# → 3 violations over 5 minutes = real issue
2. Clear Naming Convention
Recommended format:
[Environment]-[Service]-[Metric]-[Severity]
Examples:
• PROD-EC2-HighCPU-Critical
• PROD-Lambda-Errors-Warning
• DEV-RDS-Connections-Info
3. Alarm Documentation
Each alarm should have:
• Clear description
• Associated runbook (what to do when it triggers)
• Owner (who is responsible)
• Trigger history
Complete Configuration Example
Monitoring a Web Application
┌─────────────────────────────────────────────────────────────────┐
│ ALARMS FOR WEB APPLICATION │
├─────────────────────────────────────────────────────────────────┤
│ │
│ PRESENTATION LAYER (ALB) │
│ ├── ALB-HighLatency (ResponseTime > 2s) │
│ ├── ALB-High5XX (5XX > 10/min) │
│ └── ALB-UnhealthyHosts (> 0) │
│ │
│ APPLICATION LAYER (EC2/Lambda) │
│ ├── EC2-HighCPU (> 80%) │
│ ├── EC2-HighMemory (> 85%) │
│ ├── Lambda-HighErrors (> 1%) │
│ └── Lambda-NearTimeout (> 80% timeout) │
│ │
│ DATA LAYER (RDS) │
│ ├── RDS-HighCPU (> 80%) │
│ ├── RDS-LowStorage (< 20%) │
│ ├── RDS-HighConnections (> 80% max) │
│ └── RDS-ReplicaLag (> 60s) │
│ │
│ COSTS │
│ └── Billing-OverBudget (> monthly budget) │
│ │
└─────────────────────────────────────────────────────────────────┘
Exercise: Create an Alarm Plan
For an e-commerce application, define alarms for:
- Availability: The application is accessible
- Performance: Acceptable response time
- Errors: Low error rate
- Capacity: Sufficient resources
See the solution
| Category | Alarm | Metric | Threshold | Action |
|---|---|---|---|---|
| Availability | ALB-NoHealthyTargets | HealthyHostCount | = 0 | PagerDuty Critical |
| Availability | Route53-HealthCheck | HealthCheckStatus | Failed | SNS + Failover |
| Performance | ALB-HighLatency | TargetResponseTime | > 3s | SNS Warning |
| Performance | Lambda-SlowExecution | Duration | > 10s | SNS Warning |
| Errors | ALB-High5XX | HTTPCode_Target_5XX | > 50/5min | SNS Critical |
| Errors | Lambda-HighErrors | Errors | > 5% | SNS Critical |
| Capacity | EC2-HighCPU | CPUUtilization | > 80% | Auto Scaling |
| Capacity | RDS-LowStorage | FreeStorageSpace | < 10GB | SNS Warning |
Summary
- Alarms monitor metrics and trigger actions
- 3 states: OK, ALARM, INSUFFICIENT_DATA
- Configure evaluation periods to avoid false positives
- Use composite alarms for complex conditions
- Document each alarm with a runbook
Next section: Validation Quiz