Lab: Setting Up Monitoring

Objective

Configure a complete monitoring system with metrics, alarms, logs, and dashboard.

Part 1: Basic Alarms

Exercise 1.1: CPU Alarm

Create an alarm that triggers when the CPU exceeds 80% for 10 minutes:

# Complete the command
aws cloudwatch put-metric-alarm \
    --alarm-name _____ \
    --metric-name _____ \
    --namespace _____ \
    --statistic _____ \
    --period _____ \
    --threshold _____ \
    --comparison-operator _____ \
    --evaluation-periods _____ \
    --dimensions Name=InstanceId,Value=i-xxx \
    --alarm-actions arn:aws:sns:...:alerts

💡 Solution

aws cloudwatch put-metric-alarm \
    --alarm-name ec2-cpu-high \
    --metric-name CPUUtilization \
    --namespace AWS/EC2 \
    --statistic Average \
    --period 300 \
    --threshold 80 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 2 \
    --dimensions Name=InstanceId,Value=i-xxx \
    --alarm-actions arn:aws:sns:us-east-1:123456789:alerts \
    --alarm-description "CPU above 80% for 10 minutes"

Explanation:

• Period 300s × 2 evaluations = 10 minutes
• Average to avoid false positives from short spikes

Exercise 1.2: Disk Alarm

Create an alarm for low disk space (> 85% used):

Parameter	Value
Metric Name	?
Namespace	?
Threshold	?
Period	?

💡 Solution

Parameter	Value
Metric Name	disk_used_percent
Namespace	CWAgent
Threshold	85
Period	3600 (1 hour)

Note: This metric requires CloudWatch Agent installed on the instance.

Part 2: Log Configuration

Exercise 2.1: Log Group and Retention

# Create a log group with 30-day retention
aws logs create-log-group \
    --log-group-name /app/web-server

aws logs put-retention-policy \
    --log-group-name /app/web-server \
    --retention-in-days _____

💡 Solution

aws logs put-retention-policy \
    --log-group-name /app/web-server \
    --retention-in-days 30

Possible values: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653

Exercise 2.2: Metric Filter

Create a filter to count 500 errors in the logs:

aws logs put-metric-filter \
    --log-group-name /app/web-server \
    --filter-name error-500-count \
    --filter-pattern '_______________' \
    --metric-transformations \
        metricName=Error500Count,\
        metricNamespace=WebApp,\
        metricValue=1

💡 Solution

aws logs put-metric-filter \
    --log-group-name /app/web-server \
    --filter-name error-500-count \
    --filter-pattern '"HTTP/1.1" 500' \
    --metric-transformations \
        metricName=Error500Count,\
        metricNamespace=WebApp,\
        metricValue=1

# Or for JSON logs
aws logs put-metric-filter \
    --log-group-name /app/web-server \
    --filter-name error-500-json \
    --filter-pattern '{ $.statusCode = 500 }' \
    --metric-transformations \
        metricName=Error500Count,\
        metricNamespace=WebApp,\
        metricValue=1

Part 3: Log Insights Queries

Exercise 3.1: Analysis Queries

Write Log Insights queries for:

A. The last 10 errors:

# Your query here

B. Number of errors per hour over the last 24h:

# Your query here

C. Top 5 pages with 404 errors:

# Your query here

💡 Solutions

A. The last 10 errors:

fields @timestamp, @message
| filter @message like /ERROR|Exception|error/
| sort @timestamp desc
| limit 10

B. Number of errors per hour over the last 24h:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() as error_count by bin(1h)
| sort @timestamp desc

C. Top 5 pages with 404 errors:

fields @timestamp, @message
| filter @message like /404/
| parse @message /GET (?<path>\/[^ ]*)/
| stats count() as hits by path
| sort hits desc
| limit 5

Part 4: Dashboard

Exercise 4.1: Create a Dashboard

Define widgets for a production dashboard:

Widget	Type	Metric	Period
CPU Overview	?	?	?
Memory Usage	?	?	?
Request Count	?	?	?
Error Rate	?	?	?
Latency P99	?	?	?
Alarm Status	?	?	?

💡 Solution

Widget	Type	Metric	Period
CPU Overview	Line	CPUUtilization	5 min
Memory Usage	Line	mem_used_percent	5 min
Request Count	Stacked Area	RequestCount	1 min
Error Rate	Number	HTTPCode_5XX / Total × 100	5 min
Latency P99	Line	TargetResponseTime p99	1 min
Alarm Status	Alarm Status	All alarms	-

Exercise 4.2: Dashboard JSON

{
    "widgets": [
        {
            "type": "metric",
            "properties": {
                "title": "CPU Utilization",
                "metrics": [
                    ["AWS/EC2", "CPUUtilization", "InstanceId", "i-xxx"]
                ],
                "period": 300,
                "stat": "Average",
                "region": "us-east-1"
            }
        },
        {
            "type": "___",
            "properties": {
                "title": "Alarm Status",
                "alarms": [
                    "arn:aws:cloudwatch:...:alarm:cpu-high",
                    "arn:aws:cloudwatch:...:alarm:mem-high"
                ]
            }
        }
    ]
}

💡 Solution

{
    "widgets": [
        {
            "type": "metric",
            "x": 0, "y": 0, "width": 12, "height": 6,
            "properties": {
                "title": "CPU Utilization",
                "metrics": [
                    ["AWS/EC2", "CPUUtilization", "InstanceId", "i-xxx"]
                ],
                "period": 300,
                "stat": "Average",
                "region": "us-east-1",
                "view": "timeSeries"
            }
        },
        {
            "type": "alarm",
            "x": 12, "y": 0, "width": 12, "height": 6,
            "properties": {
                "title": "Alarm Status",
                "alarms": [
                    "arn:aws:cloudwatch:us-east-1:123456789:alarm:cpu-high",
                    "arn:aws:cloudwatch:us-east-1:123456789:alarm:mem-high"
                ]
            }
        }
    ]
}

Part 5: Runbook

Exercise 5.1: Create a Runbook

Complete the runbook for the "High Memory" incident:

┌─────────────────────────────────────────────────────────────┐
│   RUNBOOK: HIGH MEMORY (> 90%)                             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   INITIAL ASSESSMENT (2 min max)                            │
│   ☐ _______________________________________                 │
│   ☐ _______________________________________                 │
│                                                             │
│   DIAGNOSTIC (5 min max)                                    │
│   ☐ _______________________________________                 │
│   ☐ _______________________________________                 │
│   ☐ _______________________________________                 │
│                                                             │
│   REMEDIATION ACTIONS                                       │
│   ☐ _______________________________________                 │
│   ☐ _______________________________________                 │
│                                                             │
│   VALIDATION                                                │
│   ☐ _______________________________________                 │
│   ☐ _______________________________________                 │
│                                                             │
└─────────────────────────────────────────────────────────────┘

💡 Solution

┌─────────────────────────────────────────────────────────────┐
│   RUNBOOK: HIGH MEMORY (> 90%)                             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   INITIAL ASSESSMENT (2 min max)                            │
│   ☐ Check if multiple instances are affected                │
│   ☐ Check user errors (502, timeouts)                       │
│                                                             │
│   DIAGNOSTIC (5 min max)                                    │
│   ☐ SSH connection: free -h, top -o %MEM                    │
│   ☐ Identify the memory-consuming process                   │
│   ☐ Check application logs for memory leaks                 │
│                                                             │
│   REMEDIATION ACTIONS                                       │
│   ☐ If memory leak: Restart the application                 │
│   ☐ If legitimate load: Scale out (add instances)           │
│   ☐ Free cache if possible (drop_caches)                    │
│                                                             │
│   VALIDATION                                                │
│   ☐ Memory back < 80%?                                      │
│   ☐ Application functioning normally?                       │
│   ☐ Create ticket for post-mortem analysis                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Part 6: Complete Scenario

Context

You receive an alert: "Error rate above 5% for web-app"

Questions

1. Which metrics do you check first?
2. Which logs do you analyze?
3. Which Log Insights query do you use?
4. How do you identify the root cause?

💡 Detailed Solution

1. Metrics to check:

• HTTPCode_Target_5XX (ALB) → Confirm 500 errors
• RequestCount → Traffic volume
• TargetResponseTime → Latency
• HealthyHostCount → Instances down?
• CPUUtilization, Memory → Resources saturated?

2. Logs to analyze:

• /aws/elasticloadbalancing/ → ALB logs
• /app/web-server → Application logs
• /var/log/messages → System logs

3. Log Insights query:

fields @timestamp, @message
| filter @message like /500|ERROR|Exception/
| stats count() by bin(5m)
| sort @timestamp desc

Then to detail:

fields @timestamp, @message
| filter @message like /500/
| parse @message /(?<endpoint>\/api\/[^ ]*)/
| stats count() by endpoint
| sort count desc
| limit 10

4. Identification process:

1. Correlate the timing of the error spike with deployments
2. Identify the problematic endpoint
3. Check logs for that specific endpoint
4. Look for the error stack trace
5. Identify the cause (DB timeout, code bug, config...)

Deliverable

Document to Produce

1. Alarm Configuration (CLI commands)
2. Log Insights Queries (3 minimum)
3. Dashboard Definition (JSON or description)
4. Complete Runbook for 2 scenarios
5. Scenario Analysis with solution

Evaluation Criteria

Criteria	Points
Alarms correctly configured	/20
Log Insights queries	/20
Relevant dashboard	/20
Complete runbooks	/25
Scenario analysis	/15
Total	/100

---

Next week: MODULE 14 - FinOps