Structured Diagnostic Method
The Systematic Approach
In computing, and particularly in the cloud, troubleshooting should not be random "trial and error." A structured method will save you time and make you more efficient.
┌─────────────────────────────────────────────────────────────┐
│ 5-STEP DIAGNOSTIC METHOD │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. SYMPTOM │
│ └── What exactly is not working? │
│ Be specific! "It doesn't work" is not a valid │
│ symptom. │
│ │
│ 2. HYPOTHESES │
│ └── What are the possible causes? │
│ List them by probability. │
│ │
│ 3. TESTS │
│ └── How to verify each hypothesis? │
│ One test at a time, methodically. │
│ │
│ 4. FIX │
│ └── What action resolves the issue? │
│ Apply and validate. │
│ │
│ 5. DOCUMENTATION │
│ └── Note the solution for future reference. │
│ Update the runbook. │
│ │
└─────────────────────────────────────────────────────────────┘
Step 1: Define the Symptom
Bad vs Good Symptom
❌ BAD SYMPTOMS (too vague):
• "The server doesn't work"
• "It's broken"
• "I have an error"
• "It worked before"
✅ GOOD SYMPTOMS (specific):
• "SSH timeout after 30 seconds to IP 52.1.2.3"
• "502 Bad Gateway error on http://example.com"
• "Instance i-xxx switches to terminated after 10 seconds"
• "Web page returns error 500 since 2:30 PM"
Questions to Ask
┌─────────────────────────────────────────────────────────────┐
│ CLARIFICATION QUESTIONS │
├─────────────────────────────────────────────────────────────┤
│ │
│ WHAT? │
│ • What is the exact error message? │
│ • Which feature is affected? │
│ • Which service/instance is involved? │
│ │
│ WHEN? │
│ • When did the issue start? │
│ • Is the issue constant or intermittent? │
│ • Are there patterns (times, days)? │
│ │
│ WHERE? │
│ • Which region/AZ? │
│ • Which instance/resource? │
│ • From where is the issue observed? │
│ │
│ WHO? │
│ • All users or only some? │
│ • From certain networks/IPs? │
│ │
│ CHANGE? │
│ • What has changed recently? │
│ • Deployment? Configuration? AWS? │
│ │
└─────────────────────────────────────────────────────────────┘
Step 2: Formulate Hypotheses
EC2 Problem Categories
┌─────────────────────────────────────────────────────────────┐
│ PROBLEM CATEGORIES │
├─────────────────────────────────────────────────────────────┤
│ │
│ 🌐 NETWORK │
│ ├── Misconfigured Security Group │
│ ├── Blocking NACL │
│ ├── Incorrect Route Table │
│ ├── Missing Internet Gateway │
│ └── No public IP │
│ │
│ 🔐 SECURITY / AUTH │
│ ├── Wrong SSH key │
│ ├── Wrong user │
│ ├── .pem file permissions │
│ └── IAM (for API calls) │
│ │
│ 💻 INSTANCE │
│ ├── Instance not running │
│ ├── Status check failed │
│ ├── Service not started (Apache, SSH) │
│ └── Disk full │
│ │
│ 📦 APPLICATION │
│ ├── Application crashed │
│ ├── Incorrect configuration │
│ ├── Missing dependency │
│ └── Incorrect port │
│ │
│ ☁️ AWS │
│ ├── Regional outage │
│ ├── Degraded service │
│ └── Maintenance │
│ │
└─────────────────────────────────────────────────────────────┘
Prioritize Hypotheses
Golden rule: From simplest to most complex
From most probable to least probable
Example for "SSH timeout":
HIGH PROBABILITY (check first)
1. Security Group - port 22 closed
2. No public IP
3. Instance not running
MEDIUM PROBABILITY
4. Blocking NACL
5. Route table without route to IGW
6. Firewall on the instance
LOW PROBABILITY
7. AWS regional outage
8. SSH client bug
Step 3: Test Hypotheses
Tests for Network Issues
# 1. Check instance status
aws ec2 describe-instances --instance-ids i-xxx \
--query 'Reservations[].Instances[].State.Name'
# 2. Check public IP
aws ec2 describe-instances --instance-ids i-xxx \
--query 'Reservations[].Instances[].PublicIpAddress'
# 3. Check Security Groups
aws ec2 describe-security-groups --group-ids sg-xxx
# 4. Test network connectivity (from your machine)
ping PUBLIC_IP # ICMP (if allowed)
telnet PUBLIC_IP 22 # Test SSH port
nc -zv PUBLIC_IP 22 # Alternative to telnet
# 5. Check subnet route table
aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=subnet-xxx"
Tests for Instance Issues
# 1. Check status checks
aws ec2 describe-instance-status --instance-ids i-xxx
# 2. Retrieve system logs (if configured)
aws ec2 get-console-output --instance-id i-xxx
# 3. Check CloudWatch metrics
aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 \
--metric-name CPUUtilization \
--dimensions Name=InstanceId,Value=i-xxx \
--start-time 2024-01-01T00:00:00Z \
--end-time 2024-01-01T01:00:00Z \
--period 300 \
--statistics Average
From the Instance (if accessible)
# Check services
systemctl status sshd
systemctl status httpd
# Check open ports
netstat -tlnp
ss -tlnp
# Check disk space
df -h
# Check memory
free -m
# Check logs
tail -100 /var/log/messages
tail -100 /var/log/secure
journalctl -xe
Step 4: Apply the Fix
Rules for Fixes
┌─────────────────────────────────────────────────────────────┐
│ RULES FOR APPLYING A FIX │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. ONE CHANGE AT A TIME │
│ └── Don't change 5 things at once! │
│ └── Otherwise you won't know what solved it │
│ │
│ 2. DOCUMENT BEFORE CHANGING │
│ └── Note the current state │
│ └── Be able to roll back │
│ │
│ 3. TEST AFTER EACH CHANGE │
│ └── Check if the issue is resolved │
│ └── Ensure no new issues were created │
│ │
│ 4. VALIDATE COMPLETELY │
│ └── A quick test OK is not enough │
│ └── Test all affected use cases │
│ │
└─────────────────────────────────────────────────────────────┘
Example of Documented Fix
PROBLEM: SSH timeout to instance i-0123456789
DIAGNOSTIC:
- Instance: running ✓
- Public IP: 52.1.2.3 ✓
- Security Group: sg-abc123
→ Port 22: MISSING ❌
FIX APPLIED:
aws ec2 authorize-security-group-ingress \
--group-id sg-abc123 \
--protocol tcp \
--port 22 \
--cidr 0.0.0.0/0
VALIDATION:
ssh -i my-key.pem ec2-user@52.1.2.3
→ Successful connection ✓
DATE: 2024-12-18 14:30
TECHNICIAN: Haythem
Step 5: Document
Documentation Template
# Incident: [Short title]
## Date and Duration
- Start: YYYY-MM-DD HH:MM
- Resolution: YYYY-MM-DD HH:MM
- Total duration: X hours
## Symptom
[Precise description of the observed issue]
## Impact
- Affected users: X
- Impacted services: [list]
- Severity level: Critical/Major/Minor
## Root Cause
[Explanation of the fundamental cause]
## Timeline
- HH:MM - Problem detection
- HH:MM - Diagnostic start
- HH:MM - Cause identified
- HH:MM - Fix applied
- HH:MM - Validation and closure
## Solution Applied
[Description of the fix]
## Preventive Actions
- [ ] Action 1 to prevent recurrence
- [ ] Action 2
- [ ] Runbook update
## Lessons Learned
[What was learned from this incident]
Practical Exercise
Scenario
Context:
You launched an EC2 instance 5 minutes ago.
The instance is in "running" state.
You try to connect via SSH.
Result:
$ ssh -i my-key.pem ec2-user@52.1.2.3
ssh: connect to host 52.1.2.3 port 22: Connection timed out
Apply the diagnostic method!
Guided Response
STEP 1: SYMPTOM
"SSH timeout after 30 seconds to 52.1.2.3 port 22
from my workstation (IP: 203.0.113.50)"
STEP 2: HYPOTHESES (by probability)
1. Security Group - port 22 not open or IP not authorized
2. No route to Internet Gateway
3. NACL blocking port 22
4. Instance not fully started yet
STEP 3: TESTS
Test 1: Security Group
→ aws ec2 describe-security-groups --group-ids sg-xxx
→ Result: Port 22 open to 10.0.0.0/8 only
→ My IP 203.0.113.50 is not authorized!
STEP 4: FIX
aws ec2 authorize-security-group-ingress \
--group-id sg-xxx \
--protocol tcp \
--port 22 \
--cidr 203.0.113.50/32
VALIDATION:
ssh -i my-key.pem ec2-user@52.1.2.3
→ Successful connection!
STEP 5: DOCUMENTATION
[Fill in the template above]
Summary
| Step | Action | Typical Duration |
|---|---|---|
| 1. Symptom | Define precisely | 2-5 min |
| 2. Hypotheses | List by probability | 5-10 min |
| 3. Tests | Verify methodically | Variable |
| 4. Fix | One change at a time | Variable |
| 5. Documentation | Always document | 10-15 min |
Validation Quiz
- Why should you test one hypothesis at a time?
- What is the difference between a vague and a precise symptom?
- Where to start: probable or improbable causes?
- Why document even small incidents?
- What to do if the first fix doesn't work?
Next section: Lab: Failure Scenarios