VPC Network Troubleshooting
Introduction
Network troubleshooting is an essential skill. This section teaches you a systematic method to diagnose connectivity issues.
Diagnostic Method
┌─────────────────────────────────────────────────────────────┐
│ 5-STEP METHOD │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. IDENTIFY THE SYMPTOM │
│ └─► What exactly is not working? │
│ │
│ 2. CHECK THE PATH │
│ └─► Where should the traffic go through? │
│ │
│ 3. TEST EACH COMPONENT │
│ └─► IGW → Route Table → NACL → SG → Instance │
│ │
│ 4. ANALYZE THE LOGS │
│ └─► VPC Flow Logs, CloudWatch │
│ │
│ 5. FIX AND VALIDATE │
│ └─► Apply the fix, test again │
│ │
└─────────────────────────────────────────────────────────────┘
Diagnostic Checklist
Problem: "I can't reach my instance from the Internet"
┌─────────────────────────────────────────────────────────────┐
│ CHECKLIST INTERNET ACCESS → INSTANCE │
├─────────────────────────────────────────────────────────────┤
│ │
│ ☐ 1. Does the instance have a public IP or Elastic IP? │
│ aws ec2 describe-instances --instance-id i-xxx │
│ → Check "PublicIpAddress" │
│ │
│ ☐ 2. Does the VPC have an Internet Gateway attached? │
│ aws ec2 describe-internet-gateways │
│ → Check "Attachments" → "State: available" │
│ │
│ ☐ 3. Does the Route Table have a route to the IGW? │
│ aws ec2 describe-route-tables --route-table-id rtb-xxx │
│ → Check: 0.0.0.0/0 → igw-xxx │
│ │
│ ☐ 4. Does the NACL allow the traffic? │
│ aws ec2 describe-network-acls │
│ → Inbound AND Outbound (ephemeral ports!) │
│ │
│ ☐ 5. Does the Security Group allow the port? │
│ aws ec2 describe-security-groups --group-ids sg-xxx │
│ → Check inbound rule for the port │
│ │
│ ☐ 6. Is the instance running? │
│ aws ec2 describe-instance-status --instance-id i-xxx │
│ → State: running, Status checks: passed │
│ │
│ ☐ 7. Is the service listening on the correct port? │
│ (On the instance) netstat -tlnp | grep :80 │
│ → Service should listen on 0.0.0.0:80 │
│ │
└─────────────────────────────────────────────────────────────┘
Problem: "My instance cannot access the Internet"
┌─────────────────────────────────────────────────────────────┐
│ CHECKLIST INSTANCE → INTERNET │
├─────────────────────────────────────────────────────────────┤
│ │
│ CASE 1: Instance in PUBLIC Subnet │
│ ─────────────────────────────────── │
│ │
│ ☐ Does the instance have a public IP? │
│ ☐ Route 0.0.0.0/0 → igw-xxx exists? │
│ ☐ SG outbound allows the traffic? │
│ ☐ NACL outbound allows? │
│ ☐ NACL inbound allows responses (ephemeral ports)? │
│ │
│ CASE 2: Instance in PRIVATE Subnet │
│ ────────────────────────────────── │
│ │
│ ☐ NAT Gateway exists in public subnet? │
│ ☐ Route 0.0.0.0/0 → nat-xxx exists? │
│ ☐ NAT Gateway is in "available" state? │
│ ☐ NAT Gateway has an Elastic IP? │
│ ☐ SG outbound of the instance allows? │
│ │
└─────────────────────────────────────────────────────────────┘
Problem: "Instances in the same VPC are not communicating"
┌─────────────────────────────────────────────────────────────┐
│ CHECKLIST INTRA-VPC COMMUNICATION │
├─────────────────────────────────────────────────────────────┤
│ │
│ ☐ 1. Are the instances in the same VPC? │
│ │
│ ☐ 2. Does "local" route exist? (usually automatic) │
│ 10.0.0.0/16 → local │
│ │
│ ☐ 3. Does SOURCE Security Group allow? │
│ Inbound rule: sg-destination or subnet CIDR │
│ │
│ ☐ 4. Does DESTINATION Security Group allow? │
│ Inbound rule for the port and source │
│ │
│ ☐ 5. Do both subnets' NACLs allow? │
│ (Check both directions) │
│ │
│ ☐ 6. Are you using the PRIVATE IP? │
│ Do not use the public IP for intra-VPC │
│ │
└─────────────────────────────────────────────────────────────┘
Troubleshooting Scenarios
Scenario 1: SSH Timeout
┌─────────────────────────────────────────────────────────────┐
│ SCENARIO: SSH TIMEOUT │
├─────────────────────────────────────────────────────────────┤
│ │
│ Symptom: ssh -i key.pem ec2-user@52.1.2.3 │
│ Connection timed out │
│ │
│ Diagnostic: │
│ │
│ 1. Ping the public IP │
│ $ ping 52.1.2.3 │
│ → If timeout: issue before the instance │
│ │
│ 2. Check the Security Group │
│ $ aws ec2 describe-security-groups --group-ids sg-xxx │
│ │
│ PROBLEM FOUND: │
│ ┌────────────────────────────────────────────┐ │
│ │ Inbound Rules: │ │
│ │ Type: SSH, Port: 22, Source: 10.0.0.0/16 │ │
│ │ │ │
│ │ ❌ Source limited to VPC! │ │
│ │ Your public IP is not allowed │ │
│ └────────────────────────────────────────────┘ │
│ │
│ Solution: │
│ $ aws ec2 authorize-security-group-ingress \ │
│ --group-id sg-xxx \ │
│ --protocol tcp --port 22 \ │
│ --cidr YOUR_IP/32 │
│ │
└─────────────────────────────────────────────────────────────┘
Scenario 2: Website Unreachable
┌─────────────────────────────────────────────────────────────┐
│ SCENARIO: HTTP 502 / TIMEOUT │
├─────────────────────────────────────────────────────────────┤
│ │
│ Symptom: curl http://52.1.2.3 → timeout │
│ │
│ Diagnostic: │
│ │
│ 1. Test port from your machine │
│ $ nc -zv 52.1.2.3 80 │
│ → If "Connection refused" = SG OK, service KO │
│ → If "timeout" = SG or routing KO │
│ │
│ 2. Check Route Table │
│ $ aws ec2 describe-route-tables │
│ │
│ PROBLEM FOUND: │
│ ┌────────────────────────────────────────────┐ │
│ │ Routes: │ │
│ │ 10.0.0.0/16 → local │ │
│ │ │ │
│ │ ❌ No route to IGW! │ │
│ │ The subnet is PRIVATE │ │
│ └────────────────────────────────────────────┘ │
│ │
│ Solution: │
│ $ aws ec2 create-route \ │
│ --route-table-id rtb-xxx \ │
│ --destination-cidr-block 0.0.0.0/0 \ │
│ --gateway-id igw-xxx │
│ │
└─────────────────────────────────────────────────────────────┘
Scenario 3: Application cannot reach the DB
┌─────────────────────────────────────────────────────────────┐
│ SCENARIO: APP → DB TIMEOUT │
├─────────────────────────────────────────────────────────────┤
│ │
│ Symptom: Application error "Cannot connect to database" │
│ │
│ Configuration: │
│ • App server : 10.0.11.5 (sg-app) │
│ • DB server : 10.0.21.10 (sg-db) │
│ │
│ Diagnostic: │
│ │
│ 1. From App server, test the connection │
│ $ nc -zv 10.0.21.10 3306 │
│ → timeout │
│ │
│ 2. Check SG of the DB │
│ $ aws ec2 describe-security-groups --group-ids sg-db │
│ │
│ PROBLEM FOUND: │
│ ┌────────────────────────────────────────────┐ │
│ │ Inbound Rules: │ │
│ │ Type: MySQL, Port: 3306, Source: sg-web │ │
│ │ │ │
│ │ ❌ sg-app is not allowed! │ │
│ │ Only sg-web can access │ │
│ └────────────────────────────────────────────┘ │
│ │
│ Solution: │
│ $ aws ec2 authorize-security-group-ingress \ │
│ --group-id sg-db \ │
│ --protocol tcp --port 3306 \ │
│ --source-group sg-app │
│ │
└─────────────────────────────────────────────────────────────┘
VPC Flow Logs
Enable Flow Logs
# Create a CloudWatch log group
aws logs create-log-group --log-group-name /aws/vpc/flow-logs
# Enable Flow Logs on the VPC
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids vpc-xxx \
--traffic-type ALL \
--log-destination-type cloud-watch-logs \
--log-group-name /aws/vpc/flow-logs \
--deliver-logs-permission-arn arn:aws:iam::xxx:role/flow-logs-role
Read Flow Logs
┌─────────────────────────────────────────────────────────────┐
│ VPC FLOW LOGS FORMAT │
├─────────────────────────────────────────────────────────────┤
│ │
│ version account-id interface-id srcaddr dstaddr │
│ srcport dstport protocol packets bytes start end │
│ action log-status │
│ │
│ Example: │
│ 2 123456789012 eni-abc123 10.0.1.5 52.1.2.3 │
│ 54321 443 6 10 5000 1234567890 1234567900 │
│ ACCEPT OK │
│ │
│ Interpretation: │
│ • Source: 10.0.1.5:54321 │
│ • Destination: 52.1.2.3:443 │
│ • Protocol: 6 (TCP) │
│ • Action: ACCEPT (allowed) │
│ │
│ Possible actions: │
│ • ACCEPT: Traffic allowed │
│ • REJECT: Traffic blocked by SG or NACL │
│ │
└─────────────────────────────────────────────────────────────┘
Diagnostic Tools
Useful Commands
# Check instance status
aws ec2 describe-instance-status --instance-id i-xxx
# Check attached Security Group
aws ec2 describe-instances --instance-id i-xxx \
--query 'Reservations[].Instances[].SecurityGroups'
# Check SG rules
aws ec2 describe-security-groups --group-ids sg-xxx
# Check a subnet's Route Table
aws ec2 describe-route-tables \
--filters "Name=association.subnet-id,Values=subnet-xxx"
# Check a subnet's NACL
aws ec2 describe-network-acls \
--filters "Name=association.subnet-id,Values=subnet-xxx"
Tests From the Instance
# Test Internet connectivity
curl -I https://aws.amazon.com
# Test DNS resolution
nslookup google.com
# Test specific port
nc -zv DESTINATION_IP PORT
# View routes
ip route
# View connections
netstat -tlnp
ss -tlnp
Quick Troubleshooting Table
| Symptom | Probable Cause | Check |
|---|---|---|
| Timeout from Internet | SG, Route, IGW | Complete checklist |
| Connection refused | Service not started | netstat, systemctl |
| Host unreachable | Missing route | Route Table |
| Slow/Intermittent | Misconfigured NACL | NACL rules |
| Intra-VPC timeout | SG between instances | Cross SG rules |
Summary
┌─────────────────────────────────────────────────────────────┐
│ TROUBLESHOOTING SUMMARY │
├─────────────────────────────────────────────────────────────┤
│ │
│ ORDER OF CHECKS: │
│ │
│ 1. Instance running + status checks OK │
│ 2. Public IP present (if Internet access) │
│ 3. IGW attached to VPC │
│ 4. Route 0.0.0.0/0 → igw (public subnet) │
│ 5. NACL allows (+ ephemeral ports) │
│ 6. Security Group allows │
│ 7. Service listening on the port │
│ │
│ TOOLS: │
│ • aws ec2 describe-* (resource status) │
│ • VPC Flow Logs (traffic analysis) │
│ • nc, curl, ping (connectivity tests) │
│ │
└─────────────────────────────────────────────────────────────┘
Next section: Lab: VPC Troubleshooting