Lab: VPC Troubleshooting
Lab Objective
This lab puts you in the shoes of a cloud administrator who must diagnose and resolve network connectivity issues. You will analyze real-world scenarios and propose solutions.
This lab is scenario-based. You must identify the problem and propose the solution before looking at the answer.
Part 1: Case Study
Reference Architecture
All questions are based on this architecture:
┌─────────────────────────────────────────────────────────────────────────┐
│ E-COMMERCE ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ INTERNET │
│ │ │
│ ┌──────▼──────┐ │
│ │ IGW │ │
│ │ igw-abc123 │ │
│ └──────┬──────┘ │
│ │ │
│ ┌──────────────────────────────┼──────────────────────────────┐ │
│ │ VPC │ (10.0.0.0/16) │ │
│ │ │ │ │
│ │ ┌──────────────────────────┴────────────────────────────┐ │ │
│ │ │ PUBLIC SUBNET │ │ │
│ │ │ 10.0.1.0/24 │ │ │
│ │ │ rtb-public │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────────┐ ┌─────────────────┐ │ │ │
│ │ │ │ BASTION │ │ NAT Gateway │ │ │ │
│ │ │ │ 10.0.1.10 │ │ 10.0.1.50 │ │ │ │
│ │ │ │ sg-bastion │ │ nat-xyz789 │ │ │ │
│ │ │ │ IP: 52.1.2.3 │ │ EIP: 52.4.5.6 │ │ │ │
│ │ │ └─────────────────┘ └─────────────────┘ │ │ │
│ │ │ │ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌────────────────────────────────────────────────────────┐ │ │
│ │ │ PRIVATE APP SUBNET │ │ │
│ │ │ 10.0.11.0/24 │ │ │
│ │ │ rtb-private │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────────┐ ┌─────────────────┐ │ │ │
│ │ │ │ WEB SERVER │ │ API SERVER │ │ │ │
│ │ │ │ 10.0.11.20 │ │ 10.0.11.30 │ │ │ │
│ │ │ │ sg-web │ │ sg-api │ │ │ │
│ │ │ └─────────────────┘ └─────────────────┘ │ │ │
│ │ │ │ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌────────────────────────────────────────────────────────┐ │ │
│ │ │ PRIVATE DB SUBNET │ │ │
│ │ │ 10.0.21.0/24 │ │ │
│ │ │ rtb-database │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────────┐ │ │ │
│ │ │ │ DATABASE │ │ │ │
│ │ │ │ 10.0.21.10 │ │ │ │
│ │ │ │ sg-database │ │ │ │
│ │ │ │ MySQL:3306 │ │ │ │
│ │ │ └─────────────────┘ │ │ │
│ │ │ │ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Current Configuration
Route Tables:
rtb-public:
┌─────────────────────┬───────────────┐
│ Destination │ Target │
├─────────────────────┼───────────────┤
│ 10.0.0.0/16 │ local │
│ 0.0.0.0/0 │ igw-abc123 │
└─────────────────────┴───────────────┘
rtb-private:
┌─────────────────────┬───────────────┐
│ Destination │ Target │
├─────────────────────┼───────────────┤
│ 10.0.0.0/16 │ local │
│ 0.0.0.0/0 │ nat-xyz789 │
└─────────────────────┴───────────────┘
rtb-database:
┌─────────────────────┬───────────────┐
│ Destination │ Target │
├─────────────────────┼───────────────┤
│ 10.0.0.0/16 │ local │
└─────────────────────┴───────────────┘
Security Groups:
sg-bastion:
┌────────────────────────────────────────────────────┐
│ Inbound: SSH (22) from 0.0.0.0/0 │
│ Outbound: All traffic to 0.0.0.0/0 │
└────────────────────────────────────────────────────┘
sg-web:
┌────────────────────────────────────────────────────┐
│ Inbound: HTTP (80) from 0.0.0.0/0 │
│ Inbound: SSH (22) from sg-bastion │
│ Outbound: All traffic to 0.0.0.0/0 │
└────────────────────────────────────────────────────┘
sg-api:
┌────────────────────────────────────────────────────┐
│ Inbound: TCP (8080) from sg-web │
│ Inbound: SSH (22) from sg-bastion │
│ Outbound: All traffic to 0.0.0.0/0 │
└────────────────────────────────────────────────────┘
sg-database:
┌────────────────────────────────────────────────────┐
│ Inbound: MySQL (3306) from sg-api │
│ Outbound: All traffic to 0.0.0.0/0 │
└────────────────────────────────────────────────────┘
Part 2: Troubleshooting Exercises
Exercise 1: SSH to Bastion
Symptom: An administrator cannot connect via SSH to the bastion from their office (IP: 203.0.113.50).
$ ssh -i key.pem ec2-user@52.1.2.3
Connection timed out
Questions:
- What is the most likely problem?
- What command would you use to verify?
- What is the solution?
💡 Solution Exercise 1
Analysis:
The Security Group sg-bastion allows SSH from 0.0.0.0/0, so that's not it.
Let's check:
- ✅ Route Table rtb-public has a route to IGW
- ✅ IGW is attached
- ❓ Does the bastion have a public IP?
Verification:
aws ec2 describe-instances --instance-id i-bastion \
--query 'Reservations[].Instances[].PublicIpAddress'
Probable Problem: The bastion does not have a public IP or the IP changed after a reboot.
Solution:
# Option 1: Associate an Elastic IP
aws ec2 allocate-address --domain vpc
aws ec2 associate-address --instance-id i-bastion --allocation-id eipalloc-xxx
# Option 2: Enable auto-assign public IP on the subnet
Exercise 2: Web Server Inaccessible
Symptom: The website is not accessible from the Internet.
$ curl http://10.0.11.20
curl: (7) Failed to connect to 10.0.11.20 port 80: No route to host
Questions:
- Why is the admin using the private IP from the Internet?
- What is the real architectural problem here?
- How should the architecture be configured?
💡 Solution Exercise 2
Analysis:
The Web Server is in a private subnet (10.0.11.0/24)!
Identified problems:
- The admin is trying to access a private IP from the Internet (impossible)
- The web server does not have a public IP
- Even with a public IP, no IGW route on this subnet
Incorrect Architecture: A public-facing web server should not be in a private subnet.
Possible Solutions:
Option A (Recommended): Add a Load Balancer
Internet → ALB (public subnet) → Web Server (private subnet)
Option B: Move the Web Server to public subnet
Internet → IGW → Web Server (public subnet with public IP)
For Option A:
- Create an Application Load Balancer in the public subnet
- Configure a Target Group with the web server
- Update sg-web to accept traffic from sg-alb
Exercise 3: API Cannot Access Database
Symptom: The API server cannot connect to MySQL.
$ mysql -h 10.0.21.10 -u app -p
ERROR 2003 (HY000): Can't connect to MySQL server on '10.0.21.10' (110)
Questions:
- List all possible causes.
- How to verify each hypothesis?
- What is the most likely cause given the configuration?
💡 Solution Exercise 3
Hypotheses and verifications:
| Hypothesis | Verification | Result |
|---|---|---|
| SG API → DB | Does sg-database allow sg-api? | ❌ Allows sg-api |
| Route | Does local route exist? | ✅ Yes (10.0.0.0/16 → local) |
| MySQL running | Is MySQL service started? | To check |
| Port listening | Is MySQL listening on 0.0.0.0:3306? | To check |
Identified Problem:
Looking at sg-database:
Inbound: MySQL (3306) from sg-api
That's correct. But let's look at sg-api:
Inbound: TCP (8080) from sg-web
The sg-api does not have explicit outbound access to MySQL? No, it has Outbound: All traffic so it's OK.
Further Verification: Is the API server indeed in sg-api?
aws ec2 describe-instances --filters "Name=private-ip-address,Values=10.0.11.30" \
--query 'Reservations[].Instances[].SecurityGroups'
Probable Cause: The API server might not be attached to the correct Security Group!
Exercise 4: Web Server Cannot Install Packages
Symptom: The web server cannot perform yum update.
$ sudo yum update
Loaded plugins: priorities, update-motd, upgrade-helper
Could not retrieve mirrorlist https://... error was
12: Timeout on ...
Questions:
- Why can't the server access the Internet?
- Check the network configuration.
- Propose the solution.
💡 Solution Exercise 4
Analysis:
The Web Server is in the private subnet (10.0.11.0/24) which uses rtb-private.
rtb-private:
10.0.0.0/16 → local
0.0.0.0/0 → nat-xyz789
The route to NAT Gateway exists! Let's check:
NAT Gateway Checklist:
- Is NAT Gateway in "available" state?
aws ec2 describe-nat-gateways --nat-gateway-ids nat-xyz789
-
Is NAT Gateway in a subnet with IGW?
- NAT is in 10.0.1.0/24 (public) ✅
-
Does NAT Gateway have an Elastic IP?
- EIP: 52.4.5.6 ✅
-
Does the public subnet have the route to IGW?
- 0.0.0.0/0 → igw-abc123 ✅
If everything is OK on the NAT side, check:
-
Security Group of the NAT Gateway?
- NAT Gateway has no SG (it's a managed service)
-
Does the private subnet's NACL allow outbound traffic?
Probable Cause: Restrictive NACL on the private subnet blocking outbound traffic or incoming ephemeral ports.
Solution:
# Check the NACL
aws ec2 describe-network-acls \
--filters "Name=association.subnet-id,Values=subnet-private"
# Ensure rules allow:
# Outbound: 0.0.0.0/0 (all traffic)
# Inbound: ephemeral ports 1024-65535 (for responses)
Exercise 5: Database Not Receiving Connections
Symptom: After fixing exercise 3, the DB log shows 0 connections.
Additional Configuration Revealed:
NACL of the database subnet (acl-db):
┌────────┬──────────┬───────────┬─────────────┬────────┐
│ Rule # │ Type │ Port │ Source/Dest │ Action │
├────────┼──────────┼───────────┼─────────────┼────────┤
│ 100 │ Inbound │ 3306 │ 10.0.1.0/24 │ ALLOW │
│ * │ Inbound │ All │ 0.0.0.0/0 │ DENY │
├────────┼──────────┼───────────┼─────────────┼────────┤
│ 100 │ Outbound │ 1024-65535│ 10.0.1.0/24 │ ALLOW │
│ * │ Outbound │ All │ 0.0.0.0/0 │ DENY │
└────────┴──────────┴───────────┴─────────────┴────────┘
Questions:
- What is the problem with this NACL?
- Why are connections from the API blocked?
- Fix the NACL.
💡 Solution Exercise 5
Problem:
The NACL allows MySQL traffic (3306) only from 10.0.1.0/24 (public subnet).
But the API server is in 10.0.11.0/24 (private app subnet)!
Rule 100 Inbound: 3306 from 10.0.1.0/24 → Does not include 10.0.11.0/24
Solution:
# Add a rule for the app subnet
aws ec2 create-network-acl-entry \
--network-acl-id acl-db \
--rule-number 110 \
--protocol tcp \
--port-range From=3306,To=3306 \
--cidr-block 10.0.11.0/24 \
--rule-action allow \
--ingress
# Add the corresponding outbound rule
aws ec2 create-network-acl-entry \
--network-acl-id acl-db \
--rule-number 110 \
--protocol tcp \
--port-range From=1024,To=65535 \
--cidr-block 10.0.11.0/24 \
--rule-action allow \
--egress
Lesson: NACLs are stateless and very precise about sources/destinations.
Part 3: Synthesis Exercise
Your Mission
Document a complete troubleshooting procedure for your team.
Create a document with:
-
Diagnostic Flowchart: Decision tree for common issues
-
Verification Commands: List of essential AWS CLI commands
-
Quick Checklist: Checklist for each type of issue
-
Lessons Learned: Common mistakes and how to avoid them
Deliverable Template
# VPC Network Troubleshooting Procedure
## 1. Diagnostic Flowchart
[Insert your diagram]
## 2. Verification Commands
### Instance Status
```bash
[Your commands]
Security Groups
[Your commands]
Routes
[Your commands]
3. Checklist by Symptom
Timeout from Internet
- ...
- ...
Timeout to Internet
- ...
- ...
4. Lessons Learned
| Common Mistake | How to Avoid |
|---|---|
| ... | ... |
## Evaluation Criteria
| Criterion | Points |
|-----------|--------|
| Exercises 1-5 correctly analyzed | /50 |
| Clear and complete flowchart | /15 |
| Relevant commands | /15 |
| Usable checklist | /10 |
| Relevant lessons learned | /10 |
| **Total** | **/100** |
---
*Next week: [Web Publishing](../module_07/index.mdx)*