Skip to main content

Lab: EC2 Troubleshooting Scenarios

Lab Objectives​

This lab presents you with realistic failure scenarios. For each scenario, you will need to:

  • Analyze the symptoms
  • Formulate hypotheses
  • Propose diagnostic tests
  • Identify the solution
Theoretical Exercise

This lab is an analysis exercise. No AWS access needed - work on paper or document.

Estimated Duration​

1h30 to 2h

Scenario 1: Website Inaccessible​

Context​

Situation:
- EC2 instance launched 1 hour ago
- AMI: Amazon Linux 2023
- Type: t3.micro
- Apache installed and started
- You can connect via SSH
- But http://52.1.2.3 does not respond (timeout)

Current configuration:
- VPC: vpc-default
- Subnet: public (with IGW)
- Security Group: sg-web

Available Information​

Instance state: running
Status checks: 2/2 passed

Security Group sg-web:
Inbound Rules:
- SSH (22): 0.0.0.0/0
- HTTP (80): 10.0.0.0/8

Outbound Rules:
- All traffic: 0.0.0.0/0

Questions​

  1. What is the precise symptom?

  2. List 3 hypotheses in order of probability

  3. What is the most probable cause?

  4. What command/action fixes the problem?

  5. How to validate that it is resolved?

📋 See the solution

1. Symptom: HTTP timeout to 52.1.2.3 port 80, while SSH works.

2. Hypotheses:

  1. Security Group does not allow HTTP from the Internet (HIGH)
  2. Apache is not listening on the correct port (MEDIUM)
  3. iptables firewall on the instance (LOW)

3. Cause: The Security Group allows HTTP (port 80) only from 10.0.0.0/8 (private network), not from the Internet (0.0.0.0/0).

4. Fix:

aws ec2 authorize-security-group-ingress \
    --group-id sg-web \
    --protocol tcp \
    --port 80 \
    --cidr 0.0.0.0/0

5. Validation:

curl http://52.1.2.3
# or
Open http://52.1.2.3 in a browser

Scenario 2: SSH Unreachable​

Context​

Situation:
- New instance launched
- You downloaded the .pem key
- Instance: running, status checks OK
- Public IP: 52.2.3.4

Command executed:
$ ssh -i key.pem ec2-user@52.2.3.4
ssh: connect to host 52.2.3.4 port 22: Connection timed out

Available Information​

Instance details:
- ID: i-0123456789
- State: running
- Public IP: 52.2.3.4
- Subnet: subnet-abc123 (10.0.2.0/24)
- Security Group: sg-default

Security Group sg-default:
Inbound Rules:
- (no rule)

Outbound Rules:
- All traffic: 0.0.0.0/0

Subnet subnet-abc123:
- Route Table: rtb-main
- Routes:
  - 10.0.0.0/16 → local
  - (no other route)

Questions​

  1. Identify ALL the problems in this configuration

  2. Rank them by correction priority

  3. Propose commands to fix each problem

📋 See the solution

1. Identified Problems:

Problem A - Security Group: No inbound rule = port 22 blocked

Problem B - Route Table: No route to Internet Gateway = no Internet access

2. Priority:

  1. Route Table (without a route, even with SG OK, it won't work)
  2. Security Group

3. Fixes:

# 1. Create/attach an Internet Gateway (if not done)
aws ec2 create-internet-gateway
aws ec2 attach-internet-gateway --vpc-id vpc-xxx --internet-gateway-id igw-xxx

# 2. Add the route to IGW
aws ec2 create-route \
    --route-table-id rtb-main \
    --destination-cidr-block 0.0.0.0/0 \
    --gateway-id igw-xxx

# 3. Add SSH rule to the Security Group
aws ec2 authorize-security-group-ingress \
    --group-id sg-default \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0

Scenario 3: Instance Immediately Terminates​

Context​

Situation:
- You launch an instance
- It goes from "pending" to "running" then "terminated"
- All in less than 30 seconds
- You tried 3 times, same result

Configuration:
- AMI: ami-custom-xyz (custom AMI)
- Type: t3.large
- Region: us-east-1

Available Information​

EC2 Console → State transition reason:
"Client.VolumeLimitExceeded: Volume limit exceeded"

Your current EBS volumes:
- 45 volumes of 100 GB each
- Total: 4.5 TB

Your AMI ami-custom-xyz:
- Created 6 months ago
- Root volume: 200 GB
- Uses snapshot snap-olddata

Questions​

  1. What is the cause of the termination?

  2. How to check your account limits?

  3. What are the possible solutions? (at least 3)

📋 See the solution

1. Cause: The message "Volume limit exceeded" indicates that you have reached the EBS storage quota for your account.

2. Check limits:

aws service-quotas get-service-quota \
    --service-code ebs \
    --quota-code L-D18FCD1D

# Or via the console:
# Service Quotas → Amazon Elastic Block Store (EBS)

3. Solutions:

Solution A - Delete unused volumes:

# List unattached volumes
aws ec2 describe-volumes \
    --filters "Name=status,Values=available"

# Delete
aws ec2 delete-volume --volume-id vol-xxx

Solution B - Request a quota increase:

aws service-quotas request-service-quota-increase \
    --service-code ebs \
    --quota-code L-D18FCD1D \
    --desired-value 10000

Solution C - Use another region: Quotas are per region. Launch in eu-west-1 for example.

Solution D - Use an instance type with instance store: Some instances have local storage, not EBS.

Scenario 4: Database Unreachable​

Context​

Architecture:
- Web Instance (public subnet): 10.0.1.10
- DB Instance (private subnet): 10.0.2.20
- Both in the same VPC

Problem:
- From the Web instance, unable to connect to MySQL
- telnet 10.0.2.20 3306 → timeout

DB instance is working:
- Status checks OK
- MySQL is started (verified via Session Manager)

Available Information​

Security Group sg-web (web instance):
Inbound: HTTP 80, SSH 22
Outbound: All traffic 0.0.0.0/0

Security Group sg-database (DB instance):
Inbound: MySQL (3306) from 0.0.0.0/0
Outbound: All traffic 0.0.0.0/0

Private subnet NACL (nacl-private):
Inbound Rules:
  Rule 100: DENY TCP 3306 from 0.0.0.0/0
  Rule 200: ALLOW ALL from 0.0.0.0/0
Outbound Rules:
  Rule 100: ALLOW ALL to 0.0.0.0/0

Questions​

  1. Why does the connection fail?

  2. Explain why the Security Group rule is not enough

  3. How to fix the problem?

📋 See the solution

1. Cause: The NACL blocks port 3306 with a DENY rule (rule 100) which is evaluated BEFORE the ALLOW rule (rule 200).

2. Explanation:

  • Security Groups are evaluated together (all rules)
  • NACLs are evaluated in the order of rule numbers
  • Rule 100 (DENY 3306) is evaluated before Rule 200 (ALLOW ALL)
  • Thus, port 3306 is blocked despite the SG

3. Fix: Delete or modify rule 100 of the NACL:

# Option A: Delete the DENY rule
aws ec2 delete-network-acl-entry \
    --network-acl-id nacl-private \
    --rule-number 100 \
    --ingress

# Option B: Modify to allow from the web subnet
aws ec2 replace-network-acl-entry \
    --network-acl-id nacl-private \
    --rule-number 100 \
    --protocol tcp \
    --port-range From=3306,To=3306 \
    --cidr-block 10.0.1.0/24 \
    --rule-action allow \
    --ingress

Scenario 5: Permission Denied (Challenge)​

Context​

Situation:
- Amazon Linux 2023 instance launched with key "my-key"
- File my-key.pem present on your machine
- File permission: 400 (OK)

Command:
$ ssh -i my-key.pem ec2-user@52.5.6.7
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Tests performed:
- Security Group: port 22 open to 0.0.0.0/0
- Instance: running, status OK
- Ping: works

Additional Information​

You have two key files in your folder:
- my-key.pem (created yesterday)
- my-key.pem.bak (created last week)

The instance was launched last week.

Questions​

  1. What is probably the cause?

  2. How to verify your hypothesis?

  3. What are the options to regain access?

📋 See the solution

1. Probable Cause: You are using the wrong key. The current "my-key.pem" file (created yesterday) is not the key that was associated with the instance (launched last week).

2. Verification:

# See which key is associated with the instance
aws ec2 describe-instances --instance-ids i-xxx \
    --query 'Reservations[].Instances[].KeyName'

# Compare the fingerprint of your key
ssh-keygen -l -f my-key.pem
ssh-keygen -l -f my-key.pem.bak

3. Recovery Options:

Option A - Use the correct key:

ssh -i my-key.pem.bak ec2-user@52.5.6.7

Option B - EC2 Instance Connect (if AMI compatible): Console → Instance → Connect → EC2 Instance Connect

Option C - Systems Manager Session Manager: If the SSM agent is installed and IAM role configured.

Option D - Create an AMI and relaunch:

  1. Create AMI of the instance
  2. Launch new instance with new key
  3. Mount the volume of the old instance
  4. Modify ~/.ssh/authorized_keys

Lab Deliverables​

Troubleshooting Report (for 2 scenarios of your choice)​

For each chosen scenario, document:

# Scenario X: [Title]

## 1. Identified Symptom
[Precise description]

## 2. Formulated Hypotheses
1. [Hypothesis 1] - Probability: High/Medium/Low
2. [Hypothesis 2]
3. [Hypothesis 3]

## 3. Diagnostic Tests
- Test 1: [Command/Action] → Expected result
- Test 2: [Command/Action] → Expected result

## 4. Root Cause
[Explanation of the true cause]

## 5. Solution
[Commands or actions to fix]

## 6. Validation
[How to verify it is resolved]

## 7. Prevention
[How to avoid this problem in the future]

Evaluation Grid​

CriteriaPoints
Well-defined symptom2
Relevant and ordered hypotheses3
Correct diagnostic tests3
Identified root cause3
Correct solution3
Proposed validation2
Clarity and structure2
Total per scenario18

Congratulations! You have completed the EC2 troubleshooting lab.