End-to-End Testing and Validation

Introduction

Before declaring a deployment "in production," it is essential to validate that all components work correctly together. This section presents a comprehensive testing methodology.

Testing Methodology

┌─────────────────────────────────────────────────────────────┐
│              INFRASTRUCTURE TESTING PYRAMID                 │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│                         ▲                                    │
│                        ╱ ╲                                   │
│                       ╱   ╲                                  │
│                      ╱ E2E ╲     End-to-End Tests            │
│                     ╱───────╲    (real user)                 │
│                    ╱         ╲                               │
│                   ╱Integration╲   Integration Tests          │
│                  ╱─────────────╲  (components together)      │
│                 ╱               ╲                            │
│                ╱   Components    ╲  Unit Tests               │
│               ╱───────────────────╲ (each service)           │
│              ╱                     ╲                         │
│             ╱     Infrastructure    ╲ Config Validation      │
│            ╱─────────────────────────╲(CloudFormation, etc)  │
│                                                              │
│   Start from the bottom, gradually move up                   │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Infrastructure Validation Tests

1. VPC and Network Validation

# Test 1: VPC exists and is available
aws ec2 describe-vpcs --vpc-ids vpc-xxx \
    --query 'Vpcs[0].State'
# Expected: "available"

# Test 2: All subnets created
aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-xxx" \
    --query 'Subnets[*].[SubnetId,CidrBlock,AvailabilityZone]'
# Expected: 6 subnets listed

# Test 3: Internet Gateway attached
aws ec2 describe-internet-gateways \
    --filters "Name=attachment.vpc-id,Values=vpc-xxx" \
    --query 'InternetGateways[0].Attachments[0].State'
# Expected: "available"

# Test 4: Correct routes
aws ec2 describe-route-tables \
    --filters "Name=vpc-id,Values=vpc-xxx" \
    --query 'RouteTables[*].Routes'
# Expected: route 0.0.0.0/0 to igw for public subnets

2. Security Groups Validation

# Test 5: SG ALB allows HTTPS
aws ec2 describe-security-groups --group-ids sg-alb \
    --query 'SecurityGroups[0].IpPermissions[?ToPort==`443`]'
# Expected: rule with 0.0.0.0/0

# Test 6: SG App allows only from ALB
aws ec2 describe-security-groups --group-ids sg-app \
    --query 'SecurityGroups[0].IpPermissions[*].UserIdGroupPairs[*].GroupId'
# Expected: contains sg-alb

# Test 7: SG DB allows only from App
aws ec2 describe-security-groups --group-ids sg-db \
    --query 'SecurityGroups[0].IpPermissions[*].UserIdGroupPairs[*].GroupId'
# Expected: contains sg-app

3. EC2 Instances Validation

# Test 8: Instances running
aws ec2 describe-instance-status \
    --instance-ids i-app1 i-app2 \
    --query 'InstanceStatuses[*].[InstanceId,InstanceState.Name,SystemStatus.Status,InstanceStatus.Status]'
# Expected: all "running", "ok", "ok"

# Test 9: Instances in correct subnets
aws ec2 describe-instances --instance-ids i-app1 i-app2 \
    --query 'Reservations[*].Instances[*].[InstanceId,SubnetId]'
# Expected: private subnets (10.0.11.0/24, 10.0.12.0/24)

Connectivity Tests

Test Matrix

┌─────────────────────────────────────────────────────────────┐
│              CONNECTIVITY MATRIX                            │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│   FROM / TO    │ Internet │ ALB │ App │ DB  │ S3            │
│   ─────────────┼──────────┼─────┼─────┼─────┼───────────    │
│   Internet     │    -     │ ✅  │ ❌  │ ❌  │ ✅(public)    │
│   ALB          │    ✅    │  -  │ ✅  │ ❌  │ ❌            │
│   App          │  ✅(NAT) │ ❌  │ ✅  │ ✅  │ ✅(endpoint)  │
│   DB           │    ❌    │ ❌  │ ❌  │  -  │ ❌            │
│                                                              │
│   ✅ = Must work                                             │
│   ❌ = Must be blocked                                       │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Connectivity Test Scripts

#!/bin/bash
# test-connectivity.sh

echo "=== BlogPro Connectivity Tests ==="

# Test Internet → ALB (HTTPS)
echo -n "Test 1: Internet → ALB (HTTPS)... "
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" https://blog.example.com)
if [ "$HTTP_CODE" = "200" ]; then
    echo "✅ PASS ($HTTP_CODE)"
else
    echo "❌ FAIL ($HTTP_CODE)"
fi

# Test Internet → ALB (HTTP redirect)
echo -n "Test 2: HTTP → HTTPS redirect... "
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://blog.example.com)
if [ "$HTTP_CODE" = "301" ] || [ "$HTTP_CODE" = "302" ]; then
    echo "✅ PASS (redirect $HTTP_CODE)"
else
    echo "❌ FAIL ($HTTP_CODE)"
fi

# Test Health Check endpoint
echo -n "Test 3: Health Check... "
HEALTH=$(curl -s https://blog.example.com/health | jq -r '.status')
if [ "$HEALTH" = "healthy" ]; then
    echo "✅ PASS"
else
    echo "❌ FAIL ($HEALTH)"
fi

# Test Direct IP access (should fail - no public IP)
echo -n "Test 4: Direct IP access (should timeout)... "
timeout 5 curl -s http://10.0.11.20 > /dev/null 2>&1
if [ $? -eq 124 ]; then
    echo "✅ PASS (correctly blocked)"
else
    echo "❌ FAIL (should not be accessible)"
fi

echo "=== Tests completed ==="

Load Balancing Tests

Verify Distribution

#!/bin/bash
# test-load-balancing.sh

echo "=== Load Balancer Distribution Test ==="

# Make 20 requests and count responses from each server
declare -A SERVER_COUNT

for i in {1..20}; do
    SERVER=$(curl -s https://blog.example.com/api/server-id)
    ((SERVER_COUNT[$SERVER]++))
done

echo "Request distribution:"
for server in "${!SERVER_COUNT[@]}"; do
    echo "  $server: ${SERVER_COUNT[$server]} requests"
done

# Verify that both servers received requests
if [ ${#SERVER_COUNT[@]} -ge 2 ]; then
    echo "✅ Load balancing functional"
else
    echo "❌ Load balancing failing"
fi

Failover Test

┌─────────────────────────────────────────────────────────────┐
│              FAILOVER TEST PROCEDURE                         │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│   Step 1 : Verify initial state                             │
│   ────────────────────────────────                          │
│   $ aws elbv2 describe-target-health \                      │
│       --target-group-arn arn:aws:...:tg-app                │
│   → Both targets must be "healthy"                         │
│                                                              │
│   Step 2 : Simulate server failure                          │
│   ─────────────────────────────────────                     │
│   $ aws ec2 stop-instances --instance-ids i-app1            │
│                                                              │
│   Step 3 : Wait for health check (30-60s)                   │
│   ──────────────────────────────────────────                │
│   $ watch -n 5 'curl -s https://blog.example.com/health'   │
│   → Site must remain accessible                             │
│                                                              │
│   Step 4 : Verify traffic goes to App2                      │
│   ────────────────────────────────────────────              │
│   $ for i in {1..10}; do curl -s https://blog.example.com/ │
│       api/server-id; done                                   │
│   → All responses come from App2                            │
│                                                              │
│   Step 5 : Restore App1                                     │
│   ────────────────────────────                              │
│   $ aws ec2 start-instances --instance-ids i-app1           │
│   → After a few minutes, traffic distributed again          │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Performance Tests

Basic Load Test

# Use Apache Benchmark (ab)
# 1000 requests, 50 concurrent
ab -n 1000 -c 50 https://blog.example.com/

# Expected results:
# - Requests per second > 100
# - Time per request < 500ms (mean)
# - Failed requests: 0

Metrics to Monitor

Metric	Acceptable Threshold	Action if Exceeded
Latency P50	< 200ms	Optimize code/cache
Latency P95	< 500ms	Check DB queries
Latency P99	< 1000ms	Investigate outliers
Error rate	< 0.1%	Check logs
CPU (EC2)	< 70%	Add instances
Memory (EC2)	< 80%	Increase size

Security Tests

Security Checklist

┌─────────────────────────────────────────────────────────────┐
│              SECURITY CHECKLIST                             │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│   NETWORK                                                    │
│   ☐ No unnecessary ports open to the Internet               │
│   ☐ SSH accessible only via bastion/VPN                     │
│   ☐ Database not accessible from the Internet               │
│   ☐ HTTPS enforced (redirect HTTP→HTTPS)                    │
│                                                              │
│   SSL CERTIFICATE                                            │
│   ☐ Valid certificate (not expired)                         │
│   ☐ Complete certificate chain                              │
│   ☐ TLS 1.2+ only                                           │
│   ☐ No SSL vulnerabilities (test SSL Labs)                  │
│                                                              │
│   INSTANCES                                                  │
│   ☐ No credentials in code/user-data                        │
│   ☐ IAM roles used (no keys)                                │
│   ☐ OS up to date                                           │
│   ☐ Local firewall configured                               │
│                                                              │
│   DATABASE                                                   │
│   ☐ Encryption at rest enabled                              │
│   ☐ Encryption in transit (SSL)                             │
│   ☐ No default credentials                                  │
│   ☐ Automatic backups enabled                               │
│                                                              │
└─────────────────────────────────────────────────────────────┘

SSL Test with SSL Labs

# Use the online SSL Labs test
# https://www.ssllabs.com/ssltest/

# Or with testssl.sh
docker run --rm -ti drwetter/testssl.sh https://blog.example.com

# Expected score: A or A+

Validation Report

Report Template

# Validation Report - BlogPro

## Executive Summary
- Date: YYYY-MM-DD
- Version: 1.0
- Status: ✅ VALIDATED / ❌ NOT VALIDATED

## Infrastructure Tests
| Test | Result | Notes |
|------|--------|-------|
| VPC created | ✅ | vpc-xxx |
| Subnets (6) | ✅ | |
| IGW attached | ✅ | |
| Correct routes | ✅ | |

## Security Tests
| Test | Result | Notes |
|------|--------|-------|
| SG ALB | ✅ | HTTPS only |
| SG App | ✅ | ALB only |
| SG DB | ✅ | App only |
| SSL Grade | A | SSL Labs |

## Functional Tests
| Test | Result | Notes |
|------|--------|-------|
| Site accessible | ✅ | 200 OK |
| Health check | ✅ | healthy |
| Load balancing | ✅ | 50/50 |
| Failover | ✅ | < 60s |

## Performance Tests
| Metric | Value | Threshold | Status |
|--------|-------|-----------|--------|
| Latency P50 | 150ms | < 200ms | ✅ |
| Latency P95 | 380ms | < 500ms | ✅ |
| Throughput | 250 rps | > 100 rps | ✅ |

## Identified Issues
1. None

## Recommendations
1. Enable VPC Flow Logs for monitoring
2. Set up CloudWatch alerts

## Approval
- [ ] Tech Lead
- [ ] Security Review
- [ ] Ops Team

Test Automation

Complete Validation Script

#!/bin/bash
# validate-deployment.sh

RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m' # No Color

DOMAIN="blog.example.com"
PASS=0
FAIL=0

function test_result() {
    if [ $1 -eq 0 ]; then
        echo -e "${GREEN}✅ PASS${NC}: $2"
        ((PASS++))
    else
        echo -e "${RED}❌ FAIL${NC}: $2"
        ((FAIL++))
    fi
}

echo "========================================"
echo "   BLOGPRO DEPLOYMENT VALIDATION"
echo "========================================"
echo ""

# Test 1: Site accessible
echo "Test 1: HTTPS Accessibility"
curl -sf "https://${DOMAIN}" > /dev/null
test_result $? "Site accessible via HTTPS"

# Test 2: Health check
echo "Test 2: Health Check"
HEALTH=$(curl -sf "https://${DOMAIN}/health" | jq -r '.status' 2>/dev/null)
[ "$HEALTH" = "healthy" ]
test_result $? "Health check endpoint"

# Test 3: HTTP redirect
echo "Test 3: HTTP → HTTPS redirect"
CODE=$(curl -sf -o /dev/null -w "%{http_code}" "http://${DOMAIN}")
[ "$CODE" = "301" ] || [ "$CODE" = "302" ]
test_result $? "HTTP redirect to HTTPS"

# Test 4: SSL Certificate
echo "Test 4: Valid SSL Certificate"
curl -sf --cert-status "https://${DOMAIN}" > /dev/null
test_result $? "Valid SSL Certificate"

# Test 5: Response time
echo "Test 5: Response time < 500ms"
TIME=$(curl -sf -o /dev/null -w "%{time_total}" "https://${DOMAIN}")
[ $(echo "$TIME < 0.5" | bc) -eq 1 ]
test_result $? "Response time (${TIME}s)"

echo ""
echo "========================================"
echo "   RESULT: $PASS PASS, $FAIL FAIL"
echo "========================================"

if [ $FAIL -gt 0 ]; then
    exit 1
fi

Summary

Phase	Tests	Tools
Infrastructure	VPC, SG, Routes	AWS CLI
Connectivity	Access matrix	curl, nc, telnet
Functional	E2E, Health	curl, scripts
Performance	Load, latency	ab, hey, k6
Security	SSL, access	testssl, nmap

Return: MODULE 7 Overview

Introduction​

Testing Methodology​

Infrastructure Validation Tests​

1. VPC and Network Validation​

2. Security Groups Validation​

3. EC2 Instances Validation​

Connectivity Tests​

Test Matrix​

Connectivity Test Scripts​

Load Balancing Tests​

Verify Distribution​

Failover Test​

Performance Tests​

Basic Load Test​

Metrics to Monitor​

Security Tests​

Security Checklist​

SSL Test with SSL Labs​

Validation Report​

Report Template​

Test Automation​

Complete Validation Script​

Summary​

Introduction

Testing Methodology

Infrastructure Validation Tests

1. VPC and Network Validation

2. Security Groups Validation

3. EC2 Instances Validation

Connectivity Tests

Test Matrix

Connectivity Test Scripts

Load Balancing Tests

Verify Distribution

Failover Test

Performance Tests

Basic Load Test

Metrics to Monitor

Security Tests

Security Checklist

SSL Test with SSL Labs

Validation Report

Report Template

Test Automation

Complete Validation Script

Summary