Health Checks
What is a Health Check?
A Health Check is an automatic mechanism that regularly checks if your servers are functioning correctly. The Load Balancer uses this information to direct traffic only to healthy servers.
┌─────────────────────────────────────────────────────────────┐
│ HEALTH CHECK - CONCEPT │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ ALB │ │
│ │ │ │
│ │ Every 30 seconds, checks: │ │
│ │ "Does /health respond 200 OK?" │ │
│ │ │ │
│ └─────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌─────────────┼─────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │ Server │ │ Server │ │ Server │ │
│ │ 1 │ │ 2 │ │ 3 │ │
│ │ │ │ │ │ │ │
│ │ 200✓ │ │ 200✓ │ │ 503✗ │ │
│ │ HEALTHY│ │ HEALTHY│ │UNHEALTHY│ │
│ └────────┘ └────────┘ └────────┘ │
│ ↑ ↑ ✗ │
│ │ │ │ │
│ Receives Receives Does NOT receive │
│ traffic traffic traffic │
│ │
└─────────────────────────────────────────────────────────────┘
Health Check Parameters
Configuration
┌─────────────────────────────────────────────────────────────┐
│ HEALTH CHECK PARAMETERS │
├─────────────────────────────────────────────────────────────┤
│ │
│ PARAMETER │ DESCRIPTION │ DEFAULT │
│ ───────────────────────┼──────────────────────┼──────────│
│ Protocol │ HTTP, HTTPS, TCP │ HTTP │
│ Port │ Port to check │ traffic │
│ Path │ URL to test │ / │
│ Interval │ Frequency (sec) │ 30 │
│ Timeout │ Max delay (sec) │ 5 │
│ Healthy Threshold │ Success for healthy │ 3 │
│ Unhealthy Threshold │ Failures for unhealthy│ 2 │
│ Success Codes │ HTTP OK codes │ 200 │
│ │
│ Example: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ Protocol: HTTP │ │
│ │ Port: 80 │ │
│ │ Path: /health │ │
│ │ Interval: 30s │ │
│ │ Timeout: 5s │ │
│ │ Healthy Threshold: 2 │ │
│ │ Unhealthy Threshold: 3 │ │
│ │ Success Codes: 200-299 │ │
│ │ │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Target Lifecycle
┌─────────────────────────────────────────────────────────────┐
│ TARGET STATES │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ │
│ │ initial │ Target has just been registered │
│ └────┬─────┘ │
│ │ │
│ │ Health checks start │
│ ▼ │
│ ┌──────────┐ N consecutive failures ┌──────────┐ │
│ │ healthy │ ────────────────────────────►│ unhealthy│ │
│ │ │ │ │ │
│ │ Receives │◄──────────────────────────── │ Does NOT │ │
│ │ traffic │ N consecutive successes │ receive │ │
│ └──────────┘ └──────────┘ │
│ │
│ Other states: │
│ • draining : Target being deregistered │
│ • unused : Target not used (no traffic to it) │
│ │
└─────────────────────────────────────────────────────────────┘
Endpoint /health
Recommended Implementation
┌─────────────────────────────────────────────────────────────┐
│ ENDPOINT /health │
├─────────────────────────────────────────────────────────────┤
│ │
│ SIMPLE (basic) │
│ ──────────────── │
│ GET /health │
│ Response: 200 OK │
│ Body: {"status": "ok"} │
│ │
│ ADVANCED (checks dependencies) │
│ ───────────────────────────────── │
│ GET /health │
│ Response: 200 OK (if all is well) │
│ Body: │
│ { │
│ "status": "healthy", │
│ "timestamp": "2024-01-15T10:30:00Z", │
│ "checks": { │
│ "database": "connected", │
│ "cache": "connected", │
│ "disk": "ok (75% free)" │
│ } │
│ } │
│ │
│ Response: 503 Service Unavailable (if problem) │
│ Body: │
│ { │
│ "status": "unhealthy", │
│ "checks": { │
│ "database": "connection failed", │
│ "cache": "connected", │
│ "disk": "ok" │
│ } │
│ } │
│ │
└─────────────────────────────────────────────────────────────┘
Code Examples
Python (Flask):
from flask import Flask, jsonify
import psycopg2
app = Flask(__name__)
@app.route('/health')
def health_check():
checks = {}
is_healthy = True
# Check the database
try:
conn = psycopg2.connect(DATABASE_URL)
conn.close()
checks['database'] = 'connected'
except Exception as e:
checks['database'] = f'failed: {str(e)}'
is_healthy = False
# Check Redis cache
try:
redis_client.ping()
checks['redis'] = 'connected'
except:
checks['redis'] = 'failed'
is_healthy = False
status_code = 200 if is_healthy else 503
return jsonify({
'status': 'healthy' if is_healthy else 'unhealthy',
'checks': checks
}), status_code
Node.js (Express):
app.get('/health', async (req, res) => {
const checks = {};
let isHealthy = true;
// Check the DB
try {
await db.query('SELECT 1');
checks.database = 'connected';
} catch (e) {
checks.database = 'failed';
isHealthy = false;
}
const statusCode = isHealthy ? 200 : 503;
res.status(statusCode).json({
status: isHealthy ? 'healthy' : 'unhealthy',
checks
});
});
Best Practices
Recommended Configuration
┌─────────────────────────────────────────────────────────────┐
│ HEALTH CHECK BEST PRACTICES │
├─────────────────────────────────────────────────────────────┤
│ │
│ ✅ TO DO │
│ ────────── │
│ • Use a dedicated endpoint (/health, /healthz) │
│ • Check critical dependencies (DB, cache) │
│ • Return 200 only if truly healthy │
│ • Keep the health check light and fast │
│ • Document what the health check verifies │
│ │
│ ❌ TO AVOID │
│ ────────── │
│ • Using the homepage (/) as a health check │
│ • Health check that performs heavy operations │
│ • Always returning 200 (even if there is a problem) │
│ • Interval too short (overload) │
│ • Timeout too long (slow detection) │
│ │
│ RECOMMENDED CONFIGURATION: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ Interval: 30 seconds │ │
│ │ Timeout: 5 seconds │ │
│ │ Healthy Threshold: 2 │ │
│ │ Unhealthy Threshold: 3 │ │
│ │ Path: /health │ │
│ │ │ │
│ │ Time to detect a failure: │ │
│ │ 3 failures × 30s = 90 seconds max │ │
│ │ │ │
│ │ Time to become healthy again: │ │
│ │ 2 successes × 30s = 60 seconds min │ │
│ │ │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Types of Health Checks
| Type | What it checks | When to use |
|---|---|---|
| Shallow | App responds (200 OK) | Minimum viable |
| Deep | App + dependencies (DB, cache) | Production |
| Liveness | App is alive | Kubernetes |
| Readiness | App can receive traffic | Kubernetes |
CLI Configuration
# Create a Target Group with custom health check
aws elbv2 create-target-group \
--name tg-web-healthy \
--protocol HTTP \
--port 80 \
--vpc-id vpc-xxx \
--health-check-protocol HTTP \
--health-check-port 80 \
--health-check-path /health \
--health-check-interval-seconds 30 \
--health-check-timeout-seconds 5 \
--healthy-threshold-count 2 \
--unhealthy-threshold-count 3 \
--matcher HttpCode=200-299
# Modify an existing health check
aws elbv2 modify-target-group \
--target-group-arn arn:aws:...:tg-web \
--health-check-path /api/health \
--health-check-interval-seconds 15
# Check the state of targets
aws elbv2 describe-target-health \
--target-group-arn arn:aws:...:tg-web
Health Checks Troubleshooting
┌─────────────────────────────────────────────────────────────┐
│ HEALTH CHECK TROUBLESHOOTING │
├─────────────────────────────────────────────────────────────┤
│ │
│ SYMPTOM: Target remains "unhealthy" │
│ │
│ Checklist: │
│ │
│ ☐ 1. Security Group allows traffic from LB? │
│ → EC2 SG must allow the port from sg-alb │
│ │
│ ☐ 2. Is the application listening on the correct port? │
│ → Check with netstat -tlnp on the instance │
│ │
│ ☐ 3. Does the path /health exist and return 200? │
│ → Test with curl http://localhost/health │
│ │
│ ☐ 4. Is the timeout not too short? │
│ → If /health is slow, increase timeout │
│ │
│ ☐ 5. Are the success codes correct? │
│ → If the app returns 204, add it to success │
│ │
│ Useful commands: │
│ # On the EC2 instance │
│ curl -v http://localhost/health │
│ netstat -tlnp | grep :80 │
│ │
│ # From AWS CLI │
│ aws elbv2 describe-target-health --target-group-arn ... │
│ │
└─────────────────────────────────────────────────────────────┘
Summary
| Parameter | Recommended Value | Impact |
|---|---|---|
| Interval | 30s | Check frequency |
| Timeout | 5s | Max response delay |
| Healthy Threshold | 2 | Fast recovery |
| Unhealthy Threshold | 3 | Avoid false positives |
| Path | /health | Dedicated endpoint |
Next section: Lab Architecture HA