Skip to main content

Chapter 12.1 - Production Administration

Learning Objectives​

By the end of this chapter, you will be able to:

  • Apply production best practices
  • Configure system maintenance
  • Manage updates
  • Implement high availability
  • Document procedures
  • Create runbooks

Production Best Practices​

Security​

# 1. Users and permissions
# - Least privilege
# - No direct root
# - sudo configured

# 2. Firewall
# - Restrictive rules
# - Only necessary ports
# - Connection monitoring

# 3. Updates
# - Regular
# - Tested in staging
# - Scheduled

# 4. Monitoring
# - Continuous surveillance
# - Alerts configured
# - Centralized logs

Performance​

# 1. Resources
# - Monitoring CPU, RAM, disk
# - Threshold alerts
# - Planned scaling

# 2. Optimization
# - Only necessary services
# - Cache configured
# - Compression enabled

# 3. Maintenance
# - Regular cleaning
# - Log rotation
# - Data archiving

System Maintenance​

Planning​

# Maintenance window
# - Off-peak hours
# - Team communication
# - Rollback planned

# Maintenance checklist
# 1. Backup before
# 2. Staging tests
# 3. User notification
# 4. Execute maintenance
# 5. Post-maintenance check
# 6. Document changes

Maintenance Scripts​

#!/bin/bash
# Weekly maintenance script

LOG="/var/log/maintenance-$(date +%Y%m%d).log"

{
    echo "=== Maintenance $(date) ==="
    
    # System update
    echo "Updating packages..."
    apt update && apt upgrade -y
    
    # Cleaning
    echo "Cleaning..."
    apt autoremove -y
    apt autoclean
    
    # Log rotation
    echo "Rotating logs..."
    logrotate -f /etc/logrotate.conf
    
    # Cleaning old logs
    echo "Cleaning old logs..."
    find /var/log -name "*.log" -mtime +30 -delete
    
    # Disk space check
    echo "Disk space:"
    df -h
    
    echo "=== End of maintenance ==="
} | tee "$LOG"

Backup Strategies​

Backup Types​

# 1. Full
# - All data
# - Long, lots of space
# - Fast restoration

# 2. Incremental
# - Only changes
# - Fast, little space
# - Slower restoration

# 3. Differential
# - Since last full
# - Balance time/space

Backup Script​

#!/bin/bash
# Automatic backup

BACKUP_DIR="/backup"
SOURCE_DIR="/data"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/backup-$DATE.tar.gz"

# Create backup
tar -czf "$BACKUP_FILE" "$SOURCE_DIR"

# Verify
if [ $? -eq 0 ]; then
    echo "Backup successful: $BACKUP_FILE"
    
    # Delete backups > 7 days
    find "$BACKUP_DIR" -name "backup-*.tar.gz" -mtime +7 -delete
else
    echo "ERROR: Backup failed" | mail -s "Backup Alert" admin@example.com
    exit 1
fi

Backup with Cron​

# Crontab
# Daily backup at 2 AM
0 2 * * * /usr/local/bin/backup.sh

# Weekly full backup
0 1 * * 0 /usr/local/bin/backup-full.sh

High Availability​

Redundancy​

# 1. Multiple servers
# - Load balancing
# - Automatic failover
# - Data replication

# 2. Monitoring
# - Health checks
# - Failure detection
# - Automatic switchover

# 3. Backup
# - Multiple copies
# - Different sites
# - Restoration tests

Health Checks​

#!/bin/bash
# Server health check

CHECKS_FAILED=0

# Check services
for service in nginx mysql redis; do
    if ! systemctl is-active --quiet "$service"; then
        echo "ERROR: $service inactive"
        CHECKS_FAILED=$((CHECKS_FAILED + 1))
    fi
done

# Check disk space
DISK=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')
if [ "$DISK" -gt 90 ]; then
    echo "ERROR: Disk at ${DISK}%"
    CHECKS_FAILED=$((CHECKS_FAILED + 1))
fi

# Alert if problem
if [ "$CHECKS_FAILED" -gt 0 ]; then
    echo "ALERT: $CHECKS_FAILED check(s) failed" | \
        mail -s "Health Check Alert" admin@example.com
    exit 1
fi

Documentation​

Runbooks​

Operational procedures:

# Runbook: Service Restart

## Objective
Restart a service in production

## Prerequisites
- SSH access
- sudo permissions

## Procedure
1. Check current status
   systemctl status service-name

2. Notify team
   # Slack/Email

3. Restart
   sudo systemctl restart service-name

4. Verify
   systemctl status service-name
   journalctl -u service-name -n 50

5. Document
   # Log incident

System Documentation​

# Files to document:
# - Service configurations
# - Maintenance procedures
# - Team contacts
# - Passwords (managed securely)
# - System architecture

Automation​

Automation Scripts​

#!/bin/bash
# Production task automation

# 1. Monitoring
/usr/local/bin/monitor.sh

# 2. Backup
/usr/local/bin/backup.sh

# 3. Cleaning
/usr/local/bin/cleanup.sh

# 4. Reports
/usr/local/bin/report.sh

Production Cron​

# /etc/cron.d/production
# Monitoring every 5 minutes
*/5 * * * * root /usr/local/bin/monitor.sh

# Daily backup
0 2 * * * root /usr/local/bin/backup.sh

# Weekly cleaning
0 3 * * 0 root /usr/local/bin/cleanup.sh

# Daily report
0 8 * * * root /usr/local/bin/report.sh

Summary​

In this chapter, you learned:

Best practices: Security, performance, maintenance
Maintenance: Planning, scripts, procedures
Backup: Strategies, scripts, automation
HA: Redundancy, health checks, failover
Documentation: Runbooks, procedures
Automation: Scripts, production cron

Next Steps​

Chapter 12.2: Performance Tuning
Chapter 12.3: Complete Automation

Chapter created on: December 2024