Chapter 12.3 - Complete Automation

Learning Objectives

By the end of this chapter, you will be able to:

Create complete automation systems
Integrate cron with advanced scripts
Implement automated workflows
Manage errors and retry
Create notification systems
Document the automation

Automation Architecture

Components

Elements:

Scheduling (cron/systemd timers)
Executable scripts
Logging
Notifications
Monitoring

Automation Framework

Basic Structure

#!/bin/bash
# Automation framework

set -euo pipefail

# Configuration
SCRIPT_NAME="my-automation"
LOG_DIR="/var/log/automation"
LOCK_DIR="/var/run/automation"
NOTIFY_EMAIL="admin@example.com"

# Utility functions
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_DIR/$SCRIPT_NAME.log"
}

notify() {
    local subject="$1"
    local message="$2"
    echo "$message" | mail -s "$subject" "$NOTIFY_EMAIL"
}

cleanup() {
    rm -f "$LOCK_DIR/$SCRIPT_NAME.lock"
    log "Cleanup completed"
}

# Trap for cleanup
trap cleanup EXIT

# Lock file
LOCK_FILE="$LOCK_DIR/$SCRIPT_NAME.lock"
if [ -f "$LOCK_FILE" ]; then
    log "Script already running"
    exit 1
fi
touch "$LOCK_FILE"

# Main code
main() {
    log "Starting automation"
    
    # Tasks...
    task1
    task2
    task3
    
    log "Automation completed"
}

main "$@"

Automated Workflows

Automated Backup

#!/bin/bash
# Complete automated backup

set -euo pipefail

BACKUP_DIR="/backup"
SOURCES=("/data" "/etc" "/home")
RETENTION_DAYS=7
LOG="/var/log/backup.log"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"
}

backup_source() {
    local source="$1"
    local dest="$BACKUP_DIR/$(basename $source)-$(date +%Y%m%d).tar.gz"
    
    log "Backup: $source -> $dest"
    tar -czf "$dest" "$source" || {
        log "ERROR: Backup failed for $source"
        return 1
    }
    
    log "Success: $dest created ($(du -h $dest | cut -f1))"
}

# Backup each source
for source in "${SOURCES[@]}"; do
    backup_source "$source"
done

# Cleanup old backups
log "Cleaning backups > $RETENTION_DAYS days"
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete

log "Backup completed"

Automated Monitoring

#!/bin/bash
# Automated monitoring

THRESHOLDS=(
    "CPU:80"
    "MEM:85"
    "DISK:90"
)

check_threshold() {
    local metric="$1"
    local threshold="$2"
    local value
    
    case "$metric" in
        CPU)
            value=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | sed 's/%us,//')
            ;;
        MEM)
            value=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100}')
            ;;
        DISK)
            value=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')
            ;;
    esac
    
    if [ "$value" -gt "$threshold" ]; then
        echo "ALERT: $metric at ${value}% (threshold: ${threshold}%)"
        return 1
    fi
    return 0
}

# Check all thresholds
for threshold in "${THRESHOLDS[@]}"; do
    metric=$(echo $threshold | cut -d: -f1)
    value=$(echo $threshold | cut -d: -f2)
    check_threshold "$metric" "$value" || notify "Alert $metric" "$metric exceeded"
done

Error Management and Retry

Retry Function

retry() {
    local max_attempts="$1"
    local delay="$2"
    shift 2
    local command="$@"
    
    local attempt=1
    while [ $attempt -le $max_attempts ]; do
        if $command; then
            return 0
        fi
        
        if [ $attempt -lt $max_attempts ]; then
            log "Attempt $attempt/$max_attempts failed, retrying in ${delay}s"
            sleep "$delay"
        fi
        
        attempt=$((attempt + 1))
    done
    
    log "ERROR: All attempts failed"
    return 1
}

# Usage
retry 3 5 wget http://example.com/file.txt

Advanced Error Management

#!/bin/bash
# Complete error management

ERROR_LOG="/var/log/automation-errors.log"

handle_error() {
    local line="$1"
    local command="$2"
    local exit_code="$3"
    
    echo "[$(date)] ERROR line $line: $command (code: $exit_code)" >> "$ERROR_LOG"
    
    # Notification
    echo "Error in automation line $line" | \
        mail -s "Automation Error" admin@example.com
    
    # Logging
    log "ERROR: $command (line $line, code $exit_code)"
}

# Trap errors
trap 'handle_error $LINENO "$BASH_COMMAND" $?' ERR

# Code with error management
set -e
# ...

Notification System

Multi-Channel Notifications

#!/bin/bash
# Notification system

notify() {
    local level="$1"  # info, warning, error
    local message="$2"
    
    # Email
    echo "$message" | mail -s "[$level] Automation" admin@example.com
    
    # Slack (if configured)
    if [ -n "$SLACK_WEBHOOK" ]; then
        local color="good"
        [ "$level" = "error" ] && color="danger"
        [ "$level" = "warning" ] && color="warning"
        
        curl -X POST -H 'Content-type: application/json' \
            --data "{\"attachments\":[{\"color\":\"$color\",\"text\":\"$message\"}]}" \
            "$SLACK_WEBHOOK" 2>/dev/null || true
    fi
    
    # Log
    log "[$level] $message"
}

# Usage
notify "info" "Task started"
notify "warning" "Warning: threshold near"
notify "error" "ERROR: Task failed"

Automation Documentation

Automation Runbook

# Runbook: Automation System

## Overview
Automated system for maintenance and monitoring

## Components
- Cron: Task scheduling
- Scripts: /usr/local/bin/automation/
- Logs: /var/log/automation/
- Notifications: Email + Slack

## Scheduled Tasks
- Backup: Daily 2am
- Monitoring: Every 5 minutes
- Cleanup: Weekly Sunday 3am
- Reports: Daily 8am

## Troubleshooting
- Check logs: /var/log/automation/
- Check cron: grep CRON /var/log/syslog
- Check locks: /var/run/automation/

Script Documentation

#!/bin/bash
# Script: my-automation.sh
# Author: Admin
# Date: 2024-12
# Description: Complete maintenance automation
# Usage: Executed by cron
# Dependencies: mail, curl
# Logs: /var/log/automation/my-automation.log

Complete Example

Production Automation System

#!/bin/bash
# /usr/local/bin/automation-production.sh
# Production automation system

set -euo pipefail

# Configuration
SCRIPT_DIR="/usr/local/bin/automation"
LOG_DIR="/var/log/automation"
LOCK_DIR="/var/run/automation"
NOTIFY_EMAIL="admin@example.com"

# Create directories
mkdir -p "$LOG_DIR" "$LOCK_DIR"

# Lock
LOCK="$LOCK_DIR/production.lock"
[ -f "$LOCK" ] && exit 1
touch "$LOCK"
trap "rm -f $LOCK" EXIT

# Logging
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_DIR/production.log"
}

# Notification
notify() {
    echo "$*" | mail -s "Production Automation" "$NOTIFY_EMAIL"
}

# Tasks
log "=== Starting automation ==="

# 1. Monitoring
log "Monitoring..."
"$SCRIPT_DIR/monitor.sh" || notify "Alert: Monitoring failed"

# 2. Backup
log "Backup..."
"$SCRIPT_DIR/backup.sh" || notify "Alert: Backup failed"

# 3. Cleanup
log "Cleanup..."
"$SCRIPT_DIR/cleanup.sh"

# 4. Report
log "Generating report..."
"$SCRIPT_DIR/report.sh"

log "=== Automation completed ==="
notify "Production automation completed successfully"

Production Crontab

# /etc/cron.d/production-automation
# Monitoring every 5 minutes
*/5 * * * * root /usr/local/bin/automation/monitor.sh

# Daily backup 2am
0 2 * * * root /usr/local/bin/automation/backup.sh

# Weekly cleanup
0 3 * * 0 root /usr/local/bin/automation/cleanup.sh

# Daily report
0 8 * * * root /usr/local/bin/automation/report.sh

# Complete daily automation
0 1 * * * root /usr/local/bin/automation-production.sh

Summary

In this chapter, you learned:

Architecture: Automation system components
Framework: Reusable structure, utility functions
Workflows: Backup, monitoring, complex tasks
Errors: Retry, advanced management, logging
Notifications: Multi-channel, email, Slack
Documentation: Runbooks, script comments
Practice: Complete production system

End of Course

Congratulations! You have completed the full Linux training with advanced cron automation.

Next steps:

Practice on real servers
Implement your own automations
Continue learning and exploring

Chapter created on: December 2024

Learning Objectives​

Automation Architecture​

Components​

Automation Framework​

Basic Structure​

Automated Workflows​

Automated Backup​

Automated Monitoring​

Error Management and Retry​

Retry Function​

Advanced Error Management​

Notification System​

Multi-Channel Notifications​

Automation Documentation​

Automation Runbook​

Script Documentation​

Complete Example​

Production Automation System​

Production Crontab​

Summary​

End of Course​