Skip to main content

Chapter 11.3 - Monitoring Crons and Alerts

Learning Objectives​

By the end of this chapter, you will be able to:

  • Monitor cron execution
  • Detect task failures
  • Configure alerts for crons
  • Notify via email/Slack
  • Create monitoring dashboards
  • Implement health checks

Monitoring Crons​

Verify Execution​

# View last executions
sudo grep CRON /var/log/syslog | tail -20

# By script
sudo grep "my-script.sh" /var/log/syslog

# journalctl
sudo journalctl -u cron --since "1 hour ago"

# Check if executed today
sudo grep "my-script.sh" /var/log/syslog | grep "$(date +%b\ %d)"

Verification Script​

#!/bin/bash
# Check cron execution

SCRIPT="my-script.sh"
LOG="/var/log/syslog"
EXPECTED_TIME="02:00"  # Expected time

# Check execution today
TODAY=$(date +%b\ %d)
EXECUTED=$(grep "$SCRIPT" "$LOG" | grep "$TODAY" | wc -l)

if [ "$EXECUTED" -eq 0 ]; then
    echo "ALERT: $SCRIPT was not executed today"
    # Send alert...
fi

Failure Detection​

Return Code​

#!/bin/bash
# Script with error handling

set -e  # Stop on error

# Processing...
if ! important_command; then
    echo "ERROR: Command failed" >&2
    exit 1
fi

# Success
exit 0

Check Return Code​

#!/bin/bash
# Check failure in logs

SCRIPT="my-script.sh"
LOG="/var/log/script.log"

# Search for errors
ERRORS=$(grep -i "error\|failed\|failure" "$LOG" | wc -l)

if [ "$ERRORS" -gt 0 ]; then
    echo "ALERT: $ERRORS errors in $SCRIPT"
    # Send alert...
fi

Email Alerts​

Configuration​

# Install mail
sudo apt install mailutils  # Ubuntu/Debian
sudo yum install mailx      # Red Hat/CentOS

# Configuration
# /etc/ssmtp/ssmtp.conf (Ubuntu)
# or
# /etc/mail.rc (Red Hat)

Script with Email​

#!/bin/bash
# Script with email notification

ALERT_EMAIL="admin@example.com"
SCRIPT_NAME="my-script.sh"

# Execute script
if /usr/local/bin/my-script.sh; then
    echo "Success: $SCRIPT_NAME executed" | \
        mail -s "Success: $SCRIPT_NAME" "$ALERT_EMAIL"
else
    echo "FAILURE: $SCRIPT_NAME failed" | \
        mail -s "ALERT: $SCRIPT_NAME" "$ALERT_EMAIL"
    exit 1
fi

Crontab with Email​

# Crontab
# Send output by email
MAILTO=admin@example.com
0 2 * * * /usr/local/bin/script.sh

# Or redirect to email
0 2 * * * /usr/local/bin/script.sh 2>&1 | mail -s "Report" admin@example.com

Slack Alerts​

Slack Webhook​

#!/bin/bash
# Slack notification

SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
SCRIPT_NAME="my-script.sh"

# Message
MESSAGE="Script $SCRIPT_NAME: $(if [ $? -eq 0 ]; then echo 'Success'; else echo 'FAILURE'; fi)"

# Send
curl -X POST -H 'Content-type: application/json' \
    --data "{\"text\":\"$MESSAGE\"}" \
    "$SLACK_WEBHOOK"

Reusable Function​

#!/bin/bash
# Slack notification function

notify_slack() {
    local webhook="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
    local message="$1"
    local status="$2"
    
    local color="good"
    [ "$status" != "success" ] && color="danger"
    
    curl -X POST -H 'Content-type: application/json' \
        --data "{\"attachments\":[{\"color\":\"$color\",\"text\":\"$message\"}]}" \
        "$webhook"
}

# Usage
if my-script.sh; then
    notify_slack "Script succeeded" "success"
else
    notify_slack "Script failed" "failure"
fi

Health Checks​

Health Check Script​

#!/bin/bash
# Health check for cron

SCRIPT="my-script.sh"
LOCK_FILE="/var/run/$SCRIPT.lock"
MAX_AGE=3600  # 1 hour in seconds

# Check lock file
if [ -f "$LOCK_FILE" ]; then
    AGE=$(($(date +%s) - $(stat -c %Y "$LOCK_FILE")))
    if [ "$AGE" -gt "$MAX_AGE" ]; then
        echo "ALERT: Script locked for $AGE seconds"
        # Send alert...
        rm -f "$LOCK_FILE"
    fi
fi

Heartbeat​

#!/bin/bash
# Heartbeat for monitoring

HEARTBEAT_FILE="/var/run/my-script.heartbeat"

# Update timestamp
touch "$HEARTBEAT_FILE"

# Check in monitoring
# If heartbeat > 2 hours, alert

Monitoring Dashboard​

Report Script​

#!/bin/bash
# Daily cron report

REPORT="/var/log/cron-report-$(date +%Y%m%d).txt"

{
    echo "=== CRON REPORT $(date) ==="
    echo
    echo "=== Successful Executions ==="
    grep "my-script.sh" /var/log/syslog | grep -v "error" | wc -l
    echo
    echo "=== Failed Executions ==="
    grep "my-script.sh" /var/log/syslog | grep -i "error\|failed" | wc -l
    echo
    echo "=== Last Executions ==="
    grep "my-script.sh" /var/log/syslog | tail -10
    echo
    echo "=== Statistics ==="
    echo "Total executions: $(grep 'my-script.sh' /var/log/syslog | wc -l)"
} > "$REPORT"

# Send by email
mail -s "Cron Report $(date +%Y-%m-%d)" admin@example.com < "$REPORT"

Advanced Monitoring​

Complete Monitoring Script​

#!/bin/bash
# Complete cron monitoring

LOG_DIR="/var/log/cron-monitoring"
mkdir -p "$LOG_DIR"

# List of crons to monitor
CRONS=(
    "backup.sh"
    "cleanup.sh"
    "report.sh"
)

for cron in "${CRONS[@]}"; do
    # Check execution
    EXECUTED=$(grep "$cron" /var/log/syslog | grep "$(date +%b\ %d)" | wc -l)
    
    # Check errors
    ERRORS=$(grep "$cron" /var/log/syslog | grep -i "error\|failed" | wc -l)
    
    # Log
    echo "$(date): $cron - Executions: $EXECUTED, Errors: $ERRORS" >> \
        "$LOG_DIR/monitoring.log"
    
    # Alert if problem
    if [ "$EXECUTED" -eq 0 ] || [ "$ERRORS" -gt 0 ]; then
        echo "ALERT: $cron - Executions: $EXECUTED, Errors: $ERRORS" | \
            mail -s "Cron Alert" admin@example.com
    fi
done

Summary​

In this chapter, you learned:

Monitoring: Verify executions, detect failures
Alerts: Email, Slack, notifications
Health checks: Lock files, heartbeats
Reports: Dashboards, statistics
Scripts: Monitoring automation
Practice: Continuous monitoring, proactive alerts

Next Steps​

Module 12: Advanced Administration (with cron automation)
End of Course: Recap and Projects

Chapter created on: December 2024