Skip to main content

Chapter 11.2 - Log Management

Learning Objectives​

By the end of this chapter, you will be able to:

  • Understand the Linux logging system
  • Use journalctl (systemd)
  • Analyze /var/log
  • Configure logrotate
  • Search logs
  • Centralize logs

Logging System​

journalctl (systemd)​

Modern system logs:

# All logs
sudo journalctl

# Recent logs
sudo journalctl -n 50

# Follow in real-time
sudo journalctl -f

# By unit (service)
sudo journalctl -u sshd
sudo journalctl -u apache2

# By priority
sudo journalctl -p err
sudo journalctl -p warning

# By date
sudo journalctl --since "2024-01-01"
sudo journalctl --since "1 hour ago"
sudo journalctl --since yesterday

# By user
sudo journalctl _UID=1000

/var/log​

Traditional logs:

# System logs
/var/log/syslog        # Ubuntu/Debian
/var/log/messages      # Red Hat/CentOS

# Authentication
/var/log/auth.log      # Ubuntu/Debian
/var/log/secure        # Red Hat/CentOS

# Applications
/var/log/apache2/      # Apache
/var/log/nginx/        # Nginx
/var/log/mysql/        # MySQL

# Cron
/var/log/cron          # Red Hat/CentOS
# Ubuntu: in syslog

Log Analysis​

# Simple grep
grep "ERROR" /var/log/syslog

# Ignore case
grep -i "error" /var/log/syslog

# Context
grep -C 5 "ERROR" /var/log/syslog  # 5 lines before/after

# Multiple files
grep "ERROR" /var/log/*.log

# Recursive
grep -r "ERROR" /var/log/

Advanced Tools​

# Count occurrences
grep -c "ERROR" /var/log/syslog

# Unique lines
grep "ERROR" /var/log/syslog | sort -u

# Statistics
grep "ERROR" /var/log/syslog | awk '{print $1}' | sort | uniq -c

# Latest errors
grep "ERROR" /var/log/syslog | tail -20
# Text search
sudo journalctl -g "ERROR"

# By binary
sudo journalctl /usr/bin/sshd

# Combination
sudo journalctl -u sshd -p err --since "1 hour ago"

Log Rotation​

logrotate​

Automatic configuration:

# Configuration
/etc/logrotate.conf
/etc/logrotate.d/

# Test configuration
sudo logrotate -d /etc/logrotate.conf

# Force rotation
sudo logrotate -f /etc/logrotate.conf

Example Configuration​

# /etc/logrotate.d/my-app
/var/log/my-app/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data www-data
    sharedscripts
    postrotate
        systemctl reload my-app > /dev/null 2>&1 || true
    endscript
}

Options:

  • daily/weekly/monthly: Frequency
  • rotate N: Keep N rotations
  • compress: Compress old logs
  • missingok: No error if file missing
  • notifempty: Do not rotate if empty
  • create: Create new file with permissions

Cron Logs​

View Cron Logs​

# Ubuntu/Debian (in syslog)
sudo grep CRON /var/log/syslog

# Red Hat/CentOS
sudo tail -f /var/log/cron

# journalctl
sudo journalctl -u cron
sudo journalctl _COMM=cron

Task Logs​

# If redirection in crontab
# 0 2 * * * /script.sh > /var/log/script.log 2>&1

# View logs
tail -f /var/log/script.log

# Search errors
grep -i error /var/log/script.log

Cron Monitoring​

#!/bin/bash
# Check cron execution

# Check last execution
LAST_RUN=$(grep "script.sh" /var/log/syslog | tail -1 | awk '{print $1, $2, $3}')

if [ -z "$LAST_RUN" ]; then
    echo "ALERT: Cron script has never run"
else
    echo "Last execution: $LAST_RUN"
fi

# Check errors
ERRORS=$(grep "script.sh" /var/log/syslog | grep -i error | wc -l)
if [ "$ERRORS" -gt 0 ]; then
    echo "ALERT: $ERRORS errors detected"
fi

Log Centralization​

rsyslog​

Server configuration:

# /etc/rsyslog.conf
# Enable UDP/TCP
module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

# Store by host
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteHost

Client configuration:

# /etc/rsyslog.conf
# Send to server
*.* @192.168.1.100:514

Logstash/Filebeat​

ELK Stack:

  • Elasticsearch: Storage
  • Logstash: Processing
  • Kibana: Visualization
  • Filebeat: Collection

Useful Scripts​

Daily Analysis​

#!/bin/bash
# Daily log analysis

DATE=$(date +%Y%m%d)
REPORT="/var/log/analysis-$DATE.txt"

{
    echo "=== Log Analysis $(date) ==="
    echo
    echo "=== Errors ==="
    grep -i error /var/log/syslog | tail -20
    echo
    echo "=== Warnings ==="
    grep -i warning /var/log/syslog | tail -20
    echo
    echo "=== SSH Failed Attempts ==="
    grep "Failed password" /var/log/auth.log | wc -l
    echo
    echo "=== Top Processes ==="
    ps aux --sort=-%cpu | head -10
} > "$REPORT"

Log Cleanup​

#!/bin/bash
# Clean old logs

# Delete logs > 30 days
find /var/log -name "*.log" -mtime +30 -delete
find /var/log -name "*.gz" -mtime +90 -delete

# Clean systemd journal
sudo journalctl --vacuum-time=30d
sudo journalctl --vacuum-size=500M

Summary​

In this chapter, you learned:

journalctl: systemd logs, search, filters
/var/log: Traditional logs, applications
Search: grep, journalctl -g, advanced tools
logrotate: Automatic rotation, configuration
Cron logs: Execution monitoring, error detection
Centralization: rsyslog, ELK stack
Practice: Analysis, cleanup, monitoring

Next Steps​

Chapter 11.3: Cron Monitoring and Alerts
Module 12: Advanced Administration

Chapter created on: December 2024