Linux Log Analysis: A Complete Guide for Administrators

Introduction

Log analysis is an essential skill for any Linux administrator. Logs contain information about system, service, and application operations, helping to quickly diagnose failures, track security issues, and optimize performance. In this guide, you'll learn how to work effectively with logs in Linux, using both standard utilities and modern tools.

Prerequisites

Before you begin, ensure that:

• You have access to a Linux system (physical or virtual) with a systemd-based distribution installed (e.g., Ubuntu 22.04, CentOS 8).
• You are familiar with basic command-line operations.
• Superuser privileges may be required to access certain logs (use sudo).

Step 1: Understanding Log Structure in Linux

In Linux, logs are stored in two primary formats:

1. File-based logs: traditional text files in the /var/log/ directory. This includes kernel logs (kern.log), system logs (syslog), authentication logs (auth.log), package manager logs (dpkg.log or yum.log), and many others.
2. systemd journal: a binary journal managed by systemd-journald. Accessible via the journalctl command, stored in /var/log/journal/ (if persistent storage is enabled).

By default, many modern distributions use the systemd journal but may also write to file-based logs via services like rsyslog or syslog-ng.

To view the list of logs in /var/log/, run:

ls -la /var/log/

To view active journal disk usage:

sudo journalctl --disk-usage  # check journal size

Step 2: Using Basic Commands to View Logs

For file-based logs, use standard utilities:

• less: page through content. less /var/log/syslog
• tail: view the last lines. tail -f /var/log/syslog for real-time monitoring.
• grep: search by pattern. grep "error" /var/log/syslog
• head: view the first lines. head -n 50 /var/log/syslog

Combine commands for efficient analysis. For example, to find the last 10 errors in syslog:

grep "error" /var/log/syslog | tail -n 10

For journalctl, similar operations include:

• journalctl: view the entire journal.
• journalctl -f: follow in real time.
• journalctl -u ssh.service: logs for a specific service.
• journalctl --since "2026-02-14 09:00:00": filter by time.

Step 3: Filtering and Searching in Logs

More advanced filtering requires using grep with regular expressions or awk to extract fields.

Example: find all failed SSH attempts in auth.log:

grep "Failed password" /var/log/auth.log

Use awk to extract IP addresses:

grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -nr

This will show the count of failed attempts from each IP.

For journalctl, use built-in filters:

journalctl _COMM=sshd  # only sshd logs
journalctl PRIORITY=err  # only errors
journalctl --boot -1  # logs from the previous boot

Step 4: Security Log Analysis

Security logs are critically important. Key files include:

• /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS) for authentication.
• /var/log/faillog for failed login attempts (view via faillog).
• Firewall logs (e.g., ufw in /var/log/ufw.log).

Regularly check for suspicious activity. Automate using scripts or tools like fail2ban, which analyze logs and block IPs automatically.

Step 5: Automation and Monitoring

For continuous monitoring, use:

• logwatch: daily email reports. Install with sudo apt install logwatch (Debian/Ubuntu) or sudo yum install logwatch (RHEL). Configure in /etc/logwatch/.
• Logrotate: automatic log rotation, enabled by default. Check /etc/logrotate.conf/.
• ELK stack (Elasticsearch, Logstash, Kibana) or Grafana Loki for centralized log collection and visualization in larger environments.

Example simple script to alert on critical errors:

#!/bin/bash
if journalctl -p err --since "1 hour ago" | grep -q ".*"; then
    echo "Errors detected in logs within the last hour" | mail -s "System Errors" admin@example.com
fi

Add to cron for regular execution.

Result Verification

After completing this guide, you should be able to:

• Locate and open primary system logs.
• Use grep, awk, and journalctl for filtering.
• Diagnose common issues (e.g., service errors, failed logins).
• Set up basic log monitoring.

Test yourself: try to find all kernel errors from the last day or determine which service generates the most logs.

Potential Issues

1. Access errors: Some logs require root privileges. Use sudo or add the user to the adm (Debian/Ubuntu) or wheel (RHEL) group.
2. Large log volume: The journal can consume significant space. Configure limits in /etc/systemd/journald.conf (parameters SystemMaxUse, MaxRetentionSec). For file-based logs, configure logrotate.
3. Incorrect filters: Ensure you use correct service names or paths. For journalctl, check available fields via journalctl -F.
4. Missing logs: Some services may not write to standard logs. Check their configuration (e.g., in /etc/rsyslog.conf or for systemd services via StandardOutput).

If logs do not appear, restart the service or systemd-journald.