Log Analysis in Linux: From Basic Commands to Advanced Tools

Introduction / Why This Matters

Logs (log files) are the primary source of information about what's happening inside a Linux system. They contain records from the kernel, system services, installed applications, and security events. The ability to quickly find and analyze logs allows you to:

• Diagnose failures: Find out why a service didn't start or why an application crashed.
• Track security: Detect suspicious login attempts (in auth.log).
• Monitor performance: Identify disk errors, network problems.
• Audit actions: See what a user or system did at a specific moment.

This guide will take you from simply viewing logs to effectively searching with complex criteria.

Prerequisites / Preparation

Before you begin, ensure that:

1. You have access to a Linux terminal (locally or via SSH).
2. You possess sudo privileges (to view all systemd logs and files in /var/log/). Without sudo, you will only see your own logs.
3. Your system uses systemd (the vast majority of modern distributions: Ubuntu, Debian, Fedora, CentOS/RHEL 8+). You can check with the command pidof systemd.
4. You have basic familiarity with the command line and commands like cat, less, grep.

Step 1: Viewing the System Journal with journalctl

journalctl is the primary tool for working with systemd's Centralized Logging. It reads the binary journal that aggregates logs from all services and the kernel.

Basic commands:

# View the entire journal (starting with oldest entries)
journalctl

# View with paged output (recommended)
journalctl | less

# Follow logs in real-time (like tail -f)
journalctl -f

# View logs for a specific service (unit)
journalctl -u ssh.service

# View logs from the last 2 hours
journalctl --since 2h

# View logs from today
journalctl --since today

# View logs from a specific time range
journalctl --since "2026-02-15 09:00:00" --until "2026-02-15 10:00:00"

The --since and --until flags accept many formats: YYYY-MM-DD HH:MM:SS, yesterday, 1 hour ago.

Step 2: Filtering by Severity (Priority)

Logs have facilities and priorities. Often, you need to search for errors.

# Show only messages with 'err' priority and higher (crit, alert, emerg)
journalctl -p err

# The same result, but with an explicit range
journalctl -p err..crit

# Show only warnings
journalctl -p warning

# Full list of priorities: emerg, alert, crit, err, warning, notice, info, debug.

Step 3: Searching by Keywords and Context

The journalctl command supports built-in filters, but often it's more convenient to use grep.

# Search for mentions of "failed" (case-sensitive)
journalctl | grep failed

# Case-insensitive search
journalctl | grep -i timeout

# Search for lines containing "error" OR "Exception"
journalctl | grep -E "error|Exception"

# Show 5 lines before and after a match (context)
journalctl | grep -C 5 "segmentation fault"

# Search in a specific log file (not in journald)
grep -i "authentication failure" /var/log/auth.log

Step 4: Working with Classic Log Files in /var/log/

Not everything goes into the systemd journal. Many applications write to their own text files in /var/log/.

Key files:

• /var/log/syslog or /var/log/messages — general system messages.
• /var/log/auth.log or /var/log/secure — authentication, sudo, ssh.
• /var/log/kern.log — kernel logs.
• /var/log/dmesg — output from dmesg at boot time.
• /var/log/apt/history.log — package installation history (Debian/Ubuntu).

Commands for working with them:

# View the last 50 lines of syslog
tail -n 50 /var/log/syslog

# Monitor a file continuously
tail -f /var/log/auth.log


# Search in an old archived file (e.g., syslog.1)
zcat /var/log/syslog.1.gz | grep "service started"

# Search across multiple files
grep "connection refused" /var/log/nginx/error.log /var/log/syslog

Step 5: Analyzing Logs for Specific Services and Applications

SSH connections:

journalctl -u ssh.service
# or
grep "sshd" /var/log/auth.log | tail -50

Nginx/Apache:

# Logs are typically in /var/log/nginx/ or /var/log/apache2/
tail -f /var/log/nginx/error.log
grep "404" /var/log/nginx/access.log | wc -l  # count of 404 errors

Cron (scheduler):

# Cron logs are usually in syslog or a separate file
grep CRON /var/log/syslog

Failed services (systemd):

# Check status and recent logs for a service
systemctl status <service_name>
journalctl -u <service_name> --since "5 min ago" -p err

Critical kernel errors (dmesg):

dmesg | tail -50
dmesg | grep -i "memory\|error\|fail"

Step 6: Advanced Analysis: Statistics and Aggregation

Often you need to understand not just what happened, but which events are most frequent.

# Top 10 log sources (by PID/name) in journalctl over the last hour
journalctl --since 1h --no-pager | awk '{print $5}' | sort | uniq -c | sort -nr | head -10

# Count of errors by type (example for syslog)
grep -o -E "error|fail|exception" /var/log/syslog | sort | uniq -c | sort -nr

# Find all SSH brute-force attempts (many failed logins)
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -nr

Step 7: Saving and Exporting Logs for Deep Analysis

If you need to send logs to a developer or analyze them in another tool:

# Export the last 24 hours of systemd journal to a text file
journalctl --since yesterday > /tmp/journal_$(date +%F).txt

# Export with a size limit (e.g., 50 MB)
journalctl --since "1 day ago" --output=short-monotonic > /tmp/last_day.log

# Create a compressed archive of logs from /var/log/ (requires sudo)
sudo tar -czf /tmp/logs_archive_$(date +%F).tar.gz /var/log/

Checking Your Results

You have successfully mastered log analysis if you can:

1. Quickly find the cause of a service crash: journalctl -u <service> -p err --since "10 min ago" revealed the error.
2. Track suspicious activity: grep "Failed password" /var/log/auth.log identified the attacker's IP.
3. See what loaded the system: journalctl -b (logs from the current boot) and searching for kworker or oom-killer.
4. Generate a report: Exported logs for a specific period to a file.

Common Issues & Solutions

Issue: journalctl outputs "No journal files were found." Solution: The systemd-journald service is not running or journaling is disabled. Check systemctl status systemd-journald. Logs might be written only to /var/log/.

Issue: Permission denied when reading /var/log/ or journalctl. Solution: Run commands with sudo or as root. For journalctl, you can add your user to the systemd-journal group (sudo usermod -aG systemd-journal $USER), but this grants access to all logs.

Issue: Logs are rotated (renamed to .gz), and grep doesn't find old entries. Solution: Use zgrep for compressed files or expand them with zcat/gunzip -c.

Issue: There are too many logs to make sense of. Solution: 1. Tighten your filters (-p err, specific service, specific time). 2. Use --no-pager to output to stdout and redirect to a file for analysis in an editor. 3. Look for unique PID or UID and analyze logs only from them. 4. Configure more granular logging in the application's configuration (e.g., loglevel = WARN instead of DEBUG).