Linux Log Analysis: A Step-by-Step Guide for Beginners

Introduction / Why This Matters

Logs (journals) in Linux are the primary source of information about what's happening in the system. They contain records from the kernel, system services, and applications. The ability to quickly find the necessary data in them is a key skill for diagnosing failures, analyzing security, and monitoring performance. After this guide, you won't just be "looking at logs"; you'll be able to purposefully search for the root causes of problems.

Requirements / Preparation

1. Access to a Linux terminal (Ubuntu, CentOS, Debian, or any other distribution with systemd).
2. sudo privileges (administrator) to read most system logs. Without them, you'll only see entries related to your user.
3. Basic familiarity with the command line (navigation, running commands).
4. An understanding of which service or process is causing the problem (e.g., a web server won't start, the network won't come up).

Step 1: Start with systemd journal (journalctl)

Modern Linux distributions use systemd-journald for centralized log collection. This is the most powerful and convenient tool.

Basic commands to get started:

# Show all system logs (newest first)
sudo journalctl

# Show logs only for the current system boot
sudo journalctl -b

# Filter by a specific service (e.g., ssh)
sudo journalctl -u ssh.service

# Logs at error level (err) and higher (crit, alert, emerg)
sudo journalctl -p err

# Logs for a specific time period
sudo journalctl --since "2024-01-15 09:00:00" --until "10:30"
sudo journalctl --since 1h  # last hour
sudo journalctl --since today

# Follow logs in real-time (like tail -f)
sudo journalctl -f

💡 Tip: Combining flags is your main power. Example: sudo journalctl -u nginx --since '10 minutes ago' -p warning will show nginx warnings and errors from the last 10 minutes.

Step 2: Work with classic text logs in /var/log

Although journalctl dominates, many applications (especially third-party ones or those in containers) write to separate text files in /var/log/.

Key directories and files:

• /var/log/syslog or /var/log/messages — general system events.
• /var/log/auth.log or /var/log/secure — authentication logs (ssh, sudo, login).
• /var/log/kern.log — kernel messages.
• /var/log/nginx/, /var/log/apache2/ — web server logs.
• /var/log/dmesg — kernel message buffer (similar to dmesg).

Basic commands for working with text logs:

# View the last lines of a file (like tail)
sudo tail -n 50 /var/log/syslog

# Search for a pattern (case-sensitive)
sudo grep "failed" /var/log/syslog

# Case-insensitive search
sudo grep -i "error" /var/log/auth.log

# Search across multiple files
sudo grep "connection" /var/log/nginx/*.log

# View in real-time
sudo tail -f /var/log/syslog

# Search with context (3 lines before and after match)
sudo grep -A 3 -B 3 "segfault" /var/log/kern.log

Step 3: Use powerful tools for complex analysis

For deep analysis, utilities that work with both journalctl output and text files are useful.

awk — for field extraction

Logs often have a structured format (date, time, level, message). awk lets you work with columns.

# From journalctl, output only time and message, filtered by PID
sudo journalctl -o short-iso | awk '$5 == "1234" {print $1, $2, $3}'


# From a text log, count occurrences of a specific error
sudo grep "No space left on device" /var/log/syslog | awk '{print $1}' | sort | uniq -c

less + search — for interactive exploration

For large files, use less (or sudo less /var/log/syslog). Inside less:

• / — start search (e.g., /timeout).
• n / N — next/previous match.
• & — filter output by pattern (all lines NOT containing the pattern are hidden).
• F — follow mode (like tail -f).

journalctl as a source for other utilities

journalctl output can be piped to grep, awk, etc.

# Find all entries containing 'firewall' and save them
sudo journalctl | grep -i firewall > firewall_issues.log

# Sort logs by PID and view unique processes
sudo journalctl -o json | jq -r '_processID' | sort | uniq

(Note: the -o json and jq usage may require installing the jq package)

Step 4: Save and export important data

For long-term analysis or sending to support, you need to save logs.

# Export logs for a specific service from the last 2 hours to a file (journalctl format)
sudo journalctl -u sshd --since 2h > sshd_last_2h.log

# Export to a standardized text format (with timestamps)
sudo journalctl -u nginx --since yesterday --no-pager > nginx_yesterday.txt

# Create a compressed log archive for sending
sudo journalctl --since 2024-01-01 --until 2024-01-07 | gzip > logs_jan1-7.txt.gz

Step 5: Configure logging (prevention)

To avoid situations where logs fill up the disk, configure rotation (deletion of old logs) and verbosity level.

For systemd-journald:

Edit /etc/systemd/journald.conf:

[Journal]
# Limit total journal size
SystemMaxUse=500M
# Limit size of a single journal file
SystemMaxFileSize=50M
# Keep logs for no longer than 2 weeks
MaxRetentionSec=2week
# Verbosity level (info, debug, warning...). For production, leave as info or warning.
# MaxLevelStore=info

Apply changes: sudo systemctl restart systemd-journald.

For classic logs (logrotate):

Logs in /var/log are usually managed by the logrotate daemon. Its configuration is in /etc/logrotate.conf and /etc/logrotate.d/. Default settings are usually adequate.

Verification of Results

You have successfully learned log analysis if you can:

1. Find the latest errors for a specific service (e.g., sudo journalctl -u docker --since 1h -p err).
2. Filter logs by time and severity level.
3. Save a relevant log snippet to a file for further use.
4. Understand where to get logs for a new application (usually either from journalctl or a file in /var/log/<appname>/).

Potential Issues

Issue: "Permission denied" when reading logs

Cause: You are running the command without sudo. Solution: Always use sudo to read system logs (/var/log/*, journalctl). If sudo is unavailable, you can add the user to the adm group (sudo usermod -aG adm $USER), but this requires a re-login.

Issue: journalctl is empty or has no entries for a period

Cause 1: The journal may have been vacuumed (journalctl --vacuum-size). Cause 2: Too aggressive MaxRetentionSec settings in journald.conf. Solution: Check your storage settings. For permanent archiving of critical logs, configure their duplication to a separate text file via the service configuration (e.g., StandardOutput=syslog or file in the systemd unit).

Issue: Too much data, can't find anything

Cause: The query is too broad or the logging level is too low (debug). Solution: Strengthen your filters. Add time bounds (--since, --until), a specific service (-u), and a level (-p). If necessary, temporarily increase the service's logging level to see details, but remember to revert to info/warning afterward.

Issue: Application logs are written to an unknown location

Cause: The application may be configured to log to its own file in /var/log/ or to standard output (stdout/stderr), which systemd intercepts. Solution: 1) Check the application's documentation. 2) Find the process (ps aux | grep <app>) and look at its startup arguments. 3) Check systemd units (systemctl status <app>.service); they may indicate StandardOutput=journal or file. 4) Use sudo lsof -p <PID> | grep log to see the process's open file descriptors.

Advanced Example: Finding the Cause of a Network Outage

Suppose the network drops periodically.

1. Define the time window: When was the last outage? (uptime, command history).
2. Check the kernel and network services:

   sudo journalctl -b -k | grep -i "network\|eth\|link"  # kernel logs for current boot
   sudo journalctl -u systemd-networkd --since "2 hours ago"
   sudo journalctl -u NetworkManager --since "2 hours ago"
   

3. Check authentication (if the issue is with SSH):

   sudo grep "Failed password" /var/log/auth.log | tail -20
   

4. Check system messages:

   sudo journalctl --since "2 hours ago" -p warning..err | grep -E "(network|connection|timeout|reset)"
   

5. Correlate with other events: The outage might coincide with another service starting (sudo journalctl --since "2 hours ago" | grep "Started"), a log rotation, or a cron job.

Conclusion (Not a separate heading)

Log analysis is a practical skill that develops with experience. Start with simple commands like journalctl -u <service> -f and grep on /var/log/. Over time, you'll learn to quickly combine filters, recognize typical error patterns, and perform root cause analysis. Remember: logs almost always tell the truth, if you know how to read them.