Checking System Logs in Linux: A Complete Guide to journalctl and syslog

Introduction / Why This Matters

System logs in Linux are the primary source of information about the operating system, services, and applications. When errors occur (a service failure, network issues, hardware malfunctions), the first place to look is the logs. This guide will help you master the essential tools for viewing, filtering, and analyzing logs, significantly speeding up problem diagnosis.

What you'll gain:

• The ability to quickly find systemd logs using journalctl.
• Proficiency in working with text logs in /var/log/.
• An understanding of the differences between the systemd journal and classic syslog.
• The capability to filter logs by time, service, and severity level.

Prerequisites / Preparation

Before you begin, ensure:

1. You have access to a Linux terminal (locally or via SSH).
2. Reading system logs typically requires sudo privileges (especially for journalctl and files in /var/log/).
3. Your system uses systemd (check with systemctl). If systemd is not present, work only with text logs in /var/log/.
4. Basic command-line skills: navigation, using grep, less, tail.

Step 1: Basics of Working with journalctl (systemd Journal)

Most modern distributions (Ubuntu, Fedora, CentOS 8+) use systemd-journald to collect logs. This journal is stored in a binary format and is accessible via the journalctl utility.

View all logs

sudo journalctl

This command outputs all journal entries since boot. Use arrow keys to scroll, / to search within less, q to exit.

View recent entries

sudo journalctl -n 50  # last 50 lines
sudo journalctl -f     # follow the log in real-time (like tail -f)

Filter by time

sudo journalctl --since "2026-02-16 09:00:00" --until "2026-02-16 10:00:00"
sudo journalctl --since 1h   # last hour
sudo journalctl --since today

Filter by service

sudo journalctl -u sshd.service   # SSH service logs
sudo journalctl -u nginx.service

Filter by severity level (priority)

Levels: emerg, alert, crit, err, warning, notice, info, debug.

sudo journalctl -p err   # errors only
sudo journalctl -p warning..err  # from warning to err inclusive

View logs for the current boot

sudo journalctl -b       # current boot
sudo journalctl -b -1    # previous boot

Search by text

sudo journalctl | grep -i "failed"  # search for "failed" (case-insensitive)
sudo journalctl | grep -i "error\|fail"  # multiple patterns

Step 2: Working with Classic Text Logs (/var/log)

If your system doesn't use systemd or you need logs for specific applications, check the files in /var/log/.

Main log files

• /var/log/syslog (Debian/Ubuntu) — general system log.
• /var/log/messages (RHEL/CentOS/Fedora) — similar to syslog.
• /var/log/kern.log — kernel logs.
• /var/log/auth.log (Debian) / /var/log/secure (RHEL) — authentication, SSH, sudo.
• /var/log/dmesg — output of dmesg from the last boot.

Viewing and monitoring

sudo tail -f /var/log/syslog   # monitor updates
sudo less /var/log/auth.log    # view with navigation
sudo grep "sshd" /var/log/auth.log  # find mentions of sshd

Log rotation

Files in /var/log/ are often compressed and rotated (e.g., syslog.1, syslog.2.gz). To view compressed files:

zcat /var/log/syslog.2.gz | less

Step 3: Using dmesg for Kernel Logs

The dmesg command shows the Linux kernel's ring buffer. Useful for diagnosing hardware problems, driver errors, and boot issues.

View kernel buffer

sudo dmesg
sudo dmesg | less  # with pagination

Filter by subsystem

sudo dmesg | grep -i usb      # USB devices
sudo dmesg | grep -i eth      # network interfaces
sudo dmesg | grep -i error    # kernel errors

Real-time monitoring

sudo dmesg -w  # watch for new kernel messages

Save output to a file

sudo dmesg > dmesg_output.txt

Step 4: Application and Service-Specific Logs

Many services write logs to their own files in /var/log/. Usually, this is a directory named after the service.

Examples:

• Nginx/Apache: /var/log/nginx/access.log, /var/log/nginx/error.log

  sudo tail -f /var/log/nginx/error.log
  

• MySQL/MariaDB: /var/log/mysql/error.log

  sudo cat /var/log/mysql/error.log | grep -i "error"
  

• Docker: /var/log/docker.log or docker logs <container_id>

  docker logs --tail 100 container_name
  

Finding logs for an unknown service

ls -la /var/log/  # view structure
sudo find /var/log -name "*<service_name>*" -type f  # find files by pattern

Step 5: Advanced Filtering and Search Techniques

Combine tools for precise searching.

Search by date in text logs

sudo grep "Feb 16" /var/log/syslog  # search by date (format depends on locale)

Using journalctl with grep

sudo journalctl | grep -i "failed to start"  # find a specific phrase

Export systemd journal to a text file

sudo journalctl > full_journal.txt

View logs with color highlighting

Install ccze or use:

sudo journalctl --no-pager | ccze -A

Step 6: Configuring Log Rotation and Cleanup

Clean up old systemd journal

# Keep only the last 2 days of entries
sudo journalctl --vacuum-time=2d

# Limit journal size to 500 MB
sudo journalctl --vacuum-size=500M

Configuring journald (config file /etc/systemd/journald.conf)

[Journal]
SystemMaxUse=500M    # maximum journal size
MaxRetentionSec=1week  # retain logs for no longer than a week

After changes, restart: sudo systemctl restart systemd-journald.

Configuring logrotate for text logs

Configurations are in /etc/logrotate.d/. Example for nginx:

/var/log/nginx/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 640 www-data adm
    sharedscripts
    postrotate
        [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

Verification

After completing the steps, you should be able to:

1. View the latest systemd journal entries via journalctl -xe.
2. Find logs for a specific service (e.g., journalctl -u sshd or /var/log/auth.log).
3. Filter logs by time or error level.
4. Clean up old entries to free up space if needed.

Example check: Run sudo journalctl -p err --since "1 hour ago". If the command returns entries (or empty output if no errors exist) — the tool is working.

Common Issues

❌ "Failed to connect to bus: No such file or directory" with journalctl

Cause: systemd-journald is not running or you are not on a system with systemd (e.g., in a container). Solution: Check systemctl status systemd-journald. In containers, use text logs in /var/log/.

❌ "Permission denied" when reading logs

Cause: Insufficient permissions. Solution: Add sudo or add your user to the adm group (Debian/Ubuntu) or systemd-journal group (RHEL):
sudo usermod -aG adm $USER (requires re-login).

❌ /var/log/syslog is empty or missing

Cause: Possibly using systemd-journald without direct file output, or a syslog daemon is not configured. Solution: Check if systemd-journald is running. To save logs to a file, configure systemd-journald or install rsyslog.

❌ systemd journal is too large

Cause: By default, journald can consume a lot of space. Solution: Set limits in /etc/systemd/journald.conf (see Step 6) and run sudo journalctl --vacuum-size=200M.

❌ Can't find logs for a specific application

Cause: The application may write logs to its own directory (e.g., /opt/app/logs/) or to the systemd journal. Solution: Check the application's documentation. Use sudo journalctl | grep -i "app_name" or find files: sudo find / -name "*app*log*" 2>/dev/null.

Final Recommendations

• Priority: Start with journalctl -xe — it's the fastest way to see recent systemd errors.
• Filtering: Always narrow output by time (--since), service (-u), or level (-p). This saves time.
• Monitoring: For tracking events in real-time, use journalctl -f or tail -f /var/log/syslog.
• Documentation: Specific applications (PostgreSQL, Docker, Kubernetes) often have their own log directories — consult official documentation.
• Security: Don't leave old logs on disk indefinitely. Configure rotation, especially on production servers.

These skills will help you stop "guessing" when errors occur and quickly find the root cause. Good luck with your troubleshooting.