journalctl: Complete Guide to Working with systemd Logs

Introduction / Why You Need This

journalctl is the primary utility for viewing and analyzing logs collected by systemd-journald. It replaces multiple text log files (/var/log/syslog, /var/log/messages, dmesg), providing a single, structured, indexed source of information about all processes in the system.

With this guide, you will learn how to:

• Quickly find logs for a specific service or process.
• Filter entries by time, severity level (error/warning/info), and other fields.
• Monitor events in real-time.
• Save and export logs for further analysis.

This is an essential skill for any sysadmin, DevOps engineer, or developer debugging an application on Linux.

Requirements / Preparation

1. Operating System: Any modern Linux distribution using systemd (the vast majority).
2. Access Permissions: To view logs from all users and services, root privileges (or membership in the systemd-journal group) are typically required. Commands requiring elevated privileges in this guide are marked with sudo.
3. Understanding Log Structure: Each journal entry has numerous fields (_SYSTEMD_UNIT, _PID, _COMM, PRIORITY, __REALTIME_TIMESTAMP, etc.). We will use these for filtering.

Step 1: Basic Viewing and Navigation

A simple journalctl call without arguments will display the entire available journal, starting from the oldest entry. This can be a lot of data.

Best Practices to Start:

# Show the last 50 lines (like tail -n 50)
journalctl -n 50

# Show the last 50 lines and follow new entries (like tail -f)
journalctl -f -n 50

# Show logs from the beginning (default)
journalctl --no-pager

💡 Tip: Use the --no-pager flag if you want the output to go directly into a pipeline (pipe) or file, without paging through less.

Step 2: Filtering by Service (UNIT)

The most common scenario is to view logs for a specific daemon or service.

# Logs for the Nginx service (service name taken from the systemd unit file)
journalctl -u nginx.service

# Logs for the Docker service
journalctl -u docker.service -f

# Logs for multiple services simultaneously
journalctl -u ssh.service -u cron.service

⚠️ Important: The service name must be exact. You can see a list of all active units with systemctl list-units --type=service.

Step 3: Filtering by Severity Level (PRIORITY)

Each entry has a priority from 0 (emerg) to 7 (debug). For debugging, errors and warnings are most often needed.

# Only errors (err) and critical (crit, emerg)
journalctl -p err

# Only warnings and errors
journalctl -p warning

# Everything except debug messages (info and higher)
journalctl -p info

# Combination of levels: err, warn, notice
journalctl -p err -p warn -p notice

Priority Scale (from most to least severe): 0. emerg (system is unusable)

1. alert (action must be taken immediately)
2. crit (critical condition)
3. err (error condition)
4. warning (warning condition)
5. notice (normal but significant condition)
6. info (informational message)
7. debug (debug-level message)

Step 4: Filtering by Time

Specifying an exact time interval is key to fast searching.

# Logs from the last 10 minutes
journalctl --since "10 minutes ago"

# Logs for today (since 00:00)
journalctl --since "today"

# Logs for a specific date
journalctl --since "2026-02-15" --until "2026-02-16"

# Logs with second precision (format: "YYYY-MM-DD HH:MM:SS")
journalctl --since "2026-02-15 14:30:00" --until "2026-02-15 15:00:00"

# Logs from the last hour (combined with `-p err`)
journalctl --since "1 hour ago" -p err

💡 Tip: The time format is quite flexible. You can write "5 min ago", "yesterday", "2026-02-15".

Step 5: Combining Filters and Searching by Fields

The power of journalctl lies in combining filters and searching by structured fields (prefix _). This is much more precise than grep.

# Nginx service logs with error level from the last 30 minutes
journalctl -u nginx.service -p err --since "30 minutes ago"

# Logs for a specific process by PID
journalctl _PID=1234

# Logs for a specific executable
journalctl _COMM=/usr/bin/ssh

# Logs from a specific user (UID)
journalctl _UID=1000

# Logs related to a specific session
journalctl _SESSION=2

Popular Fields for Filtering:

• _SYSTEMD_UNIT — equivalent to -u.
• _PID — Process ID.
• _UID — User ID.
• _GID — Group ID.
• _COMM — Command (executable name).
• _EXE — Full path to the executable.
• SYSLOG_IDENTIFIER — Identifier, often matching the program name.

Searching by Text (if field filters don't help)

Use standard grep to search the free text of the MESSAGE field.

# Search for all mentions of 'Connection refused' (case-insensitive)
journalctl | grep -i "connection refused"

# Search with context (2 lines before and after)
journalctl | grep -C 2 "timeout"

# Search and count matches
journalctl | grep -c "Failed password"

Step 6: Output Management and Export

Changing Output Format

By default, many metadata fields are displayed. You can change the format.

# Short format: only time and message
journalctl -o short

# Verbose format (with all fields)
journalctl -o verbose

# JSON (convenient for scripts and parsing)
journalctl -u nginx.service -o json --since "1 hour ago" > nginx_logs.json

# JSON-pretty (readable JSON)
journalctl -o json-pretty

# Show only the message field (very concise)
journalctl -o cat

Limiting Output Size

# Show the last N entries (most common option)
journalctl -n 100

# Limit output by time (e.g., last 2 hours)
journalctl --since "2 hours ago"

Saving Logs to a File

# Save ssh service logs from the last day to a file
journalctl -u ssh.service --since "yesterday" > ssh_logs.txt

# Save to JSON for later analysis by a script
journalctl -p err --since "2026-02-01" -o json > errors_feb.json

Step 7: Managing the Journal Itself (Vacuuming, Rotation)

The systemd journal can take up significant space by default (up to 10% of disk). Management:

# Show current journal statistics (size, entry count)
journalctl --disk-usage

# Clean (delete) all old entries, keeping only the last N megabytes
sudo journalctl --vacuum-size=500M

# Clean entries older than a specified time
sudo journalctl --vacuum-time=3d  # older than 3 days
sudo journalctl --vacuum-time=1w  # older than 1 week

# Force journal rotation (create a new file)
sudo journalctl --rotate

# Combination: rotation + cleanup of entries older than 1 day
sudo journalctl --rotate && sudo journalctl --vacuum-time=1d

⚠️ Important: Cleanup (--vacuum) only works on files in /var/log/journal. If the journal is stored only in memory (/run/log/journal), it will be cleared upon reboot.

Verification

1. Basic Test: Run journalctl -n 5 -f. You should see the last 5 log lines and new entries as they appear (e.g., when starting a service).
2. Filtering: Run journalctl -u ssh.service -p info. The output should contain only info level and higher entries from the SSH service.
3. Export: Run journalctl --since "1 hour ago" -o json > test.json. The test.json file should contain a valid JSON array of entry objects.
4. Management: Run sudo journalctl --disk-usage to check the current size. After sudo journalctl --vacuum-size=100M, the size should decrease.

If all the listed commands execute without errors and the output matches expectations — you have successfully mastered the basics of working with journalctl.

Potential Problems

1. "Failed to open directory /var/log/journal: Permission denied"

Cause: The current user lacks permissions to read system logs. Solution: Use sudo before the command or add the user to the systemd-journal group: sudo usermod -aG systemd-journal $USER (requires re-login).

2. No logs for the needed service (journalctl -u ... is empty)

Cause A: The service did not use stdout/stderr for logging (uses a separate file). Solution: Check the service configuration (/etc/systemd/system/<name>.service), the StandardOutput option. Logs might be going to a separate file, not journald. Cause B: The service was not running during the period in question. Solution: Ensure the service is active (systemctl status <name>). Check the time interval (--since).

3. "No journal files were found."

Cause: systemd-journald is configured to store logs only in memory (/run/log/journal), and you rebooted the system. Solution: For persistent storage, create the directory: sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal. Logs will start being saved after restarting the systemd-journald service or rebooting.

4. Too much data, search is slow

Solution: Always narrow the time interval (--since) and priority level (-p). Use field-based filtering (_PID, _SYSTEMD_UNIT) instead of grep, as searching indexed fields is much faster.

5. Need to monitor logs, but journalctl -f doesn't show new entries

Cause: You started journalctl with a very old --since or without -f, and the output is "stuck" at the beginning. Solution: Press Ctrl+C and restart with -f and possibly -n 0 (don't show old entries): journalctl -u service.name -f -n 0.

Advanced Scenarios and Examples

Finding the Last Reboots and Their Causes

# Find all reboot entries (message from systemd)
journalctl -b -1 -u systemd-logind.service | grep -i "system reboot" -A5 -B5

-b -1 — logs from the previous boot. This helps find what happened before a system crash.

Analyzing Causes of Service Failures

# Find all failed service starts
journalctl -p err | grep -i "failed"

# Or more precisely, by the unit result field
journalctl _SYSTEMD_UNIT=nginx.service _SYSTEMD_INVOCATION_ID=$(systemctl show nginx.service -p InvocationID --value)

Comparing Logs Between Two Boots

# Compare the difference in logs between the current and previous boot
journalctl -b -0 > current_boot.log
journalctl -b -1 > previous_boot.log
diff -u previous_boot.log current_boot.log | less

Real-time Monitoring with Color Highlighting

journalctl doesn't support colors out of the box, but you can pipe it to grep --color:

journalctl -f | grep --color -E "error|failed|exception|WARNING"

Use in Scripts (without paging)

Always use --no-pager in scripts so the command doesn't wait for a key press.

#!/bin/bash
# Check if there were nginx errors in the last 15 minutes
if journalctl -u nginx.service --since "15 minutes ago" -p err --no-pager | grep -q .; then
    echo "Errors detected in nginx!"
    # Can send a notification or restart the service
    systemctl restart nginx
fi