Mastering systemd-journald Log Analysis: The Complete journalctl Guide

Introduction / Why This Is Needed

Systemd-journald is the central log collection and storage daemon in modern systemd-based systems. It records messages from the kernel, systemd units, and applications into a binary journal, providing centralization, integrity, and structured data.

Analyzing these journals allows you to:

• Quickly diagnose service and application failures.
• Track event history with microsecond precision.
• Correlate logs from different sources (kernel, systemd, user applications).
• Save space through compression and rotation.

This guide will transform you from a beginner who only knows journalctl into a confident user capable of finding a needle in a haystack.

Requirements / Preparation

Before you begin, ensure that:

1. Your system uses systemd (check with: pidof systemd). Most distributions since 2015.
2. You have root privileges (or sudo) to access all logs. Without them, you will only see your own entries.
3. The journal is active (usually it is). Check with: systemctl status systemd-journald.
4. There is enough disk space for storing and rotating logs (by default, 10% of the /var partition).

💡 Tip: Configure the journal size via /etc/systemd/journald.conf (parameters SystemMaxUse, SystemMaxFileSize) if space is limited.

Step 1: Basic Viewing and Navigation

The first thing to master is simple journal reading.

# Show all entries (from oldest to newest)
journalctl

# Show the last 50 entries (like tail)
journalctl -n 50

# Page through output (automatic for large output)
journalctl | less

# In less: space — forward, b — backward, / — search, q — quit

What you see: Each line contains a timestamp, host, source (e.g., sshd[1234]), and message. By default, output is sorted by time (oldest -> newest).

Step 2: Filtering by Time

This is the most common use case: "What happened an hour/day/week ago?". Use the --since and --until flags.

# Logs from the last hour
journalctl --since "1 hour ago"

# Logs for today (since 00:00)
journalctl --since today

# Logs for a specific date and time
journalctl --since "2024-01-15 09:00:00" --until "2024-01-15 10:00:00"

# Logs for yesterday
journalctl --since yesterday

# Combine: from February 1st to now
journalctl --since "2024-02-01"

Time formats: YYYY-MM-DD HH:MM:SS, relative (5 min ago, 2 hours ago), keywords (today, yesterday).

Step 3: Filtering by Services and Sources

Looking for why nginx crashed or what sudo did? Filter by unit files, PID, or priority.

# Logs for a specific systemd service
journalctl -u nginx.service

# Logs by process PID
journalctl _PID=1234

# Logs by priority (err, warning, info, debug)
journalctl -p err    # errors only
journalctl -p warning..err  # from warning to err (inclusive)

# Logs from a specific executable
journalctl /usr/bin/ssh

# Logs with a specific identifier (e.g., kernel)
journalctl -k  # same as journalctl _TRANSPORT=kernel

Combine filters:

# nginx errors from the last 2 hours
journalctl -u nginx.service -p err --since "2 hours ago"

# Kernel logs since the last reboot
journalctl -k -b -1

Step 4: Real-Time Monitoring

For "on-the-fly" debugging, use the follow mode.

# Follow new entries (like tail -f)
journalctl -f

# Follow new entries for a specific service
journalctl -u docker.service -f

# Follow with a priority filter (errors only)
journalctl -f -p err

Hotkeys: Ctrl+C — exit. Ctrl+Z — suspend (then fg to resume).

Step 5: Deep Analysis and Export

When you need to save logs or search for complex patterns.

# Export the entire journal to a text file (without paging)
journalctl --no-pager > full_journal.txt

# Export with filtering (only ssh errors from the past week)
journalctl -u sshd.service -p err --since "1 week ago" --no-pager > ssh_errors.txt

# Search text in logs (case-insensitive)
journalctl | grep -i "timeout"

# Show entry fields (for debugging filters)
journalctl -o verbose -n 1  # verbose output for one entry

# Show only messages (without metadata)
journalctl --output=cat -n 20

Output formats (-o):

• short (default) — concise.
• verbose — all fields, including _PID, _UID, _GID.
• json / json-pretty — for machine processing.
• export — binary format for systemd-journal-remote.

Step 6: Journal Size Management and Cleanup

Over time, the journal can occupy gigabytes. Manage rotation and clean up manually.

# Check current journal disk usage
journalctl --disk-usage

# Force journal rotation (create a new file)
journalctl --rotate

# Clean logs (delete all old entries)
journalctl --vacuum-time=1d    # keep only the last day
journalctl --vacuum-size=100M  # keep no more than 100 MB
journalctl --vacuum-files=3    # keep the 3 most recent files

# View journald configuration
cat /etc/systemd/journald.conf

⚠️ Important: Cleanup permanently deletes data. Make backups (journalctl --rotate && journalctl > backup.log) before vacuuming in production.

Check Your Result

You have successfully mastered analysis if you can:

1. Find the latest errors for a service within a specified period:

   journalctl -u your_service.service -p err --since "2 hours ago"
   

2. Export logs to send to a developer:

   journalctl --since "yesterday" --no-pager > logs_for_dev.txt
   

3. Track an event in real-time when restarting a service:

   journalctl -u your_service.service -f
   

4. Assess disk usage and clean up old entries if necessary.

Possible Issues

Additional Resource: Full documentation — man journalctl, man journald.conf.