Complete Guide to Managing systemd-journald Logs

Introduction / Why This Is Needed

Systemd-journald is a centralized logging system in modern Linux distributions. It replaces the classic syslog and stores records in a binary format, enabling structured search and indexing. However, without management, logs can consume gigabytes of disk space, slow down the system, and complicate analysis. This guide will show you how to control log growth, quickly find relevant events, and configure automatic rotation.

Requirements / Preparation

Before starting, ensure that:

• You have sudo privileges or access to the root user.
• Your system uses systemd (check with: systemctl --version).
• The journal is stored in persistent mode (files in /var/log/journal/). If the directory doesn't exist, create it: sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald.
• The journalctl utility is installed (typically included in the systemd package).

Step 1: Viewing Logs and Basic Search

The systemd-journald journal stores all system events, including service output, kernel messages, and init-system activity. To view logs, use:

# Output all entries with the newest at the end (default)
journalctl

# View with pagination (arrow keys ↑/↓, q to exit)
journalctl | less

# Display in real-time (similar to tail -f)
journalctl -f

# Show only kernel messages
journalctl -k

Note: To exit less, press q. To interrupt -f, press Ctrl+C.

Step 2: Filtering by Time, Service, and Priority

Most useful filters:

# Logs from today
journalctl --since today

# For a specific period (format: "YYYY-MM-DD HH:MM:SS")
journalctl --since "2024-02-15 09:00:00" --until "2024-02-15 18:00:00"

# Only errors (priority err, crit, alert, emerg)
journalctl -p err

# Logs of a specific service (e.g., nginx)
journalctl -u nginx.service

# Combined filter: nginx errors from the last 2 hours
journalctl -u nginx.service -p err --since "2 hours ago"

Additional options:

• -n 50 — show the last 50 entries.
• --no-pager — output without pausing (useful for scripts).
• -o json or -o json-pretty — output in JSON for parsing.

Step 3: Clearing Old Logs

The journal can grow uncontrollably. Clearing is possible by time or size:

# Delete entries older than 7 days
sudo journalctl --vacuum-time=7d

# Keep no more than 200 MB of the newest entries
sudo journalctl --vacuum-size=200M

# Keep only logs from the last 24 hours
sudo journalctl --vacuum-time=1d

⚠️ Important: The --vacuum-* operation deletes all entries matching the criteria, even if they belong to different services. For selective clearing, use filters with --vacuum-time, but exercise caution.

If standard clearing doesn't free space (e.g., due to "locked" files), apply:

sudo systemctl stop systemd-journald
sudo rm -rf /var/log/journal/*  # full clear
sudo systemctl start systemd-journald

Step 4: Configuring Automatic Rotation

Edit the configuration file /etc/systemd/journald.conf:

# Enable persistent storage (if not already active)
Storage=persistent

# Maximum total journal size (K, M, G allowed)
SystemMaxUse=500M

# Maximum size of a single journal file
SystemMaxFileSize=50M

# Automatically delete entries older than N days
MaxRetentionSec=30d

# Compress old entries (saves space)
Compress=yes

# Split logs by service (useful for diagnostics)
SplitMode=uid

Applying changes:

sudo systemctl restart systemd-journald
sudo systemctl status systemd-journald  # check status

Checking current limits:

journalctl --disk-usage  # current size
journalctl --list-boots  # list of boots (stored by default)

Step 5: Advanced Search and Analysis

For deep analysis, use filter combinations and the grep utility:

# Find all mentions of "timeout" in cron service logs
journalctl -u cron.service | grep -i timeout


# Count errors by service from the last day
journalctl --since yesterday -p err --no-pager | \
  grep -oP '(?<=^.*?service: ).*?(?=\.)' | \
  sort | uniq -c | sort -nr

# Export logs to a text file (for support submission)
journalctl --since "1 hour ago" --no-pager > /tmp/journal_snapshot.txt

Tip: For complex queries, use the __UID (user ID) or _PID (process ID) fields:

journalctl _PID=1234  # logs of a specific process
journalctl _UID=1000  # logs for user with UID 1000

Verification

1. Journal size — run journalctl --disk-usage. The value should match SystemMaxUse (if configured) or be within reasonable limits (< 1 GB for a server).
2. Log availability — check for files in /var/log/journal/ (if Storage=persistent).
3. Rotation correctness — after limits are triggered, old files should archive (extension .journal~) or be deleted.
4. Search functionality — run a test query, e.g., journalctl -u systemd-journald --since "1 hour ago".

Possible Issues

Next Steps:

• Explore boot monitoring with journalctl -b -1 (logs from the previous boot).
• Set up log forwarding to external systems (e.g., Elasticsearch) via systemd-journal-remote.
• Use journalctl -o export to convert logs to standard formats (CSV, XML).