System Log Analysis in Linux with journalctl

Introduction / Why This Is Needed

Journalctl is a utility for viewing and managing systemd logs, which replaces classic system logs (syslog). It collects messages from the kernel, services, and user applications into a single binary journal, providing centralized and structured access to Linux logs.

With this guide, you will be able to:

• Quickly identify causes of service failures.
• Track events in real time.
• Filter huge volumes of logs by time, service, or priority level.
• Export data for analysis in third-party tools.

This skill is critically important for system administrators, developers, and anyone working with Linux servers.

Requirements / Preparation

Before you begin, ensure that:

1. Your system uses systemd (check with pidof systemd). Almost all modern distributions (Ubuntu, CentOS, Debian, Fedora) use systemd by default.
2. You have access to a terminal with standard user privileges. To view logs for some services or clear the journal, sudo privileges may be required.
3. The systemd package is installed (usually pre-installed).

Basic Journal Viewing

Step 1: Opening the Journal

Simply type in the terminal:

journalctl

By default, you will see all entries, starting from the oldest, with pagination (Page Up/Down keys for navigation). Press q to exit.

💡 Tip: If pagination gets in the way, add --no-pager: journalctl --no-pager. This prints the entire journal at once.

Step 2: Viewing Recent Entries

To see the most recent events (similar to tail -f without following), use:

journalctl -n 50

The -n (or --lines) flag displays the specified number of most recent lines. Without a number, it shows 10.

Filtering Logs

Step 3: Filtering by Time

Very often you need to find logs for a specific period. Journalctl understands flexible date formats:

# Logs for today
journalctl --since today

# Logs for the last 2 hours
journalctl --since "2 hours ago"

# Logs between two points in time
journalctl --since "2024-02-15 09:00:00" --until "2024-02-15 10:00:00"

# Logs for a specific date
journalctl --since yesterday --until today

Step 4: Filtering by Service (Unit)

If an issue is related to a specific service (e.g., nginx or sshd), filter by its name:

journalctl -u nginx.service
journalctl -u sshd --since today

You can combine this with time filters. You can find the unit name via systemctl list-units.

Step 5: Filtering by Priority Level

The systemd journal marks messages with priority levels: debug, info, notice, warning, error, crit, alert, emerg. By default, it shows info and higher.

# Only errors and critical messages
journalctl -p err..emerg

# Only warnings and higher
journalctl -p warning

# Level from debug to notice
journalctl -p debug..notice

Short form: journalctl -p 3 (where the number is the priority code, 0=emerg, 1=alert, 2=crit, 3=err, etc.).

Step 6: Text Search

To search for a specific word or phrase in messages:

journalctl | grep "Permission denied"

Or use journalctl's built-in search (case-sensitive):

journalctl -g "connection error"

The -g (or --grep) flag searches the MESSAGE field. For case-insensitive search, add -i: journalctl -gi error.

Advanced Features

Step 7: Real-time Monitoring

To see new entries as they arrive (like tail -f), use:

journalctl -f

Combining this with filters is very useful, e.g., journalctl -u docker -f — monitoring only Docker logs.

Step 8: Output Management and Formatting

You can customize the output format for scripts or a more readable view:

# Show only time and message (convenient for grep)
journalctl -o short-iso

# Show all entry fields in JSON (for parsing)
journalctl -o json

# Show only messages without metadata
journalctl -o cat

Step 9: Exporting and Saving Logs

To archive logs or send them to support, save the journal to a file:

# Entire journal
journalctl > full_log.txt

# Part of the journal (e.g., last day's logs for the cron service)
journalctl -u cron --since "1 day ago" > cron_log.txt

# Compressed (gzip)
journalctl --since "1 week ago" | gzip > weekly_logs.gz

Verifying the Result

After running the commands, you should see the filtered journal entries on screen. Ensure that:

1. Time ranges match expectations (check the first and last lines).
2. The service filter (-u) returns logs for that specific service (look for its name in the output).
3. When using -f, new lines appear automatically.

If the output is empty, possible reasons are:

• No entries exist for the specified period.
• The filter is too strict (e.g., incorrect unit name).
• The priority level does not include the needed messages.

Potential Issues

Issue 1: "Failed to open journal: Permission denied"

Cause: Your user does not have permission to read the entire journal. Some logs (especially from the kernel or system services) are only accessible to root. Solution: Use sudo before the command: sudo journalctl -u sshd. Or configure permissions via sudo visudo (be careful!).

Issue 2: Journal is too large, command runs slowly

Cause: By default, journalctl reads the entire journal, which can be gigabytes in size. Solution: Always apply filters (--since, -u, -p) to limit the volume. You can also specify the number of lines -n. To clean up old journal data: sudo journalctl --vacuum-time=7d (removes entries older than 7 days).

Issue 3: Can't see logs from my user application

Cause: The application does not write to systemd-journald (uses its own file or standard output). Solution: Ensure the application is configured to log to stdout/stderr, which systemd intercepts. For daemons, check the unit settings (StandardOutput=journal). Or look for logs in traditional locations (/var/log/).

Issue 4: journalctl -f does not show new entries

Cause: You ran -f without a service filter, and new entries have a priority lower than info (e.g., debug), which is suppressed by default. Solution: Specify the priority explicitly: journalctl -f -p debug or set the priority for the whole system via /etc/systemd/journald.conf.