journal-space-exhausted error in systemd: causes and fixes

What the `journal-space-exhausted` Error Means

The journal-space-exhausted (or No space left on device) error occurs in systemd-journald when the disk partition designated for storing logs becomes full. systemd-journald is a service that collects and stores system logs in a binary format. When space runs out, journald stops writing new entries and may output error messages to the console or to logs (if they can still be written). This can lead to the loss of important diagnostic information and, in some cases, to failures in other services that depend on logging.

The full error text in logs may look like this:

systemd-journald[123]: Failed to write entry (4 of 4) to /var/log/journal/.../system.journal: No space left on device

systemd-journald[123]: /var/log/journal/.../system.journal: No space left on device

Causes

Insufficient space on the disk partition where logs are stored. By default, journald writes to /var/log/journal (on persistent storage) and /run/log/journal (in RAM, if enabled). If the partition fills up, the error is inevitable.
Lack of log rotation. If no limits on size or retention time are configured, logs can grow indefinitely.
Very high logging level. The default logging level is INFO, but if set to DEBUG or TRACE, log volume increases dramatically.
Failures in services that generate massive amounts of logs (e.g., recurring errors in a daemon).
The /var or / partition is filled with other data, not just logs. This could be due to old packages, caches, temporary files, etc.

Solutions

Method 1: Clean Up Old Journals

The quickest way is to delete some old entries to free up space. systemd-journald provides the journalctl command with --vacuum-size and --vacuum-time options.

Steps:

Determine the current size of the journals:
```
journalctl --disk-usage
```
Example output: Archived and active journals take up 1.2G on disk.
Clean the journals, keeping, for example, only the last 3 days:
```
sudo journalctl --vacuum-time=3d
```
Or limit the total journal size to 100 MB:
```
sudo journalctl --vacuum-size=100M
```
The command will delete the oldest entries until the size is below the specified limit.
Check how much space was freed:
```
journalctl --disk-usage
```

⚠️ Important: Cleanup deletes data permanently. Ensure you don't need old logs for incident investigation.

Method 2: Configure Size Limits in the Configuration File

To prevent the issue from recurring, configure journald parameters in the configuration file.

Steps:

Open the configuration file:
```
sudo nano /etc/systemd/journald.conf
```
Or use any text editor.

Find and uncomment (remove the # at the start of the line) the following parameters, setting reasonable values:

[Journal]
# Limits the total size of archived journals on disk
SystemMaxUse=100M
# Limits the size of the journal in RAM (if Storage=volatile or Storage=auto and /run/log/journal exists)
RuntimeMaxUse=50M
# Maximum time to keep entries (e.g., 30 days)
MaxRetentionSec=30d
# Limits the size of an individual journal file
SystemMaxFileSize=10M

More on parameters: man journald.conf.

Save the file and restart the service:

sudo systemctl restart systemd-journald

Check the status:
```
sudo systemctl status systemd-journald
```
Ensure there are no errors.

Method 3: Disable Journaling (Not Recommended)

If systemd journaling is not needed (e.g., on embedded systems with extremely limited resources), it can be disabled. Do not do this on production servers without a compelling reason, as you will lose system logs necessary for diagnostics.

Steps:

Edit /etc/systemd/journald.conf:
```
[Journal]
Storage=none
```

Delete existing journals (if any):

sudo rm -rf /var/log/journal
sudo rm -rf /run/log/journal

Restart journald:

sudo systemctl restart systemd-journald

Verify the service runs without errors:
```
sudo systemctl status systemd-journald
```

Method 4: Move Journals to a Separate Partition

If the /var partition is constantly filling up, you can move the journals to a separate disk or partition. This is especially useful on servers with limited space on the system partition.

Steps:

Stop the journald service:
```
sudo systemctl stop systemd-journald
```
Create a mount point for the new partition, e.g., /mnt/journal. Mount the partition (ensure it's formatted with a filesystem that supports binary files, e.g., ext4). Or use a separate directory on another partition.
Copy existing journals to the new location:
```
sudo cp -r /var/log/journal/* /mnt/journal/
```
If journals are stored in /run/log/journal, they are temporary and don't need copying.
Remove old journals from /var/log/journal (or move them if you want a backup):
```
sudo rm -rf /var/log/journal/*
```
Create a symbolic link or edit the mount. It's recommended to mount the new partition on /var/log/journal. To do this, edit /etc/fstab:
```
/dev/sdXN /var/log/journal ext4 defaults 0 2
```
Replace /dev/sdXN with your partition.
Remount all partitions:
```
sudo mount -a
```
Start journald:
```
sudo systemctl start systemd-journald
```
Verify journals are written to the new location:
```
journalctl --disk-usage
```
And check the mount:
```
df -h /var/log/journal
```

Prevention

To avoid the journal-space-exhausted error recurring, follow these recommendations:

Configure journal size limits in /etc/systemd/journald.conf immediately after system installation. Use parameters like SystemMaxUse, RuntimeMaxUse, and MaxRetentionSec. Example configuration:
```
[Journal]
SystemMaxUse=100M
RuntimeMaxUse=50M
MaxRetentionSec=30d
```
This prevents unlimited journal growth.

Regularly check disk usage for partitions storing journals (usually /var). Add a cron job, e.g., daily:

#!/bin/bash
USAGE=$(df /var/log/journal | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$USAGE" -gt 90 ]; then
    logger "systemd journals occupy more than 90% of space on /var/log/journal"
    # You can add automatic cleanup
    journalctl --vacuum-size=50M
fi

Use a separate partition for /var/log/journal if the system partition has limited space. This isolates journals from other data.
Avoid logging levels DEBUG or TRACE on production systems unless required. Set the default level to INFO via systemctl or in service configurations.
Monitor system logs for frequent error messages that might indicate a problem before disk space runs out. Use tools like journalctl -f or centralized logging.
Regularly clean temporary files and package caches (apt clean, yum clean all) to free up space on the /var partition.

For more information on configuring journald, refer to the manual:

man journald.conf

F.A.Q.

Why does the journal-space-exhausted error occur?

How to check how much space systemd journals occupy?

Can systemd journaling be disabled?

How to configure automatic cleanup of old logs?

Hints

Check which partition is full

Clean old journals

Configure size limits in configuration

Restart systemd-journald service

Set up regular cleanup (optional)