Error 'No space left on device': inode exhaustion in Linux

What the ENOSPC Error Means

The No space left on device error (code ENOSPC) in Linux usually indicates a lack of disk space. However, in the context of inodes, it means the inode count is exhausted on the file system, not physical disk space.

An inode (index node) is a structure storing file metadata (size, permissions, timestamps, data pointers). Each file or directory occupies one inode. When a file system is created, a fixed number of inodes is set, which cannot be changed without recreating the partition.

Symptoms:

• Inability to create new files or directories, even if df -h shows free space.
• Errors during package installation, program compilation, web server or database operation.
• Commands like touch, mkdir, apt install fail with No space left on device.

Causes

1. Large number of small files — application logs, cache, temporary files, backups, email.
2. Faulty application behavior — infinite file creation (e.g., due to a script error or attack).
3. File system with a small inode count — formatting with the -i parameter (e.g., mkfs.ext4 -i 2048) creates few inodes per megabyte.
4. Hidden system files — kernel files, core dumps, temporary files from system services.
5. Files with long names or special characters — in some file systems, these may consume additional inode? No, each file uses exactly one inode, but long names may occupy more data space, not inode.
6. File system corruption — rare case where inodes are marked as used but not linked to files.

Solutions

Method 1: Diagnosis and Inode Usage Analysis

First, determine which partition has exhausted inodes and find the "culprits".

1. Check inode usage per partition:

   df -i
   

   
   Example output:

   Filesystem      Inodes IUsed  IAvail Use% Mounted on
   /dev/sda1       6553600 6553599     1 100% /
   

   
   If IUsed is close to the total inode count (Inodes), the problem is on that partition.
2. For ext4/xfs, determine total inode count:

   sudo tune2fs -l /dev/sda1 | grep -i "inode count"
   

   
   Or for xfs:

   sudo xfs_info / | grep imaxpct
   

3. Find directories with the most files:

   sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20
   

   
   This command shows the 20 directories with the most files (without crossing to other partitions via -xdev).
4. Drill down into the problematic directory:

   sudo find /var/log -type f | wc -l   # example for /var/log
   

⚠️ Important: Only delete files after confirming they are unnecessary. It is recommended to back up important data first.

Method 2: Clean Temporary Files and Caches

Most often, inodes are exhausted due to accumulated temporary data.

1. Clean temporary directories:

   sudo rm -rf /tmp/*
   sudo rm -rf /var/tmp/*
   

   
   Ensure no open files exist in these directories (use lsof | grep /tmp).
2. Clean package manager cache:
   • For Debian/Ubuntu:

     sudo apt clean
     

   • For RHEL/CentOS/Fedora:

     sudo yum clean all
     

   • For Arch Linux:

     sudo pacman -Scc
     

3. Delete old logs: Check /var/log size and delete archived logs (e.g., *.gz, *.old) that are no longer needed:

   sudo find /var/log -name "*.gz" -type f -delete
   sudo find /var/log -name "*.old" -type f -delete
   

   
   Do not delete active logs without stopping the relevant services (e.g., journalctl --rotate for systemd).
4. Clear browser and application caches (if located in home directories):

   rm -rf ~/.cache/*
   

Method 3: Remove Unnecessary Small Files

If the issue is in a specific directory (e.g., /var/spool/mail or /srv), delete outdated data.

1. Find and delete files by pattern:

   # Delete backups
   sudo find / -name "*.bak" -type f -delete
   # Delete core dumps
   sudo find / -name "core.*" -type f -delete
   # Delete temporary application files (example for PHP)
   sudo find /var/www -name "*.tmp" -type f -delete
   

2. Archive and delete old data: If you need to keep files but free inodes, compress them into an archive (the archive occupies one inode):

   tar -czf /backup/old-logs.tar.gz /var/log/old-logs/
   sudo rm -rf /var/log/old-logs/
   

3. Check mail directory (if mail server exists):

   sudo find /var/mail -type f | wc -l
   

   
   Delete or archive old emails.

Method 4: Move Data to Another Partition

If the problematic partition (/) is small and contains many small files, move some data.

1. Create a mount point and mount a new partition:

   sudo mkdir /mnt/newdisk
   sudo mount /dev/sdb1 /mnt/newdisk
   

2. Move a directory with many files:

   sudo rsync -avx /var/log/ /mnt/newdisk/logs/
   

   
   Ensure data is copied, then delete the originals:

   sudo rm -rf /var/log/*
   

   
   And create a symlink:

   sudo ln -s /mnt/newdisk/logs /var/log
   

3. Add an entry to /etc/fstab for automatic mounting after reboot.

Method 5: Recreate the File System with More Inodes (Last Resort)

If the partition can be formatted (e.g., a separate data disk), recreate it with increased inode count.

1. Make a full data backup to another storage device.
2. Unmount the partition:

   sudo umount /dev/sdb1
   

3. Format with the -i parameter (more inodes per megabyte):

   sudo mkfs.ext4 -i 4096 /dev/sdb1   # 4096 inodes per 1MB (default ~16384 for 1KB-block)
   

   For xfs, inodes are allocated dynamically, so this issue occurs less frequently.
4. Restore data and update /etc/fstab.

💡 Tip: When formatting ext4, use -T largefile4 for directories with large files (reduces inode count) or -T small for many small files (increases).

Prevention

1. Regular monitoring:

   # Add to crontab (daily)
   0 2 * * * df -i | grep -E '/( |$)' | awk '$5+0 >= 80 {print "WARNING: " $0}'
   

2. Logging: Configure log rotation (logrotate) with file count limits and compression of old logs.
3. Limit application caching: For web servers (nginx, Apache), configure cache cleanup. For mail servers — limit email retention.
4. Proper formatting: When creating partitions for systems with many small files (e.g., mail, virtualization), use:

   sudo mkfs.ext4 -T largefile4 /dev/sdX   # for large files
   sudo mkfs.ext4 -T small /dev/sdX        # for small files
   

   
   Or explicitly specify -i (bytes-per-inode). Smaller -i value → more inodes.
5. Use xfs or btrfs for new systems where inodes are allocated dynamically (xfs) or flexibly (btrfs). Note that btrfs has its own specifics.
6. Set up monitoring via Zabbix/Prometheus: Collect the df -i metric and set alerts when inode usage exceeds 85%.
7. Regular cleanup: Add cron jobs to clean /tmp and package caches:

   0 3 * * * find /tmp -type f -mtime +7 -delete
   0 4 * * * apt clean -y
   

8. Avoid creating many files in a single directory: For high-load systems (e.g., web logs), use date-based or hash-based directory structures.