Symptoms

Running the logrotate error generates an error indicating that there is no space: logrotate -f /etc/logrotate.conf error: error creating output file /var/log/firewall: No space left on device error: error creating output file /var/log/mail.err: No space left on device error: error creating output file /var/log/wtmp-nnnnnnnn: No space left on device The df command shows available space: df -h Filesystem Size Used Avail Use% Mounted on /dev/sda5 7.9G 4.2G 3.4G 56% / devtmpfs 5.8G 248K 5.8G 1% /dev tmpfs 5.8G 0 5.8G 0% /dev/shm /dev/sda1 114M 53M 56M 49% /boot /dev/sda3 256G 19G 237G 8% /data01 /dev/sda7 1.5G 524M 899M 37% /var Attempting to create a file on the /var file system also results in a space error: touch /var/log/test touch: cannot touch '/var/log/test': No space left on device

Cause

The file system has run out of inodes. Rerunning the df command with the "-i" switch can confirm this: df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda5 524288 106337 417951 21% / devtmpfs 0 0 0 - /dev tmpfs 1513794 1 1513793 1% /dev/shm /dev/sda1 30120 49 30071 1% /boot /dev/sda3 267691072 5438 267685634 1% /data01 /dev/sda7 97536 97536 0 100% /var This is generally caused by either a fragmented file system, lots of very small files, temp files, or a large mail queue.

Resolution

Note: These commands must be run as root. The du command can be run to see what is taking up the space: du -x -h /var --max-depth=1 Example output showing that the /var/spool directory takes up most of the space within /var: 8.0K /var/state 4.0K /var/X11R6 4.0K /var/crash 45M /var/lib 4.0K /var/named 4.0K /var/opt 12K /var/yp 380M /var/spool 132K /var/run 4.0M /var/cache 25M /var/adm 16K /var/lost+found 8.0K /var/tmp 28K /var/lock 37M /var/log 56K /var/games 489M /var The du command can then be run again further down the directory tree: du -x -h /var/spool --max-depth=1 Further investigation is required depending on what is taking up the space. If assistance is required, open a Service Request (SR) with the Dell Technologies Avamar Support team. In this example: the issue was many very small files. The largest directory was /var/spool/postfix/maildrop, and contained thousands of small files: ls -l /var/spool/postfix/maildrop| wc -l 96559 Each of the files contained the following: admin : /etc/sudoers is mode 0777, should be 0440 The permissions and ownership for the sudoers file were updated: chmod 0440 /etc sudoers chown root:root /etc/sudoers The files in the /var/spool/postfix/maildrop directory were removed, and the issue was resolved.

Support Cases