OPERATIONAL DEFECT DATABASE
...
...
This article provides information to troubleshoot vCenter Server Appliance (VCSA) high virtual disk space. Errors appear as messages in vCenter Server similar to: vSphere UI Health Alarm", "Log disk exhaustion on vcenter name" Database Health Alarm", "Core and Inventory Disk Exhaustion on vcenter name" By default, the error messages begin when the disk reaches 80% full Please note that in version 6.7, the /storage/archive disk can be up to 100% full by design Earlier 6.7 versions may still present an error in the vCenter GUI. These can be safely ignored. By default, when the some disks reach 95% full, the main vCenter service, vmware-vpxd, is turned off automatically This is to protect the database from corruption. Thus the problem may cause vCenter Server to be inaccessible with 503 Service Unavailable errors. There are many possible causes for specific disk partitions running out of space. Some causes include: Partitions are not sized correctly Large files in the partitions Tables in the database consuming space Services are failing to clean up files
Identify the full partition Through the vCenter GUI: Log In to the vCenter Server Appliance Management Interface (VAMI) Select Monitor > Disks Note which disks are showing 78% or higher usage Please keep in mind that in 6.7 and higher versions, the "archive" disk being 80% or more full can be safely ignored Through the vCenter command line: Log into the vCenter Server Appliance as root via SSH or the vCenter virtual machine Console. Note: If you're unable to log in to vCenter Appliance and suspect this is due to the /dev/sda3 "root partition" being full, refer to the following article steps 1 to 7 to gain Single user mode shell access: How to reset the lost or forgotten root password in vCenter Server Appliance 6.5 and above. Type shell to switch to the BASH Shell. (for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access Find out which partitions are full enough to trigger warnings. 78% or more full by running the command: df -hHere's an alternative command which only shows you the partitions which are getting full:df -h |awk '0+$5 >= 78 {print}'Note: We're looking for "78" because In many cases if the partition is at 78-79%, it may be exceeding the 80% needed to trigger warnings, then recovering back to the lower percentage. Similarly if it is at 93%, it may be exceeding the 95% needed to trigger vmware-vpxd service to shut down, then recovering back to the lower percentage. Select the article links for partitions that are displaying as full from this list FilesystemMounted onResolution/dev/sda3"/" /dev/sda3 - root partition 100% full due to Audit.log files not being rotated in vCenter Server Appliance Root Partition full due to dnsmasq.log files in vCenter Server Appliance or Platform Service Controller /dev/mapper/log_vg-log/storage/logTroubleshooting vCenter Appliance /storage/log directory is 80% or more full/dev/mapper/core_vg-core/storage/core/storage/core partition full due to core.netlogond.xxxxx files in vCenter Server Appliance/dev/mapper/db_vg-db/storage/db /storage/db full or nearly full in VCSA/dev/mapper/seat_vg-seat/storage/seat/storage/seat disk 100% full on vCenter Server Appliance 6.x/7.x/dev/mapper/archive_vg-archive/storage/archiveNote: This partition being full is normal and expected in vCenter versions 6.7 and above "File system /storage/archive is low on storage space" warning in vCenter Server Appliance Further troubleshooting: If the previous links did not help with the problem, Identify large files in the partition High disk space usage in vCenter Server Appliance starts with identifying which partition is impacted with high disk space usage. Connect to vCenter Server Appliance through SSH or through the vCenter VM console Type shell to switch to the BASH Shell. (for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access Use df -h command to get the disk space usage stats of the vCenter Server Appliance Note: If you are not able to access vcsa bash shell, refer Unable to log in to the vCenter Server Appliance shell using root account even after password reset Once the partition with low disk space is identified, below steps can be followed to identify the large files created in the respective partitions, except for the partitions store database files. For partitions and mount points of vCenter Server Appliance, refer: List of VMDKs for a vCenter Server Appliance 6.0 - Size Mount point and Purpose List of VMDKs for a vCenter Server Appliance 6.5 - Size Mount point and Purpose List of VMDKs/Partitions for a vCenter Server Appliance 6.7 - Size Mount point and Purpose List of VMDKs/Partitions for a vCenter Server Appliance 7.0 - Size Mount point and Purpose List of VMDKs/Partitions for a vCenter Server 8.0 - Size Mount point and Purpose Use the below command within the affected partition to identify top 20 files with high disk usage: du -a <Path> | sort -n -r | head -n 20 For example: du -a /storage/log/ | sort -n -r | head -n 20 Change directories to any listed here which are suspiciously large, and look for large files using the command: ls -lhaWe can also do a more powerful search using the more complex command:find ./ -type f -size +100M |less If large files are not the issue it may be that numerous smaller files are causing a problem. The following command will show you the subdirectories containing the most files:find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 > 100' | sort -k2,2nr |less Please note, this command can take a few minutes to run!Before deciding to remove any files to clear space, ensure you have a reliable backup for your vCenter. Note: The *.tgz files are archived log bundle files and can be removed. Resizing the vCenter virtual disks If all else fails, it can be that normal activity on the vCenter has become greater than the existing disk size supports. You may therefore decide to resize the virtual disk: Notice that in the VAMI the identity of the exact virtual disk to enlarge is shown in Monitor > Disks Otherwise, please refer to the "List of VMDKs/Partitions" articles linked above Please keep in mind that you cannot change disk size on the vCenter VM if there is a snapshot of the VM It is critical to ensure you have a good backup of vCenter before resizing any virtual disks on it Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.0 Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.5 and above
Monitor and prevent vCenter Appliance disks from filling
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
