OPERATIONAL DEFECT DATABASE
...
...
vCenter Server Appliance (vCSA) reports errors similar to: vSphere UI Health Alarm", "Log disk exhaustion on vcenter nameDatabase Health Alarm", "Core and Inventory Disk Exhaustion on vcenter name vCenter Server may be inaccessible with 503 Service Unavailable errors.vSphere Appliance Management Interface (VAMI) Monitor > Disk shows /storage/log at 75% full or more: At 75% of continuous disk use the space will trigger Yellow status warningsAt 85% the space will trigger Red critical alarmsAt 95% the vpxd service will shut down to prevent file and database corruption
This article provides information on how to troubleshoot and resolve the /storage/log partition being full on vCenter Appliance.
Potential causes include: vCenter Server log bundles not being cleared after generationVery high frequency events filling up logsServices such as the Apache Tomcat Java Servlet service failing to clean up files/storage/log partition set too small
Under default settings, when the partition reaches 75% full of continuous disk use for at least 10 minutes: You will see alarms being triggered. When the partition reaches 95% full at any point The operating system will prevent the main vmware-vpxd service from starting to try to prevent file corruption. Please keep in mind that this does not require it to be at 95% for a continuous period so the threshold might be crossed, shut down vpxd service, and then the use may recede,Deleting critical files may prevent the vCenter Server Appliance from working.Resizing vCenter Appliance virtual disks carries the risk of data corruption. WARNING: Ensure you have a good backup of vCenter Appliance before deleting files or resizing disks.
Verify whether one of these known issues is not causing the problem: Affected VersionsAssociated Knowledgebase Article link6.0 /storage/log partition full due to cloudvm-ram-size.log file rotation is not working in vCenter Server Appliance 6.0 before Update 3, 6.5 before Update 1 /storage/log partition full due to SSO log files are not compressed in vCenter Server Appliance/storage/log partition full due to SSO log files are not compressed in vCenter Server Appliance 7.0 before Update 1C vCenter Appliance /storage/log partition full due to excessive pod-startup.log files 7.0 before Update 3c vmafdd.log is not being compressed which eventually leads to "log disk exhaustion" warning on the vCenter (83238) (vmware.com)vCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories (85475) (vmware.com) 7.0 before Update 3o, 8.0 before Update 1 /storage/log filling up with imfile-state files | rsyslogd (87669) (vmware.com) 7.0, 8.0, not resolved in any newer version vCenter Server Appliance 7.0.x /storage/log partition runs out of space due to VMware Analytics service log file (analytics-runtime.log.stderr) (85468) 7.0 Update 1 through Update 2. Fixed in Update 3 vCenter has a large number of localhost_access log files generated under /storage/log/vmware/eam/web/ (85249) 7.0 Update 1 and newer releases. Fixed in 8.0. vCenter has a large number of localhost_access log files generated under /storage/log/vmware/eam/web/ (85249) 7.0 Update 2. Fixed in Update 3. The /storage/log volume is filling up in vCenter 7.0 U2 due to growing sps-runtime.log.stderr (83671) (vmware.com) All versions /storage/log partition full due to Large Java dump files created under /storage/log/vmware/perfcharts in vCenter Server Appliance /storage/log becomes 100% full when exporting the vCenter Appliance log bundles vCenter Appliance /storage/log partition full due to excessive pod-startup.log files VCSA /storage/log/ partition runs out of space due to java hprof dumps created by VMware Analytics service If the known issues don't identify the problem, look for excessive Tomcat log files in the partition. Connect to the vCenter Server Appliance through SSH or through the vCenter VM consoleType shell to switch to the BASH shell for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access Use the following command to list the Apache Tomcat service log files under the below paths: /storage/log/vmware/sso/tomcat//storage/log/vmware/eam/web//storage/log/vmware/lookupsvc/tomcat/ (vCenter 7.0 only) If there are a lot of these you may delete them to free up space ls -lha catalina*logrm catalina*log Check whether space has significantly improved or not. If it has, the drive might be too small for the log traffic. You might increase the size of the /storage/log virtual disk per vCenter Server Appliance disk space is full df -h Check for and remove excessive PostgreSQL service health alarm log files In /storage/log/vmware/sso/tomcat/, /storage/log/vmware/eam/web/ & /storage/log/vmware/lookupsvc/tomcat/ (vCenter 7.0 only) directories, run the commands: ls -lha localhost_access*rm localhost_access* Check again whether space has significantly improved or not. If it has, the drive might be too small for the log traffic. You might increase the size of the /storage/log virtual disk per vCenter Server Appliance disk space is full Check for and remove excessive Storage Profile Service access logs cd /var/log/vmware/vmware-spsls -lha sps-access*logrm sps-access*log Check again whether space has significantly improved or not. If it has, the drive might be too small for the log traffic. You might increase the size of the /storage/log virtual disk per vCenter Server Appliance disk space is full If the previous steps did not Identify the problem, follow vCenter Server Appliance disk space is full in the "Further troubleshooting" section. Warning: Before deleting any files from the vCenter Appliance, ensure you have a good backup of vCenter
vCenter Server Appliance disk space is fullvCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
