Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
100% capacity used for /dev/sda3.Size of audit.log file is very large and /var/log/audit folder consumes majority of the space.Saved logs from log rotate policy reference a date that is not in line with the policy.Unable to connect to the vCenter Server as services are not started.Running /etc/cron.daily/logrotate manually rotates logs as expected.Accessing vSphere Web Client might fail with error: 503 service unavailable
This article provides steps to reduce the audit.log size.
To resolve this issue, truncate the audit.log file and verify the cron job is working correctly. Truncate audit.log Log in to the vCenter Server Appliance through SSH.Run this command to enable access the Bash shell: shell.set --enabled true Type shell and press Enter.Navigate to the /var/log/audit folder with this command: cd /var/log/audit For 6.7: cd /var/spool/anacron/ Run this command to verify the issue is with the audit.log file being too large (a few GBs): ls -lh For example: ls -lh total 3.5G -rw------- 1 root root 3.5G Feb 3 16:55 audit.log -rw------- 1 root root 445K Apr 8 2016 audit.log-20160408.bz2 -rw------- 1 root root 447K Apr 9 2016 audit.log-20160409.bz2 Truncate (clean the content without deleting the file) the audit.log file with this command: truncate -s 0 audit.log Verify that the cron job to rotate the audit.log is running Run this command to see when the cron job was last ran successfully: ls -l /var/spool/cron/lastrun/ For example: ls -l /var/spool/cron/lastrun/ total 0-rw------- 1 root root 0 Apr 22 2016 cron.daily-rw------- 1 root root 0 Apr 22 2016 cron.hourly-rw------- 1 root root 0 Apr 21 2016 cron.weekly Determine if the cron job was last updated long time ago. Normally, this should be daily.Run this command to check for credential failures running the cron job: grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head For example:grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head2016-11-07T00:20:01.617180+00:00 vcenter /usr/sbin/cron[18972]: Authentication token is no longer valid; new one required2016-11-07T00:20:01.617183+00:00 vcenter /usr/sbin/cron[18974]: Authentication token is no longer valid; new one required Run this command to check if the root password has expired: chage -l root For example:chage -l rootPassword change requested. Choose a new password.Old Password:New password: Change the root password as prompted.Verify the root account password has been changed: chage -l rootFor example:chage -l rootMinimum: 0Maximum: 365Warning: 7Inactive: -1Last Change: Feb 03, 2017Password Expires: Feb 03, 2018Password Inactive: NeverAccount Expires: Never Restart all vCenter Server services. service-control --stop --allservice-control --start --all Note: Run the below command to change the root password to never expire: #chage -m 0 -M 99999 -I -1 -E -1 root
VMware Skyline Health Diagnostics for vSphere - FAQvCenter Server Appliance Maintenance Series: Troubleshooting disk space related issues由于 Audit.log 文件未进行轮换，vCenter Appliance root 分区已满EAM Service fails to start after vCenter Server reboot/storage/log directory is full in vCenter Server Appliance 6.0