Loading...
Loading...
Due to a bug in ConfigStore API, stale data related to block devices might not be deleted in time from the ESXi ConfigStore database and causing an out of space condition. As a result, write operations to ConfigStore start to fail. In the backtrace, you see logs such as:2022-12-19T03:51:42.733Z cpu53:26745174)WARNING: VisorFSRam: 203: Cannot extend visorfs file /etc/vmware/configstore/current-store-1-journal because its ramdisk (configstore) is full.
ConfigStore SetVitalDataInstances API sets vital configuration by overwriting existing data, sometimes this leads to empty rows in configstore database (stale data).
Fix has been submitted to 7.0.3 P08, 8.0.1 P02, 8.0.2.
Follow the below mentioned steps.Option 1:Use configstore-recovery python script attached to the KB article. Copy the script to the host and run python configstore-recovery The script performs following steps:1. Temporarily increase the configstore ramdisk size to 64MB (initial size is 32MB)2. Clean stale/empty data from the configstore DB.3. Perform VACUUM on the configstore DB. The VACUUM command rebuilds the database file, repacking it into a minimal amount of disk space.4. Revert configstore ramdisk size to 32MBLogs from the script are captured in /var/run/log/syslog.log Option 2:Manually recover the host by following the below steps: 1. Temporarily increase the configstore ramdisk size to 64MB (initial size is 32MB).a. Get configstore ramdisk group ID using:vsish -e set /sched/groupPathNameToID host system visorfs ramdisks configstoreb. Set configstore ramdisk max memory to 64 using:vsish -e set /sched/groups/<GID>/memAllocationInMB max=64c. Verify configstore ramdisk memory allocation using:vsish -e get /sched/groups/<GID>/memAllocationInMBExample:[root@sc2-10-186-8-178:~] vsish -e set /sched/groupPathNameToID host system visorfs ramdisks configstore1627[root@sc2-10-186-8-178:~][root@sc2-10-186-8-178:~] vsish -e get /sched/groups/1627/memAllocationInMBmemsched-allocation {min:32max:32shares:-3minLimit:-1units: 4 -> mb}[root@sc2-10-186-8-178:~] vsish -e set /sched/groups/1627/memAllocationInMB max=64[root@sc2-10-186-8-178:~][root@sc2-10-186-8-178:~] vsish -e get /sched/groups/1627/memAllocationInMBmemsched-allocation {min:32max:64shares:-3minLimit:-1units: 4 -> mb}[root@sc2-10-186-8-178:~]2. Forcefully purge any stale device entries currently on the host. This gives a chance to purge any recently unmapped devices (< 7days) to get purgedesxcli storage core device purge -f3. Delete 'esx/storage/devices_access' configuration using configstorecliconfigstorecli config current delete -c esx -g storage -k devices_access --all4. Reboot the host (do not force reboot).reboot
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.