OPERATIONAL DEFECT DATABASE
...
...
The ESXi host reports not responding in vCenter ServerLogging in to the vSAN host through Log Insight or through the vSphere Web Client fails.In the vobd.log file, you see the entries similar to: [VisorfsCorrelator] 98549997090us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantraces--2016-11-13T19h09m03s166.gz because its ramdisk (vsantraces) is full.[VisorfsCorrelator] 98548366158us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'vsantraces' is full. As a result, the file /vsantraces/vsantraces--2016-11-13T19h09m03s166.gz could not be written. vsantraces ramdisk appears to be full or nearly full on the ESXi: vdf -h ----- Ramdisk Size Used Available Use% Mounted on root 32M 2M 29M 8% -- etc 28M 232K 27M 0% -- opt 32M 0B 32M 0% -- var 48M 516K 47M 1% -- tmp 256M 12K 255M 0% -- iofilters 32M 0B 32M 0% -- shm 1024M 0B 1024M 0% -- hostdstats 114M 2M 111M 1% -- vsantraces 300M 287M 12M 95% -- Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
If no scratch location is configured for vsantraces files, the traces are stored on the ramdisk.The default ramdisk size for vsantraces is 300MB. When the usage reaches around 95%, additional files fail to be created.Under certain conditions, some vsantrace entries may get logged too frequently, eventually exhausting the capacity of the volume holding the trace files.
The ramdisk gets full which could lead to a hung host or the host becoming non-responsive in vCenterThe vsantraces files are used for vSAN diagnostics so if vsantraces stops logging due to no space then historical data is lost for troubleshooting vSAN issues.
It is possible in vSAN that the ramdisk for vSAN trace information may grow and fill the allotted ramdisk. vsantraces can be redirected to another location for logging.vSAN traces can grow to any size you select, but VMware recommends around 600 MB if available.If persistent logging is configured for the cluster, we can redirect vsantraces here using these steps.1) Connect to each ESXi server in the cluster using SSH.2) Get the current vsantrace config by running esxcli vsan trace for vSAN 6.7 and lowerYou see output similar to:vSAN Traces Directory: /vsantracesNumber Of Files To Rotate: 8Maximum Trace File Size: 180 MBLog Urgent Traces To Syslog: trueFor vSAN 7.x and higher run cat /etc/vmware/vsan/vsantraced.conf | grep -v "#"You see output similar to:VSANTRACED_ROTATE_MAX_FILES=10VSANTRACED_ROTATE_FILE_SIZE=8VSANTRACED_URGENT_ROTATE_MAX_FILES=4VSANTRACED_URGENT_ROTATE_FILE_SIZE=8VSANTRACED_LAST_SELECTED_VOLUME="/vsantraces"VSANOBSERVER_MAX_MB_SIZE="38"3) To configure vsantraces to log elsewhere, run this command: [root@localhost:~] esxcli vsan trace set Usage: esxcli vsan trace set [cmd options] Note: This command is not thread safe. Cmd options: -f|--numfiles: Log file rotation for vSAN trace files. -p|--path: Path to store vSAN trace files. -r|--reset: When set to true, reset defaults for VSAN trace files. -s|--size: Maximum size of vSAN trace files in MB.If the Scratch folder has already been redirected, we can also redirect vsantraces into the same folder. There may be unique folders for each host that already exist on your storage for this if previously configured. In this case, create a folder called vsantraces so these can be separated out per-Host. If one folder per Host is not there, create one folder for each. To create one folder for each host, for the corresponding vSphere version, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x (1033696)Note: Log-bundles should be separated into their own host-folders, otherwise, once the first host gets the locks on the scratch location it prevents the other hosts in the cluster from logging.After we have a folder for each Host and the logging is separated, we can redirect traces using these commands: mkdir /vsantraces esxcli vsan trace set -f 10 -s 60 -p /vsantracesFor example:If files are located at /vmfs/volumes/Datastore/scratch/localhost/, run these commands: mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/ esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/Note: The preceding commands are just an example. The location may vary based on the environment.This automatically begins sending trace files to the new location, no further configuration is needed.Delete or move older vsantraces files from the vsantraces folder /vsantraces/ located on the ramdisk1) Navigate to the vsantraces folder by running cd var/log/vsantraces2) List the contents of the folder by running ls-ahlSample output:[root@esxi1:/vsantraces] ls -ahltotal 264512drwxr-xr-x 1 root root 512 Jun 4 20:04 .drwxr-xr-x 1 root root 512 Jun 4 20:16 ..-rw-r--r-- 1 root root 254.8K Jun 4 17:22 vsanObserver--2023-06-04T17h02m00s.gz-rw-r--r-- 1 root root 254.9K Jun 4 17:44 vsanObserver--2023-06-04T17h24m00s.gz-rw-r--r-- 1 root root 254.7K Jun 4 18:06 vsanObserver--2023-06-04T17h46m00s.gz-rw-r--r-- 1 root root 254.6K Jun 4 18:28 vsanObserver--2023-06-04T18h08m00s.gz-rw-r--r-- 1 root root 254.8K Jun 4 18:50 vsanObserver--2023-06-04T18h30m00s.gz-rw-r--r-- 1 root root 254.6K Jun 4 19:12 vsanObserver--2023-06-04T18h52m00s.gz-rw-r--r-- 1 root root 254.6K Jun 4 19:34 vsanObserver--2023-06-04T19h14m00s.gz-rw-r--r-- 1 root root 254.7K Jun 4 19:56 vsanObserver--2023-06-04T19h36m00s.gz-rw-r--r-- 1 root root 208.5K Jun 4 20:16 vsanObserver--2023-06-04T19h58m01s.gz-rw-r--r-- 1 root root 20.0M Jun 2 07:56 vsantraces--2023-06-01T23h01m33s187.gz-rw-r--r-- 1 root root 20.0M Jun 2 16:51 vsantraces--2023-06-02T07h56m21s853.gz-rw-r--r-- 1 root root 20.0M Jun 3 01:48 vsantraces--2023-06-02T16h51m11s433.gz-rw-r--r-- 1 root root 20.0M Jun 3 10:45 vsantraces--2023-06-03T01h48m01s609.gz-rw-r--r-- 1 root root 20.0M Jun 3 19:39 vsantraces--2023-06-03T10h45m44s438.gz-rw-r--r-- 1 root root 20.0M Jun 4 04:28 vsantraces--2023-06-03T19h39m32s771.gz-rw-r--r-- 1 root root 20.0M Jun 4 13:24 vsantraces--2023-06-04T04h28m00s889.gz-rw-r--r-- 1 root root 15.4M Jun 4 20:17 vsantraces--2023-06-04T13h24m46s788.gz-rw-r--r-- 1 root root 192 Jun 4 13:24 vsantraces.index-rw-r--r-- 1 root root 8.0M May 15 03:33 vsantracesClom--2023-05-11T15h15m00s791.gz-rw-r--r-- 1 root root 8.0M May 18 15:48 vsantracesClom--2023-05-15T03h33m29s948.gz-rw-r--r-- 1 root root 8.0M May 22 04:02 vsantracesClom--2023-05-18T15h48m59s091.gz-rw-r--r-- 1 root root 8.0M May 25 15:50 vsantracesClom--2023-05-22T04h02m55s293.gz-rw-r--r-- 1 root root 8.0M May 29 02:54 vsantracesClom--2023-05-25T15h50m08s307.gz-rw-r--r-- 1 root root 8.0M Jun 1 05:55 vsantracesClom--2023-05-29T02h54m10s389.gz-rw-r--r-- 1 root root 8.0M Jun 4 18:09 vsantracesClom--2023-06-01T05h55m47s005.gz-rw-r--r-- 1 root root 206.3K Jun 4 20:16 vsantracesClom--2023-06-04T18h09m27s740.gz-rw-r--r-- 1 root root 192 Jun 4 18:09 vsantracesClom.index-rw-r--r-- 1 root root 620.8K Jun 4 03:29 vsantracesDOMObj--2023-03-13T17h23m25s376.gz-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesDOMObj.index-rw-r--r-- 1 root root 1.0M Jun 3 17:10 vsantracesIODiag--2023-06-02T18h31m21s576.gz-rw-r--r-- 1 root root 1.0M Jun 4 15:56 vsantracesIODiag--2023-06-03T17h10m06s623.gz-rw-r--r-- 1 root root 195.4K Jun 4 20:17 vsantracesIODiag--2023-06-04T15h56m16s897.gz-rw-r--r-- 1 root root 72 Jun 4 15:56 vsantracesIODiag.index-rw-r--r-- 1 root root 2.0M May 17 13:09 vsantracesLSOM--2023-03-13T17h23m25s299.gz-rw-r--r-- 1 root root 575.1K Jun 4 20:11 vsantracesLSOM--2023-05-17T13h09m23s998.gz-rw-r--r-- 1 root root 48 May 17 13:09 vsantracesLSOM.index-rw-r--r-- 1 root root 15 Mar 13 17:23 vsantracesLSOMVerbose--2023-03-13T17h23m25s333.gz-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesLSOMVerbose.index-rw-r--r-- 1 root root 10.0M May 29 00:22 vsantracesUrgent--2023-05-25T05h05m25s105.gz-rw-r--r-- 1 root root 10.0M May 31 12:01 vsantracesUrgent--2023-05-29T00h22m35s242.gz-rw-r--r-- 1 root root 10.0M Jun 4 10:11 vsantracesUrgent--2023-05-31T12h01m34s418.gz-rw-r--r-- 1 root root 1.0M Jun 4 20:17 vsantracesUrgent--2023-06-04T10h11m46s702.gz-rw-r--r-- 1 root root 96 Jun 4 10:11 vsantracesUrgent.index3) To delete older files run the following:rm vsantraces*2023-05*.gzrm vsanObserver--2023-05*.gzrm vsantracesUrgent--2023-05*.gz4) To move them run the following:mv vsantraces*2023-05*.gz /<new-location-of-vsantraces>mv vsanObserver--2023-05*.gz /<new-location-of-vsantraces>mv vsantracesUrgent--2023-05*.gz /<new-location-of-vsantraces> Note: If you're not comfortable performing the delete/move of older vsantraces logs engage vSAN support for assistance.
If no additional storage is available, limit the size of vsantraces to 200MB. This allows a bit of overhead for additional files and ensures that we do not fill the ramdisk.For vSAN 6.7 and lower this can be done by running this command on all vSAN hosts in the cluster:esxcli vsan trace set -f 10 -s 20This would make the max files setting '10' and the file size setting 20 MBs. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBsFor vSAN 7.x and higher this can be done by running this command on all vSAN hosts in the cluster:esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10This would make the max files setting '10' and the file size setting 10 MBs for both the vsantraces and the urgenttraces. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBsThese changes are immediate and older files will be rolled off as smaller ones take their places over time.No services require a restart to enable this change.Delete older vsantraces files from the vsantraces folder /vsantraces/ as per the Resolution section.
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
