Symptoms
Running esxtop displays abnormal DAVG/cmd and KAVG/cmd values.Running the below command on the affected ESXi hosts display results as below:
vm-support -p -d 1800 -i 5 -w /vmfs/volumes/<VolumeNAme>/
Note: The below results are just examples. Values may vary in different scenarios.
Sample of LUNs that showed abnormal values:
LUN ATS ATSFnaa.60a980003830386d4c3f4a43464b6733 436046 41naa.60a980003830386d4c3f4a43464b6753 438974 385naa.60a980003830386d4c3f4a43464b6755 436387 24naa.60a980003830386d4c3f4a43464b6757 438984 169naa.60a980003830386d4c3f4a43464b6759 435740 23naa.60a980003830386d4c3f4a43464b6762 435732 25naa.60a980003830386d4c3f4a43464b6764 437415 30naa.60a980003830386d4c3f4a43464b6766 435739 29
Sample of LUNs that did not show abnormal values:
LUN ATS ATSFnaa.60a980003830386d4c3f4a43464b6835 785 0naa.60a980003830386d4c3f4a43464b6837 629 0naa.60a980003830386d4c3f4a43464b6839 629 0naa.60a980003830386d4c3f4a43464b6841 630 0naa.60a980003830386d4c3f4a43464b6843 629 0naa.60a980003830386d4c3f4a43464b6845 629 0naa.60a980003830386d4c3f4a43464b6847 644 0naa.60a980003830386d4c3f4a43464b6849 633 0
Repeat offenders are typically busier LUNs in general.
Example of a few LUNs:BADNETAPP Fibre Channel Disk (naa.60a980003830386d4c3f4a43464b6831)/> cat /storage/scsifw/devices/naa.60a980003830386d4c3f4a43464b6831/statsStatistics { Successful Commands:25656883 Failed Commands:597 Blocks Read:1591656653 Failed Blocks Read:0 Blocks Written:1596779952 Failed Blocks Written:0 Read Operations:12494250 Failed Read Operations:0 Write Operations:12695671 total ATS cmds:436086 total ATS failures:30 total zero cmds:139 total zero failures:0 total blocks zeroed:284672GOODNETAPP Fibre Channel Disk (naa.60a980003830386d4c3f4a43464b6833)cat /storage/scsifw/devices/naa.60a980003830386d4c3f4a43464b6833/statsStatistics { Successful Commands:46839 Failed Commands:569 Blocks Read:828213 Failed Blocks Read:0 Blocks Written:1802 Failed Blocks Written:0 Read Operations:12857 Failed Read Operations:0 Write Operations:1802...total ATS cmds:639 total ATS failures:0 total zero cmds:0 total zero failures:0 total blocks zeroed:0Pattern seen for effected LUNs:CMDS/s: 0.19DAVG/cmd: 18446744073709552.00KAVG/cmd: -18446744073709552.00CMDS/s: 0.38DAVG/cmd: 9223372036854776.00KAVG/cmd: -9223372036854776.00
Resolution
This issue is resolved in ESXi 6.5 U2, available at VMware Downloads.
