Symptoms

ESXi 6.0 may not failover to an alternate available path after encountering a Permanent Device Loss (PDL) condition on the active path.Purpose Upgrading to ESXi 6.0 Update 2 resolves this issue.

Cause

An inadvertent change in PDL multipathing behavior in ESXi 6.0 results in alternative working paths for a LUN not being checked if a PDL condition/error is detected. When encountering a PDL condition on the active path, the ESXi host initiates a health check against the remaining paths but does not fail over if another path is responsive/healthy. The correct response is to failover to one of the healthy working paths. As a result, the host is no longer able to issue I/O to these LUNs until the ESXi host is rebooted.

Resolution

This issue is resolved in ESXi 6.0 Update 2 and later, available at VMware Downloads. To work around this issue if you do not want to upgrade, restart the ESXi host. Related Information You experience these additional symptoms: In the /var/log/vmkernel.log file on the ESXi host, you see path-evaluation activity followed by I/O failures associated with SCSI sense code H:0x8: cpu6:32909)WARNING: NMP: nmp_PathDetermineFailure:2961: Cmd (0x2a) PDL error (0x5/0x25/0x0) - path vmhba2:C0:T5:L2 device naa.514f0c5ec2000008 - triggering path evaluation cpu6:32909)WARNING: NMP: nmp_PathDetermineFailure:2961: Cmd (0x2a) PDL error (0x5/0x25/0x0) - path vmhba2:C0:T5:L2 device naa.514f0c5ec2000008 - triggering path evaluation cpu6:32909)WARNING: NMP: nmp_PathDetermineFailure:2961: Cmd (0x28) PDL error (0x5/0x25/0x0) - path vmhba2:C0:T5:L2 device naa.514f0c5ec2000008 - triggering path evaluation cpu6:32909)WARNING: NMP: nmp_PathDetermineFailure:2961: Cmd (0x28) PDL error (0x5/0x25/0x0) - path vmhba2:C0:T5:L2 device naa.514f0c5ec2000008 - triggering path evaluation [ ... ] cpu28:36724)NMP: nmp_ThrottleLogForDevice:3286: Cmd 0x2a (0x43a61055c5c0, 36134) to dev "naa.514f0c5ec2000008" on path "vmhba1:C0:T6:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL cpu14:36168)NMP: nmp_ThrottleLogForDevice:3286: Cmd 0x89 (0x439e11581700, 32806) to dev "naa.514f0c5ec2000008" on path "vmhba2:C0:T7:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. Lack of failover to the alternate path result in aborted I/O, this can cause LUN availability issues which in turn cause virtual machine failures.The PDL condition may be encountered on a subset (but not all) paths during non-disruptive upgrade events on certain storage platforms.The LUN and datastore do not return to availability after this event until the original path is available again or the ESXi host is rebooted. Question Is it OK to approve pre-check conditions and proceed with XtremIO 4.0 upgrade if vSphere 6.0 hosts are not running Update 2? Facts XtremIO VMware vSphere ESXi 6.0 Update 2 NMP (native multi-pathing) PowerPath VNXe NDU (non-disruptive upgrade) Answer As part of the XtremIO 4.0 pre-upgrade check there is an EMC recommendation not to proceed on vSphere 6.0 hosts using NMP multi-pathing unless customer is running Update 2. Logical Build of the VB540 should be using PowerPath for multipathing on all ESXi host blades. Only AMP hosts would not be using PowerPath and they should be using either local disk or VNXe shared storage. This is directly related to EMC KB 000463259 and VMware KB 2144657 which documents a Data Unavailability issue related to VMware NMP native multi-pathing and path failover during NDU non-disruptive upgrade to storage controllers.

Support Cases