Symptoms

One or more NFS datastores go into an APD (all paths down) state on a single host at a time. This could happen to different datastores on different hosts or possibly the same datastore on multiple hosts. Generally its random and intermittent and is solved by either downing and upping the ethernet ports on the ESXi host or rebooting it. It does not necessarily always happen to the same datastore or the same host.The key feature of this problem is that the affected datastore or NFS export is still accessible from other hosts. If the datastore is down on all hosts its not as likely to be this problem. If its not solved by downing the network ports or rebooting the host, its also not going to be this issue.This affects both VNX1, VNX2, and eNAS products.

Cause

VMware support may advise setting the NFS.MaxQueueDepth to 64, EMC does not have a recommendation for this value currently. However, It is not likely to resolve this specific issue.Engineering has identified a bug in the way that we handle our TCP send window calculation in some situations. Essentially what happens is that at a certain point the VNX sets the TCP send window value to 0 inappropriately. This disallows the VNX from sending any new data to the host its communicating with on that connection. The VNX can still acknowledge incoming data on the TCP layer, but will not be able to send any NFS responses. To our knowledge at this point we have only seen this behavior affecting ESXi NFS datatsores due to a specific way that ESXi performs its TCP acknowledgments at times. At certain points instead of sending along an acknowledgement with its next data packet ESXi uses an additional separate acknowledgement as soon as new data is received from the VNX even if it has data in its transmission queue. This behavior makes the DM believe that the transfer is unidirectional and puts it into header prediction mode. If the ESXi TCP acknowledgement behavior stays consistent while transferring more than 2GB of data from the DM, the DM will slowly reduce the TCP send window to 0 making that particular TCP connection able to only send data in one direction (from the host to the array.). If the datamover receives a data packet with a new ACK number within that 2GB transfer, or there is any packet loss that causes a retransmission then the problem is not encountered. ESXi runs a heartbeat on the datastore to determine if it's still available. This heartbeat is a GetAttr request to a specific file on the datatstore. If it fails a certain number of times the ESXi host marks the datastore as APD. Since the VNX cant reply to the GetAttr requests from the ESXi host while its TCP send window is set to 0 It marks the datastore as inaccessible. For whatever reason ESXi does not attempt to reset the connection, which would also resolve this problem. That is why doing a reboot or downing and upping the network ports on the host works to restore access. The TCP send window is calculated separately for each connection. So other datastores will remain online provided they have not hit the same condition. The datastore itself isn't the problem, so other hosts should still be able to access it unless they have hit the same condition with their connection to this particular datastore.This issue can be confirmed if there is a packet trace that covers the datastore going from an online to an offline state.

Resolution

The TCP send window calculation behaviour will be fixed in a future release of code for both 7.1 and 8.1 code versions (VNX1, VNX2, and eNAS). There is a hotfix available currently, If a fix is needed immediately contact support to request it along with reboot/failover outage scheduled.

Support Cases