Symptoms

By running esxcli/localcli vsan health cluster list marks the network health check as red, pointing out "hosts with connectivity issues" Figure 1: vSan health cluster listTo view further details, run the command below: esxcli vsan health cluster get -t “Hosts with connectivity issues” Hosts with communication issues Host ----------------- 159.70.xxx.xxx vCenter does not show any vSAN issue. This is a communication issue between hosts only, NOT vCenter <-> Hosts.Figure 2: vSan skyline healthScenario 1:SSL certificate issues can be seen from /var/run/log/vsanmgmt.log. /var/run/log/vsanmgmt.log error vsand[10450841] [opID=0d02566e VsanVimHelpers::GetVsanVersionNamespace] Failed to test vsan vmodl version with error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:728) on 159.70.xxx.xxx Scenario 2: In a Stretched cluster with Witness Traffic Separation (WTS) setup, you see errors in vsanmgmt.log: 2024-02-07T01:37:25.992Z error vsand[11443311] [opID=Thread-424 VsanVimHelpers::GetVsanVersionNamespace] Failed to test vsan vmodl version with error [Errno 101] Network is unreachable on 10.25.x.xxx 2024-02-07T01:37:25.992Z warning vsand[11443311] [opID=Thread-424 VsanVimHelpers::GetVsanVersionNamespace] Retry retrieving vsan vmodl version, 7 2024-02-07T01:37:30.994Z error vsand[11443311] [opID=Thread-424 VsanVimHelpers::GetVsanVersionNamespace] Failed to test vsan vmodl version with error [Errno 101] Network is unreachable on 10.25.x.xxx 2024-02-07T01:37:30.994Z error vsand[11443311] [opID=Thread-424 VsanHealthHelpers::PerHostTryFetchHostInfo] Failed to fetch host info Traceback (most recent call last): File "/usr/lib/vmware/vsan/perfsvc/VsanHealthHelpers.py", line 1529, in PerHostTryFetchHostInfo File "/usr/lib/vmware/vsan/perfsvc/VsanHealthHelpers.py", line 1599, in PerHostFetchHostInfo File "/usr/lib/vmware/vsan/perfsvc/cliutils.py", line 393, in PerfsvcPeerSoapAdapter File "/usr/lib/vmware/vsan/perfsvc/cliutils.py", line 44 1, in PerfsvcPeerSoapAdapterHelper File "/usr/lib/vmware/vsan/perfsvc/VsanVimHelpers.py", line 243, in GetVsanVersionNamespace PyCppVmomi.vmodl.fault.HostNotReachable: (vmodl.fault.HostNotReachable) {} the host reported is witness node: [root@xxxx:~] esxcli vsan health cluster get -t 'Hosts with connectivity issues' Hosts with connectivity issues redChecks if API calls from VC to a host are failing while the host is in connected state. Ask VMware: http://www.vmware.com/esx/support/askvmware/index.php?eventtype=com.vmware.vsan.health.test.hostconnectivityHosts with communication issues Host ---------------- 10.25.5.xxx Later this error is marked as host connectivity issue in the LCM advisory report: Figure 2: Advisory report (Screenshot from LAB device) 

Cause

Scenario 1: SSL certificate issuesScenario 2: When data node and witness node have configured dedicated witness vmknics, the witness traffic is separated from vSAN traffic logically, when users configure witness traffic and vSAN traffic into a real different network, the witness traffic and vSAN traffic are separated.If the health 'Hosts with connectivity issues' is performed at VC(vSAN skyline health check), VC accesses the data node and witness node using mgmt traffic.If the health 'Hosts with connectivity issues' is performed at Host, Host access data node and witness node using vSAN traffic, for the above situation, vSAN traffic and witness traffic are separated, and witness node has no vSAN traffic, the check fails because host cannot access witness node using vSAN traffic.

Resolution

For Scenario 1:If custom certificates are present, ask the customer to get new certificates for the hosts. DO NOT Apply the steps in this KB.If self-signed certificates are present, follow the steps below to replace them:Renewing and refreshing the hosts' certificate. Steps: Select vCenter then, the Host.Select on Configure -> System -> Certificate.Select on renew (wait for it to complete).Select on Refresh CA Certificates.Do the above two steps on each host identified with SSL issues.You can also perform it on all hosts so they get new certs simultaneously. Figure 4: Cert configuration pageFor Scenario 2:The error can be safely ignored, and it will not block the LCM. VMware stated it is fixed in 8.0 U3.

Support Cases