Symptoms

The following symptoms may occur: Node is in VSAN cluster partitionVSAN network partition alarm is shown in Skyline healthNode is showing network partitioned, although it can ping other nodes VSAN VMK port using command: # vmkping -I vmk3 localcli vsan cluster get Cluster Information: Enabled: true Current Local Time: 2019-03-03T07:40:03Z Local Node UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a Local Node Type: NORMAL Local Node State: MASTER Local Node Health State: HEALTHY Sub-Cluster Master UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a Sub-Cluster Backup UUID: Sub-Cluster UUID: 5292b573-6d0c-d8e2-b233-aa15b43a3740 Sub-Cluster Membership Entry Revision: 0 Sub-Cluster Member Count: 1 Sub-Cluster Member UUIDs: 5b90382f-a7d7-8dc2-4358-246e96e1013a Sub-Cluster Membership UUID: aa767b5c-5cc5-a904-8224-246e96e1013a Unicast Mode Enabled: true Maintenance Mode State: OFF Config Generation: 2a3be2d8-0006-4b0f-a87a-887ed8242d79 3 2019-03-03T06:39:42.986 [root@xxxx:/tmp] esxcfg-vmknic -l | grep -i vmk3 vmk3 20336 IPv4 172.x.x.x 255.xx.2xx.0 172.x.x.x 00:50:56:66:aa:ed 1500 65535 true STATIC defaultTcpipStack vmk3 20336 IPv6 xxxx::xxxx:xxxx:xxxx:xxxx 64 00:50:56:66:aa:ed 1500 65535 true STATIC, PREFERRED defaultTcpipStack [root@xxxx:~] vmkping -I vmk3 172.x.x.x PING 172.16.200.10 (172.16.200.10): 56 data bytes 64 bytes from 172.x.x.x : icmp_seq=0 ttl=64 time=6.226 ms 64 bytes from 172.x.x.x : icmp_seq=1 ttl=64 time=2.879 ms 64 bytes from 172.x.x.x : icmp_seq=2 ttl=64 time=0.134 ms --- 172.x.x.x ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.134/3.080/6.226 ms [root@xxxxx:~] vmkping -I vmk3 172.xx.xx.11 PING 172.x.x.x (172.x.x.x ): 56 data bytes 64 bytes from 172.x.x.x: icmp_seq=0 ttl=64 time=0.200 ms 64 bytes from 172.x.x.x: icmp_seq=1 ttl=64 time=0.163 ms 64 bytes from 172.x.x.x: icmp_seq=2 ttl=64 time=0.152 ms --- 172.x.x.x ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.152/0.172/0.200 ms

Cause

Unicast agent list is invalid or incomplete.One or more hosts cannot communicate with other VSAN hosts in the cluster.

Resolution

1. Run this command on all hosts before changing the unicast agent list. esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates 2. Add the missing unicast agent entries. esxcli vsan cluster unicastagent add -t node -u -U -a -p -i Example: esxcli vsan cluster unicastagent add -t node -u 57ea62b0-9df6-6532-110c-9457a55a37d0 -U true -a XX.XXX.XXX.XX-p 12321 -i vmk3 [root@xxxxx:~] esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name ------------------------------------ --------- ---------------- ------------- ----- ---------- 5b90378f-0b0b-2da4-933b-246e96e101b6 0 true 172.x.x.x 12321 5b90380e-927b-73ce-159f-246e96e1018a 0 true 172.x.x.x 12321 [root@xxxx:~] [root@xxxxxx:~] esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name ------------------------------------ --------- ---------------- ------------- ----- ---------- 5b90378f-0b0b-2da4-933b-246e96e101b6 0 true 172.x.x.x 12321 5b90382f-a7d7-8dc2-4358-246e96e1013a 0 true 172.x.x.x 12321 [root@xxxxx:~] [root@xxxx:/tmp] esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name ------------------------------------ --------- ---------------- ------------- ----- ---------- 5b90380e-927b-73ce-159f-246e96e1018a 0 true 172.x.x.x 12321 5b90382f-a7d7-8dc2-4358-246e96e1013a 0 true 172.x.x.x 12321 [root@xxxxx:/tmp] [root@xxxx:~] esxcli vsan cluster get Cluster Information Enabled: true Current Local Time: 2019-03-03T07:48:55Z Local Node UUID: 5b90382f-a7d7-8dc2-4358-246e96e1013a Local Node Type: NORMAL Local Node State: AGENT Local Node Health State: HEALTHY Sub-Cluster Master UUID: 5b90378f-0b0b-2da4-933b-246e96e101b6 Sub-Cluster Backup UUID: 5b90380e-927b-73ce-159f-246e96e1018a Sub-Cluster UUID: 5292b573-6d0c-d8e2-b233-aa15b43a3740 Sub-Cluster Membership Entry Revision: 3 Sub-Cluster Member Count: 3 Sub-Cluster Member UUIDs: 5b90378f-0b0b-2da4-933b-246e96e101b6, 5b90380e-927b-73ce-159f-246e96e1018a, 5b90382f-a7d7-8dc2-4358-246e96e1013a Sub-Cluster Membership UUID: d519d05b-a227-5621-66dc-246e96e101b6 Unicast Mode Enabled: true Maintenance Mode State: OFF Config Generation: 2a3be2d8-0006-4b0f-a87a-887ed8242d79 3 2019-03-03T06:39:42.986 3. Once the unicast agent list has been fixed on all hosts, run the below command on all hosts to set IgnoreClusterMemberListupdates back to its default setting of 0. esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates See official Broadcom KB Network partition caused by an invalid/incomplete unicast agent list on vSAN host(s)

Support Cases