Loading...
Loading...
Impacted Dell Hardware: VPLEX Series VPLEX-VS2 VPLEX VS6 VPLEX-MetroIP Impacted Dell Software: GeoSynchrony 6.0 Patch 1 GeoSynchrony 6.0 Patch 2 GeoSynchrony 6.0 Service Pack 1 GeoSynchrony 6.0 Service Pack 1 Patch 1 GeoSynchrony 6.0 Service Pack 1 Patch 2 Users using VPLEX VS6 or VS2 hardware in a Metro IP configuration, running on GeoSynchrony 6.0.X code levels, experience poor performing rebuild, and random extreme spikes in WAN latency. Rebuild throughput remains around 10 MB/s. This issue is susceptible for any Metro IP systems running GeoSynchrony 6.0.X versions. Regardless of the hardware being VS2 or VS6
User recently created distributed devices, triggering full rebuilds. Rebuilds for larger devices can show completion ETAs of 15 days or more. It has been determined through internal Dell benchmark testing that VPLEX Metro IP configurations with GeoSynchrony 6.0.X process fewer IOPS when VPLEX MTU is set to 9000 and transfer size is set to greater than 1MB. VPLEX CPU utilization remains within acceptable limits for all directors. WAN latency spikes are observed from source site to target site. WAN latency values are acceptable from target site to source site Root Cause: At larger I/O size, the sender is more aggressive in transmitting data through the network stack. This increases the probability of packet drops leading to retransmits and in turn causes a decrease in WAN throughput. Also, the current protocol does not react to packet drops in a timely manner relying primarily on timeouts exacerbating the performance issue. At a lower I/O size, the load on the stack and network is not high enough to cause packet drops and performance degradation. This issue is susceptible for any Metro IP systems running 6.0.X version. Irrespective of the hardware being VS2 or VS6 hardware.
Permanent Resolution: There is no fix for this issue. Refer to the below actions. VPLEX Engineering recommends for Metro clusters using IP WAN to ensure that the following two (2) criteria are met: Note: The below steps are valid on both VS2 and VS6 hardware running GeoSynchrony 6.0.X. Set VPLEX IP WAN ports (A2-XG**/B2-XG**) to an MTU of 1500 NOTE: The network does NOT have to be set to 1500 MTU. ONLY the VPLEX IP WAN-COM ports must be set to 1500, the network MTU can stay at the current value if already larger than 1500. To set the MTU to 1500: Verify connectivity: VPlexcli:/> connectivity validate-wan-com Disable port-group-0 on Cluster-1 & 2: VPlexcli:/> cd /clusters/cluster-1/cluster-connectivity/port-groups/port-group-0 VPlexcli:/clusters/Cluster-1/cluster-connectivity/port-groups/port-group-0> set enabled all-disabled VPlexcli:/> cd /clusters/cluster-2/cluster-connectivity/port-groups/port-group-0 VPlexcli:/clusters/Cluster-2/cluster-connectivity/port-groups/port-group-0> set enabled all-disabled Modify the MTU size in the subnet: VPlexcli:/> cd /clusters/cluster-1/cluster-connectivity/subnets/cluster-1-SN00 VPlexcli:/clusters/Cluster-1/cluster-connectivity/subnets/cluster-1-SN00> set mtu 1500 VPlexcli:/> cd /clusters/cluster-2/cluster-connectivity/subnets/cluster-2-SN00 VPlexcli:/clusters/cluster-2/cluster-connectivity/subnets/cluster-1-SN00> set mtu 1500 Enable port group 0: VPlexcli:/> cd /clusters/cluster-1/cluster-connectivity/port-groups/port-group-0 VPlexcli:/clusters/cluster-1/cluster-connectivity/port-groups/port-group-0> set enabled all-enabled VPlexcli:/> cd /clusters/cluster-2/cluster-connectivity/port-groups/port-group-0 VPlexcli:/clusters/Cluster-2/cluster-connectivity/port-groups/port-group-0> set enabled all-enabled Disable port-group-1 on Cluster-1 & 2: VPlexcli:/> cd /clusters/cluster-1/cluster-connectivity/port-groups/port-group-1 VPlexcli: /clusters/cluster-1/cluster-connectivity/port-groups/port-group-1> set enabled all-disabled VPlexcli:/> cd /clusters/cluster-2/cluster-connectivity/port-groups/port-group-1 VPlexcli: /clusters/cluster-2/cluster-connectivity/port-groups/port-group-1> set enabled all-disabled Modify the MTU size in the subnet: VPlexcli:/> cd /clusters/cluster-1/cluster-connectivity/subnets/cluster-1-SN01 VPlexcli:/clusters/Cluster-1/cluster-connectivity/subnets/Cluster-1-SN01> set mtu 9000 VPlexcli:/> cd /clusters/cluster-2/cluster-connectivity/subnets/cluster-2-SN01 VPlexcli:/clusters/Cluster-2/cluster-connectivity/subnets/Cluster-1-SN01> set mtu 9000 Enable the port group: VPlexcli:/> cd /clusters/cluster-1/cluster-connectivity/port-groups/port-group-1 VPlexcli:/clusters/Cluster-1/cluster-connectivity/port-groups/port-group-1> set enabled all-enabled VPlexcli:/> cd /clusters/cluster-2/cluster-connectivity/port-groups/port-group-1 VPlexcli:/clusters/Cluster-2/cluster-connectivity/port-groups/port-group-1> set enabled all-enabled Validate connectivity, again: VPlexcli:/> connectivity validate-wan-com Ensure rebuild transfer size is set to a value below 1M, for example 992K. To set transfer size, Set the transfer-size for all distributed devices to 992K run the below commands. VPlexcli:> cd /distributed-storage/distributed-devices VPlexcli:/distributed-storage/distributed-devices> rebuild set transfer-size * 992k NOTE: Valid values must be multiples of 4K. VPLEX minimum transfer size is 40K
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.