Loading...
Loading...
Certain SDCs, with a heavier read IO load, will disconnect briefly from SDSs. - Events as seen from the MDM events.txt: 2021-07-10 22:31:22.274 SDC_DISCONNECTED_FROM_SDS_IP WARNING SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.173 of SDS fs03sds058; ID: f34f1b7c00000039 2021-07-10 22:31:23.897 SDC_CONNECTED_TO_SDS_IP INFO SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.173 of SDS fs03sds058; ID: f34f1b7c00000039 2021-07-10 23:14:42.041 SDC_DISCONNECTED_FROM_SDS_IP WARNING SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.112 of SDS fs01sds013; ID: f34f1b4f0000000c 2021-07-10 23:14:44.041 SDC_CONNECTED_TO_SDS_IP INFO SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.112 of SDS fs01sds013; ID: f34f1b4f0000000c 2021-07-10 23:21:13.722 SDC_DISCONNECTED_FROM_SDS_IP WARNING SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.134 of SDS fs02sds027; ID: f34f1b5d0000001a 2021-07-10 23:21:15.561 SDC_CONNECTED_TO_SDS_IP INFO SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.134 of SDS fs02sds027; ID: f34f1b5d0000001a 2021-07-10 23:31:09.561 SDC_DISCONNECTED_FROM_SDS_IP WARNING SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.147 of SDS fs02sds040; ID: f34f1b6a00000027 2021-07-10 23:31:11.353 SDC_CONNECTED_TO_SDS_IP INFO SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.147 of SDS fs02sds040; ID: f34f1b6a00000027 2021-07-11 20:57:22.105 SDC_DISCONNECTED_FROM_SDS_IP WARNING SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.172 of SDS fs03sds057; ID: f34f1b7b00000038 2021-07-11 20:57:23.709 SDC_CONNECTED_TO_SDS_IP INFO SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.172 of SDS fs03sds057; ID: f34f1b7b00000038 2021-07-11 21:02:20.985 SDC_DISCONNECTED_FROM_SDS_IP WARNING SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.181 of SDS fs03sds066; ID: f34f1b8400000041 2021-07-11 21:02:21.996 SDC_CONNECTED_TO_SDS_IP INFO SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.181 of SDS fs03sds066; ID: f34f1b8400000041 2021-07-11 21:04:58.553 SDC_DISCONNECTED_FROM_SDS_IP WARNING SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.145 of SDS fs02sds038; ID: f34f1b6800000025 2021-07-11 21:05:00.473 SDC_CONNECTED_TO_SDS_IP INFO SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.145 of SDS fs02sds038; ID: f34f1b6800000025 - Events as seen from the SDC side: 2021-07-11T21:04:58.553170+00:00 sdc1001 kernel: [19011355.478437] PowerFlex netCon_IsKaNeeded:4321 :CON ffff8c3eb591b900 peer 192.168.1.145 didn't receive message for 30 iterations. Marking as down 2021-07-11T21:04:58.552199+00:00 sdc1001 kernel: [19011355.491770] [23305989309] Disconnected from SDS with ID f34f1b6800000025 2021-07-11T21:04:59.156173+00:00 sdc1001 kernel: [19011355.598778] [23305989416] Connected to SDS with ID f34f1b6800000025 2021-07-11T21:04:59.557004+00:00 sdc1001 kernel: [19011356.492319] PowerFlex mapVolIO_Do_CK:1447 :Mit: ffff8c3d42cc9400. Retrying IO Type READ. Failed comb: 433800a0234. offsetInComb 5017600. SizeInLB 2048. SDS_ID f34f1b6800000025. Comb Gen 1. Head Gen 181. 2021-07-11T21:04:59.557019+00:00 sdc1001 kernel: [19011356.510597] PowerFlex mapVolIO_Do_CK:1461 :Mit: ffff8c3d42cc9400. Vol ID 0xfa49a360000009f. Last fault Status IO_FAULT_NOT_PRI(12).Last error Status ERROR(1) Retry count (1) chan (5) - Using iperf to test SDC to SDS connectivity shows large swings of throughput. A good iperf run shows steady and consistent throughput: # iperf3 -c 192.168.1.145 Connecting to host 192.168.1.145, port 5201 [ 4] local 10.1.3.98 port 50318 connected to 192.168.1.145 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 2.41 GBytes 20.7 Gbits/sec 254 1.43 MBytes [ 4] 1.00-2.00 sec 2.02 GBytes 17.3 Gbits/sec 38 1.20 MBytes [ 4] 2.00-3.00 sec 2.46 GBytes 21.2 Gbits/sec 0 1.40 MBytes [ 4] 3.00-4.00 sec 1.26 GBytes 10.8 Gbits/sec 18 1.42 MBytes [ 4] 4.00-5.00 sec 2.15 GBytes 18.4 Gbits/sec 17 1.22 MBytes [ 4] 5.00-6.00 sec 2.40 GBytes 20.6 Gbits/sec 2 1.00 MBytes [ 4] 6.00-7.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.28 MBytes [ 4] 7.00-8.00 sec 2.28 GBytes 19.6 Gbits/sec 5 1.44 MBytes [ 4] 8.00-9.00 sec 874 MBytes 7.33 Gbits/sec 234 1.46 MBytes [ 4] 9.00-10.00 sec 814 MBytes 6.83 Gbits/sec 2407 1.48 MBytes Impact - Less than optimal application performance on the SDC hosts - Possible IO errors under heavy load on the SDC hosts
In this case, the SDC hosts kernel tuning parameters had been changed away from the OS defaults, which caused the TCP stack to not be able to work optimally under heavy loads.
Remove the sysctl kernel tuning parameters and set them back to default. Consult with the OS vendor on what the various kernel tuning parameters do and how possible changes can affect the OS the SDCs are running on. Impacted Versions VxFlex OS 3.0.x PowerFlex v3.5.x PowerFlex v3.6.x Fixed In Version N/A, Not a PowerFlex issue
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.