Loading...
Loading...
Fault sets are in use, and multiple SDSs are exiting PMM at the same time. ESXi logs displaying latency against a PowerFlex volume: 2025-03-26T19:20:58.764Z Wa(180) vmkwarning: cpu33:2098795)WARNING: ScsiDeviceIO: 1772: Device eui.##### performance has deteriorated. I/O latency increased from average value of 350 microseconds to 28802 microseconds. SDS trace logs showing the PMM migration for an example comb - it takes roughly 27 milliseconds to complete the internal migration for comb 746b009a00b6. 2025/03/26 14:20:58.730516 MED:7f19cf3cedb0:contCmd_EndMigration:06164: CombId 746b009a00b6 - Start rc = SUCCESS 2025/03/26 14:20:58.730757 MED:7f19cf3cedb0:raidComb_EndExitPmmInter:02999: Comb 746b009a00b6: PMM becomes Rebuild (PMM remote 41a8b5e300000023), (Old rebuild remote 41a8b5df00000012). 2025/03/26 14:20:58.757184 MED:7f19cf3cedb0:contCmd_EndMigration:06197: CombId 746b009a00b6 - Done rc = SUCCESS Impact Temporary volume latency observed by SDC clients while exiting PMM
During the exit from PMM, the combs are quiesced, preventing I/O operations. Latency is primarily introduced in the final phase of the raidComb_EndExitPmmInter function, where the system hardens bitmaps. While this operation is typically fast, in this case, the delay is more pronounced due to the large number of combs involved, likely caused by a full fault set exiting PMM simultaneously. This increases the likelihood of encountering multiple combs in the hardening stage, amplifying the impact on latency observed by SDC clients.
Possible solutions to prevent this issue: Use Instant Maintenance mode (IMM). Only exit PMM on a single SDS at a time. Impacted Versions PowerFlex 3.x PowerFlex 4.x Fixed In Version PowerFlex 4.5.5
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.