Loading...
Loading...
Both controllers enter an unresponsive state. From the ME5 event logs: A1643 2025-04-22 15:02:09 658 Warning The system health is degraded and cannot support firmware upgrade. (number of pre-firmware upgrade tests failed: 1, pre-firmware upgrade failed tests: 0x0000000000000002 ) A1642 2025-04-22 15:01:57 658 Warning The system health is degraded and cannot support firmware upgrade. (number of pre-firmware upgrade tests failed: 1, pre-firmware upgrade failed tests: 0x0000000000000002 ) A1641 2025-04-22 15:01:45 658 Warning The system health is degraded and cannot support firmware upgrade. (number of pre-firmware upgrade tests failed: 1, pre-firmware upgrade failed tests: 0x0000000000000002 ) A1640 2025-04-22 14:55:26 81 Informational Kill was released (that is, the partner controller was allowed to boot up), automatic. A1639 2025-04-22 14:53:26 71 Informational Failover completed. (failed or shutdown controller: B) A1638 2025-04-22 14:53:26 77 Informational Write-back cache was initialized for controller B. Write-back data was found. A1637 2025-04-22 14:53:26 71 Informational Failover was initiated. (failed or shutdown controller: B) A1636 2025-04-22 14:53:25 194 Informational Auto-write-through trigger event: partner processor down. A1635 2025-04-22 14:53:25 188 Informational Write-back cache was disabled. A1634 2025-04-22 14:53:25 84 Warning Killed partner controller. (reason: Heartbeat lost)
The primary cause of this issue was unauthorized or aggressive network scanning on the iSCSI network. Key contributing factors include: Lack of Network Isolation: iSCSI traffic was not properly segregated from general enterprise network traffic, making it susceptible to scans. Bursty Traffic Handling: iSCSI is sensitive to high packet rates, and security scans generated excessive requests, leading to resource exhaustion. OID Table Saturation: The storage system’s Object Identifier (OID) table became full due to the flood of connection attempts, causing core failures. (Oid table full errors.) Disruption of inter-core communication (Unresponsive core logs).
To prevent recurrence, implement the following best practices: Isolate the iSCSI Network Ensure iSCSI traffic runs on a dedicated, isolated network (separate VLAN or physical network). Restrict access to only authorized initiators. Avoid Scanning iSCSI Ports Security scans should exclude iSCSI networks to prevent disruptions. If scanning is mandatory, perform it only during maintenance windows with prior coordination. Implement Proper Access Controls Use CHAP authentication for iSCSI initiators. Configure firewall rules to block unauthorized access to iSCSI ports (typically TCP 3260). Monitor for Anomalous Traffic Deploy network monitoring to detect unusual traffic patterns that may indicate scanning or attacks. Best Practices for Secure iSCSI Deployment: Logical Segmentation: Use dedicated VLANs or physical networks for iSCSI to minimize exposure. Management Port Separation: Only expose storage management interfaces to the general network, keeping iSCSI ports isolated. Regular Firmware Updates: Ensure storage controllers and NICs are running the latest firmware to handle traffic efficiently. By following these measures, organizations can maintain high availability and security for their iSCSI storage infrastructure while avoiding performance degradation due to external scans. Best Practice Recommendation: Implementing iSCSI on isolated VLANs or physical networks ensures better performance and protects storage infrastructure from unintended disruption. Prevent scanning tools from targeting this network segment to avoid flooding logs and crashing cores. Proper segmentation, security policies, and scan exclusions are critical for a healthy storage environment.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.