Symptoms

For dual SP arrays, one SP of the storage system goes into service mode and the whole system cannot be operated through management interfaces, including CLI, UI, REST API, and SMI-S. This may also manifest as SPs rebooting alternately until both SPs end up in Service Mode.A Unity array with both SPs in service mode will not service I/O, so this would be a Data Unavailable (DU) situation. For VSA, the single SP may reboot into service mode or just stay in normal mode, losing management in either case. The whole system cannot be operated through management interfaces, including CLI, UI, REST API, and SMI-S.SSH or IPMI should work. IPMI works always, SSH may only work after the array was stabilized. This problem is found on OE version 4.0.0.x and is fixed in OE version 4.0.1.x.

Cause

The log file /nbsnas/http/logs/mod_jk.log, which records every request from UI and REST, resides in a file system mounted on /nbsnas of the primary SP. Without a log rotation mechanism, bloating of this file continues to consume the available space of the file system. Other internal consumers start to fail after no space is left on the file system. One of the SPs goes into service mode when detects repeated failures of those components. It was observed in the lab that when this happens and services try to failover to the secondary SP, it too experiences the same symptoms. The SPs reboot a few times alternately, and eventually both go into service mode. Customers see this problem if: always use UI or REST API to configure the storage system, or open the UI in the browser and leave it there without closing. With only UI access, normally It takes a few months for customers to see this problem. If customers use REST API to query data from storage system frequently, this issue happens more quickly. A second issue was found in which upgrading to Unity OE 4.0.1.8320161 may exacerbate the issue as it may duplicate the log file in question during the NDU, therefore accelerating the process. You can confirm if so by checking the space consumption on /nbsbas. If space consumption is minimal or low, you did NOT experience this issue during NDU and therefore nothing else is required.4.0.1.x codes already contain the fix for the main problem, so the log rotation itself is working correctly.If the partition is showing a very high used percentage, then the responsible logs files may have to be deleted (requires Dell support).Example of how to check space usage and what logs to delete can be found in the notes section. Dell has decided to remove Unity OE 4.0.1.8320161 for Unity and UnityVSA from support.emc.com. A revised Unity OE release (4.0.1.8404134) was published in September 2016.

Resolution

To resolve this issue, it is necessary for Technical Support to gain root access to the array.Contact Unity Technical Support and mention this KB article: 489057

Support Cases