Symptoms
The most likely event or error code for this issue is: 0x00304404Description: Node has been physically removed or shut down.Other possible event codes:
0x00307701: XENV is not active.0x00304203: Node has stopped.0x00302b04: The Node has stopped.0x00300D06: The cluster service has stopped. 0x0030c601: The appliance has stopped servicing IOs.
A node reboot can trigger other secondary alerts or dial homes, such as:
Port link failure alerts (Event code 0x00307404). More details in PowerStore Alerts: Port Link Failure.Port health state alerts (Event codes: 0x00305302, 0x00305303, 0x00305402, 0x00305403). More details in PowerStore Alerts: Node Port Health States
Cause
A PowerStore node may reboot unexpectedly due to various reasons.Each unexpected reboot should be investigated separately.See the Additional Info section below for details on what is needed for this investigation.
Resolution
A few options exist to check for unexpected node reboots.
Checking alerts and events from the PowerStore Manager (GUI)
Check the events and alerts that could indicate an unexpected node reboot:
Within PowerStore Manager, check the Monitoring section, and look at the details under the ALERTS and EVENTS tabs.Look for timestamps, error or event codes, messages, and so on. In order to make your searches clearer, use the filter options from within the ALERTS and EVENTS tabs:
Checking for dump files
Check for the existence of system dump files around the time of the errors. Kernel dumps are not included in Data Collects.Log in to the cluster over ssh and run svc_dc list_dumpsYou can also try to find dump files from PowerStore Manager. For more details see PowerStore: How to generate and collect various logs from PowerStore.To login to the nodes over ssh, find the cluster or node IP within PowerStore Manager under Settings > Network IPs. Log in with your preferred ssh client using the service user account and the respective service user password (defined during the setup of your system).
Checking the uptime on both nodes
Run the command uptime on both nodes. This will tell you how long the node had been up for and help confirm possible reboots.This is also useful as some unexpected reboots may not produce a dump file.
Other indicators
A gap in the Performance graphs in PowerStore Manager may also indicate a Node reboot. This should be used for guidance only, and you must confirm with more evidence as suggested above. Performance graphs are available either from Dashboard > PERFORMANCE, or Hardware > Appliance X > Performance.
