Symptoms

PowerStore Manager UI or CLI (RestAPI and PSTCLI) is not available and may return the following errors: PowerStore Manager UI is not accessible (users cannot log in) after an NDU, node reboot or, appliance power cycle:Attempting to log in fails with error: "Service unavailable. Please retry in a few minutes. (0xE04040010004)" or "503 Service Unavailable"If going through the Initial Configuration Wizard (ICW) using the service port, the user may see a message saying "Loading...." On the Cluster Details step which continues for a time. Eventually the UI may display "The required appliance was not found or Request timed out." As a result of this issue: For unconfigured appliances Impossible to access the PowerStore Manager login page or login into the REST API.Connection Utility cannot show configured and unconfigured appliances in such environment due to absence of discovery IP on appliances. For configured appliances Impossible to access the PowerStore Manager login page or login into the REST API.Add Appliance operation is not possible. Access system using ssh, if ssh is not enabled use service ports as outlined in PowerStore: SSH and PowerStore Manager access using the Service LAN Ports.Unable to issue pstcli commands. [SVC:service@XXXXXXX-B user]$ pstcli -d -u admin -session Password (for host localhost): The system was unable establish a secure connection to the storage server. No IPv4 169.254/16 IP bonded on disc0: [SVC:service@XXXXXXX-A user]$ ip a show | grep disc0 -A 3 11: disc0@eno1: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff inet6 fe80::18df:bcff:fe34:2b4a/64 scope link valid_lft forever preferred_lft forever [SVC:service@XXXXXXX-B user]$ ip a show | grep disc0 -A 3 11: disc0@eno1: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether bb:bb:bb:bb:bb:bb brd ff:ff:ff:ff:ff:ff inet6 fe80::18df:bcff:fe34:2b4a/64 scope link valid_lft forever preferred_lft forever

Cause

This issue only affects PowerStoreOS 3.0 and above. During Node boot up, the avahi-autoipd service will randomly pick up an IP in 169.254.0.0/16 range, test if any other device uses this IP using arping, and then binds it to the disc0 interfaceA disc0 interface IP address is required to load the Control Path HTTP stack.Due to an unexpected ARP response from a network device in the management network environment, the avahi-autoipd service fails to obtain an IP address, which prevents the Control Path HTTP stack from loading and results in loss of WebUI and PSTCLI access.

Resolution

Workaround: Identify the device that is sending unexpected ARP responses to PowerStore.The issue can be resolved by stopping the switch or device acting as proxy ARP in the management network. Do this by either of the following ways: Update switch firmware to the version which contains fix for the issue (relevant for Cisco switches as described in https://quickview.cloudapps.cisco.com/quickview/bug/CSCul01316)If the "proxy ARP" is enabled on the switch for Zeroconf/Avahi IP range of 169.254.0.0/16, it is suggested to be disabled or reconfigured to exclude this range as it should not be used for proxy ARP. Consult with the customer's network administrator to verify that disabling this setting should not impact any other network service.Consult with the customer's network administrator on the best way to perform this change. Isolate PowerStore's management network to a different Access or Native VLAN on the switch. Once all the above are done, verify disc0 has an IPv4 address in the range of 169.254.0.0/16 [SVC:service@XXXXXXX-A user]$ ip a show | grep disc0 -A 3 11: disc0@eno1: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether ea:44:78:0a:05:c0 brd ff:ff:ff:ff:ff:ff inet 169.254.60.18/16 brd 169.254.255.255 scope link disc0:mc valid_lft forever preferred_lft forever [SVC:service@XXXXXXX-B user]$ ip a show | grep disc0 -A 3 11: disc0@eno1: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 1a:df:bc:34:2b:4a brd ff:ff:ff:ff:ff:ff inet 169.254.64.234/16 brd 169.254.255.255 scope link disc0:mc valid_lft forever preferred_lft forever Restart controlpath [SVC:service@XXXXXXX-B user]$ svc_container_mgmt restart CP Waiting for container restart Container controlpath restart returned: 0 Container controlpath is back up Waiting for stack to load Waiting for stack to load Waiting for stack to load Waiting for stack to load [SVC:service@XXXXXXX-B user]$ If unable to identify a network device sending unexpected ARP responses, or any assistance is required for this issue, contact Technical Support or your Authorized Service Representative, and quote this Dell Knowledge Base article ID.Fix:This issue has been fixed in the PowerStoreOS v3.6.0.0(See the MDT-498676 listed in the release note for this operating system version.)

Support Cases