Symptoms

Under certain conditions, the LOx NVRAM cards that are used in Gen5 nodes can lose their firmware flash contents. When this issue occurs, the card shows in OneFS boot sequence as "Generic NVMe Device" and OneFS no longer recognizes it as a valid NVRAM device. Gen5 nodes that are affected by this issue fail to boot with the following error: The node does not have the expected NVRAM device /dev/mnv0. Contact EMC Customer Support immediately. During the boot process, the NVRAM device is identified as generic instead of LOx in console output: nvme0: <Generic NVMe Device> at device 0.0 on pci9

Cause

Isilon Engineering has identified an issue with the firmware file system on this card. If firmware slot 3 is the active slot, this can cause the card firmware to become corrupted when the node is rebooted.

Resolution

Isilon engineering has released Node Firmware Package 10.1.3 which contains an updated version of the LOx card firmware (rp180d01) to proactively remediate this issue. Once this firmware has been applied to a node, it is no longer susceptible to this issue. NOTE: This process initiates a rolling reboot of each node as it is being updated. If you are unable to update firmware or reboot nodes now, contact Isilon Technical Support for assistance with applying a short-term remediation, and mention this KB. NOTE: Do not reboot the nodes before, or in preparation for, updating their firmware. Try to avoid any reboots before the updated firmware has been applied. Anytime a node is rebooted while susceptible to this issue there is a risk of firmware superblock corruption. If this issue occurs, the node seems to boot fine, but any subsequent reboot (such as when the firmware update is installed) leads to an unbootable node even if the firmware update installed successfully. Step 1: Remediate affected nodes.If your cluster contains one or more nodes that have already experienced this issue and are in a nonbooting state due to the "Node does not have expected NVRAM device" error, they must be restored to a usable state before remediation can continue. Use the instructions provided in KB 79815: Gen5 node LOx NVRAM card not detected on boot, shows as 'Generic NVMe Device' instead to bring the cluster nodes back to booting state. Once all affected nodes have been restored to service, return to this document to apply the updated firmware. Step 2: Install the latest Node Firmware Package and update firmware. NOTE: This process initiates a rolling reboot of each node as it is being updated. The LOx card firmware must be updated to Node Firmware Package 10.1.3 (LOx FW rp180d01) or newer on all affected nodes in the cluster to remediate the issue. To do this, install the latest Node Firmware Package on the cluster, and run a node firmware update. The latest Node Firmware Package can be found on the support.emc.com site. Once the node firmware update is complete, the Node Firmware Package should be uninstalled. Download, installation, and firmware update instructions can be found in the accompanying Release Notes document for each NFP. Step 3: Gather logs.When all updates are finished, gather logs using the following command and upload them to EMC Isilon Technical Support for use in future troubleshooting, if needed. isi_gather_info

Support Cases