Loading...
Loading...
GPU unresponsiveness may occur after 60+ days of continuous uptime on Blackwell GPUs with NVLink. Based on the GPU driver version, the issue may manifest with the following errors:An nvidia-smi command unresponsiveness may occur, causing all GPU-related jobs to stall or freeze.With Driver R580, an XID 150 error may appear, accompanied by the message "MSE Degraded, GPU Reset Required."This error will be visible from nvidia-bug-report.sh output, dmesg output, or in the Linux OS messages file.MSE (Management Service Engine) is a firmware service in charge of NVLink management. It is responsible for servicing both NVLinkmanagement packets as well as in-band and out-of-band NVLink status queries (link status, error counters, etc).This issue is workload agnostic and can happen on systems that are idle or actively running jobs over 60+ days.This issue occurs because NVLink management microcode in the GPU firmware where a counter was monitoring total execution time of NVLink message processing tasks. The microcode lacked a check for overflow such thatafter an extended uptime (60+ days), caused undefined NVLink task scheduling behavior, leading to GPU driver unresponsiveness with XID 150 (R580 driver and later).Refer to NVIDIA release PID 1150364: This states that customers may experience GPU stalls or unresponsiveness after 60+ days of continuous uptime. To gain access to PID 1150364 you must have anhttps://partners.nvidia.comaccount.
Any HPE systems with any of the following GPUs:GB200 NVL72GB300 NVL72GB200 NVL4HGX B200 and B300 8-GPU
The following firmware releases includes the updated microcode that resolves this issue:GB200 NVL72 : 1.3.6 or laterGB300 NVL72 : 1.0.6 or laterGB200 NVL4 : TBDHGX B200 and B300 8-GPU: 1.4.3 or laterUntil the fix is available and deployed, NVIDIA recommends that customers proactively reset all GPUs at least once every 60+ days. Alternatively, reactively reboot only failed nodes.
Operating Systems Affected:OS Independent
Click on a version to see all relevant bugs
Hewlett Packard Enterprise Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.