Loading...
Loading...
Issue affects versions 19.10.0.x, 19.11.0.x, 19.12.0.x and 19.13.0.0. NetWorker Server Protection Policy Expiration process runs, and the media database Deletion messages are seen in the daemon.log at the time of the hang: 71193 12/04/2025 2:47:43 AM 0 0 0 6512 6508 0 BackupServer nsrd NSR info Media Info: Building list of all save sets on disk volume 'VOL1' for device 'rd=DEV1'. 71193 12/04/2025 2:47:43 AM 0 0 0 6512 6508 0 BackupServer nsrd NSR info Media Info: Deleted 157 GB from save set XXXXX on volume VOL1.001 71193 12/04/2025 2:47:43 AM 0 0 0 6512 6508 0 BackupServer nsrd NSR info Media Info: Deleted 314 GB from save set XXXXX on volume VOL1.001 71193 12/04/2025 2:47:43 AM 0 0 0 6512 6508 0 BackupServer nsrd NSR info Media Info: Deleted 388 GB from save set XXXXX on volume VOL1.001 nsrwatch monitoring run against the nsrmmdbd process shows a "Potential Hang Detected" and does not recover. Daemon.log shows devices not responding: Linux: /nsr/logs/daemon.raw Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw NetWorker: How to use nsr_render_log to render .raw log files 12/04/2025 02:50:01 AM 2 5 0 5808 6500 0 BackupServer nsrd NSR warning Operation 791 on device DEV1 is not responding. Daemon.log shows nsrd refusing to accept RPC connections. This may happen 1 or 2 hours after nsrmmdbd stopped responding. 12/04/2025 4:40:22 AM 5 2 107 13876 21656 0 BackupServer nsrmon RPC critical Check whether the client services are running on the host '127.0.0.1'.173680 12/04/2025 4:40:23 AM 3 2 107 26788 7196 0 BackupServer nsrmon RPC error RPC client handle: No connection could be made because the target machine actively refused it. Any command run to the NetWorker server media database, such as mminfo is unresponsive.
A media database deadlock can occur during save set deletion in the Server Protection Policy Expiration task. This issue stems from the NetWorker Server's media database. Under high load, simultaneous delete, backup, and clone operations can cause a media database deadlock. This prevents further updates to the database. The only way to clear the deadlock is to restart the NetWorker daemons. With nsrmmdbd unresponsive, eventually nsrd is unable to continue to create RPC connections and NetWorker backup operations are no longer able to start.
A code fix is available in the following NetWorker releases: 19.11.0.6 19.12.0.2 19.13.0.1 What to do in order to reduce the frequency of the unresponsiveness: Minimizing the number of backups and clone operations active during the Server Protection Policy Expiration process reduces the impact, and where possible running in a window when no backups and clones are active. Stagger the start time of each nsrworkflow to minimize the number of actions (specially clones) that are triggered simultaneously. Ensure the SQLITE MMDB variables specified in the NetWorker Performance Optimization guide have been applied Disable clone cache by creating /nsr/debug/disableclonecache file in the NetWorker server.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.