Loading...
Loading...
NetWorker Virtual Edition (NVE) NvePlatformOsRollup fails, reporting "One or more nodes are not running SLES failed" The NVE /data01/avamar/repo/temp/NvePlatformOsRollup*/tmp/workflow.log reports: YYYY-MM-DD HH:mm:SS (TZ) 47180850010780 INFO: - - - - - - - END YYYY-MM-DD HH:mm:SS (TZ) 47180850010780 INFO: "uname -a", exit status=0 (success) YYYY-MM-DD HH:mm:SS (TZ) 47180850010780 INFO: os=unknown version=unknown update=unknown arch=x86_64 ... YYYY-MM-DD HH:mm:SS (TZ) 47180850010780 INFO: post: - One or more nodes are not running SLES failed The /data01/avamar/repo/temp/NvePlatformOsRollup*/tmp/err.log shows that it failed to run uname -a : YYYY-MM-DD HH:mm:SS (-TZ) 47198950636200 ERROR: "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@< NVE_IP_Address > uname -a", exit status=255 (error)
NOTE: the NVE rollup can fail with "One or more nodes are not running SLES failed" for multiple reasons. This article focuses on one specific cause; the NVE's /usr/local/avamar/var/probe.xml does not contain the correct IP address. If the below information does not apply to your NVE, see the Additional Information section for articles regarding other causes. During the rollup, the NVE runs several commands to validate its authenticity. If uname -a does not return the correct output. The rollup fails with error "One or more nodes are not running SLES failed." The rollup's workflow log reports that it fails to get the system information. It attempts to use the wrong IP address to connect to the NVE and check the operating system details prior to updating. The /data01/avamar/repo/temp/NvePlatformOsRollup*/tmp/workflow.log shows YYYY-MM-DD HH:mm:SS (TZ) 47106469567160 INFO: post: "ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@ 123.123.9.16 uname -a", exit status=255 (error) YYYY-MM-DD HH:mm:SS (TZ) 47106469567160 INFO: address=123.123.9.16 remote=ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@123.123.9.16 os=unknown version=unknown update=unknown arch=unknown YYYY-MM-DD HH:mm:SS (TZ) 47106469567160 INFO: - One or more nodes are not running SLES succeeded The IP address shown is not the current IP address of the NVE appliance. It may be an older IP address used previously. nve:~ # ifconfig -a eth0 Link encap:Ethernet HWaddr 00:50:56:BB:B4:63 inet addr: 123.123.0.16 Bcast:123.123.0.255 Mask:255.255.255.0 inet6 addr: fe80::250:56ff:febb:b463/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2713 errors:0 dropped:0 overruns:0 frame:0 TX packets:1309 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:492699 (481.1 Kb) TX bytes:1463814 (1.3 Mb)lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:21317 errors:0 dropped:0 overruns:0 frame:0 TX packets:21317 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5696590 (5.4 Mb) TX bytes:5696590 (5.4 Mb) The rollup is getting the wrong IP address from the /usr/local/avamar/var/probe.xml . nve:~ # cat /usr/local/avamar/var/probe.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <dpn> <module name="nve" userinput_domain="networker.lan" userinput_search="networker.lan"> <node type="single-node server"> <network-interface id="0" userinput_ifname="eth0"> <address value=" 123.123.9.16 " userinput_netmask="255.255.255.0" userinput_customhostname="nve.networker.lan"/> <uses allow="replication,internal,backup,management"/> </network-interface> </node> </module> </dpn> When running the same command from the workflow, it does not return uname -a output. ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@ WRONG_IP_FROM_WORKFLOW.LOG uname -a Example: nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@ 123.123.9.16 uname -a nve:~ # To further validate that this is the cause. Using localhost in the command returns the uname output: nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@localhost uname -a Enter passphrase for key '/root/.ssh/rootid': HIDDENT_ROOTID_PASSPHRASE Linux nve 4.12.14-122.283-default #1 SMP Thu Nov 27 17:17:01 UTC 2025 (cddd1eb) x86_64 x86_64 x86_64 GNU/Linux NOTE: If uname output is not returned when using localhost , then there may also be an issue with the NVE's rootid passphrase, see: NVE: NvePlatformOsRollup fails "One or more nodes are not running SLES" due to corrupted rootid passphrase
CAUTION: The NVE collects the IP address from probe.xml at the start of the rollup, once the IP address is collected, it does not "recollect." In order to resolve this scenario, you must have a valid NVE snapshot from before the upgrade to revert to. Creating a snapshot before the upgrade is best practice, see: How to Install Platform Security Rollup for NetWorker Virtual Edition . Reverting the NVE to a snapshot will rollback the NVE to the point-in-time the snapshot was created. Any successful NetWorker backups performed since the snapshot was created are lost. If you do not want to lose these backups, collect the bootstrap save set details ( mminfo -B ) a NetWorker Server Disaster Recovery ( nsrdr ) must be performed after reverting to the snapshot and successfully applying the rollup. Carefully read the below instructions before reverting to any snapshot. If you did not create a snapshot before applying the rollup. A new NVE must be deployed with the same hostname and configuration as the existing NVE. After deploying and completing post-deployment configuration, an nsrdr can be performed to recover the configuration and backups from the old NVE to the new one. Perform a Server Protection -> Server backup of the NVE . Open an SSH session to the NVE, log in as admin . Switch to root : sudo su - Collect bootstrap save set details: mminfo -B (Save the output of this command). Revert the NVE to the snapshot taken before the rollup. Reconnect to the NVE over SSH and switch to root: sudo su - Confirm the current IP address of the NVE: ifconfig -a or ip a If /usr/local/avamar/var/probe.xml exists, open it with a text editor and ensure that the configuration values (IP address, search domain, hostname) are all correct: vi /usr/local/avamar/var/probe.xml Replace the highlighted values with the correct ones for your NVE: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <dpn> <module name=" SERVER_SHORT_NAME " userinput_domain=" DOMAIN.DOMAIN " userinput_search= " DOMAIN.DOMAIN "> <node type="single-node server"> <network-interface id="0" userinput_ifname="eth0"> <address value=" IPv4_ADDRESS " userinput_netmask= " SUBNET_ADDRESS " userinput_customhostname=" FULLY_QUALIFIED_DOMAIN_NAME "/> <uses allow="replication,internal,backup,management"/> </network-interface> </node> </module> </dpn> Example: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <dpn> <module name="nve" userinput_domain="networker.lan" userinput_search="networker.lan"> <node type="single-node server"> <network-interface id="0" userinput_ifname="eth0"> <address value="123.123.0.16" userinput_netmask="255.255.255.0" userinput_customhostname="nve.networker.lan"/> <uses allow="replication,internal,backup,management"/> </network-interface> </node> </module> </dpn> Ensure that the file is owned by admin: chown admin:admin /usr/local/avamar/var/probe.xml Ensure that the file permissions are correct: chmod 644 /usr/local/avamar/var/probe.xml Ensure that the /etc/ssh/sshd_config has the correct IP address defined in the Match Address field (if defined): NVE: NvePlatformOsRollup fails "One or more nodes are not running SLES" due to incorrect SSHD_CONFIG settings Before attempting the rollup, ensure that the SSH command from the workflow returns the correct uname -a output: ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@ NVE_IP_ADDRESS uname -a NOTE: You may be prompted to enter to the root passphrase. Example: nve:~ # ssh -q -i /root/.ssh/rootid -x -q -o ConnectTimeout=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no root@123.123.0.16 uname -a Enter passphrase for key '/root/.ssh/rootid': Linux nve 4.12.14-122.201-default #1 SMP Tue Mar 12 13:01:34 UTC 2024 (b92bd5a) x86_64 x86_64 x86_64 GNU/Linux WARNING: If this command does not return the uname -a output (either no output or an error), OS updating will not be performed when the next rollup is applied. Additional Perform the NVE rollup. After the rollup succeeds create another temporary snapshot of the NVE. If you need to recover backups that were completed before the snapshot rollback, perform a nsrdr : NetWorker: NetWorker Server Disaster Recovery (NSRDR) Once all actions are completed and the NVE is returned to full operation, the NVE VM snapshots can be removed.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.