Symptoms

The VM backup workflow shows the VMs in a waiting to run state. The backup action details in the NetWorker Management Console show: NOTE: The "vProxy" column remains empty, indicating that a vProxy has not been picked up by the workflow. vProxies are showing enabled: Yes under Devices->VMware ProxiesThere are no "vProxy is unavailable" messages in the NMC Alerts window.The NetWorker server can correctly resolve the vProxy FQDN, shortname, and IP address.The vProxy can correctly resolve the NetWorker server FQDN, shortname, and IP address.Port 9090 is open between the NetWorker server and vProxy appliance. Each system can communicate with the other over this port.

Cause

Before the VM backup session can be established, the NetWorker server must identify and connect to an available vProxy appliance. Code optimizations went in to NetWorker 19.9.0.4 and 19.10.0.1 through NETWORKER-85156.

Resolution

If the NetWorker server is 19.9.0.3 or older, upgrade to 19.9.0.4/19.10.0.1 or later. The vProxy appliances should also be upgraded to the latest version supported by NetWorker. Upgrading the vProxy appliance ensures that the latest security fixes are applied. New vProxy releases also bundle updated ddboost libraries, VMware VDDK version, and VMware Tools. NetWorker and vProxy package downloads are available through: https://www.dell.com/support/home/product-support/product/networker/overview NetWorker: How to Upgrade NetWorker and Best Practices Prior to UpgradeNVP vProxy: How To Upgrade the NVP vProxy Appliance using nsrvproxy_mgmt Ensure that the environment is utilizing DNS. NetWorker server, vProxies, Data Domain systems, VMware vCenters, and VMware ESXi hosts must all be using FQDNs resolvable in DNS. Resouces configured in NetWorker should be added using FQDN. The FQDN, shortname, and IP addresses should all resolve correctly. Any systems utilizing hosts file entries must have correct IP addresses, FQDNs, and aliases populated correctly.Dual stack (IPv4 and IPv6) is not supported by the vProxy appliance, one or the other must be used. In IPv6 enabled VMware environment, the following components should not have any unreachable IPv4 entries in the DNS server: NetWorker server FQDNvProxy appliance FQDNData Domain FQDNvCenter FQDNESXi FQDN The FQDNs listed above should return only AAAA records from the DNS and should not have any unreachable IPv4 records in the DNS.See: NVP vProxy: VM backups sitting in Waiting to Run state in an environment using hosts file entries or dual stack (IPv4 and IPv6) configuration vProxy Selection Debugging: There are several potential causes for the vProxy not being picked up by the workflow. This KB provides a method for further debugging the vProxy selection process when there is no identifiable issue with communication between the NetWorker server and vProxy appliance, and no observed issues with vProxy availability. See the Additional Info section for diagnosing some of the potential causes: vProxy is not enabled.vProxy cannot resolve NetWorker server.NetWorker server cannot resolve vProxy. Port 9090 communication issue. 1. On the NetWorker server, create an empty file called vmbackup_logging with no extension in the NetWorker tmp directory: Linux: nve:~ # touch /nsr/tmp/vmbackup_logging nve:~ # ls -l /nsr/tmp | grep vmbackup -rw------- 1 root root 0 May 23 11:00 vmbackup_logging Windows (Administrator PowerShell): PS C:\Users\Administrator.AMER> New-Item -Path "C:\Program Files\EMC NetWorker\nsr\tmp\vmbackup_logging" -ItemType File Directory: C:\Program Files\EMC NetWorker\nsr\tmp Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 5/23/2024 11:04 AM 0 vmbackup_logging PS C:\Users\Administrator.AMER> NOTE: The commands shown above assume the default NetWorker installation path was used. If NetWorker was installed in another location, modify the commands according to your install location. Monitor the NetWorker server's rendered daemon.raw for "select_vproxy" vProxy messages. Example: nve:~ # tail -10 /nsr/logs/daemon.log 0 05/23/24 11:15:25 nsrd NSR notice 05/23/24 11:15:25.083659 Vmproxy queue: Processing requests in pending queue (priority=2, count=2) for transport mode 0, datastores 'esx02-datastore', vCenter 'vcsa.amer.lan' 0 05/23/24 11:15:25 nsrd NSR notice 05/23/24 11:15:25.083747 Vmproxy queue: Selecting vProxy for request (id=1, manager_id=1716477021) in pending queue (count=2) 0 05/23/24 11:15:25 nsrd NSR notice 05/23/24 11:15:25.083771 select_vproxy: all full '1'. 0 05/23/24 11:15:25 nsrd NSR notice 05/23/24 11:15:25.083790 select_vproxy: vProxy not found. 0 05/23/24 11:15:25 nsrd NSR notice 05/23/24 11:15:25.083809 process_pending: err: 53014. 0 05/23/24 11:15:26 nsrd NSR notice 05/23/24 11:15:26.084025 Vmproxy queue: Processing requests in pending queue (priority=2, count=2) for transport mode 0, datastores 'esx02-datastore', vCenter 'vcsa.amer.lan' 0 05/23/24 11:15:26 nsrd NSR notice 05/23/24 11:15:26.084113 Vmproxy queue: Selecting vProxy for request (id=1, manager_id=1716477021) in pending queue (count=2) 0 05/23/24 11:15:26 nsrd NSR notice 05/23/24 11:15:26.084135 select_vproxy: all full '1'. 0 05/23/24 11:15:26 nsrd NSR notice 05/23/24 11:15:26.084154 select_vproxy: vProxy not found. 0 05/23/24 11:15:26 nsrd NSR notice 05/23/24 11:15:26.084183 process_pending: err: 53014. Linux: /nsr/logs/daemon.raw Windows: C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw NetWorker: How to use nsr_render_log Wait for the workflow to fail; if the workflow does not fail and must be manually cancelled, wait 10-15 minutes then cancel the job. If no immediate cause can be determined from the debug settings applied, contact Dell NetWorker support and provide the following: Rendered daemon.log: NetWorker: How to use nsr_render_log Zipped policy/workflow logs for the workflow where the issue was observed: Linux: /nsr/logs/policy/POLICY_NAME/WORKFLOW_NAME Windows: C:\Program Files\EMC NetWorker\nsr\logs\policy\POLICY_NAME\WORKFLOW_NAME NSRGET from the NetWorker server: NetWorker: How to Use the NSRGet NetWorker Data Collection Tool To remove debugging delete the vmbackup_logging file from /nsr/tmp

Support Cases