Symptoms

The daemon.raw shows that nsrvim process takes ~205 minutes to complete MM/DD/YY 06:11:28 1 5 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg NSR notice MM/DD/YY 06:11:28.664803 Running nsrvim.... MM/DD/YY 06:11:28 1 3 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg RAP notice job 'nsrvim' progress message: nsrvim starting on my-nsr.my-domain.com (process 29683). MM/DD/YY 06:11:28 1 3 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Connecting to NetWorker on 'my-nsr.my-domain.com'. MM/DD/YY 06:11:28 1 3 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Querying NSR hypervisor resource 'my-vcenter.my-domain.com' MM/DD/YY 06:11:28 1 3 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Connecting to service at https://my-vcenter.my-domain.com/sdk ... MM/DD/YY 09:36:02 1 3 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Querying for categories from infrastructure services daemon. MM/DD/YY 09:36:02 1 3 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Updating NSR hypervisor resource 'my-vcenter.my-domain.com' MM/DD/YY 09:36:02 1 3 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg RAP notice job 'nsrvim' progress message: Finished updating NSR hypervisor resource 'my-vcenter.my-domain.com' MM/DD/YY 09:36:03 1 3 0 2734675712 414 0 my-nsr.my-domain.com nsrdisp_nwbg RAP notice job 'nsrvim' progress message: nsrvim exiting on my-nsr.my-domain.com (process 29683). The VMware Protection Policy Backup action shows: MM/DD/YYY 10:00:02 AM Action backup vmware-vproxy 'backup' has initialized as 'vproxysave job' with job id 65059338 MM/DD/YYY 10:00:03 AM Calling the nsrvim program to collect the inventory data. MM/DD/YYY 10:00:03 AM Setting default timeout 1800. MM/DD/YYY 10:00:03 AM Using a timeout of 1800 seconds for the nsrvim request. Minimum timeout is 360 seconds. Maximum timeout is 3600 seconds. MM/DD/YYY 10:30:22 AM Failed to run nsrvim, error: Dispatcher: Request timed out. MM/DD/YYY 10:30:22 AM Dispatcher request timed out. MM/DD/YYY 10:30:22 AM Failed to run nsrvim. Will retry in 360 seconds. MM/DD/YYY 10:36:22 AM Calling the nsrvim program to collect the inventory data. MM/DD/YYY 10:36:22 AM Setting default timeout 1800. MM/DD/YYY 10:36:22 AM Using a timeout of 1800 seconds for the nsrvim request. Minimum timeout is 360 seconds. Maximum timeout is 3600 seconds. MM/DD/YYY 11:06:40 AM Failed to run nsrvim, error: Dispatcher: Request timed out. MM/DD/YYY 11:06:40 AM Dispatcher request timed out. MM/DD/YYY 11:06:40 AM Action backup vmware-vproxy 'backup' with job id 65059338 is exiting with status 'failed', exit code 1 MM/DD/YYY 11:06:40 AM Action has finished with failures. The nsrvim Debug 5 execution shows it took ~494 minutes to complete successfully. [root@my_nsr ~]# nsrvim --names my-vcenter.my-domain.com -v -D 5 10/25/22 13:50:55.611491 lg_inet_getaddrinfo(): ENTER 10/25/22 13:50:55.611847 lg_inet_getaddrinfo(): input host=localhost 10/25/22 13:50:55.611861 lg_inet_getaddrinfo(): input service=NULL 10/25/22 13:50:55.611880 lg_inet_getaddrinfo(): input hints-flags=0x0002 AI_PASSIVE=0 AI_NUMERICHOST=0 AI_NUMERICSERV=0 AI_CANONNAME=1 10/25/22 13:50:55.611890 lg_inet_getaddrinfo(): pass-through ... calling external libc getaddrinfo() ... 10/25/22 13:50:55.613508 lg_inet_getaddrinfo(): output cannonname = localhost 10/25/22 13:50:55.613532 lg_inet_getaddrinfo(): output addr = 127.0.0.1 ... 10/25/22 13:51:02.139553 No disks spec found - whole VM work item. 10/25/22 13:51:02.139565 Exit get_clients_in_group 10/25/22 22:04:53.079037 retrieved Folder:Datacenters:group-d1 ... 10/25/22 22:04:54.310944 Unregistered session id:1, fd:3, idx:0, ssn_max_pollfd:0, 0(512) 10/25/22 22:04:54.311001 nsrvim finished with code: 0 [root@my_nsr ~]#

Cause

When the nsrvim exceeds the nsrvim timeout limit the VMware Protection Policy backup action fails to start. The nsrvim execution is taking longer than the default nsrvim timeout to complete against the affected vCenter server. The VMware Protection Policy backup action default nsrvim timeout is 1800 seconds (30 minutes), and the maximum configurable timeout is 3600 seconds (60 minutes).The evaluation of the vCenter vpxd logs show that trivia logging is enabled. As per VMware KB Article 1001584, trivia logging can cause a performance impact if left for a significant amount of time.

Resolution

Revert to the default vCenter logging after troubleshooting has been completed in the vSphere environment.

Support Cases