Symptoms

The NetWorker VMware Protection (NVP) integration is configured with the vProxy Appliance.The virtual machine (VM) backups intermittently fail with VDDK "Error 1: Unknown error."The backups seem to complete successfully when the NSR Hypervisor "Deployed in cloud" option is selected, but this is not a cloud deployment.The backup session log shows: YYYY-MM-DD HH:MM:SS INFO: [@(#) Build number: 122] Data Mover: Hard disk 1: Preparing for data movement. YYYY-MM-DD HH:MM:SS ERROR: [@(#) Build number: 122] Data Mover: Hard disk 1: VDDK: Failed to open source VMDK '[MY-DATASTORE] MY-VM-NAME/MY-VM-NAME.vmdk': VDDK Error: 1: Unknown error. YYYY-MM-DD HH:MM:SS ERROR: [@(#) Build number: 122] Data Mover: Hard disk 1: Failed to initialize Block Reader. The vbackupd-vddk.log shows: YYYY-MM-DD HH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_ConnectEx: Establish connection using hotadd:nbd. YYYY-MM-DD HH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_Connect: Establish connection. YYYY-MM-DD HH:MM:SS NOTICE: VDDK INFO VixDiskLib: Resolve host. YYYY-MM-DD HH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_Connect: Failed to start session. Other error encountered: HTTP error response: Service Unavailable at 4838. YYYY-MM-DD HH:MM:SS NOTICE: VDDK INFO 2023-03-28T14:06:18.966Z error -[02910] [Originator@6876 sub=Default] Cannot use advanced transport modes for xxx.xxx.xxx.xxx/moref=vm-3589/snapshot-4324: Other error encountered: HTTP error response: Service Unavailable. YYYY-MM-DD HH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_ConnectEx: Advanced transport modes not available for opening moref=vm-3589. (Cannot use advanced transport modes for xxx.xxx.xxx.xxx/moref=vm-3589/snapshot-4324: Other error encountered: HTTP error response: Service Unavailable.) The vSphere vCenter vpxd*.log shows: YYYY-MM-DD HH:MM:SS-04:00 info vpxd[06382] [Originator@6876 sub=vpxLro opID=294c6a23] [VpxLRO] -- BEGIN task-1444391 -- snapshot-4324 -- vim.vm.Snapshot.remove -- 5275149d-19b4-1037-b078-198deed7fd29(5269d6ec-0568-eef0-1dba-ba2008dacfd8) YYYY-MM-DD HH:MM:SS-04:00 error vpxd[07882] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000 YYYY-MM-DD HH:MM:SS-04:00 error vpxd[06374] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000 YYYY-MM-DD HH:MM:SS-04:00 error vpxd[07865] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000 YYYY-MM-DD HH:MM:SS-04:00 error vpxd[06303] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000 YYYY-MM-DD HH:MM:SS-04:00 info vpxd[06388] [Originator@6876 sub=vpxLro opID=lf70lc8q-d3bs-h5:70059312-fc] [VpxLRO] -- FINISH lro-7249334 YYYY-MM-DD HH:MM:SS-04:00 info vpxd[32419] [Originator@6876 sub=vpxLro opID=lf70lc8q-d3bt-h5:70058396-c] [VpxLRO] -- FINISH lro-7249335 YYYY-MM-DD HH:MM:SS-04:00 warning vpxd[06356] [Originator@6876 sub=vmomi.soapStub[4633] opID=TaskLoop-host-31] SOAP request returned HTTP failure; , >), /vpxa>, method: waitForUpdates; code: 500(Internal Server Error) YYYY-MM-DD HH:MM:SS-04:00 info vpxd[06382] [Originator@6876 sub=vpxLro opID=294c6a23] [VpxLRO] -- FINISH task-1444391 Location of vCenter Server log files

Cause

The NVP solution uses HTTP SOAP requests in the backup workflow to complete required tasks.The vCenter vpxd.log shows the vProxy HTTP SOAP request reaches the vCenter server.However, the vCenter to ESXi operation returns an HTTP return code 500 (Internal Server Error). The HTTP return code between the vCenter and ESXi host is due to the vCenter server running out of the default 2000 HTTP sessions.

Resolution

The vCenter Server default SOAP session count can be increased using the vCenter vpxd.cfg file. These steps must be performed by the vCenter Administrator. If required, contact VMware Support for more information about the maxSessionCount setting. Open an SSH session to the vCenter Appliance.Create a backup of the vpxd.cfg file: cp /etc/vmware-vpx/vpxd.cfg /var/core/vpxd.cfg.bak Stop the vpxd service: service-control --stop vmware-vpxd Change the maxSessionCount to a higher value (ex: 3000, 4000, or 6000) in the vpxd.cfg file: ... 6000 ... ... Start the vpxd service: service-control --start vmware-vpxd If evaluation is required on the out of SOAP messages, VMware Support must be engaged.

Support Cases