Symptoms

The NetWorker VMware Protection (NVP) vProxy was found to be unresponsive and had to be shut down and restarted to restore it to operation.The vProxy /opt/emc/vproxy/runtime/logs/vrapid/vrapid-engine.log shows a gap in time when the migration was in progress, then show its processes being restarted. 2017/09/14 19:41:56 INFO: The state of the vProxy is: READY 2017/09/14 19:41:56 TRACE: Retrieving process status successful 2017/09/15 00:33:27 NOTICE: Logging to '/opt/emc/vproxy/runtime/logs/vrapid/vrapid-engine.log' on host 'vProxy_name' 2017/09/15 00:33:27 INFO: Daemon logs will be rotated after 1048576 bytes and kept for 365 days. The VMware logs (vmware.log) for the vProxy show a failed migration. VMware responded to the failed migration by forcing the vProxy to shut down: 2017-09-14T19:39:11.352Z| vmx| I125: MigrateVMXdrToSpec: type 1 unsharedSwap 0 memMinToTransfer 0 cpuMinToTransfer 0 numDisks 0 numStreamIps 1 2017-09-14T19:39:11.352Z| vmx| I125: Received migrate 'to' request for mid id ################, src ip <###.###.###.###>, dst ip <###.###.###.###>. 2017-09-14T19:39:11.352Z| vmx| I125: MigrateSetInfo: state=1 srcIp=<###.###.###.###> dstIp=<###.###.###.###> mid=################ uuid=########-####-####-####-############ priority=low 2017-09-14T19:39:11.352Z| vmx| I125: MigrateSetState: Transitioning from state 0 to 1. ... 2017-09-14T19:42:13.579Z| vcpu-0| I125: Msg_Post: Error 2017-09-14T19:42:13.579Z| vcpu-0| I125: [msg.disklib.CAPACITY_MISMATCH] The capacity of the parent virtual disk and the capacity of the child disk are different 2017-09-14T19:42:13.579Z| vcpu-0| I125: [msg.disk.noBackEnd2] Cannot open either the disk '/vmfs/volumes/DATASTORE-UUID/Datastore_Folder/Disk_File.vmdk' or the redo log '. 2017-09-14T19:42:13.579Z| vcpu-0| I125: [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running. ... 2017-09-14T19:42:14.062Z| vmx| I125: Transitioned vmx/execState/val to poweredOff 2017-09-14T19:42:14.062Z| vmx| I125: VMXVmdbCbVmVmxExecStateVal: no more client connections. 2017-09-14T19:42:14.062Z| vmx| I125: VMX idle exit 2017-09-14T19:42:14.062Z| vmx| I125: VMIOP: Exit 2017-09-14T19:42:14.073Z| vmx| I125: Vix: [1860575 mainDispatch.c:843]: VMAutomation_LateShutdown() 2017-09-14T19:42:14.073Z| vmx| I125: Vix: [1860575 mainDispatch.c:793]: VMAutomationCloseListenerSocket. Closing listener socket. 2017-09-14T19:42:14.074Z| vmx| I125: Flushing VMX VMDB connections 2017-09-14T19:42:14.074Z| vmx| I125: VigorTransport_ServerCloseClient: Closing transport 32279F40 (err = 0) 2017-09-14T19:42:14.074Z| vmx| I125: VigorTransport_ServerDestroy: server destroyed. 2017-09-14T19:42:14.078Z| vmx| I125: VMX exit (0). 2017-09-14T19:42:14.078Z| vmx| I125: AIOMGR-S : stat o=2170 r=3339 w=262 i=795 br=54706176 bw=62685184 2017-09-14T19:42:14.078Z| vmx| I125: OBJLIB-LIB: ObjLib cleanup done. 2017-09-14T19:42:14.078Z| vmx| W115: VMX has left the building: 0. NOTE: The vmware.log can be downloaded from the vProxy Virtual Machine's (VM) datastore folder in VMware vSphere. Even if the migration was successful, a migration of the vProxy while it is in operation can cause the backup or recovery to fail. Any of the following symptoms can occur with the VMs that were "in-progress" when the vProxy was migrated: The Migration Status flag being left set as false See Article: NVP-vProxy: Error disabling storage migration, retrying, error: VDDK Error: 20005: The operation has been disabled by the guest operating system. The Annotation fields on the VM left showing a backup is in progress. See Article: NVP vProxy: Failed to lock Virtual Machine for backup: Another vProxy operation is active on the VM Client disks attached to the vProxy See Article: NVP-vProxy: How to use health check tool ProxyHC on vProxy appliance VMware snapshots created for the vProxy backup left unconsolidated as a result of client disks left attached to the vProxy.

Cause

The vProxy VM was migrated while it was backing up. In this specific case, the migration failed and VMware was unable to recover from the error and forced the vProxy to shut down.

Resolution

Migration is a VMware process that is external to the NetWorker vProxy and its operations. Do not migrate the vProxy while it is performing backups or recoveries as this can cause those operations to fail. If the VMware configuration is using Distributed Resource Scheduler (DRS) or Storage (SDRS), the vProxies must be excluded.If VMware High Availability (HA) is configured, VM monitoring of the vProxy should be disabled to prevent a forced restart if the vProxy fails to respond to the HA monitoring check.If a migration of the vProxy must be performed, ensure that there are no backups or recoveries running. The vProxy should be idle when migrated. NOTE: If DRS or SDRS is enabled in VMware and the vProxy is not excluded the following issue may be observed during VM restores: NVP vProxy: VM Disk is Removed or Restored as an Empty Drive For more information regarding the migration error, open a support case with VMware to investigate the problem.

Support Cases