Symptoms

vCenter SSL certificate was replaced/updated on the vCenter server.All vProxy image based backup performed through the vCenter fail.The VM session logs will show the following error: YYYY-MM-DD HH:MM:SS INFO: [@(#) Build number: ###] There are 2 certificates available at VCENTER_HOSTNAME. First one will be used. YYYY-MM-DD HH:MM:SS ERROR: [@(#) Build number: ###] Failed to disable storage migration for virtual machine "vm-###": VDDK Error: 1: Unknown error. YYYY-MM-DD HH:MM:SS ERROR: [@(#) Build number: ###] Error disabling storage migration for virtual machine "VM_NAME". YYYY-MM-DD HH:MM:SS INFO: [@(#) Build number: ###] Set custom attribute 'Dell EMC vProxy Session' value for object vm-### to ''. YYYY-MM-DD HH:MM:SS INFO: [@(#) Build number: ###] Unlocked virtual machine. YYYY-MM-DD HH:MM:SS INFO: [@(#) Build number: ###] vProxy locks reset successfully. The vProxy /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-vddk.log may show: YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO VixDiskLib : VixDiskLib_PrepareForAccess : Disable Storage VMotion failed. Error 1 (Unknown error) (Other error encountered: SSL Exception: Verification parameters: YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO PeerThumbprint : 2D:5E:84:C7:C7:41:8A:19:9E:02:F9:BB:B1:BD:CD:0C:4E:B3:AB:30 YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO ExpectedThumbprint : 4e:68:1d:93:99:36:53:6a:ec:cd:8f:ae:0b:08:16:ef:75:89:79:1c YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO ExpectedPeerName : vCenter_Name YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO The remote host certificate has these problems: YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO * Host name does not match the subject name(s) in certificate. YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO YYYY MM DDTHH:mm:SSZ NOTICE: VDDK INFO * unable to get local issuer certificate) at 5001. Reviewing the session logs: NetWorker Management Console (NMC): The session log can be reviewed by opening the 'backup action details' from the 'Monitoring' tab, selecting a failed VM backup, then clicking 'show messages', and then 'Get full Log'.vProxy: The session logs can be reviewed from a SSH session on the vProxy, connect to the vProxy as admin and switch to root. Completed/Failed session logs can be found under /opt/emc/vproxy/runtime/logs/recycle/vbackupd/

Cause

vProxy version 4.3.0-34 or older is used. The vProxy sends current vCenter certificate thumbprint to VDDK library. When the vCenter certificate is renewed this thumbprint is no longer valid and VDDK returns an error. In this case vProxy gets a new certificate from vCenter and generates a new thumbprint but older VDDK library (7.0.0-15832853 or older) doesn’t accept it.

Resolution

Solution: Upgrade the vProxies to 4.3.0-36 or newer. Starting 4.3.0-36, the VDDK library version is 7.0.3-19513565 which does not require the above workaround to be performed when a certificate change occurs in the vCenter. NOTE: VDDK versions are updated periodically on vProxy releases. NVP vProxy: How To Upgrade the NVP vProxy Appliance using nsrvproxy_mgmt Workaround: 1. Ensure that the following process is used for replacing the SSL certificate on the vCenter server [VMware KB]: https://kb.vmware.com/s/article/2097936 2. From the NetWorker Management Console's Devices->VMware Proxies tab make note of the configuration settings of the vProxy(s) from the vProxy properties. Delete the vProxy(s). Deleting the vProxy should unregister it from the NetWorker server.3. From the vSphere client reboot the vProxy(s). If the vProxy is not rebooted the issue may persist once added back.4. From the NetWorker Management Console, return to the VMware Proxies tab and add the vProxy(s) back to the NetWorker server matching the configuration settings. Adding the vProxy back will re-register the vProxy5. Run the backups which previously failed.

Support Cases