Loading...
Loading...
The NetWorker VMware Protection (NVP) integration is configured with the vProxy Appliance. Backups were taken using the "hot add" transport mode. Also, the overall disk throughput is good. The throughput ( DiskStats ) can be found in the VM backup session logs: NetWorker server: Linux: /nsr/logs/policy/ POLICY-NAME / WORKFLOW-NAME / SESSION-ID-VM_Name-DATE. log Windows (Default): C:\Program Files\EMC NetWorker\nsr\logs\policy\ POLICY-NAME \ WORKFLOW-NAME \ SESSION-ID-VM_Name-DATE. log The NetWorker server policy logs follow the jobsdb retention (72 hours). Logs older than the retention period are purged; however, the vProxy retains these logs for longer. vProxy appliance: /opt/emc/vproxy/runtime/logs/recycle/vbackupd/ DATE /BackupVMSessions- UUID .log NOTE: The vProxy performs hot add backups by default, even when Network Block Device (NBD) is enabled, the vProxy requests hot add first. If all hot add sessions are consumed, NBD sessions may be used. Hot add backups typically have better performance than NBD. Some operations are taking longer than expected, such as "disabling storage migration" with up to 20 mins and VDDK connections reaching a VDDK wait of 24 mins. In vProxy /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-vddk.log , the following warning is frequently observed: YYYY-MM-DDTHH:MM:SSZ NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SSZ warning -[02967] [Originator@6876 sub=vimaccess] cannot get thumbprint: SSL error code '151441516', exception: 'Wrong X.509 Certificate format' YYYY-MM-DDTHH:MM:SSZ NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SSZ warning -[02982] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3 YYYY-MM-DDTHH:MM:SSZ NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SSZ warning -[16072] [Originator@6876 sub=vimaccess] GetResponse: Failed to execute CURL command, treat CEIP as disabled: Couldn't connect to server. Also, there was no " VDDK INFO VixDiskLib: VixDiskLib_Disconnect: Disconnect " messages reported at the end of disks backup.
The issue began after VMware added a new vCenter from a different domain using Linked Mode, inaccessible to vProxy and the backup service account. After enabling debug on vbackupd engine and VDDK, we observed that vProxy was trying to connect to the newly linked vCenter server when querying the CEIP status on vCenter as indicated by the following snippet. As this vCenter server is not reachable from vProxy, some operations appeared to be delayed by the failure to establish this communication. YYYY-MM-DDTHH:MM:SSZ NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SSZ info -[19317] [Originator@6876 sub=vimaccess] retrieve endpoint url: 'https:// <new vCenter server> :443/analytics/ceip/api/state' . thumbprint ''
Adding a new entry or an alias in vProxy's /etc/hosts file to resolve the newly linked vCenter to the IP address of the original vCenter server where vProxy is deployed (fake IP). This allows the vProxy to connect to its vCenter server during CEIP calls. Upon this modification, the "Couldn't connect to server" message was no longer reported into vbackupd-vddk.log. Also, disk closure and disconnect messages were properly logged.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.