Symptoms

During a NetWorker VMware Protection (NVP) VM backup, the backup fails and the following error is logged in the VM backup session log: NetWorker Server: Linux: /nsr/logs/policy/POLICY_NAME/WORKFLOW_NAME/JOBID_VM-NAME_TIMESTAMP.logWindows (Default): C:\Program Files\EMC NetWorker\nsr\logs\policy\POLICY_NAME\WORKFLOW_NAME\JOBID_VM-NAME_TIMESTAMP.log NOTE: The NetWorker server policy logs adhere to the jobsdb retention (default 72 hours), logs older than the retention window are automatically removed. The backup session logs can also be found on the vProxy appliance and are not impacted by the jobsdb retention. vProxy Appliance: /opt/emc/vproxy/runtime/logs/recycle/vbackupd/DATE/BackupVmSessions-SESSION_ID.log YYYY-MM-DDTHH:mm:SS ERROR: [NETWORKER-BUILD] UUID-VCENTER-NAME:VM-MOREF: HTTP Get request failed to download config file '[DATASTORE-NAME] VM-NAME/VM-NAME.vmx' using URL 'https://VCENTER-NAME/folder/VM-NAME/VM-NAME.vmx?dcPath=DATACENTER-NAME&dsName=DATASTORE-NAME'. HTTP Status code: 500. YYYY-MM-DDTHH:mm:SS ERROR: [NETWORKER-BUILD] Failed to download VM config file "[DATASTORE-NAME] VM-NAME/VM-NAME.vmx" into saveset file "VM-MOREF-config-file-0.cfg". The /var/log/hostd.log on the VMs (ESXi) host reports the following: YYYY-MM-DDTHH:mm:SS Wa(164) Hostd[79374775]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/UUID/VM-NAME/VM-NAME.vmx] File - failed to get objectId, '/vmfs/volumes/UUID/VM-NAME/VM-NAME.vmx': Operation not supported (11) This error coincides with when the HTTP 500 error is observed during the VMs backup. 

Cause

The HTTP 500 (Internal Server Error) in the VM backup session log is returned from the VMware environment. The vProxy backup workflow sends an HTTP GET request to get the VM's configuration files; however the vCenter is responding with HTTP 500 (Internal Server Error)The hostd error message "Operation not supported (11)" typically indicates an issue accessing or manipulating the specified virtual machine file in VMware.There are several causes which could factor into this error; however, most are at the ESXi level: File permissions -The ESXi host does not have the necessary permissions to access the file.Datastore Accessibility - Connectivity issues between the ESXi host and datastore.Management Agents - The ESXi host's hostd and vpxa agents are encountering issues.Disk space issues on the ESXi hostVM configuration file issuesESXi host requires updates

Resolution

WARNING: This issue is occurring at the VMware level so it is highly recommended to engage VMware support. The following steps can be used to aid in diagnosing the issue. The VMware Administrator must perform any action in the VMware environment. NetWorker VMware Protection (NVP) Permissions Verification: The ProxyHC utility can be used to validate backup access permissions. The ProxyHC utility is not provided on the vProxy appliance by default.See NVP-vProxy: How to use health check tool ProxyHC on vProxy appliance ./ProxyHC perm Correct any missing permissions issues identified by ProxyHC. NOTE: ProxyHC does not verify that the ESXi host has permissions to access the VM files. ProxyHC verifies that the user account used to perform NetWorker VMware Protection has the appropriate permissions. The permissions are documented in the NetWorker VMware Integration Guide, available through https://www.dell.com/support/home/product-support/product/networker/docs. This issue can occur when VMware encryption is enabled, but the vCenter user account for NetWorker lacks cryptographic permissions: NetWorker NVP-vProxy: Backup error: Failed to download the VM configuration file. HTTP Status code: 500When VMware encryption is enabled, the user account must have the following permissions: Cryptographic operations > Add diskCryptographic operations > Direct accessCryptographic permissions > Register VM The VMware administrator must perform permission verification of the files on the ESXi host. Datastore Accessibility: From the VMware vSphere Client, check the Datastore tab for any accessibility errors. In case there is an issue where connectivity is intermittent, check the vSphere Event Console for any errors or failures regarding the datastore access. Management Agents: Check the ESXi host's hostd and vpxa agent files for any errors. ComponentLogDescriptionESXi host agent log/var/log/hostd.logContains information about the agent that manages and configures the ESXi host and its virtual machinesvCenter Server agent log/var/log/vpxa.logContains information about the agent that communicates with vCenter Server (if vCenter Server manages the host) In some instances, restarting the agents can resolve the issue: /etc/init.d/hostd restart /etc/init.d/vpxa restart ESXi disk space issues: Ensure that there is sufficient disk space available on the ESXi host. Use the vdf -h command to check disk space. VM configuration file corruption: Verify access to the .VMX file outside of the backup workflow. Using the .vmx file URL from the VM session log, use one or more of the following options to validate the VMX file's integrity: Re-register the VM. Right-click the VM and click Remove from Inventory (DO NOT DELETE FROM DISK!)Browse the Datastore, locate the VM's .vmx file, and add it back to the inventory. On the vProxy appliance open an SSH session, run the following curl command. Replace the vCenter username with the user account used to perform VM backups. Replace the URL with the URL identified in the VM session log: curl -v -k --user VCENTER_USER_ACCOUNT "VMX_URL" Example: admin@nsr-vproxy02:~> curl -v -k --user administrator@vsphere.local "https://vcsa.amer.lan/folder/05b58d65-4eef-c1d2-5070-00505606604d/rhel-client03.amer.lan.vmx?dcPath=vSAN%2520Datacenter&dsName=vsanDatastore" Enter host password for user 'administrator@vsphere.local': * Trying 192.168.9.111:443... * Connected to vcsa.amer.lan (192.168.9.111) port 443 (#0) ... < HTTP/2 200 ... CONTENTS OF .VMX File * Connection #0 to host vcsa.amer.lan left intact NOTE: HTTP 200 indicates a successful GET request of the vmx file. The output also contains the contents of the VM's vmx file. Verify that the contents of the file are good and there are no signs of corruption. If the curl command returns any other HTTP status (404, 500, and so forth) VMware support must be engaged. From a web browser that has access to the vCenter address. Enter/Paste the URL from the VM session log. Enter the credentials of the vCenter user account used for backups: NOTE: If it is successful, the vmx file downloads. Verify that the contents of the file are good and there are no signs of corruption. If the browser returns any other HTTP errors (404, 500, and so forth), VMware support must be engaged. VMware vCenter and ESXi versioning: Consult the NetWorker compatibility matrix for your NetWorker version. https://elabnavigator.dell.com/eln/modernHomeAutomatedTiles?page=NetWorkerIn the NetWorker All Components guide, see NetWorker NVP (Proxy) Compatibility Matrix.Use the latest vProxy major release supported by your NetWorker version and compatible with your ESXi version.Ideally the vCenter Server and ESXi hosts should be on the same version and update release. Workaround: Either of the following workarounds can be tested; however, each option may only serve as a temporary solution: Perform a Host and Storage VMware vSphere vMotion of any VM reporting these failures. After vMotioning the VM, perform a backup from NetWorker.Reboot the ESXi hosts which contain the VMs reporting these issues during backups. After the ESXi host has come back online, confirm if VM backups are successful for VMs residing on this host. In either case, VMware support is recommended to root-cause this issue. 

Support Cases