Symptoms

The NetWorker VMware Protection (NVP) integration is configured with the vProxy appliance. The virtual machine (VM) backups intermittently fail to open the VM's source VMDK with the hotadd transport. The scheduled VMware Protection Workflow execution will impacts different VMs during every execution.The vProxy backup session log shows: YYYY-MM-DDTHH:MM:SS INFO: [@(#) Build number: 66] Data Mover: Hard disk 1: Preparing for data movement. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Data Mover: Hard disk 1: VDDK: Failed to open source VMDK "[Datastore] VM_name/VM_name-000002.vmdk": VDDK Error: 13: You do not have access rights to this file. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Data Mover: Hard disk 1: Failed to initialize Block Reader. YYYY-MM-DDTHH:MM:SS INFO: [@(#) Build number: 66] Data Mover: Hard disk 1: Cleaned up after data movement. YYYY-MM-DDTHH:MM:SS INFO: [@(#) Build number: 66] Data Mover: Hard disk 3: Preparing for data movement. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Failed to backup virtual disk "Hard disk 1". YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Data Mover: Hard disk 3: VDDK: Failed to open source VMDK "[Datastore] sVM_name/VM_name_2-000002.vmdk": VDDK Error: 13: You do not have access rights to this file. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Data Mover: Hard disk 3: Failed to initialize Block Reader. YYYY-MM-DDTHH:MM:SS INFO: [@(#) Build number: 66] Data Mover: Hard disk 3: Cleaned up after data movement. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Failed to backup virtual disk "Hard disk 3". The /opt/emc/vproxy/runtime/logs/vbackupd/vbackupd-vddk.log shows: YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_OpenEx: Open a disk. YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SS error -[21154] [Originator@6876 sub=transport] Mounting 42229cff-3484-fb68-ed83-674d7a152728 using transport hotadd failed : . YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO YYYY-MM-DDTHH:MM:SS error -[21154] [Originator@6876 sub=transport] Cannot use mode hotadd to access [Datastore] VM_name/VM_name_2-000002.vmdk: Cannot mount using this method. (Mounting 42229cff-3484-fb68-ed83-674d7a152728 using transport hotadd failed : .) YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO VixDiskLib: Unable to locate appropriate transport mode to open disk. Error 13 (You do not have access rights to this file) (Mounting 42229cff-3484-fb68-ed83-674d7a152728 using transport hotadd failed : .) at 5294. YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_OpenEx: Cannot open disk [Datastore] VM_name/VM_name_2-000002.vmdk. Error 13 (You do not have access rights to this file) (Mounting 42229cff-3484-fb68-ed83-674d7a152728 using transport hotadd failed : .) at 5434. YYYY-MM-DDTHH:MM:SS NOTICE: VDDK INFO VixDiskLib: VixDiskLib_Open: Cannot open disk [Datastore] VM_name/VM_name_2-000002.vmdk. Error 13 (You do not have access rights to this file) at 5478.YYYY-MM-DDTHH:MM:SS INFO: [@(#) Build number: 66] Data Mover: Hard disk 1: Preparing for data movement. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Data Mover: Hard disk 1: VDDK: Failed to open source VMDK "[Datastore] VM_name/VM_name-000002.vmdk": VDDK Error: 13: You do not have access rights to this file. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Data Mover: Hard disk 1: Failed to initialize Block Reader. YYYY-MM-DDTHH:MM:SS INFO: [@(#) Build number: 66] Data Mover: Hard disk 1: Cleaned up after data movement. YYYY-MM-DDTHH:MM:SS INFO: [@(#) Build number: 66] Data Mover: Hard disk 3: Preparing for data movement. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Failed to backup virtual disk "Hard disk 1". YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Data Mover: Hard disk 3: VDDK: Failed to open source VMDK "[Datastore] sVM_name/VM_name_2-000002.vmdk": VDDK Error: 13: You do not have access rights to this file. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Data Mover: Hard disk 3: Failed to initialize Block Reader. YYYY-MM-DDTHH:MM:SS INFO: [@(#) Build number: 66] Data Mover: Hard disk 3: Cleaned up after data movement. YYYY-MM-DDTHH:MM:SS ERROR: [@(#) Build number: 66] Failed to backup virtual disk "Hard disk 3". There are 100 VM backup sessions running on the vCenter, additional VMs may show in a queued state.

Cause

There are too many concurrent hotadd sessions during the scheduled VMware Protection Workflow execution. The "Performance and scalability" section of the Dell EMC NetWorker VMware Integration guide recommends less than 100 concurrent backup sessions per vCenter Server.

Resolution

Reduce the number of concurrent hotadd sessions during the scheduled VMware Protection Workflow execution, by removing vProxies or changing the available hotadd sessions per vProxy.

Support Cases