Loading...
Loading...
During the snapshot removal step of a Veeam Backup & Replication task, the source vSphere VM loses connectivity temporarily.
Veeam does not remove the snapshot itself; Veeam sends an API call to the vSphere environment to have the action performed. The snapshot removal process significantly reduces the total IOPS that can be delivered by the VM due to additional locks on the VMFS storage resulting from increased metadata updates, as well as the added I/O load of the snapshot removal process itself. In most environments, if you are already over 30-40% IOP load for your target storage, which is not uncommon with a busy SQL/Exchange server, then the snapshot removal process will easily push that into the 80%+ mark and likely much higher. Most storage arrays will see a significant latency penalty once IOPs get into the 80%+ mark, which will, of course, be detrimental to application performance.
The following test should be performed when connectivity to the VM is not sensitive, for instance, during off-peak hours. To isolate the vSphere snapshot removal event, Veeam suggests the following isolation test: Create a snapshot on the VM in question. Leave the snapshot on the VM for the duration of time that a Veeam job runs against that VM. Remove the snapshot. Observe the VM during the snapshot removal. While performing the test above, if you observe the same connectivity issues as during the Veeam job run, the problem is likely to exist within the vSphere environment itself. Review the following list of troubleshooting steps and known issues. If none of the following options resolve the issue, we recommend contacting Broadcom support directly regarding the problem with removing the snapshot.
If the VM being stunned is stored on an NFS 3.0 Datastore, see the section below about a Known Issue with NFS 3.0 and Hotadd. Check for snapshots on the VM while no Veeam job is running and remove any that are found.Veeam Backup & Replication can back up a VM that has snapshots present. However, it has been observed that when vSphere attempts to remove a snapshot created during a Veeam job operation, and a snapshot was already present on the VM before the Veeam job, snapshot stun may occur. Check for orphaned snapshots on the VM. Reference: Finding and listing virtual machine snapshots Reduce the number of concurrent tasks that are occurring within Veeam. This will reduce the number of active snapshot tasks on the datastores. Move the VM to a datastore with more available IOPS, or split the VM's disks across multiple datastores to more evenly distribute the load. If the VM's CPU resources spike heavily during Snapshot consolidation, consider increasing the CPU reservation for that VM. Ensure you are on the latest build of your current version of vSphere, hypervisors, VMware Tools, and SAN firmware when applicable. Move the VM to a host with more available resources. If possible, change the time of day that the VM gets backed up or replicated to a time when the least storage activity occurs. Use a workingDir to redirect Snapshots to a different datastore than the one the VM resides on. Reference: Creating snapshots in a different location than default virtual machine directory for VMware ESXi and VMware ESX
NFS 3.0 Stun: Resolved and Unresolved Variants The snapshot stun issue affecting NFS 3.0 datastores with Virtual Appliance (HotAdd) transport mode was resolved in VMware ESXi 8.0 Update 2b. A related issue remains unresolved on ESXi 8.0 Update 2b and later: a VM residing on an NFSv3 datastore can become unresponsive during snapshot removal when Changed Block Tracking (CBT) is enabled. Broadcom documents this as a limitation of NFSv3 locking for which no fix is currently available. The solution and workarounds below apply to both variants. Broadcom KB323118: Virtual machines residing on NFS storage become unresponsive during a snapshot removal operation Broadcom KB428200: Virtual machines residing on NFSv3 storage become unresponsive during a snapshot removal operation when CBT is enabled
Related Broadcom Articles Snapshot removal stops a virtual machine for long time (323397) Virtual machines residing on NFS storage become unresponsive during a snapshot removal operation (323118) High VM Stun time during snapshot deletion or SVmotion failure on ESXi 6.7U2 or later (317708) Virtual machine becomes unresponsive or inactive when taking memory snapshot (321376) Finding and listing virtual machine snapshots (344559) A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts (343375) VM snapshot stun times correlate with the number of virtual disks (337998) Virtual machines residing on NFSv3 storage become unresponsive during a snapshot removal operation when CBT is enabled (428200)
Veeam Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.