Symptoms

The Linux image backup for the IDPA system is "Completing with Exception" due to the quiescing problem.This example the vm was the internal ACM vm that comes with IDPA solution From the vmware linux image log of the backup the following can be seen:avvcbimage Info : VM Config Info: VMX version 4(F), ctk enabled(T), fault tolerant (F), Guest OS SUSE Linux Enterprise 12 (64-bit), Cloud UUID = NoUUID. version=9avvcbimage Info : VM Guest Info: GuestOS Family = linuxGuest, Guest Full Name = SUSE Linux Enterprise 12 (64-bit), Guest OS State = running, Tools Running = guestToolsRunning, Tools Version Status = guestToolsUnmanagedavvcbimage Info : Creating snapshot 'Avamar-', quiesceFS=1avvcbimage Info : SnapshotManager: createSnapshot for vmMoref = vm-21avvcbimage Info : SnapshotManager: POST to /snapshotmanager/vm/vm-21/action/createSnapshot, payload({ "snapshotName": "Avamar-", "snapshotDesc": "Created by Avamar workorder Management_VM_Backup-Management_VM_Backup- on ", "quiesceFileSystem": "true"...avvcbimage Warning : vSphere Task failed (quiesce, snapshot error=45): 'An error occurred while saving the snapshot: Failed to quiesce the virtual machine.'.avvcbimage Error : Snapshot 'Avamar- ' creation for VM '[vsanDatastore] DataProtection-ACM/DataProtection-ACM.vmx' task creation encountered a quiesce problemavvcbimage Warning : The VM could not be quiesced prior to snapshot creation and this backup will not be used as a base for subsequent CBT backups if successful.avvcbimage Warning : Consider using the pre-freeze and post-thaw scripts to facilitate VM quiescingavvcbimage Warning : This error might caused by the pre-freeze script returning a non-zero error code....After 1st snapshot task fails, the backup proceeds by taking second snapshot but with quiesce feature disabled.avvcbimage Info : VM Disk Consolidation for '[vsanDatastore] DataProtection-ACM/DataProtection-ACM.vmx' task still in progress, sleep for 3 secavvcbimage Info : Creating snapshot 'Avamar-', quiesceFS=0avvcbimage Info : SnapshotManager: createSnapshot for vmMoref = vm-21avvcbimage Info : SnapshotManager: POST to /snapshotmanager/vm/vm-21/action/createSnapshot, payload({ "snapshotName": "Avamar-", "snapshotDesc": "Created by Avamar workorder Management_VM_Backup-Management_VM_Backup- on ", "quiesceFileSystem": "false"The backup finishes but shows as completed with exception ( code 10020) In the workorder log (.alg) the following is seen: 2019-04-03 00:12:42 Management_VM_Backup-Management_VM_Backup-Warning : Workorder "Management_VM_Backup-Management_VM_Backup-epoch" non-zero exit status 'code 170: completed with errors, client log should be examined'

Cause

1. To see what is causing the quiescing to fail, set the debug logging for the vmtools. Following steps in https://kb.vmware.com/s/article/1007873 to set the debug logging on the guest operating system and then restart vmware tools. 2. Do a new backup to recreate quiesce failure. 3. Review debug tools logs ( quiesce information should be in the vmsvc log and have a [vmbackup] prefix ).In this example we can the issue was was caused by a partially deleted /var/log/brs directory. This directory could not be fully removed due background process still had directory open. This partially deleted folder will prevent Linux filesystem freeze with ioctls functions. [ debug] [vmbackup] Submitted backup start task.[ debug] [vmbackup] *** VmBackupSyncDriverStart[ debug] [vmsvc] SyncDriver: Skipping remote filesystem, name=systemd-1, mntpt=/proc/sys/fs/binfmt_misc.[ debug] [vmsvc] SyncDriver: Calling backend 0.[ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...[ debug] [vmsvc] SyncDriver: opening path '/var/lib/named/proc'.[ debug] [vmsvc] SyncDriver: freezing path '/var/lib/named/proc'.[ debug] [vmsvc] SyncDriver: freeze on '/var/lib/named/proc' returned: 95 (Operation not supported)[ debug] [vmsvc] SyncDriver: opening path '/var/log/logbundles'.[ debug] [vmsvc] SyncDriver: freezing path '/var/log/logbundles'.[ debug] [vmsvc] SyncDriver: successfully froze '/var/log/logbundles'.[ debug] [vmsvc] SyncDriver: opening path '/var/log/brs (deleted)'.[ debug] [vmsvc] SyncDriver: failed to open '/var/log/brs (deleted)': 2 (No such file or directory)[ warning] [vmbackup] Error freezing filesystems.

Resolution

Closer examination of the /var/log/brs the directory with can see problems opening this directory from other commands. like lsof and stat also show issue. IDPA-ACM:/var/log # lsof|grep brslsof: WARNING: can't stat() xfs file system /var/log/brs (deleted) Output information may be incomplete.IDPA-ACM:/var/log # ls -aldrwxr-xr-x 0 root root 6 Feb 27 2018 brsThe ACM Virtual Machine was rebooted to kill background acm process. After reboot the /var/log/brs directory was recreated by ACM process and the backup was able to complete without further quiesce issuesIDPA-ACM:/var/log # ls -aldrwxr-xr-x 2 root root 6 Apr 15 10:30 brs

Support Cases