Symptoms
VMware Image backups on vCenter 6.5 or later failing with the following FATAL error message when performing a snapshot with quiesce feature disabled.
avvcbimage Info : Creating snapshot 'Avamar-', quiesceFS=FALSE
avvcbimage Info : Using vSphere CreateSnapshotTaskEX: snapshot name (Avamar-)
avvcbimage Info : Snapshot 'Avamar-' creation for VM '[Avamar] winvm/winvm.vmx' task still in progress, sleep for 3 sec
avvcbimage Warning : vSphere Task failed (application quiesce): An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.
avvcbimage Warning : Snapshot 'Avamar-' creation for VM '[Avamar] winvm/winvm.vmx' task creation encountered a quiesce problem
avvcbimage Error : Crash level consistent snapshot quiesce failure.
avvcbimage FATAL : [IMG0003] The VMX '[Avamar] winvm/winvm.vmx' could not be snapshot
Cause
Avamar proxy is not using a service called 'SnapShot Manager'. When this service is not being used, the logs show the following:
avvcbimage Info : NOT using SnapShot Manager - using proxy method of calling vSphere API functions
SnapShot Manager is a service running on Avamar utility nodes and is accessible via TCP 8543 (https://avamar.example.com:8543/snapshotmanager/). When this service is used by all proxies, all snapshot operations (create, delete, consolidate) tasks are centralized to the Avamar server. When this service is not used, the proxy takes its own snapshot operations via standard vSphere API.There is an Avamar software defect with the vSphere API quiesce_fs=false snapshot mechanism introduced in:
Avamar 18.2 proxy with hotfix 309856 (or later)Avamar 19.1 proxy
Avamar Snapshot Manager may be unavailable due to two potential causes:
The Network or firewall in which case, the following is seen in the logs:
avvcbimage Info : GET: curl_easy_perform -failed. Path: /snapshotmanager/about/, Curl Error: 7
Or
avvcbimage Info : GET: curl_easy_perform -failed. Path: /snapshotmanager/about/, Curl Error: 28
User has disabled or removed the Snapshot Manager option in the avvcbimageAll.cmd file on proxy in which case, the following is seen:
admin@182proxy:~> grep snapshotManager /usr/local/avamarclient/var/avvcbimageAll.cmd
##--snapshotManager=10.2X0.1X6.1X1
Or
admin@182proxy:~> grep snapshotManager /usr/local/avamarclient/var/avvcbimageAll.cmd
admin@182proxy:~>
Resolution
Workaround: Confirm that Avamar Snapshot Manager is being used by:
Verifying that Avamar proxy can connect Avamar service on TCP 8543
OR
Re-registering proxy or adding back this line back to
/usr/local/avamarclient/var/avvcbimageAll.cmd
without any comment #:
--snapshotManager==
Resolution:This issue has been resolved in the Avamar 19.1+ proxy cumulative hotfix(s). Please install the latest new update for your version to resolve.
