Symptoms

VMware image backups failing with error code 10007.avtar logs end suddenly without error. avtar Info : Status 2016-05-22 05:32:31, 110.6 GB (194.6 MB, 0.17% new) 233MB 2% CPU 7277985792 input bytes, VMFiles/2/virtdisk-flat.vmdkavtar Info : - Changeblock #92 offset=80951115776, bytes=268435456avtar Info : - Changeblock #184 offset=85034139648, bytes=131072avtar Info : Status 2016-05-22 05:47:38, 115.8 GB (194.6 MB, 0.17% new) 233MB 2% CPU 11380670464 input bytes, VMFiles/2/virtdisk-flat.vmdkavtar Info : - Changeblock #276 offset=89156222976, bytes=262144avtar Info : - Changeblock #368 offset=91305017344, bytes=131072avtar Info : - Changeblock #460 offset=93791191040, bytes=2359296avtar Info : Status 2016-05-22 06:02:45, 126.8 GB (194.6 MB, 0.16% new) 239MB 2% CPU 15121465344 input bytes, VMFiles/2/virtdisk-flat.vmdkavtar Info : - Changeblock #552 offset=97889288192, bytes=655360avtar Info : Status 2016-05-22 06:17:49, 131.5 GB (194.6 MB, 0.16% new) 249MB 2% CPU 19517751296 input bytes, VMFiles/2/virtdisk-flat.vmdk avvcbimage log shows the backup failing due to a 'Signal 9 exit - KIlled' issue. avvcbimage Error : [IMG0010] PAX stream append ([vmclient] vmclient/vmclient01.vmdk) at sector offset 200669952 length 128 sectors=65536 failed (32)avvcbimage Info : Process 4018 (/usr/local/avamarclient/bin/avtar) finished (Signal 9 exit - Killed)avvcbimage Warning : CTL workorder "backup_5pm-1463868002597_c8a07604f6a5f0c5dcdfc871a5b3f5a8d4d77441" non-zero exit status 'Signal 9 exit - Killed'avvcbimage Info : Starting graceful (staged) termination, PAX stream append failed (wrap-up stage)avvcbimage Info : isExitOK()=157avvcbimage Info : Cancel detected(miscellaneous error), isExitOK(0).avvcbimage Info : bytes submitted was 102743015424avvcbimage Error : [IMG0010] pax_container::endfile(virtdisk-flat.vmdk,102743015424) returned problem:32avvcbimage Error : [IMG0010] pax_container::enddir returned problem:32avvcbimage Info : isExitOK()=157avvcbimage Info : Cancel detected(miscellaneous error), isExitOK(0).avvcbimage Info : backup ([vmclient] vmclient/vmclient01.vmdk) terminated 135168.00 MBavvcbimage Info : isExitOK()=157avvcbimage Error : [IMG1003] Backup of [vmclient] vmclient/vmclient01.vmdk failedavvcbimage Error : [IMG0011] Avtar exited with 'Signal 9 exit - Killed'avvcbimage Info : Final summary, cancelled/aborted 1, snapview 0, exitcode 157: miscellaneous error

Cause

/var/log/messages of the proxies shows that oom-killer is killing Avamar (such as avtar and avvcbimage) and VMware processes throughout the backup window. Killing the Avamar processes kills the VMware image backup.grep "oom-killer" /var/log/messages proxy1 kernel: [4772630.893189] avvcbimage invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4773061.607740] avvcbimage invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4774722.062882] avtar.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4775034.371673] avagent.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4775034.417619] avtar.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4775229.334239] avvcbimage invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4775920.675742] avvcbimage invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4776254.395983] avvcbimage invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4776692.716554] avtar.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4778083.647001] avvcbimage invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0, oom_score_adj=0proxy1 kernel: [4778545.598976] vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Avamar VMware proxies found to be undersized in the memory allocated to them. For this example, only ~1GB of memory is available to the proxy.free -m total used free shared buffers cachedMem: 990 219 770 0 21 117-/+ buffers/cache: 80 909Swap: 139 44 95

Resolution

Increase the amount of memory available to the Avamar proxies.Recommended to reboot the Avamar proxies after increasing the memory due to the other possible processes killed by to oom-killer.

Support Cases