Symptoms

VMware image backups fail with error code 10007 and avtar logs end suddenly without an error. avtar Info <8688>: Status 2016-05-22 05:32:31, 110.6 GB (194.6 MB, 0.17% new) 233MB 2% CPU 7277985792 input bytes, VMFiles/2/virtdisk-flat.vmdk avtar Info <10698>: - Changeblock #92 offset=80951115776, bytes=268435456 avtar Info <10698>: - Changeblock #184 offset=85034139648, bytes=131072 avtar Info <8688>: Status 2016-05-22 05:47:38, 115.8 GB (194.6 MB, 0.17% new) 233MB 2% CPU 11380670464 input bytes, VMFiles/2/virtdisk-flat.vmdk avtar Info <10698>: - Changeblock #276 offset=89156222976, bytes=262144 avtar Info <10698>: - Changeblock #368 offset=91305017344, bytes=131072 avtar Info <10698>: - Changeblock #460 offset=93791191040, bytes=2359296 avtar Info <8688>: Status 2016-05-22 06:02:45, 126.8 GB (194.6 MB, 0.16% new) 239MB 2% CPU 15121465344 input bytes, VMFiles/2/virtdisk-flat.vmdk avtar Info <10698>: - Changeblock #552 offset=97889288192, bytes=655360 avtar Info <8688>: Status 2016-05-22 06:17:49, 131.5 GB (194.6 MB, 0.16% new) 249MB 2% CPU 19517751296 input bytes, VMFiles/2/virtdisk-flat.vmdk <end of log> The avvcbimage log shows the backup failing due to a 'Signal 9 exit - Killed' issue. avvcbimage Error <0000>: [IMG0010] PAX stream append ([vmclient] vmclient/vmclient01.vmdk) at sector offset 200669952 length 128 sectors=65536 failed (32) avvcbimage Info <6688>: Process 4018 (/usr/local/avamarclient/bin/avtar) finished (Signal 9 exit - Killed) avvcbimage Warning <6690>: CTL workorder "backup_5pm-1463868002597_c8a07604f6a5f0c5dcdfc871a5b3f5a8d4d77441" non-zero exit status 'Signal 9 exit - Killed' avvcbimage Info <9772>: Starting graceful (staged) termination, PAX stream append failed (wrap-up stage) avvcbimage Info <40654>: isExitOK()=157 avvcbimage Info <16022>: Cancel detected(miscellaneous error), isExitOK(0). avvcbimage Info <9746>: bytes submitted was 102743015424 avvcbimage Error <0000>: [IMG0010] pax_container::endfile(virtdisk-flat.vmdk,102743015424) returned problem:32 avvcbimage Error <0000>: [IMG0010] pax_container::enddir returned problem:32 avvcbimage Info <40654>: isExitOK()=157 avvcbimage Info <16022>: Cancel detected(miscellaneous error), isExitOK(0). avvcbimage Info <12032>: backup ([vmclient] vmclient/vmclient01.vmdk) terminated 135168.00 MB avvcbimage Info <40654>: isExitOK()=157 avvcbimage Error <0000>: [IMG1003] Backup of [vmclient] vmclient/vmclient01.vmdk failed avvcbimage Error <0000>: [IMG0011] Avtar exited with 'Signal 9 exit - Killed' avvcbimage Info <16038>: Final summary, cancelled/aborted 1, snapview 0, exitcode 157: miscellaneous error

Cause

Issue Description The /var/log/messages file on the Avamar proxy hosts shows that the Linux Out of Memory (OOM) killer is terminating Avamar processes (such as avtar, avagent, and avvcbimage) and VMware processes during the backup window. When these Avamar processes are killed, VMware image backups fail. Verification To identify OOM-killer activity, run the following command on the proxy: grep "oom-killer" /var/log/messages Example output: proxy1 kernel: [4772630.893189] avvcbimage invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 proxy1 kernel: [4773061.607740] avvcbimage invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 proxy1 kernel: [4774722.062882] avtar.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 proxy1 kernel: [4775034.371673] avagent.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 proxy1 kernel: [4775229.334239] avvcbimage invoked oom-killer: gfp_mask=0x200d2, order=0, oom_score_adj=0 proxy1 kernel: [4778545.598976] vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 These entries confirm that the Avamar VMware proxy is under memory pressure and invoking the OOM killer during backup operations. Root Cause The Avamar VMware proxy is undersized in terms of allocated memory. In this example, the proxy has approximately 1 GB of total memory, which is insufficient for image-level backups. Memory Verification To check the available memory on the proxy, run: free -m Example output: total used free shared buffers cached Mem: 990 219 770 0 21 117 -/+ buffers/cache: 80 909 Swap: 139 44 95 This output confirms that the proxy has limited memory available, leading to frequent OOM-killer activity and VMware image backup failures.

Resolution

Increase the memory allocated to the Avamar proxy servers. After increasing the memory, reboot the Avamar proxies. Rebooting is recommended because the Linux OOM killer may have terminated other processes due to previous memory pressure.

Support Cases