Loading...
Loading...
Environment: Avamar client running AIX version 6.x and 7.x. Avamar server with DataDomain backend storage Avamar backups of EPIC CACHE database to DataDomain take a long time to complete and may over-run the available backup window 2017-02-16 10:24:38 avtar Info <5156>: Backup #27 timestamp 2017-02-16 10:26:11, 18 files, 21 directories, 4,051 GB (9 files, 7.922 GB, 0.20% new) . . 2017-02-16 10:24:38 avtar Info <6083>: Backed-up 4,051 GB in 1586.17 minutes: 153 GB/hour (1 files/hour) File processing performance is good at 150GB/hour but CPU usage is unexpectedly low throughout the duration of the backup 2017-02-15 09:54:49 avtar Info <8688>: Status 2017-02-15 09:54:49, 1 files, 228.2 GB (0 bytes, 0.00% new) 89MB 11% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:54:59 avtar Info <8688>: Status 2017-02-15 09:54:59, 1 files, 228.4 GB (0 bytes, 0.00% new) 89MB 9% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:55:09 avtar Info <8688>: Status 2017-02-15 09:55:09, 1 files, 228.7 GB (0 bytes, 0.00% new) 89MB 9% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:55:19 avtar Info <8688>: Status 2017-02-15 09:55:19, 1 files, 229.0 GB (0 bytes, 0.00% new) 89MB 11% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:55:29 avtar Info <8688>: Status 2017-02-15 09:55:29, 1 files, 229.2 GB (0 bytes, 0.00% new) 89MB 10% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:55:39 avtar Info <8688>: Status 2017-02-15 09:55:39, 1 files, 229.5 GB (0 bytes, 0.00% new) 89MB 9% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:55:49 avtar Info <8688>: Status 2017-02-15 09:55:49, 1 files, 229.8 GB (0 bytes, 0.00% new) 89MB 13% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:55:59 avtar Info <8688>: Status 2017-02-15 09:55:59, 1 files, 230.0 GB (0 bytes, 0.00% new) 89MB 8% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:56:09 avtar Info <8688>: Status 2017-02-15 09:56:09, 1 files, 230.2 GB (0 bytes, 0.00% new) 89MB 6% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:56:19 avtar Info <8688>: Status 2017-02-15 09:56:19, 1 files, 230.5 GB (0 bytes, 0.00% new) 89MB 12% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:56:29 avtar Info <8688>: Status 2017-02-15 09:56:29, 1 files, 230.8 GB (0 bytes, 0.00% new) 89MB 11% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:56:39 avtar Info <8688>: Status 2017-02-15 09:56:39, 1 files, 231.0 GB (0 bytes, 0.00% new) 89MB 5% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:56:49 avtar Info <8688>: Status 2017-02-15 09:56:49, 1 files, 231.2 GB (0 bytes, 0.00% new) 89MB 9% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:57:00 avtar Info <8688>: Status 2017-02-15 09:57:00, 1 files, 231.4 GB (0 bytes, 0.00% new) 89MB 9% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:57:10 avtar Info <8688>: Status 2017-02-15 09:57:10, 1 files, 231.6 GB (0 bytes, 0.00% new) 89MB 7% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:57:20 avtar Info <8688>: Status 2017-02-15 09:57:20, 1 files, 231.8 GB (0 bytes, 0.00% new) 89MB 7% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:57:30 avtar Info <8688>: Status 2017-02-15 09:57:30, 1 files, 232.1 GB (0 bytes, 0.00% new) 89MB 10% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:57:40 avtar Info <8688>: Status 2017-02-15 09:57:40, 1 files, 232.3 GB (0 bytes, 0.00% new) 89MB 7% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:57:50 avtar Info <8688>: Status 2017-02-15 09:57:50, 1 files, 232.5 GB (0 bytes, 0.00% new) 89MB 9% CPU (1 open files) /epic/prod02/er1/CACHE.DAT 2017-02-15 09:58:00 avtar Info <8688>: Status 2017-02-15 09:58:00, 1 files, 232.7 GB (0 bytes, 0.00% new) 89MB 7% CPU (1 open files) /epic/prod02/er1/CACHE.DAT
Somewhat atypically for a database type backup, avtar appears to be limited by the amount of data it can ingest rather than how quickly it can process data. This is made clear by the low CPU usage reported by avtar, which is mostly idle whilst the backup progresses. The reason for this is how avtar's internal threads work together. There are two threads that run in parallel within avtar Thread #1 (T1) reads the data from the disk and queues for thread #2. It's performance is IO dependent. Thread #2 (T2) reads the data from T1 and segments/dedupes this data by communicating with DD. It's performance is CPU dependent. T2 performance also depends on the network and DD load since once processed, the data needs to be shipped over the network to the target storage. If T2 has 6 buffers in the queue that it has not read/processed as yet due to hold ups on the network or target storage then T1 activity will be held Basically, Avamar throttles T1 if T2 has too much unprocessed data and during this held time, T1 cannot read any data from the disk. Similarly, if T1 has not queued anything, T2 has to wait. For an ideally configured system, T2 should not have to wait and it should always have the data made available by T1. If a system runs in an efficient manner then we should see that the run time of T2 is over 85% of the run time of the program. In addition, the CPU utilization should be near 1 CPU core. The backup will perform best if thread 2 is kept busy. Essentially thread #2 is spending a lot of its time waiting for data from thread #1
Suggestions to improve performance: Check that the target storage device is not overloaded If the target storage device (for example DataDomain) is overloaded the backup performance may be limited by the ingestion capability of that device. Avamar: How to enable and interpret avtar COMSTATS logging to diagnose communication issues Check that the network is not a bottleneck If the data transmitted by the Avamar client to DataDomain is sent at a rate similar or equivalent to the network throughput then it is possible that the network is the bottleneck. If the network is a bottleneck then tuning on the client side will not help the backup complete any faster. The following articles may be helpful in determining network throughput between Avamar and Data Domain Avamar: How to run iperf between two Avamar grids to test network throughput performance How To Run iPerf from Avamar Proxy to Data Domain Increase the ddr-queue-size Increase the ddr-queue-size parameter from the default of 6 to something higher, for example, between 20 and 40. Even higher values can be used. This is an avtar parameter. The trade off is that additional memory will be consumed for the queue / buffer. This will potentially reduce the time that one or the other process sits idle whilst it waits on the other. Improve disk performance & efficiency To help ensure that thread 1 does not bottleneck thread 2 Use higher performance disks such as SSD or configure the disk subsystem with more spindles Experiment with mount options on the client machine. Reconfigure the LUN/Volume layout on the sixteen 1TB disks/LUNs and ensure that the parallel reads from five different DBs are hitting different spindles. Enable caching at the storage level Enable caching in the client OS so the OS performs sequential reads ahead of the actual reads done by avtar T1 - this will enable T1 s read requests to be filled immediately in a sustained smooth way rather than on-demand Client Operating System performance: (AIX/Linux/Windows) The performance of T2 is based on the processing capability of a single core (**). AIX P8 could possibly do 150 MBPS for a finely tuned system. Linux/Windows performance may be different. Specific recommendations for AIX systems: Make sure the files ystems are not mounted with CIO (concurrent I/O) or DIO (direct I/O) options. Remount the filesystems without this options if they are set. Make sure the following TCP parameters are set at the following values or higher: tcp_recvspace=655360 rfc1323=1 udp_recvspace=655360 sb_max=1310720 no -p -o sack=1 Confirm the parameters by running ifconfig -a, the parameters are only in effect if they are seen associated to the NIC that is sending data to DataDomain
Click on a version to see all relevant bugs