Symptoms
The Avamar server has a storage performance testing feature which periodically measures the performance of the underlying storage. An Avamar Virtual Edition system reports warnings about storage performance. This occurs even if the storage is performing acceptably. For example, the warning messages below are generated despite the average read performance values being high at >2500MB/per sec.
2016/02/01-09:40:00.56257 {0.0} [perfbeat.11:197] WARN: perfbeat::outoftolerance mask=[] average=2582.79 limit=258.2795 mbpersec=231.38
2016/02/01-09:44:53.56439 {0.0} [perfbeat.10:195] WARN: perfbeat::outoftolerance mask=[] average=2604.99 limit=260.4991 mbpersec=221.23
2016/02/01-09:45:28.61315 {0.0} [perfbeat.7:189] WARN: perfbeat::outoftolerance mask=[] average=3480.14 limit=348.0138 mbpersec=67.87
2016/02/01-09:49:53.56891 {0.0} [perfbeat.10:195] WARN: perfbeat::outoftolerance mask=[] average=2604.99 limit=260.4991 mbpersec=208.90
2016/02/01-10:00:21.65498 {0.0} [perfbeat.5:185] WARN: perfbeat::outoftolerance mask=[] average=3593.12 limit=359.3115 mbpersec=287.52
2016/02/01-10:14:46.54139 {0.0} [perfbeat.6:187] WARN: perfbeat::outoftolerance mask=[] average=3629.08 limit=362.9077 mbpersec=167.07
2016/02/01-10:15:14.63836 {0.0} [perfbeat.3:181] WARN: perfbeat::outoftolerance mask=[] average=3576.72 limit=357.6724 mbpersec=248.07
2016/02/01-10:19:46.54991 {0.0} [perfbeat.6:187] WARN: perfbeat::outoftolerance mask=[] average=3629.08 limit=362.9077 mbpersec=133.98
2016/02/01-10:24:25.52382 {0.0} [perfbeat.4:183] WARN: perfbeat::outoftolerance mask=[diskio] average=3512.50 limit=351.2502 mbpersec=225.29
Cause
The perfbeat storage performance testing feature measures performance. It assesses the performance against the average measured over a period of time. Warnings about the storage are relative to the system itself. They do not advise of an absolute performance level. For example, an AVE system running on fast SAN storage may be measured as having a favorable average performance. A temporary dip in this performance due to increased load on the SAN results in warnings. This occurs even though the reduced performance is perfectly acceptable. To summarize, when perfbeat WARN messages are seen, consider the storage to be running slower than usual.
Resolution
Review the Avamar Virtual Edition Guide. Check the recommended minimum benchmarks for the size of the virtual system where the messages are observed. Below is an example from the Avamar Virtual Edition v7.2 Guide.
The average values are typically well above 2500 MB/sec as seen in the table above. This is far higher than the minimum throughput requirements specified in the guide. The warning messages in the log show that the values are approaching the minimums.This is expected during load spikes on the SAN. If temporary, no action is required. Monitor the situation and ensure that disk performance does not dip below the minimum recommended throughput for an extended time period.
