Symptoms
Exchange backup fail with DDR errors similar to the following below:2019-04-02 07:31:26 avtar Warning : Calling DDR_WRITE returned result code:(5040) calling system(), returns nonzero message:DDRIO_Write::WriteToDDR: ddp_write failed[ 1072] [5668] Tue Apr 02 07:31:26 2019 ddp_write() failed Offset 0, BytesToWrite 600, BytesWritten 0 Err: 5040-DDBoost reauthentication failure, NFS err = 1[ 1072] [20940] Mon Apr 01 21:01:28 2019 ddp_stat() failed, Path avamar-123456789//STAGING/123456abcde/BACKUP-98765xxWXYZ/container.2.cdsf.trace, Err: 5004-nfs lookup failed (nfs: No such file or directory)[ 1072] [20940] Mon Apr 01 21:00:43 2019...2019-04-02 07:31:26 avtar Error : DDRInstance::Invoke - ddrmgr write failure result code: 50402019-04-02 07:31:26 avtar Error : endDataSlices2: encountered an error with flushing buffered writesThis issue may occur even after increasing the OST ABANDON TIMEOUT value to a higher number than the default 10800 seconds (3 hours) in the Avamar grid.
Cause
This issue more likely to be seen if the DB backup size is very large (multiple terabytes).From the ddrmaint log, we could see token authentication failure due to token expiration:>04/02 07:31:26.325 (tid 0x7f5f3541d9xx): ost_validate_token: Invalid Token - Token End time 1554202326 >04/02 07:31:26.325 (tid 0x7f5f3541d9xx): DDBOOST verify user Token failed: ost_validate_token: Invalid Token - Token Expired>04/02 07:31:26.325 (tid 0x7f5f3541d9xx): nfsprocx_ost_mnt_3_svc: connection failed permission from plugin 192.168.x.x version 3.4By default, the token expiration time is 36000 seconds (10 hours). When it exceeds the expiration time and backup is still active, request is sent to Data Domain to extend the token.In this case, the token extend request failed.
Resolution
As a workaround, split the backup jobs into separate datasets and policies, each backing up a subset of the databases so that the backup size will be smaller. The smaller backup jobs then will complete in a time frame before the DDR token expires.For a more permanent solution, open new service request to upgrade Data Domain operating system from v6.0.x to at least 6.1.1.x or higher.
