Loading...
Loading...
1. Daemon logs show that a clone operation failed due to a termination request. 5/2/2023 4:54:35 PM 0 0 0 3304 3300 0 nwserver . emc . com nsrd NSR info Error Alert: Clone operation failed with error: 'NSRCLONE stopped due to a termination request. 5/2/2023 4:56:07 PM 0 0 0 3304 3300 0 nwserver . emc . com nsrd NSR info Error Alert: Clone operation failed with error: 'NSRCLONE stopped due to a termination request. 2. Clone logs show that there was an error performing the clone of client save set. The file copy operation failed with an NFS I/O error. 5/2/2023 4:07:38 PM Unable to perform clone of client saveset. Error: proxy_clone_rs failed. 5/2/2023 4:07:38 PM Failed to obtain filecopy status for ssid=2320332870, cloned 0 bytes out of 214485172228: Retrieving the status of a file copy operation failed [5009] ([13344] [16888] Tue May 02 16:07:38 2023 ddp_filecopy_status() failed, start_offset[0], length[0], Err: 5009-filecopy operation failed (nfs: I/O error)). 5/2/2023 4:07:38 PM Failed to obtain filecopy status for ssid=2991432354, cloned 0 bytes out of 386350972932: Retrieving the status of a file copy operation failed [5009] ([13344] [17300] Tue May 02 16:07:38 2023 ddp_filecopy_status() failed, start_offset[0], length[0], Err: 5009-filecopy operation failed (nfs: I/O error)). 3. Source DD logs indicate that there is no space left on the device and that the container set is out of space. 05/02 16:07:37.563176 [7f182be641f0] filecopy failed for <file: task idx 214037, /data/col1/nwserver/BACKUPDEV01/15/57/653f07b8-00000006-c64d9d05-644d9d05-0b3b1500-1139205a, destination: 192.168.9.106>, errno=28, errstr=No space left on device 05/02 01:07:40.212021 [10bd5490] CRITICAL: MSG-CM-00002: /../vpart:/vol1/col1/cp1/cset: Container set 21fa5af1e27ac4be:c2a3fa5b19a4ba85 out of space. 05/02 01:07:40.212057 [10c4b110] repl: ctx 572:0 No space left on device 4. The destination DD system is running out of space and is unable to complete the operation. 05/02 01:07:40.211978 [10c4b110] WARNING: MSG-REPL-00403: repl ctx 572:0: server error in send_segs: Out of space 05/02 01:07:40.212010 [10bd5490] DDR_SPACE: set for limit "write", space_id 0x4, full(before) 0x0, full(after) 0x4, full_count 1 05/02 01:07:40.212021 [10bd5490] CRITICAL: MSG-CM-00002: /../vpart:/vol1/col1/cp1/cset: Container set 21fa5af1e27a4abe:c2a3fa5b19a4ba85 out of space. 05/02 01:07:40.212057 [10c4b110] repl: ctx 572:0 No space left on device
The symptoms indicate a problem with the device's storage capacity. The failure to perform the clone operation, along with the "No space left on device" error in the DD logs, suggests that there is insufficient space to perform the operation. The Data Domain command filesys show space indicates that the system is nearing capacity.
Perform a Data Domain cleaning job on the affected Data Domain to see if sufficient space is reclaimed. By default, this job is performed weekly on Tuesdays. Investigate potential causes of high space usage on the Data Domain: NetWorker: Reclaiming space from Data Domain Guide Check NetWorker backups for data which may no longer be needed and can be deleted: NetWorker: How to delete Multiple or Individual SSIDs Adjust browse and retention policies on future backup and clone operations: NetWorker: How to change the Browse/Retention Policy NOTE: To investigate space issues further both NetWorker and Data Domain support teams can be engaged to review potential issues on the Data Domain or for any issues where NetWorker is not cleaning expired data.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.