Symptoms

1. Daemon logs show that a clone operation failed due to a termination request. 5/2/2023 4:54:35 PM 0 0 0 3304 3300 0 nwserver.emc.com nsrd NSR info Error Alert: Clone operation failed with error: 'NSRCLONE stopped due to a termination request. 5/2/2023 4:56:07 PM 0 0 0 3304 3300 0 nwserver.emc.com nsrd NSR info Error Alert: Clone operation failed with error: 'NSRCLONE stopped due to a termination request. 2. Clone logs show that there was an error performing the clone of client saveset. The file copy operation failed with an NFS I/O error. 5/2/2023 4:07:38 PM Unable to perform clone of client saveset. Error: proxy_clone_rs failed. 5/2/2023 4:07:38 PM Failed to obtain filecopy status for ssid=2320332870, cloned 0 bytes out of 214485172228: Retrieving the status of a file copy operation failed [5009] ([13344] [16888] Tue May 02 16:07:38 2023 ddp_filecopy_status() failed, start_offset[0], length[0], Err: 5009-filecopy operation failed (nfs: I/O error)). 5/2/2023 4:07:38 PM Failed to obtain filecopy status for ssid=2991432354, cloned 0 bytes out of 386350972932: Retrieving the status of a file copy operation failed [5009] ([13344] [17300] Tue May 02 16:07:38 2023 ddp_filecopy_status() failed, start_offset[0], length[0], Err: 5009-filecopy operation failed (nfs: I/O error)). 3. Source DD logs indicate that there is no space left on the device and that the container set is out of space. 05/02 16:07:37.563176 [7f182be641f0] filecopy failed for , errno=28, errstr=No space left on device 05/02 01:07:40.212021 [10bd5490] CRITICAL: MSG-CM-00002: /../vpart:/vol1/col1/cp1/cset: Container set 21fa5af1e27ac4be:c2a3fa5b19a4ba85 out of space. 05/02 01:07:40.212057 [10c4b110] repl: ctx 572:0 No space left on device 4. The destination dd system is running out of space and is unable to complete the operation. 05/02 01:07:40.211978 [10c4b110] WARNING: MSG-REPL-00403: repl ctx 572:0: server error in send_segs: Out of space 05/02 01:07:40.212010 [10bd5490] DDR_SPACE: set for limit "write", space_id 0x4, full(before) 0x0, full(after) 0x4, full_count 1 05/02 01:07:40.212021 [10bd5490] CRITICAL: MSG-CM-00002: /../vpart:/vol1/col1/cp1/cset: Container set 21fa5af1e27a4abe:c2a3fa5b19a4ba85 out of space. 05/02 01:07:40.212057 [10c4b110] repl: ctx 572:0 No space left on device

Cause

The symptoms indicate a problem with the device's storage capacity.The failure to perform the clone operation, along with the "No space left on device" error in the DD logs, suggests that there is insufficient space to perform the operation.

Resolution

Delete unnecessary files: Start by identifying and deleting any unnecessary files or snapshots. You can use a file system management tool to locate and delete the files.Check the available space: Check the amount of available space in the file system using the "df -h" command.Identify the full container set: Identify the container set that is out of space by looking at the error message. The container set ID is listed in the message.Run cleaning To recover space, you must run a cleaning process on the container set that is out of spaceVerify space recovery after the cleaning process completes. Verify that space has been recovered by checking the available space using the "df -h" command.

Support Cases