Loading...
Loading...
Some save sets previously failed to clone using a script or automatically. 'ghost' save set may show without volume name with mminfo (see below). Check that of destination filesystem shows saveset file, but NOT 'notes' file. scanner command of saveset fails (see below). Further attempts to clone the saveset fail with error: Err: 5004-nfs lookup failed (nfs: No such file or directory) mminfo CLI output shows failed clone saveset with 'E' flag: # mminfo -q ssid=928093938 -r ssid,cloneid,ssretent,clretent,ssflags,clflags,copies ssid clone id retent clretent ssflags clflg copies 928093938 1532073714 08/24/2018 08/20/2018 vrF 2 928093938 1532252170 08/24/2018 08/24/2018 vrF E <--- This is the incomplete clone. However, once we add the 'volume' switch, only 1 line is reported: # mminfo -q ssid=928093938 -r ssid,cloneid,ssretent,clretent,ssflags,clflags,copies,volume ssid clone id retent clretent ssflags clflg copies volume 928093938 1532073714 08/24/2018 08/20/2018 vrF 2 source-volume-name ^ ---2 copies yet only 1 shows in the output. Scanner CLI output: 8909:scanner: using 'DD-name_destination-device-path' as the device name 8936:scanner: scanning Data Domain disk destination-volume-name on DD-name_destination-device-path 8939:scanner: Data Domain disk destination-volume-name already exists in the media index 107505:scanner: Opening the file '/nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd' failed [5004] ([57525] [140043448723232] Wed Aug 1 13:18:50 2018 ddp_open_file() failed for File: //nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd, Err: 5004-nfs lookup failed (nfs: No such file or directory) ). 107505:scanner: Opening the file '/nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd' failed [5004] ([57525] [140043448723232] Wed Aug 1 13:18:50 2018 ddp_open_file() failed for File: //nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd, Err: 5004-nfs lookup failed (nfs: No such file or directory) ). 8761:scanner: done with Data Domain disk destination-volume-name nsrclone CLI output: 80470:nsrclone: Following volumes are needed for cloning 80471:nsrclone: source-volume-name (DD) 100303:nsrclone: Cloning savesets from storage node 'nw-server-name' 142165:nsrclone: Number of clone threads per client is set to 256. Max number of savesets per thread is set to 0. Max client threads is set to 0. 104860:nsrclone: Concurrent proxy clone started for service type DD. Cloning save sets from the volume source-volume-name Total client count in a volume source-volume-name is 1 77426:nsrclone: Cloning save sets from the volume source-volume-name 104861:nsrclone: Total client count in a volume source-volume-name is 1 104864:nsrclone: Number of clone threads per client set to 1. Max number of savesets per thread is set to 0. Max client threads is set to 30 for volume source-volume-name 174908:nsrclone: Saving the backup data in the pool 'destination-pool'. 175019:nsrclone: Received the media management binding information on the host 'nw-server-name'. 174910:nsrclone: Connected to the nsrmmd process on the host 'nw-server-name'. 32379:nsrclone: Cloning save set index:705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd (ssid/cloneid 928093938/1532073714) 75360:nsrclone: Cloning of size 1 KB is in progress... 72985:nsrclone: Source volume 'source-volume-name' ready for cloning 75675:nsrclone: Failed to initiate the save nsrmmd to create a clone instance. Direct File Save failed for /nw-server-name/destination-device-path/active/df304e8e-00000006-375196f2-5b5196f2-23ce16ac-34ae8c2d on volume destination-volume-name 75267:nsrclone: Failed to initiate mmd session for ssid 928093938. 75712:nsrclone: Cloning failed for save set : index:705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd (ssid 928093938) 77276:nsrclone: Total size cloned so far: 0 KB.141764:nsrclone: Unable to perform clone of client saveset. Error: proxy_clone_rs failed.104863:nsrclone: Concurrent proxy clone completed for sevice type DD. 67790:nsrclone: Total size cloned: 0 KB. 77275:nsrclone: DD cloning failed Proxy clone failed to clone all or few savesetsSome of the clones failed Some of the clones failed
DD has Retention Lock enabled. This prevented the incomplete original save set from being removed, and prevents the next clone attempt from over-writing the data in the destination path.
Workaround: From the scanner attempt above, find the path of the saveset on the destination: //nw-server-name/destination-device-path/08/04/notes/705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd We need the path of the saveset itself, which will be one level above 'notes', for example: //nw-server-name/destination-device-path/08/04/ Then add the DD mtree path, which always starts /data/col1/mtree-name . 'mtree list' on the DD CLI shows the Mtrees configured. Typically the Mtree name matches the host NetWorker server name. Then add the long saveset name to the end. So in this example, the full path becomes: /data/col1/nw-server-name/destination-device-path/08/04//705d0dc6-00000004-5982f3fd-599ed855-190c442c-4c9896fd We must use this path to disable retention lock on the destination Data Domain system: On the destination DD CLI, run: mtree retention-lock revert [path] (path from above) This asks for the sysadmin password. Clone the saveset again - this should now work without error because it can now write to the destination path. mminfo queries for the cloned savesets should now work correctly. Repeat for any other affected savesets.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.