Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
The output of esxcli vsan debug object list can give VMDKS that have been moved between Namespaces the appearance of being (Missing), thus making them seem Orphaned or Unassociated.For example:[root@vSAN-1:~] esxcli vsan debug object listObject UUID: 19b10b5d-48fd-733d-a2f0-005056263753 Version: 7 Health: healthy Owner: vSAN-1.gsslabs.org Size: 10.00 GB Used: 0.01 GB Policy: CSN: 2 stripeWidth: 1 cacheReservation: 0 proportionalCapacity: 0 spbmProfileName: vSAN Default Storage Policy forceProvisioning: 0 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad hostFailuresToTolerate: 1 spbmProfileGenerationNumber: 0 Type: vdisk Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk (Missing) Group UUID: 10b10b5d-3071-b7cb-a2be-005056263753 Directory Name: N/A
The purpose of this article is to provide a detailed example of how this behavior can occur, along with corrective measures available using ESXi's vSAN Object Tool, or objtool.
When moving vSAN VMDK Descriptor Files from one Namespace directory to a different Namespace directory, the Object Path Attribute is not automatically updated.By default, vSAN does not move VMDK Descriptor Files from one Namespace to a different Namespace.This is expected product behavior. The Object Path Attribute of vSAN Objects must be updated explicitly with no automated methods currently available.
Currently there is no mechanism to prevent this behavior from happening as vSAN does not move VMDK Descriptor files between Namespace Directories. However, the need to move files between Namespaces should be a relatively rare scenario.
See step 5 in the Related Information section.
This behavior can be reproduced when VMDK Descriptor files are moved manually using the Datastore Browser in the vSphere Client UI, when using "mv" commands in ESXi's command line interface, and when VMDKS are moved using vSphere API's such as MoveDatastoreFile_Task. In the example scenario below, we'll walk through the process of reproducing and correcting this behavior with two example Virtual Machines: "Virtual Machine 1" & "Virtual Machine 2": "Virtual Machine 1" Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753"Virtual Machine 2" Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-0050562637531. First, we can obtain the Object UUID of "Virtual Machine 1.vmdk" by simply cat'ing the VMDK Descriptor file - [root@vSAN-1:~] cat "/vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk"# Disk DescriptorFileversion=4encoding="UTF-8"CID=fffffffeparentCID=ffffffffcreateType="vmfs"# Extent descriptionRW 20971520 VMFS "vsan://19b10b5d-48fd-733d-a2f0-005056263753"# The Disk Data Base#DDBddb.adapterType = "lsilogic"ddb.geometry.cylinders = "1305"ddb.geometry.heads = "255"ddb.geometry.sectors = "63"ddb.longContentID = "0046fa2bcb13ce19e6ba05cffffffffe"ddb.thinProvisioned = "1"ddb.uuid = "60 00 C2 92 46 38 6d 04-ae 07 eb 8b a2 fe af 7a"ddb.virtualHWVersion = "14"2. Next, we can move "Virtual Machine 1.vmdk" to Virtual Machine 2's Directory using a "mv" command: [root@vSAN-1:~] mv "/vmfs/volumes/vsanDatastore/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk" "/vmfs/volumes/vsanDatastore/6fb10b5d-5bd1-f2aa-219a-005056263753"3. Using the vSphere Client UI, we can attach "Virtual Machine 1.vmdk" to Virtual Machine 2: 4. Reviewing the output of esxcli vsan debug object list, we can now see that the VMDK Descriptor is reported as (Missing), even though it still exists:[root@vSAN-1:~] esxcli vsan debug object listObject UUID: 19b10b5d-48fd-733d-a2f0-005056263753 Version: 7 Health: healthy Owner: vSAN-1.gsslabs.org Size: 10.00 GB Used: 0.01 GB Policy: CSN: 2 stripeWidth: 1 cacheReservation: 0 proportionalCapacity: 0 spbmProfileName: vSAN Default Storage Policy forceProvisioning: 0 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad hostFailuresToTolerate: 1 spbmProfileGenerationNumber: 0 Type: vdisk Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk (Missing) Group UUID: 10b10b5d-3071-b7cb-a2be-005056263753 Directory Name: N/AThis is because the Path Attribute for this Object still references the previous directory from Virtual Machine 1. This can be confirmed via an "objtool getAttr" (Object Tool Get Attribute) command: * Command Syntax: /usr/lib/vmware/osfs/bin/objtool getAttr -u <Object UUID>[root@vSAN-1:~] /usr/lib/vmware/osfs/bin/objtool getAttr -u 19b10b5d-48fd-733d-a2f0-005056263753Object Attributes --UUID:19b10b5d-48fd-733d-a2f0-005056263753Object type:vsanObject size:10737418240User friendly name:(null)HA metadata:(null)Allocation type:ThinPolicy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+0) (\"spbmProfileName\" \"vSAN Default Storage Policy\"))Object class: vdiskObject capabilities: NONEObject path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdkGroup uuid: 10b10b5d-3071-b7cb-a2be-005056263753Container uuid: (null)5. We must update the Path Attribute using an "objtool setAttr" (Object Tool Set Attribute) command. * Command Syntax: /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>[root@vSAN-1:~] /usr/lib/vmware/osfs/bin/objtool setAttr -u 19b10b5d-48fd-733d-a2f0-005056263753 -d "/vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753/Virtual Machine 1.vmdk"Object set attribute succeededRe-running esxcli vsan debug object list, we can now confirm the VMDK Descriptor File is no longer reported as (Missing):[root@vSAN-1:~] esxcli vsan debug object listObject UUID: 19b10b5d-48fd-733d-a2f0-005056263753 Version: 7 Health: healthy Owner: vSAN-1.gsslabs.org Size: 10.00 GB Used: 0.01 GB Policy: stripeWidth: 1 spbmProfileGenerationNumber: 0 cacheReservation: 0 spbmProfileName: vSAN Default Storage Policy forceProvisioning: 0 proportionalCapacity: 0 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad hostFailuresToTolerate: 1 CSN: 2 Type: vdisk Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753/Virtual Machine 1.vmdk (Exists) Group UUID: 10b10b5d-3071-b7cb-a2be-005056263753 Directory Name: N/ANOTE: When using "esxcli vsan debug object list" in certain vSAN versions it shows only 100 Objects and it may happen that you don't see the one you are looking for. You will need to add the flag "--all" however, this could impact the host functionality so, it is recommended to run the command and send the output to a txt file. Here it is "esxcli vsan debug object list --all > /tmp/objects.txt". Then you can "less" the file to search for your specific object.If the vSAN DOM Object UUID is known you can use the following syntax: esxcli vsan debug object list -u <DOM Object UUID>