Symptoms

While trying to create a disk group (DG) on an ESXi host, the deployment can fail with the following error: Failed to create multiple disk group for host XXXXX Dayone.log: 2021-10-01-17:25:42 microservice.ms-day1-bringup "2021-10-01 17:25:42,251 [INFO] <Dummy-5635:139774922186312> progress_converter.py _load_foreach_items_from_resource() (113): use cached foreach items for instance a4bee741-5f45-48de-b33f-31b45b76d039." 2021-10-01-17:25:42 microservice.ms-day1-bringup "2021-10-01 17:25:42,255 [INFO] <Dummy-5635:139774922186312> progress_converter.py _get_skipped_hosts() (222): Get skipped-hosts from Config Service, result: None" 2021-10-01-17:25:43 microservice.do-storage "2021-10-01 17:25:43,086 [INFO] <Dummy-534:140441273922376> request.py log_request() (31): [REQUEST START] Request API: /do-storage/v1/storage/tasks/50a6c8b5-0627-43d4-8c1f-ddc449cd7174 GET" 2021-10-01-17:25:43 microservice.do-storage "2021-10-01 17:25:43,087 [INFO] <Dummy-534:140441273922376> request.py log_response() (83): [REQUEST END] The request finished successfully. Response status: 200 OK" 2021-10-01-17:25:43 microservice.do-storage "[2021-10-01 17:25:43,427: ERROR/MainProcess] Create multi DGs failed." 2021-10-01-17:25:43 microservice.do-storage "[2021-10-01 17:25:43,427: ERROR/MainProcess] (vim.EsxCLI.CLIFault) {" 2021-10-01-17:25:43 microservice.do-storage " dynamicType = <unset>," 2021-10-01-17:25:43 microservice.do-storage " dynamicProperty = (vmodl.DynamicProperty) []," 2021-10-01-17:25:43 microservice.do-storage " msg = 'EsxCLI.CLIFault.summary'," 2021-10-01-17:25:43 microservice.do-storage " faultCause = <unset>," 2021-10-01-17:25:43 microservice.do-storage " faultMessage = (vmodl.LocalizableMessage) []," 2021-10-01-17:25:43 microservice.do-storage " errMsg = (str) [" 2021-10-01-17:25:43 microservice.do-storage " 'Unable to add device: Failed to reserve disk naa.500xxxxxxxx with exception: Failed to read vSAN intent log Failed to complete read device: /vmfs/devices/disks/naa.5002538b10a02970 bytes read: 0 of 512'" In the dayone.log of VxRail Manager, the VxRM is attempting to create the DG. However, this fails due to an inability to read a device and the vSAN intent log. Diving deeper into the storage health, confirms a significant number of failures when the device is read. 

Cause

The cause of this issue may be a failed storage device (Disk or SSD) on the affected ESXi host. Note: A failure to create a disk group does not indicate a hardware failure in every scenario.

Resolution

To resolve this issue: To determine the slot number of the disk, follow VxRail: Identify a failed disks physical slot from ESXi CLI.Run the following commands on the mentioned ESXi host to review reported read and sense errors, and Permanent Device Lost (PDL). vdq -qH (Check for PDL)esxcli vsan storage list localcli storage core device stats get (Outputs read and write errors per device)cat var/run/log/vmkernel.log | grep "0x2a,\|0x28,\|0x8a,\|0x88," |cut -d ' ' -f13|sort|uniq -c|sort -rn (Reported sense errors)Within the output reference, note the naa number of the device called out during the failure. If the device is reporting read errors try the following:Reseat the affected device and retry the build.If needed, replace the affected device. 

Support Cases