Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Uploads to catalog fail with the error "Cannot complete login due to an incorrect user name or password. - Cannot complete login due to an incorrect user name or password."The upload "Imported Media File" task may also hang indefinitely without an error being observed.Catalog uploads previously worked successfully without issue.Within the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log you observe an error similar to the following: 2022-07-20T11:40:26,879 | ERROR | task-service-activity-pool-156497 | FutureUtil | Error creating Media | requestId=<REQUEST_UUID>,request=POST https://vcloud.example.com/api/catalog/<CATALOG_UUID>/action/upload,requestTime=1595364143157,remoteAddress=<IP>:<PORT>,userAgent=Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:77.0) Gecko/20100...,accept=application/*+xml;version 34.0 vcd=<VCD_UUID>,task=<TASK_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_ACTIVITY_UUID>)com.vmware.ssdc.util.LMException: Cannot complete login due to an incorrect user name or password. at com.vmware.ssdc.util.LMException.wrap(LMException.java:128) at com.vmware.vcloud.val.filetransfer.impl.FileTransferToRemoteHostActivity$TransferToRemoteHostPhase.invokeCancelableOperation(FileTransferToRemoteHostActivity.java:110) at com.vmware.vcloud.common.activity.AbstractActivity$CancelablePhase.invoke(AbstractActivity.java:365) at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175) at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Cannot complete login due to an incorrect user name or password. at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl$TransferWrapper.execute(NfcClientFacadeImpl.java:176) at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl.putFile(NfcClientFacadeImpl.java:233) at com.vmware.vcloud.val.filetransfer.impl.FileTransferToRemoteHostActivity$TransferToRemoteHostPhase.invokeCancelableOperation(FileTransferToRemoteHostActivity.java:107)Caused by: HTTP error on PUT https://vcloud.example.com/folder/<DATASTORE_UUID>/media/<ORGANIZATION_VDC_UUID>/<ORGANIZATION_UUID>/media-<MEDIA_UUID>.iso?dcPath=example-dc&dsName=example-dcc01-vsan: 500 Within the vCenter Server /var/log/vmware/vxpd/vpxd.log the following error is visible: 2022-07-20T11:40:18.454Z warning vpxd[04567] [Originator@6876 sub=Default] [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 14 from server: There is no disk space available -- NfcFssrvrWriteCB: Failed to write 1024000 bytes @ 38912000 : There is no space left on the device Within the ESXi /var/log/vpxa.log the following error is visible: 2022-07-20T11:40:20.017Z warning vpxa[2104498] [Originator@6876 sub=Libs] [NFC ERROR] NfcFileRawPwrite: Failed to write xxxxx bytes at offset xxxxxx from file '/vmfs/volumes/vsan:<VSAN_UUID>/<DATASTORE_UUID>/media/<ORGANIZATION_VDC_UUID>/<ORGANIZATION_UUID>/media-<MEDIA_UUID>.iso': Success (@&!*@*@(msg.fileio.success)Success)2022-07-20T11:40:20.017Z warning vpxa[2104498] [Originator@6876 sub=Libs] [NFC ERROR] NfcFssrvrWriteCB Received error:14, fileError:102022-07-20T11:40:20.018Z warning vpxa[2102551] [Originator@6876 sub=Libs] [NFC ERROR] NfcFssrvrSendResponse: Found an error in the nfc session: There is no disk space available (NFC_NO_DISKSPACE)2022-07-20T11:40:20.018Z warning vpxa[2102551] [Originator@6876 sub=Libs] [NFC ERROR] Sending Nfc error 14: There is no disk space available -- NfcFssrvrWriteCB: Failed to write xxxxx bytes @ xxxxxx : There is no space left on the device Within the ESXi /var/log/vmkernel.log the following error is visible: 2022-07-20T11:40:44.573Z cpu11:2105484)WARNING: Res3: 7542: '<REQUEST_UUID>': [rt 1] No Space - did not find enough resources after second pass! (needed: 1, found: 0)2022-09-22T11:40:44.573Z cpu11:2105484)Res3: 7543: '<REQUEST_UUID>': [rt 1] Resources t 0, e 0, np 192, wb 0, b 0, RCs u 0, i 0, nf 1305, pe 0, oe 02022-07-20T11:40:44.573Z cpu11:2105484)BC: 410: write to media-<MEDIA_UUID>.iso (f530 28 3 xxxxx 537c4b6e xxxxx 504b1096 xxxxx 2a 0 0 0 0 0) 1048576 bytes failed: No space left on device
This issue occurs when the vSAN namespace object is completely full. All catalog media images (ISO files) are uploaded by Cloud Director as file objects into a directory structure under a single folder, which results in the namespace filling up. A namespace object has a maximum size of 255 GB.
VMware does not recommend using vSAN storage for catalogs in Cloud Director. Other storage solutions should be utilized for catalogs.
The following architecture guides also discuss this topic:https://download3.vmware.com/vcat/vmw-vcloud-architecture-toolkit-spv1-webworks/index.html#page/Cloud%20Automation%20and%20Orchestration/Architecting%20a%20vCloud%20Director%20Solution/Architecting%20a%20vCloud%20Director%20Solution.1.051.htmlhttps://download3.vmware.com/vcat/vmw-vcloud-architecture-toolkit-spv1-webworks/index.html#page/Storage%20and%20Availability/Architecting%20VMware%20vSAN%206.2/Architecting%20Virtual%20SAN%206.2.2.105.html