Symptoms
Deployment, redeploy, or upgrade of HCX fleet appliances may fail with the following error:Redeploy of Interconnect Appliance failed. Interconnect Service Workflow InterconnectRedeploy failed. Error:["Interconnect Service Workflow GenerateAndPostConfig failed. Error: [\"Failed to upload file to guest @ /tmp/newConfig.proto for vm.
Cause: Failed to authenticate with the guest operating system using the supplied credentials.\"]"]
UpgradeAppliance Failed. Reason:Interconnect Service Workflow InterconnectRedeploy failed. Error: ["Redeploy of Interconnect Appliance failed. Interconnect Service Workflow InterconnectRedeploy failed. Error:["Interconnect Service Workflow GenerateAndPostConfig failed. Error: [\"Failed to upload file to guest @ /tmp/newConfig.proto for vm.
Cause: Failed to authenticate with the guest operating system using the supplied credentials.\"]"]
Purpose
Identify a condition that may be blocking the deployment of HCX fleet appliance VMs and provide a procedure to clear it.
Cause
The failure in the deployment process is due to the vpxa and hostd logging level set to "trivia" on the ESXi hosts.The increased load on the hostd service to generate the detailed log messages introduces a delay in getting the initial configuration applied to the VM which leads to the race condition that causes the deployment to fail.
Impact / Risks
New Service Mesh configuration will not be able to deploy any fleet appliances so Migration and Network Extension services will be unavailable.Appliances for an existing Service Mesh may get disrupted during redeployment or upgrade.
Resolution
Reduce the logging level on the ESXi hosts in the deployment cluster.
In the vCenter UI, go to ESXi host ---> Configure ---> System ---> Advanced System SettingsSearch for the following logging sources:
Vpx.Vpxa.config.log.level
config.HostAgent.log.level
Change the value from "trivia" to "info" for both
Restart the ESXi management agents using below CLI:
/etc/init.d/hostd restart/etc/init.d/vpxa restart
Deploy new Service Mesh or redeploy existing appliances
