Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
HCX Migration workflow will fail to execute and below errors/exceptions can be seen in app-engine logs:Error 1: 2022-12-22 09:34:51.157 UTC [ReplicationTransferService_SvcThread-15145, Ent: DEFAULT, , TxId: TxId: 8acda383-c4e3-44e1-9805-bb34b859bdf9] ERROR c.v.h.s.r.jobs.SetupTarget- Job (b98f4467-0309-474c-bbbd-3aa415e91470) failed with exception java.lang.RuntimeException: InterconnectServiceConfigJob failed Error 2: 2022-12-22 10:53:41.586 UTC [VmotionService_SvcThread-1669, Ent: HybridityAdmin, , TxId: c1fc0607-1355-47f5-bd09-689d2b908255] ERROR c.v.v.h.m.common.MigrationUtil- Job (2511a82f-7247-4eee-bf33-4b012e8827ee) failed with exception System Error. Error while logging into Mobility Agent..fault.InvalidLogin.summary 2022-12-22 00:39:15.338 UTC [VmotionService_SvcThread-255582, Ent: HybridityAdmin, , TxId: 4e070f8f-fb36-4dd2-bd2a-6252bce10c1d] ERROR c.v.v.h.m.common.MigrationUtil- Job (184868dc-a4b0-4f06-8b37-0f91427e7875) failed with exception System Error. Error while logging into Mobility Agent.. null. During migration failure, when user try to redeploy IX appliance as an interim recovery using standard method, the redeploy operation will also fail and below error/exceptions can be seen in HCX app-engine log:Error 3: 2022-12-22 10:50:26.576 UTC [InterconnectService_SvcThread-502341, SM:servicemesh-7f412c6e-2788-499b-b9dd-3964c356e376, IX:0fd27cb0-c163-460d-a795-09124cedfae4, J:799fe681, , TxId: TxId: f7366114-e531-4557-b7af-2e3dc1186716] WARN c.v.v.h.s.i.GenerateAndPostConfiguration- Failed to post the config using VIX for the appliance f5bedd76-1d6b-4966-b10e-67da474e4ed4. Retrying the operation. java.lang.RuntimeException: Failed to upload file to guest @ /tmp/newConfig.proto for vm ServiceMesh-IX-I1 Caused by: com.vmware.vim.binding.vim.fault.InvalidGuestLogin: Failed to authenticate with the guest operating system using the supplied credentials. Note: The above exception(as highlighted in "Error 3") can also be seen for NE appliances while performing standard redeploy OR enabling new stretch for a given segment/PG.Location of App Engine log: HCX Manager: /common/log/admin/app.log
This resource is to inform about below symptoms and how to recover that: ANY migration workflow failureIX/NE standard redeployment failureNetwork Stretch Job failure
Migration workflow will be affected in the given state of IX appliance.The impact is applicable to IX & NE both appliances.Upgrade/Redeploy/Resync workflow related to IX/NE appliances won't be serviced.New Network Extension services won't be serviced.Existing Network Extension datapath will remain unaffected.
This is fixed in HCX 4.6.0 onwards.
As a potential workaround, one of the step from below can be performed to remediate the "root" password and its validity for the corresponding IX/NE appliances.STEP 1(UI Based): Verify the root password expiration validity for IX or NE appliances. Go to HCX Manager admin shell >> CCLI >> list >> Locate IX OR NE appliance Id >> go <IX or NE id> >> SSH admin@vmware-hcx [ ~ ]$ ccli Welcome to HCX Central CLI [admin@vmware-hcx] list |-----------------------------------------------------------------------------------| | Node | Id | Address | State | Selected | |-----------------------------------------------------------------------------------| | hcx-NE-I1 | 0 | X.X.X.X:9443 | Connected | | |-----------------------------------------------------------------------------------| | hcx-IX-I1 | 1 | X.X.X.X:9443 | Connected | | |-----------------------------------------------------------------------------------| [admin@vmware-hcx] go 1 Switched to node 1. [admin@vmware-hcx:hcx-IX-I1] ssh Welcome to HCX Central CLI root@hcx-IX-I1 [ ~ ]# chage -l root Last password change : Oct 06, 2022 Password expires : Jan 06, 2022 >>>>>> Password inactive : never Account expires : never Minimum number of days between password change : 0 Maximum number of days between password change : 90 Number of days of warning before password expires : 7 Refer to HCX Connector/Cloud vCenter Plugin OR Standalone UI where IX/NE appliance creds has been expired. Go to Interconnect >> Service Mesh >> Appliances >> Click on the IX or NE appliance >> Click "CHANGE PASSWORD" Enter "root" and "admin" credentials in the below UI and then "UPDATE". Note: User can choose the custom password for "root" & "admin" account.Once "UPDATE" option is clicked, a "Task" will be initiated in the backend to update the root/admin credentials and its validity. Please refer SM >> Appliances >> Task section.Note: The root/admin creds will be changed/modified by HCX manager once SM appliances gets upgraded or redeployed in the future.STEP 2(CLI Based): HCX Manager uses appliance certs to establish SSH connection with its corresponding fleet appliances. Note: Root password is not used to establish the SSH connection. We can run below command in the respective IX or NE root shell via CCLI, to keep the existing password and just to update its validity. passwd -x -1 root root@hcx-IX-I1 [ ~ ]# passwd -x -1 root passwd: password expiry information changed. root@hcx-IX-I1 [ ~ ]# chage -l root Last password change : Oct 06, 2022 Password expires : never >>>>>>>>> Password inactive : never Account expires : never Minimum number of days between password change : 0 Maximum number of days between password change : -1 Number of days of warning before password expires : 7 STEP 3: User can also perform "force redeploy" option on IX/NE appliances in a given Service Mesh (SM) to help recover migration and network extension workflow respectively.Note: The force redeploy option will impact the traffic flow over an existing extended network data path.
IMPORTANT: We have also noticed a similar error/exceptions caught up during Migration, Network Extension & Redeployment workflow, but that has a different underlying cause which requires ONLY "Force Redeploy" option to remediate and fix.Refer KB HCX - IX/NE appliance configuration failed. Reason: Failed to upload file to guest for more details. 2022-07-18 13:26:57.852 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- GenerateAndPostConfig Running in state: POST_CONFIG_VIX 2022-07-18 13:26:58.199 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- About to push config file /common/logs/admin/GW_CONFIG_10.24.242.61-1658150818081.proto to appliance vm-523955 2022-07-18 13:26:58.399 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] ERROR c.v.v.h.s.i.GenerateAndPostConfiguration- Update config on cloud gateway failed: File Upload is unsuccessful java.lang.Exception: File Upload is unsuccessful