Symptoms

When upgrade to VxRail 7.0.37x, LCM failed at the finalizing task VcCertTask: Error: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times. From lcm-web.log 2022-08-04 23:56:38,433 INFO [LCM] [lcm-core-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:49] Start to execute task : VcCertTask. 2022-08-04 23:56:38,433 INFO [LCM] [lcm-core-0] c.v.l.t.SimpleUpgradeTaskExecutor [SimpleUpgradeTaskExecutor.java:71] Task VcCertTask, execute timeout: 1200 seconds. 2022-08-04 23:56:38,433 INFO [LCM] [upgrade-task-0] c.e.m.m.e.DynamicReloadableResourceBundleMessageSource [DynamicReloadableResourceBundleMessageSource.java:17] Try to translate message: Update vCenter certificate. 2022-08-04 23:56:39,421 INFO [LCM] [upgrade-task-0] c.d.v.l.d.p.r.d.NanoServiceDO [NanoServiceDO.java:63] Call nano-service to update VC cert in VXM. ==> unix.http://127.0.0.1/rest/vxm/internal/operation/v1/vxm/download-vc-certs/execute, req body: {"vc_info":{"host":"attvcenter.mossadams.ma","username":"administrator@vsphere.ma","password":"******","port":443},"auto_accept_vc_cert":true} 2022-08-04 23:56:44,189 ERROR [LCM] [upgrade-task-0] c.v.c.c.VcCertService [VcCertService.java:39] Error happened when update VC cert in vxm.500 INTERNAL SERVER ERROR: "{"result": {"error": {"code": "E3100_Security_CERT_03", "params": ["attvcenter.mossadams.ma", "unable to get local issuer certificate"], "message": "Failed to validate the certificate of attvcenter.mossadams.ma, the reason is: unable to get local issuer certificate"}}}" 2022-08-04 23:59:32,437 INFO [LCM] [lcm-core-0] c.e.m.m.u.r.BaseVirtualApplianceUpgradeProfilePool [BaseVirtualApplianceUpgradeProfilePool.java:749] Performing UPGRADE at progress 46! 2022-08-04 23:59:32,437 INFO [LCM] [lcm-core-0] c.d.v.l.d.p.r.d.VxrailSystemDO [VxrailSystemDO.java:134] operation status cache hit LcmUpgrade-824fec9a-2a8f-437a-b778-7d152ed5afae 2022-08-04 23:59:32,543 INFO [LCM] [lcm-core-0] c.d.v.l.d.p.r.d.VxrailSystemDO [VxrailSystemDO.java:180] operation status cache update 213 LcmUpgrade-824fec9a-2a8f-437a-b778-7d152ed5afae 2022-08-04 23:59:32,543 WARN [LCM] [lcm-core-0] c.e.m.m.u.s.VirtualApplianceUpgradeService [VirtualApplianceUpgradeService.java:994] LCM Exception occurred during upgrade for virtual appliance 70372188 - {} com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times. at com.vce.lcm.api.LCMServiceImpl.handleUpgradeException(LCMServiceImpl.java:1745) at com.vce.lcm.api.LCMServiceImpl.performUpgrade(LCMServiceImpl.java:450) at com.emc.mystic.manager.upgrade.service.VirtualApplianceUpgradeService.runUpgrade(VirtualApplianceUpgradeService.java:972) at com.emc.mystic.manager.upgrade.service.DeployAndUpgradeRestServiceImpl.performVirtualApplianceUpgrade(DeployAndUpgradeRestServiceImpl.java:1748) at com.emc.mystic.manager.upgrade.service.PureUpgradeRestServiceImpl.runUpgrade(PureUpgradeRestServiceImpl.java:98) at com.emc.mystic.manager.upgrade.service.DeployAndUpgradeRestServiceImpl.lambda$performUpgrade$1(DeployAndUpgradeRestServiceImpl.java:781) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.vce.lcm.exception.LCMInternalException: Error happened when update VC cert in vxm for 5 times. at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:85) at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:127) at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:133) at com.vce.lcm.core.composite.finalize.CompositeUpgradeFinalizeService.finalize(CompositeUpgradeFinalizeService.java:53) at com.vce.lcm.api.LCMServiceImpl.performCompositeUpgrade(LCMServiceImpl.java:1544) at com.vce.lcm.api.LCMServiceImpl.performUpgrade(LCMServiceImpl.java:448) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times. at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205) at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:72) ... 14 common frames omitted Caused by: com.vce.lcm.exception.LCMException: Error happened when update VC cert in vxm for 5 times. at com.vce.commons.core.VcCertService.updateVcCertInVxm(VcCertService.java:51) at com.vce.lcm.core.composite.finalize.VcCertTask.perform(VcCertTask.java:38) at com.vce.lcm.task.UpgradeTask.execute(UpgradeTask.java:58) at com.vce.lcm.task.SimpleUpgradeTaskExecutor.lambda$execute$0(SimpleUpgradeTaskExecutor.java:65) ... 4 common frames omitted From short.term.log Error returned for the cert verify task："unable to get local issuer certificate" verifying site certificate using: openssl s_client -connect :443 -verify_return_error -brief -CApath /var/lib/vmware-marvin/trust/lin -verify_hostname ... "2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.616574231Z stdout F common.exceptions.OperationException: OperationException: {""message"": ""Exception(b'depth=0 C = US, ST = WA, L = Seattle, O = Moss Adams LLP, OU = IT, CN = attvcenter.mossadams.ma nverify error:num=20:unable to get local issuer certificate n140230863038272:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: n',)"", ""bundle"": """", ""prefix"": ""common.exceptions.OperationException"", ""key"": """", ""field"": """", ""error_code"": ""0"", ""exit_code"": ""-1"", ""params"": []}" "2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.616582687Z stdout F 2022-08-04 23:59:01,616 [INFO] certificate.py validate_site() (184): error reason in Exception(b'depth=0 C = US, ST = WA, L = Seattle, O = Moss Adams LLP, OU = IT, CN = \nverify error:num=20:unable to get local issuer certificate\n140230863038272:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:\n',) start pos and end pos: 109, 167" "2022-08-04 23:59:02,597" microservice.nano-service "2022-08-04T23:59:01.61659078Z stdout F 2022-08-04 23:59:01,616 [ERROR] atomic.py execute() (204): Step GenerateVCCerts raise exception {""code"": ""E3100_Security_CERT_03"", ""params"": ["""", ""unable to get local issuer certificate""], ""message"": ""Failed to validate the certificate of , the reason is: unable to get local issuer certificate""}"

Cause

VC Certificate Chain is not a complete one, for example, certs/lin/00de08c9.1 downloaded from VC missing the .0 cert file.

Resolution

1. Logon VCSA, cd /etc/vmware-vpx/docRoot/certs/, rename the *.1 cert file to .0: # mv 00de08c9.1 00de08c9.02. Logon VxM, cd /var/lib/vmware-marvin/trust/lin, rename the *.1 cert file to .0 # mv 00de08c9.1 00de08c9.03. Run the command below on VxRail Manager to verify if it is succeeded. If so you will see similar outputs like the followings, then re-try the upgrade.#openssl s_client -connect :443 -verify_return_error -brief -CApath /var/lib/vmware-marvin/trust/lin -verify_hostname CONNECTION ESTABLISHED Protocol version: TLSv1.2 Ciphersuite: ECDHE-RSA-AES256-GCM-SHA384 Peer certificate: CN = c2-vc.rackL01.local, C = US Hash used: SHA512 Signature type: RSA Verification: OK Verified peername: c2-vc.rackL01.local Supported Elliptic Curve Point Formats: uncompressed:ansiX962_compressed_prime:ansiX962_compressed_char2 Server Temp Key: ECDH, P-256, 256 bits

Support Cases