Symptoms

When running the shut-down wizard, the validate step is failing: Verify and ensure that the current cluster has no TANZU service. Error: Internal error occurs Confirm that Tanzu is not present in the environment.The wizard fails with this same error even after a restart of the service virtual machines (VMs).VxRail Manager Web.log 2023-07-14 23:35:10.445+0000 ERROR [pool-4-thread-1] com.emc.mystic.manager.commons.emc.precheck.ParameterizedPrechecker ParameterizedPrechecker.preCheck:31 - Unknow exception thrown from abstract prechecker com.vmware.vim25.eam.RuntimeFaultFaultMsg: EAM is still loading from database. Please try again later. at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?] at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:117) ~[jaxws-rt-2.3.3-b01.jar:2.3.3-b01] at com.sun.xml.ws.client.sei.StubHandler.readResponse(StubHandler.java:223) ~[jaxws-rt-2.3.3-b01.jar:2.3.3-b01] at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:176) ~[jaxws-rt-2.3.3-b01.jar:2.3.3-b01] at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:263) ~[jaxws-rt-2.3.3-b01.jar:2.3.3-b01] at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89) ~[jaxws-rt-2.3.3-b01.jar:2.3.3-b01] at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:62) ~[jaxws-rt-2.3.3-b01.jar:2.3.3-b01] at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:131) ~[jaxws-rt-2.3.3-b01.jar:2.3.3-b01] at com.sun.proxy.$Proxy417.queryAgency(Unknown Source) ~[?:?] at com.emc.mystic.manager.commons.vmware.VimOperator.isTanzuEnv(VimOperator.java:4511) ~[mystic.manager.commons.vmware-7.0.451.jar:? ] at com.emc.mystic.manager.web.service.precheck.TanzuServiceExistChecker.handlePrecheck(TanzuServiceExistChecker.java:41) ~[classes/: ?] at com.emc.mystic.manager.commons.emc.precheck.ParameterizedPrechecker.preCheck(ParameterizedPrechecker.java:27) ~[mystic.manager.commons.emc-7.0.451.jar:?] at com.emc.mystic.manager.web.service.PrecheckServiceImpl.lambda$explicitPrecheck$3(PrecheckServiceImpl.java:173) ~[classes/:?] at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?] at com.emc.mystic.manager.web.service.PrecheckServiceImpl.lambda$explicitPrecheck$4(PrecheckServiceImpl.java:146) ~[classes/:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2023-07-14 23:35:10.490+0000 INFO [tomcat-http--17] com.emc.mystic.manager.web.controller.StatusControllerImpl StatusControllerImpl.status:75 - Get status from operation status, id = 6df807d9-6db1-4877-9c21-685df6ef71a2 2023-07-14 23:35:10.505+0000 INFO [pool-4-thread-1] com.emc.mystic.manager.datadomain.service.VxRailSystemDomainServiceImpl VxRailSystemDom ainServiceImpl.updateOperationStatus:325 - Operation status updated. id: 788

Cause

There is an EAM certificate problem present in the vCenter. This is affecting the EAM service, which is necessary for the Shut Down validation. 

Resolution

Instructions: Take a snapshot of vCenter.Log in to the vCenter Server Appliance using SSH.Run this command to enable access to the Bash shell: shell.set --enabled true Type shell and press Enter.Run this command to retrieve the vpxd-extension solution user certificate and key: mkdir /certificate /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key Run this command to update the extension's certificate with vCenter Server. python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u Administrator@vsphere.local Note: If this produces the error "Hostname mismatch, certificate is not valid for 'localhost'," change 'localhost' to the FQDN or IP of the vCenter. The process is checking this value against the SAN entries of the certificate. Note: The default user and domain is Administrator@vsphere.local. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Administrator@domain.local password Restart EAM and start the rest of the services with these commands: service-control --stop vmware-eam service-control --start --all In certain situations, you might receive the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'." This error can be safely ignored if you are getting the error after the message "Successfully updated certificate for "com.vmware.vim.eam" extension" as this message confirms that Extension certificate updated successfully with vCenter Server.

Support Cases