OPERATIONAL DEFECT DATABASE
...
...
VMware vSphere Profile-Driven Storage Service and VMware Update Manager service failing to startPermissions on the vCenter server object is also inaccessible on the web client sps.log: <YYYY-MM-DDTHH:MM:SS> [main] INFO opId=sps-Main-360774-610 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - Service solution user token acquired successfully. Expiration time: Wed May 15 20:40:38 UTC 2019<YYYY-MM-DDTHH:MM:SS> [main] INFO opId=sps-Main-360774-610 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - SPS solution user initialized successfully<YYYY-MM-DDTHH:MM:SS> [pool-1-thread-1] WARN opId=sps-Main-360774-610 com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler - Error while trying to do reloginjava.lang.IllegalStateException: Client initialization is not complete! at com.google.common.base.Preconditions.checkState(Preconditions.java:174) at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask.get(ConnectionInitializationTask.java:80) at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getClient(VpxdClientManagerImpl.java:169) at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getCurrentSession(VpxdClientManagerImpl.java:180) at com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler.retry(RequestRetryHandler.java:76) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.setException(MethodInvocationHandlerImpl.java:525) at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156) at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)<YYYY-MM-DDTHH:MM:SS> [main] ERROR opId=sps-Main-360774-610 com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - VPXD client login failed.<YYYY-MM-DDTHH:MM:SS> [main] ERROR opId=sps-Main-360774-610 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service at com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException.fromEx(VpxdException.java:53) at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:159) at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:129) at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:34) at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask$CallableTemplate.call(ConnectionInitializationTask.java:118) at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.initialize(VpxdClientManagerImpl.java:106) at com.vmware.sps.StorageMain.commonInitialization(StorageMain.java:188) at com.vmware.sps.StorageMain.main(StorageMain.java:67)Caused by: (vim.fault.NoPermission) { faultCause = null, faultMessage = null, object = ManagedObjectReference: type = Folder, value = group-d1, serverGuid = 4082b4f9-c668-4546-aba1-9f1cfed71f07, privilegeId = System.View} vmware-vum-server-1.log: <YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,92] Ufa shutdown manager stopping... first make sure this thread is enlisted.<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=ThreadPool] Thread enlisted<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,97] Ufa shutdown manager stopping... Total number of callbacks: 1<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=VcIntegrity] Error on logout (ignored): Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem, timeout, or service overload.<YYYY-MM-DDTHH:MM:SS> warning vmware-vum-server[21223] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fa4dc03d230, h:15, <TCP '127.0.0.1 : 50448'>, <TCP '127.0.0.1 : 80'>>, e: 111(Connection refused)<YYYY-MM-DDTHH:MM:SS> warning vmware-vum-server[21162] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fa5105085c0, h:15, <TCP '127.0.0.1 : 50450'>, <TCP '127.0.0.1 : 80'>>, e: 111(Connection refused)<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[21220] [Originator@6876 sub=VcIntegrity] Error on logout (ignored): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,109] Ufa shutdown manager stopped<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[21166] [Originator@6876 sub=ThreadPool] Thread delisted vpxd.log: <YYYY-MM-DDTHH:MM:SS> verbose vpxd[35323] [Originator@6876 sub=[SSO] opID=sps-Main-226767-398-34] [UserDirectorySso] NormalizeUserName(<SSO Domain>\vpxd-extension-25b86efe-fc6e-49a1-93d9-1b4b8694aa44, false) res: <SSO Domain>\vpxd-extension-25b86efe-fc6e-49a1-93d9-1b4b8694aa44<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=AuthorizeManager opID=sps-Main-226767-398-34] [Auth]: User <SSO Domain>\vpxd-extension-25b86efe-fc6e-49a1-93d9-1b4b8694aa44<YYYY-MM-DDTHH:MM:SS> verbose vpxd[35323] [Originator@6876 sub=Vmomi opID=sps-Main-226767-398-34] Invoke error: vim.SessionManager.loginByToken session: 528a0ce5-c69e-d8a6-bb4f-43448412dfe6 Throw: vim.fault.NoPermission<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=vpxLro opID=sps-Main-226767-398-34] [VpxLRO] -- FINISH lro-40<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=Default opID=sps-Main-226767-398-34] [VpxLRO] -- ERROR lro-40 -- SessionManager -- vim.SessionManager.loginByToken: vim.fault.NoPermission:--> Result:--> (vim.fault.NoPermission) {--> faultCause = (vmodl.MethodFault) null
The issue is seen if there are duplicate entries for the default administrator
Unable to add permissions on the vCenter server objects
This issue is resolved in vCenter Server 6.7 Update 3l build 17138064 and 7.0 Update 3c build 19234570 or later
** Ensure to take a functional backup of the nodes *** Overview of Backup and Restore options in vCenter Server 6.x (2149237)https://kb.vmware.com/s/article/2149237 log in to the VCSA using sshstop the service using the following command service-control --stop vmware-vpxdservice-control --stop vmware-content-library log in to the database using the following command: command: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres Execute the following steps to validate and remove the duplicate entries VCDB=# select * from vpx_access;Sample:id | principal | role_id | entity_id | flag | surr_key-----+---------------------------+---------+-----------+------+----------1 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 18 | SSO.VCENTER\Administrator | -1 | 7 | 1 | 2901 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 11 Additionally validate the entries where principal value is <SSO Domain>\Administrator VCDB=# select * from vpx_access where principal like '%Administrator%'; id | principal | role_id | entity_id | flag | surr_key-----+-----------------------------+---------+-----------+------+---------- 1 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 1 209 | SSO.VCENTER\Administrator | -1 | 2 | 1 | 3 601 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 7 Delete the entries except id = 1 command: delete from vpx_access where id=<ID from above output>;Note: command needs to be executed for all the additional entries start the service using the following command: service-control --start vmware-vpxdservice-control --start vmware-content-libraryservice-control --start vmware-spsservice-control --start vmware-updatemgr Log in to the web client and validate the functionality
VMware Skyline Health Diagnostics for vSphere - FAQ
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
