OPERATIONAL DEFECT DATABASE
...
...
1. Performance degradation in vcenter ui after patching the vcenter from 7.0.2.00500 to Version: 7.0.3.00600, things like edit settings take too much time to load 2. The ui freezes and moving around the inventory is very slow. 3. you tried kb https://kb.vmware.com/s/article/87837 and it does not fix the issue 4. logs : In the dataservice.log:[2022-08-05T18:42:19.251Z] [WARN ] http-nio-5090-exec-100 70001522 100009 200004 ProfilingLogService [queryId: q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-362290:ee06ca2d-a130-4a03-bedc-0fc123f8166b:1206883735] The Data Service detected slow execution:The query batch execution took too long: 120002 milliseconds. However in vpxd.log q-1968 the query completed at same second. 2022-08-05T18:40:19.255Z info vpxd[09376] [Originator@6876 sub=vpxLro opID=q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-362290:ee06ca2d-a130-4a03-bedc-0fc123f8166b:1206883735:01-61] [VpxLRO] -- BEGIN lro-62629 -- ResourceModel -- vim.dp.ResourceModel.queryBatch -- 528f9b01-c819-7f63-3cb8-939e54e8be5c(525cefe1-5aab-74d9-73ff-1e70fd01dfdf)2022-08-05T18:40:19.255Z info vpxd[09376] [Originator@6876 sub=vpxLro opID=q-1968:h5ui-getProperties:urn:vmomi:VirtualMachine:vm-362290:ee06ca2d-a130-4a03-bedc-0fc123f8166b:1206883735:01-61] [VpxLRO] -- FINISH lro-62629 In envoy-access.logif you run - grep "127.0.0.1:47988 127.0.0.1:8085" envoy-access-xxxx.log | cut -d' ' -f15-17and look at the results you will find85195 1 8519483579 2 8357681663 2 8166072373 2 7237068990 1 6898864241 1 6423944551 1 4454942367 16 4235141689 1 4168733888 1 3388628108 1 2810612291 2 12288The first number is the "total duration of the request+response", the second number is time spent in envoy and the third number is time spent in vpxd.Some high values will be grouped togetherAlso you will find a large number of http2 requestsI.e :less -i envoy-access-*.log | grep -c "external-vecs/http2"47701
1.Fix such performance issue after the patching.
1. Vcenter performance degradation and freezes.
This is resolved in vCenter Server Update 3i (build number 20845200).
1. Take snapshots of your enviroment.2. Make a backup of /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd-rhttpproxy-endpoint.conf, preferably under /root and NOT under the same directory3. Remove "http2:" from the following line:from/sdk http2:local 8085 redirect allowto/sdk local 8085 redirect allow4. Execute either command (the second one gives a bit more guarantee that the endpoint would be reloaded but will make the rhttpproxy service temporary unavailable).kill -HUP `pidof rhttpproxy`service-control --restart rhttpproxy5. Consider restarting the envoy front proxy (will disconnect all current connections (e.g. UI) and make all VC users reconnect)service-control --restart envoyif the workaround does not seem to have some effect, it may be easily reverted by reverting /sdk endpoint to http2 by copying the backed up conf file on top of the original file and re-doing steps (4) and (5).
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
