Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
This issue is seen in a scaled setup.From the SDDC Manager UI, on navigating to Update/Patches tab the error "Retrieving all scheduled and in-progress bundles failed. Http failure response" 504 Gateway Time-out is seen.Following LCM APIs are taking time (sometimes more than one minute around 1.45 minutes to 2 minutes). The API response time worsens when called concurrently. - /v1/upgrades - /lcm/inventory/upgrades
This document will list the steps to apply the HotPatch to fix the above issue.
In a scaled setup the API calls time out.
Hotpatch attached to the knowledge base article has been provided to address the issue.Please follow the below steps to apply the hotpatch: Take Snapshot of SDDC Manager (by logging Management VC)Download the vcf31021-hp-JULY-2021.tar file to /home/vcfLogin to SDDC Manager VM using vcf user and change to "root" usercd /home/vcf/Extract the vcf31021-hp-JULY-2021.tar file (tar -xvf vcf31021-hp-JULY-2021.tar)Back up the current binaries (jars) root@sddc-manager [ /home/vcf ]# mkdir backuproot@sddc-manager [ /home/vcf ]# cd backup/ LCM jar back up: Stop the lcm service. systemctl stop lcm Take a back up of current lcm binaries (Jars) cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-adapters-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-adapters-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-app-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-app-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-common-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-common-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-orch-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-orch-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-services-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-services-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d)cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar$(date +%Y%m%d)cp /opt/vmware/vcf/lcm/lcm-app/lib/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar /home/vcf/backup/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar.$(date +%Y%m%d) SDDC Manager UI back up: Stop the sddc-manager-ui-app service systemctl stop sddc-manager-ui-app Take a back up of current UI files cp -r /opt/vmware/vcf/sddc-manager-ui-app/ /home/vcf/backup/sddc-manager-ui-app.$(date +%Y%m%d)/ Verify the backed up LCM jars and SDDC Manager UI files at /home/vcf/backup Replace the new LCM jars and SDDC Manager UI RPM and start the service: cd /home/vcf/vcf31021-hp-JULY-2021 Replace lcm jars cp lcm-adapters-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-adapters-3.10.2-vcf31021RELEASE.jarcp lcm-app-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-app-3.10.2-vcf31021RELEASE.jarcp lcm-common-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-common-3.10.2-vcf31021RELEASE.jarcp lcm-orch-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-orch-3.10.2-vcf31021RELEASE.jarcp lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jarcp lcm-services-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-services-3.10.2-vcf31021RELEASE.jarcp lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jarcp lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jarcp caffeine-2.7.0.jar /opt/vmware/vcf/lcm/lcm-app/lib/caffeine-2.7.0.jarcp spring-context-support-5.2.3.RELEASE.jar /opt/vmware/vcf/lcm/lcm-app/lib/spring-context-support-5.2.3.RELEASE.jar Make sure replaced lcm jars has below permission and ownership. cd /opt/vmware/vcf/lcm/lcm-app/lib chown vcf_lcm:vcf caffeine-2.7.0.jarchown vcf_lcm:vcf spring-context-support-5.2.3.RELEASE.jarchmod 750 caffeine-2.7.0.jarchmod 750 spring-context-support-5.2.3.RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf lcm-adapters-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf lcm-app-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf lcm-common-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf lcm-orch-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf lcm-services-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf caffeine-2.7.0.jar-rwxr-x--- 1 vcf_lcm vcf spring-context-support-5.2.3.RELEASE.jar Example Commands : # ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-adapters-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 191004 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-adapters-3.10.2-vcf31021RELEASE.jar# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-app-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 85067 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-app-3.10.2-vcf31021RELEASE.jar# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-common-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 57424 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-common-3.10.2-vcf31021RELEASE.jar# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-orch-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 56967 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-orch-3.10.2-vcf31021RELEASE.jar# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 167907 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-rest-api-controller-3.10.2-vcf31021RELEASE.jar# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-services-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 352472 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-services-3.10.2-vcf31021RELEASE.jar# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 96078 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-sql-dal-client-3.10.2-vcf31021RELEASE.jar#ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 744570 Jul 2 10:00 /opt/vmware/vcf/lcm/lcm-app/lib/lcm-vmware-primitive-impl-3.10.2-vcf31021RELEASE.jar# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/caffeine-2.7.0.jar-rwxr-x--- 1 vcf_lcm vcf 846706 Jul 2 10:34 /opt/vmware/vcf/lcm/lcm-app/lib/caffeine-2.7.0.jar# ls -ltr /opt/vmware/vcf/lcm/lcm-app/lib/spring-context-support-5.2.3.RELEASE.jar-rwxr-x--- 1 vcf_lcm vcf 182130 Jul 2 10:52 /opt/vmware/vcf/lcm/lcm-app/lib/spring-context-support-5.2.3.RELEASE.jar Update the lcm properties files. Take backup of /opt/vmware/vcf/lcm/lc-app/conf/application-prod.properties and /opt/vmware/vcf/lcm/lcm-app/conf/jpa-prod.properties cp -fp /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties /home/vcf/backup/application-prod.properties.$(date +%Y%m%d) cp -fp /opt/vmware/vcf/lcm/lcm-app/conf/jpa-prod.properties /home/vcf/backup/jpa-prod.properties.$(date +%Y%m%d) Add the below content to /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties file. lcm.inventory.upgrade.cache.initial.capacity=2lcm.inventory.upgrade.cache.maxsize=4lcm.inventory.upgrade.cache.expiry.seconds=300 Add the below content to /opt/vmware/vcf/lcm/lcm-app/conf/jpa-prod.properties filelcm.datasource.socket.timeout=30 Upgrade the SDDC Manager UI RPM: cd /home/vcf/vcf31021-hp-JULY-2021 rpm -Uvh sddc-manager-ui-app-3.10.2-18290131.noarch.rpm Start the LCM service. #systemctl start lcm Please Wait for two minutes for service to come up. You can check the status using # curl localhost/lcm/about | json_pp % Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed100 212 0 212 0 0 4510 0 --:--:-- --:--:-- --:--:-- 4608{"buildJdk" : "1.8.0_221","serviceId" : "defaultUUID","name" : "LCM","buildDate" : "2021-05-11 15:21:24 UTC","author" : "VMware","builtBy" : "mts","createdBy" : "Apache Maven 3.5.0","version" : "3.10.2-vcf31021RELEASE-18015031"} Start the SDDC manager UI service: systemctl restart sddc-manager-ui-db.service (wait for 1 minute) systemctl start sddc-manager-ui-app (wait for two minutes for the service to come up) You can check the status using $ curl localhost/sddc-manager-ui-app/about | json_pp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 240 100 240 0 0 48000 0 --:--:-- --:--:-- --:--:-- 48000 { "builtBy" : "mts", "dbSchemaVersion" : 1, "createdBy" : "Apache Maven 3.5.0", "author" : "VMware", "name" : "SDDC-MANAGER-UI-APP", "version" : "3.10.2-vcf31021RELEASE-18290131", "id" : "4a0a3a15-496f-4aa1-9040-d11c3eb8df43", "buildDate" : "2021-07-08T12:48:46Z" } Apply the VersionAlias for SDDC_MANAGER_UI (as version of the SDDC Manager UI is changed) Take backup of /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml file. cp -fp /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml /home/vcf/backup/VersionAlias.yml.$(date +%Y%m%d) Run the following curl command to add SDDC_MANAGER_UI specific version alias information. curl 'http://localhost/v1/system/settings/version-aliases/SDDC_MANAGER_UI/3.10.2-vcf31021RELEASE-18015030' -X PUT -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"aliases" : [ "3.10.2-vcf31021RELEASE-18290131" ], "forceUpdate" : true}' Verify that the below content is added /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml file. SDDC_MANAGER_UI: - base: 3.10.2-vcf31021RELEASE-18015030 alias: - 3.10.2-vcf31021RELEASE-1829013 NOTE: Having Multiple SDDC Manager UI sessions usage might cause delay in UI response in larger VCF deployments. Recommendation is to have one or two SDDC Manager UI sessions.Also the following nginx configuration needs to be done( if not done already) Take a backup of /etc/nginx/nginx.conf Add/Modify the below routes in http section. location ~ ^/v1/(bundles|upgradables|upgrades|system)(.*) { proxy_read_timeout 600; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_pass http://127.0.0.1:7400/v1/$1$2$is_args$args; } location /lcm/ { proxy_read_timeout 600; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_pass http://127.0.0.1:7400; } #adding timeout for lcm internal api after /lcm/ route location ~ ^/internal/lcm/(.*) { proxy_read_timeout 600; proxy_connect_timeout 600; proxy_send_timeout 600; proxy_pass http://127.0.0.1:7500; } Add/Modify the below routes in https section # lcm v1 public APIs location ~ ^/v1/(bundles|upgradables|upgrades|system)(.*) { proxy_read_timeout 600; proxy_connect_timeout 600; proxy_send_timeout 600; auth_basic "closed site"; auth_basic_user_file /etc/nginx/.htpasswd; proxy_pass http://127.0.0.1:7400/v1/$1$2$is_args$args; } #Added this after sddc-manager-ui route location ~ ^/internal/lcm/(.*) { proxy_read_timeout 600; proxy_connect_timeout 600; proxy_send_timeout 600; auth_basic "closed site"; auth_basic_user_file /etc/nginx/.htpasswd; proxy_pass http://127.0.0.1:7500; } Save the file.Verify the syntax: nginx -tRestart nginx: systemctl restart nginx