OPERATIONAL DEFECT DATABASE
...
...
vROPS cloud proxy shows offline in Aria Operation Product UI after successful cluster upgrade from 8.6.x to 8.10.x but CP version is still showing 8.6.2 in UI and cprc-cli command shows CP version 8.10.2 and CP was offline and not collecting data. Below events can be seen in Cloud Proxy logs in haproxy-traffic.log file: 2023-03-14T15:34:23+00:00 localhost haproxy[4014]: backend PrxyRC_BE has no server available! 2023-03-14T15:34:24+00:00 localhost haproxy[4014]: x.x.x.x:58764 [14/Mar/2023:15:34:21.489] PrxyRC_FE~ PrxyRC_BE/VROPS_4 0/0/-1/-1/3021 503 222 - - SC-- 4/4/0/0/3 0/0 "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1" 2023-03-14T15:34:25+00:00 localhost haproxy[5451]: Proxy PrxyRC_FE started. 2023-03-14T15:34:25+00:00 localhost haproxy[5451]: Proxy PrxyRC_BE started. 2023-03-14T15:34:25+00:00 localhost haproxy[5451]: Proxy PrxyRC_UNSECURE_FE started. 2023-03-14T15:34:25+00:00 localhost haproxy[5451]: Proxy PrxyRC_UNSECURE_BE started. 2023-03-14T15:34:25+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_0 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 10ms. 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 2023-03-14T15:34:29+00:00 localhost haproxy[5454]: x.x.x.x:60922 [14/Mar/2023:15:34:26.651] PrxyRC_FE~ PrxyRC_BE/VROPS_1 0/0/-1/-1/3025 503 222 - - SC-- 2/2/1/0/3 0/0 "POST /casa/authorize HTTP/1.1" 2023-03-14T15:34:30+00:00 localhost haproxy[5454]: x.x.x.x:60934 [14/Mar/2023:15:34:27.005] PrxyRC_FE~ PrxyRC_BE/VROPS_2 0/0/-1/-1/3027 503 222 - - SC-- 2/2/0/0/3 0/0 "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1" 2023-03-14T15:34:35+00:00 localhost haproxy[5454]: x.x.x.x:60922 [14/Mar/2023:15:34:32.710] PrxyRC_FE~ PrxyRC_BE/VROPS_3 0/0/-1/-1/3023 503 222 - - SC-- 3/3/1/0/3 0/0 "POST /casa/authorize HTTP/1.1" 2023-03-14T15:34:35+00:00 localhost haproxy[5454]: x.x.x.x:60944 [14/Mar/2023:15:34:32.741] PrxyRC_FE~ PrxyRC_BE/VROPS_4 0/0/-1/-1/3028 503 222 - - SC-- 3/3/0/0/3 0/0 "POST /casa/authorize HTTP/1.1" 2023-03-14T15:34:40+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_1 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 4ms. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 2023-03-14T15:34:48+00:00 localhost haproxy[5454]: x.x.x.x:60922 [14/Mar/2023:15:34:45.773] PrxyRC_FE~ PrxyRC_BE/VROPS_2 0/0/-1/-1/3025 503 222 - - SC-- 3/3/0/0/3 0/0 "POST /casa/authorize HTTP/1.1" 2023-03-14T15:34:55+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_2 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 5ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 2023-03-14T15:35:01+00:00 localhost haproxy[5454]: x.x.x.x:60944 [14/Mar/2023:15:34:58.825] PrxyRC_FE~ PrxyRC_BE/VROPS_3 0/0/-1/-1/3023 503 222 - - SC-- 3/3/0/0/3 0/0 "POST /casa/authorize HTTP/1.1" 2023-03-14T15:35:10+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_3 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 2023-03-14T15:35:14+00:00 localhost haproxy[5454]: x.x.x.x:60922 [14/Mar/2023:15:35:11.875] PrxyRC_FE~ PrxyRC_BE/VROPS_4 0/0/-1/-1/3025 503 222 - - SC-- 3/3/0/0/3 0/0 "POST /casa/authorize HTTP/1.1" 2023-03-14T15:35:25+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_4 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 5ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 2023-03-14T15:35:25+00:00 localhost haproxy[5454]: backend PrxyRC_BE has no server available!
Provide guidance or recommendations to address the "Cloud Proxy Offline" issue after vROPS upgrade from 8.6.x to 8.10.x versions.
The reason is that customer applied a custom certificate on the vROps cluster. As the version was 8.6, Certificates were not automatically transferred to CP (it is in place starting from 8.10), so after the cluster upgrade CP was not able to do a certificate renewal, which caused that connections from CP to vROps cluster to fail due to untrusted certificates.
CP will show in Offline state in vROPS Product UI and CP version will be 8.6.x. However, using cprc-cli -s command, CP version will report as 8.10.xCP collection will stop.
This issue will not occur in Aria Operations (vROPS) version 8.10.x
Note: Please take Snapshot of the Cloud Proxy(s) before implementing the Workaround. Below steps were implemented to Resolve the issue for Custom vROPS SSL Certs: Open SSH sessions on vROPS Primary Node and Cloud Proxy Node and stop below services on CP: service vmware-casa stopservice httpd-north stopservice haproxy stopservice collector stop Copy contents of web_chain.pem and cacert.pem files in cd /storage/vcops/user/conf/ssl (in vROPS Master) using cat /storage/vcops/user/conf/ssl/web_chain.pem and /storage/vcops/user/conf/ssl/cacert.pemBackup server.ca.pem file in cd /storage/vcops/user/conf/ssl (in Cloud Proxy) using "cp server.ca.pem server.ca.pem_bak" and create a file named server.root.ca.pem using "cp server.ca.pem server.root.ca.pem" command in cd /storage/vcops/user/conf/ssl (in Cloud Proxy) and paste the contents of web_chain.pem file in server.ca.pem file and contents of cacert.pem in server.root.ca.pem file in CP.At this point vROPS Cluster's web_chain.pem content should have been copied to server.ca.pem in Cloud Proxy and content of vROPS Cluster's cacert.pem should have been copied to server.root.ca.pem in Cloud Proxy.Assign appropriate file ermission using below command: chmod 644 server.ca.pemchmod 644 server.root.ca.pem Then assign file ownership using below commands: chown admin:admin server.ca.pemchown admin:admin server.root.ca.pem Reboot Cloud Proxy using reboot -f command
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
