Symptoms

Upgrade fails with unable to connect to VC: HTTP transport error: javax.net.ssl.SSLHandshakeExceptionThe environment example: External vCenter vCenter Server 7.0 Update 2d (7.0.2.00500) Build 18455184 X-Large environment (24vCPU/56GB memory) Datacenters = 127 Clusters = 245 (128 vSAN Clusters and some additional ESXi hosts (not vSAN) NumHosts = 499 Datastores = 744 NumVirtualMachines = 2748 NumSessions = 1614 lcm-web.log 2022-03-29 03:27:35,692 INFO [LCM] [lcm-core-0] c.v.c.c.ConnectionHelper [ConnectionHelper.java:44] VC connection timed out, reconnecting. 2022-03-29 03:27:35,692 INFO [LCM] [lcm-core-0] c.v.c.c.ConnectionHelper [ConnectionHelper.java:46] Connecting to vcsa05.XXXXX.com username vxmgr@vsphere.local 2022-03-29 03:27:35,837 ERROR [LCM] [lcm-core-0] c.v.c.c.c.s.ConnectionFactoryImpl [ConnectionFactoryImpl.java:128] unable to connect to VC com.sun.xml.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:132) at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:223) at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:145) at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:139) at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:1136) at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:1050) at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:1019) at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:877) at com.sun.xml.ws.client.Stub.process(Stub.java:463) at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:191) at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:108) at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:92) ... Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1598) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1426) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246) at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:119) ... 42 common frames omitted Caused by: java.io.EOFException: SSL peer shut down incorrectly at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:483) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418) ... 51 common frames omitted vCSA envoy service filtered out https connection due to exceed max. ##### /var/log/vmware/envoy/envoy on VCSA 2022-03-29T03:27:35.683Z warning envoy[3698] [Originator@6876 sub=filter] [C1869144] remote https connections exceed max allowed: 2048 2022-03-29T03:27:35.683Z warning envoy[3698] [Originator@6876 sub=filter] [C1869144] closing connection TCP　 2022-03-29T03:27:35.826Z warning envoy[3684] [Originator@6876 sub=filter] [C1869146] remote https connections exceed max allowed: 2048 2022-03-29T03:27:35.826Z warning envoy[3684] [Originator@6876 sub=filter] [C1869146] closing connection TCP　

Cause

The vCenter Server Appliance (vCSA) has exceeded the maximum number of https remote connections. The envoy service on VCSA has rejected the new connection from VxRail manager.

Resolution

Check the current number of remote https connections by using the following command on the vCSA. netstat -tnep | grep envoy | grep ":443"| wc -l If this confirms the issue, increase the maxRemoteHttpsConnections value.1. Log in to vCSA by SSH using root user.2. Modify /etc/vmware-rhttpproxy/config.xml with vi editor. Add a new line for maxRemoteHttpsConnections in the L4Filter clause.Its default is 2048. And new value has to be tuned for each environment. 4096 12288 3. Restart rhttpproxy service by the following command. service-control --restart vmware-rhttpproxy This command does not affect existing connections.

Support Cases