Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
In certain circumstances, remote https connections to vCenter may become exhausted.By default, these connections are set to a maximum of 2048 connections. One possible contributor to the limit being reached:When using VDDK 7.0.0 in a backup solution, it increases the amount of connections to vCenters Envoy proxy service. When this limit is reached, it may result in failed backups as part of those backup workflows. To confirm that the limit is being reached, please do the following: VDDK logging (file path depends on backup software):2022-02-09T15:15:47.444+02:00 error -> [297506] [Originator@6876 sub=IO] HandshakeCb; <S SL(<io_obj p:0x00007fb7d806d1b8, h:32, <TCP '10.239.17.203 : 41704'>, <TCP '10.241.113.4 : 443'>>)>; error: N7Vmacore15SystemExceptionE(Connection reset by peer: The conne ction is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem, timeout, or service overload.) VC Envoy logging (/var/log/vmware/envoy/envoy.log):2022-02-09T13:15:47.443Z warning envoy[40146] [Originator@6876 sub=filter] [C15632615] remote https connections exceed > max allowed: 20482022-02-09T13:15:47.443Z warning envoy[40146] [Originator@6876 sub=filter] [C15632615] closing connection TCP<10.239.17.203:41704, 10.241.113.4:443>
This article provides a way for customer to remediate the VC connection limit problem.Note: This article does NOT apply to local http/https connections happening within the vCenter Appliance such as service to service connections over the local loopback interface. (e.g. vsan-health making calls to VUM)
The limit is more easily reached relates to vCenters Envoy proxy service since vCenter 7.0.x.When using VDDK 7.0.0 in a backup solution, it increases the amount of connections to the Envoy proxy service, increasing the likelihood that the maxRemoteHttpsConnection limit is reached. When this limit is reached, it may result in failed backups as part of those backup workflows.
None
To avoid or remediate this scenario, a number of workarounds exist which can be applied in isolation or in parallel as needed: Upgrade to 7.0 U1 or later VDDK version to avail of code improvements Increase the maximum RemoteHttpsConnections limit. See following steps: To change envoy's proxy limit for external inbound HTTPS connections you need to change config options according to vCenter version: For vCenter Server 7.0.2 and later, change/add this config option “maxRemoteHttpsConnections” in /etc/vmware-rhttpproxy/config.xml <config> <envoy> <L4Filter> <maxRemoteHttpsConnections>3072</maxRemoteHttpsConnections> </L4Filter> </envoy></config> For vCenter Server 7.0.1(Recommended minimum version) , change/add the config options “maxHttpConnections” and “maxHttpsConnections” in /etc/vmware-rhttpproxy/config.xml For vCenter Server 7.0.0, change the config option “max_http_connections” and “max_https_connections” in /etc/vmware-envoy/config.yaml Note vCenter Server 7.x has multiline comments in the /etc/vmware-rhttpproxy/config.xml file so you will need to uncomment the desired option when you change it's value e.g. Before: <L4Filter> <!-- <maxRemoteHttpsConnections>2048</maxRemoteHttpsConnections>After: <L4Filter> <maxRemoteHttpsConnections>3072</maxRemoteHttpsConnections> <!--After you have updated and saved the file, you need to reload the configuration by one of the two ways:1). Restart rhttpproxy service:$ service-control --restart rhttpproxy2). Send SIGHUP signal to rhttpproxy service:$ kill -1 `pidof rhttpproxy` NOTE: While no hard limit exists to the maximum configurable limit, it is recommended to increase with caution as an overly extended limit may impact VC performance due to excessive connections and subsequent resource consumption. Increasing to 3072 is verified and suitable for most cases. If customer need to increase to number > 3072, it’s necessary to check too aggressive connection usage in the backup solution design.