Symptoms

Application unable to establish new network connections for HTTPs requests.

Cause

ECS nodes have a 1000 connection limit per node. If this limit is exceeded, dataheadsvc shows the following error: 169.xxx.2.9 2021-02-24T21:13:08,792 [qtp85102332-247-acceptor-1@4fda771a-TrafficMetricsNetworkTrafficServerConnector@49de2cc4{HTTP/1.1,[http/1.1]}{10.xxx.xxx.xxx:9020}] INFO ConnectionLimit.java (line 177) Connection Limit(1000) reached for [TrafficMetricsNetworkTrafficServerConnector@49de2cc4{HTTP/1.1,[http/1.1]}{10.xxx.xxx.xxx:9020}, TrafficMetricsNetworkTrafficServerConnector@4e293c35{SSL,[ssl, http/1.1]}{10.xxx.xxx.xxx:9021}] To see more information, further parse the logs with svc_log to count the entries for "Connection Limit (1000) reached" in the last 30 minutes.Commands: # svc_log dataheadsvc "Connection Limit(1000) cleared" -start "30 minutes ago" -sh -st minute # svc_log dataheadsvc "Connection Limit(1000) cleared" -start "24 hour ago" -sh -st hour Example: admin@node1:~># svc_log dataheadsvc "Connection Limit(1000) cleared" -start "30 minutes ago" -sh -st minute svc_log v1.0.24 (svc_tools v1.8.6.0) Started 2021-02-26 18:47:10 Running on nodes: Time range: 2021-02-24 21:00:00 - 2021-02-24 21:30:59 Filter string(s): 'Connection Limit(1000) cleared' Show nodename(s): True Search reclaim logs (if any): False Count of message occurrences per minute: # Log Time Entries ------------------------------ 2021-02-24 21:01:xx 0 2021-02-24 21:02:xx 0 2021-02-24 21:03:xx 0 2021-02-24 21:04:xx 41 2021-02-24 21:05:xx 134 2021-02-24 21:06:xx 8 2021-02-24 21:07:xx 10 2021-02-24 21:08:xx 11 2021-02-24 21:09:xx 0 2021-02-24 21:10:xx 0 2021-02-24 21:11:xx 0 2021-02-24 21:12:xx 412 2021-02-24 21:13:xx 650 2021-02-24 21:14:xx 0 2021-02-24 21:15:xx 0 2021-02-24 21:16:xx 0 2021-02-24 21:17:xx 0 2021-02-24 21:18:xx 0 2021-02-24 21:19:xx 0 2021-02-24 21:20:xx 0 2021-02-24 21:21:xx 0 2021-02-24 21:22:xx 0 2021-02-24 21:23:xx 0 2021-02-24 21:24:xx 0 2021-02-24 21:25:xx 0 2021-02-24 21:26:xx 0 2021-02-24 21:27:xx 0 2021-02-24 21:28:xx 0 2021-02-24 21:29:xx 0 2021-02-24 21:30:xx 0 Total: 1266

Resolution

This issue is generally an application issue, how the application opens and closes network connections. The user's application or load balancer should be handling the connections and should close the connections once complete. Application should always try to resolve the issue on their side first before changing ECS as this causes more load on the ECS.To help in identifying the issue, use netstat during the event to get the IPs that have established connections.Command: # sudo ss | grep 9020 Example: admin@ecsnode1:~> sudo ss | grep 9020 ESTAB 0 0 [::ffff:10.xx.xxx.71]:9020 [::ffff:10.xxx.xxx.9]:56134

Support Cases