OPERATIONAL DEFECT DATABASE
...
...
The application is unable to establish new network connections for HTTPs requests.
ECS nodes have a 1000 connection limit per node. If this limit is exceeded, dataheadsvc shows the following error: 169.xxx.2.9 2021-02-24T21:13:08,792 [qtp85102332-247-acceptor-1@4fda771a-TrafficMetricsNetworkTrafficServerConnector@49de2cc4{HTTP/1.1,[http/1.1]}{10.xxx.xxx.xxx:9020}] INFO ConnectionLimit.java (line 177) Connection Limit(1000) reached for [TrafficMetricsNetworkTrafficServerConnector@49de2cc4{HTTP/1.1,[http/1.1]}{10.xxx.xxx.xxx:9020}, TrafficMetricsNetworkTrafficServerConnector@4e293c35{SSL,[ssl, http/1.1]}{10.xxx.xxx.xxx:9021}] To see more information, parse the logs with svc_log to count the entries for "Connection Limit (1000) reached" in the last 30 minutes. Commands: # svc_log dataheadsvc "Connection Limit(1000) cleared" -start "30 minutes ago" -sh -st minute # svc_log dataheadsvc "Connection Limit(1000) cleared" -start "24 hour ago" -sh -st hour Example: admin@node1:~># svc_log dataheadsvc "Connection Limit(1000) cleared" -start "30 minutes ago" -sh -st minute svc_log v1.0.24 (svc_tools v1.8.6.0) Started 2021-02-26 18:47:10Running on nodes: <All nodes> Time range: 2021-02-24 21:00:00 - 2021-02-24 21:30:59 Filter string(s): 'Connection Limit(1000) cleared' Show nodename(s): True Search reclaim logs (if any): FalseCount of message occurrences per minute: # Log Time Entries ------------------------------2021-02-24 21:01:xx 0 2021-02-24 21:02:xx 0 2021-02-24 21:03:xx 0 2021-02-24 21:04:xx 41 2021-02-24 21:05:xx 134 2021-02-24 21:06:xx 8 2021-02-24 21:07:xx 10 2021-02-24 21:08:xx 11 2021-02-24 21:09:xx 0 2021-02-24 21:10:xx 0 2021-02-24 21:11:xx 0 2021-02-24 21:12:xx 412 2021-02-24 21:13:xx 650 2021-02-24 21:14:xx 0 2021-02-24 21:15:xx 0 2021-02-24 21:16:xx 0 2021-02-24 21:17:xx 0 2021-02-24 21:18:xx 0 2021-02-24 21:19:xx 0 2021-02-24 21:20:xx 0 2021-02-24 21:21:xx 0 2021-02-24 21:22:xx 0 2021-02-24 21:23:xx 0 2021-02-24 21:24:xx 0 2021-02-24 21:25:xx 0 2021-02-24 21:26:xx 0 2021-02-24 21:27:xx 0 2021-02-24 21:28:xx 0 2021-02-24 21:29:xx 0 2021-02-24 21:30:xx 0Total: 1266
This issue is generally an application issue with how the application opens and closes network connections. The user's application or load balancer should handle the connections and should close the connections once complete. The application should always try to resolve the issue on their side first before changing ECS as this causes more load on the ECS. To help identify the issue, use netstat during the event to get the IPs that have established connections. Command: # sudo ss | grep 9020 Example: admin@ecsnode1:~> sudo ss | grep 9020ESTAB 0 0 [::ffff:10.xx.xxx.71]:9020 [::ffff:10.xxx.xxx.9]:56134 To see the number of connections in current time, use this command: admin@ecsnode1:~> sudo ss -tn sport = :9020 | wc -l
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
