Symptoms
In OneFS 7.2.1.x and earlier these messages were only visible when debug logging was enabled.In OneFS versions 8.0.0.7 and 8.1.0.2 and later /var/log/messages can be spammed with the following types of messages:2019-01-16T04:07:47+01:00 Cluster-12 /boot/kernel.amd64/kernel: TCP: [128.221.254.36]:46426 to [128.221.254.34]:7117; syncache_socket: Socket create failed due to limits or memory shortage
Cause
This type of message should normally only be logged when network debug logging is enabled. The message relates to transient memory management issues associated with clearing the isi_stats_d persistence database.NOTE: There are variations of this message whereby other protocols can produce this message as well. For example, if there are too many connection attempts at any given time for NFS, port 2049 may be displayed instead of 7117. This would require an adjustment of workflow to the cluster to reduce the influx of connection attempts. Such as reducing the amount of clients connecting or (potentially) adding hardware.
Resolution
In OneFS 8.1 and later the following sysctl is erroneously enabled by default (set to 1):isi_sysctl_cluster net.inet.tcp.log_newconn_failedDisabling it by setting the sysctl to 0 reverts the behaviour so that these messages are hidden from regular logging:
# isi_sysctl_cluster net.inet.tcp.log_newconn_failed=0
