Loading...
Loading...
Users may experience intermittent latency or timeouts when attempting to access shares stored on an Isilon cluster; access is restored with no intervention within seconds/minutes. Authentication and/or share enumeration may take multiples of 5 seconds (e.g. 25 seconds, for example) to complete, but on periodic intervals (~15 minutes per node precisely). One node may experience the issue, while others do not. When using SmartConnect round-robin, the issue may appear more frequent as each node experiences this issue independently of each other. NOTE: This KB covers one possible cause for slow SMB authentication and share enumeration; other causes may exist. The recommendation for resolution here is a means of ruling out one probable cause. The following may appear in the isi_cbind_d logs around the time of the issues: isi_cbind_d[76119]: [0x800703400]bind: CBIND_send_query(1161) Warning: Stallset dns has no available stalls Which can be found by running: # isi_for_array 'zegrep Stallset /var/log/isi_cbind_d.log*'
By default, every 15 minutes we will expire our AD LDAP DC connection proactively. The DC connection expiration happens at the time the connection is used (i.e. during authentication), which will start the process of: Lock the AD DC connection Mutex (this blocks all requests until a new DC is selected) Expire the existing connection Enumerate a list of DC's to connect to (DNS SRV record; _ldap._tcp.dc._msdcs.domain.com ) Resolve DNS names to IP address from the list of DC's (A record DNS lookups) Send CLDAP ping to all DC's; wait for fastest responders (stop sending/receiving after 10 ms from first response) Intelligently select from the responding DC's (semi-random using historical DC statistics) It is during the DNS A record lookup phase (4, above) where delays may be introduced. By default on a PowerScale cluster the groupnet DNS cache is enabled ( isi_cbind_d ). When isi_cbind_d (DNS cache daemon) is unable to service DNS lookups (both fails to respond to the request and fails to reach out to an external DC), the kernel DNS resolver will failover to the next available DNS server via /etc/resolv.conf (or more specifically, the DNS servers configured for the groupnet in question), which incurs a 5 second timeout/delay for each A record query. The total delay attributed to DNS lookups is equal to 5 times the number of A records we need to resolve. If the record count exceeds 12 records, this will trigger a 60 second timeout and reset from the client. Errors will be seen on the client relating to inability to reach the cluster or share. NOTE: The most common culprit in this situation is attributed to isi_cbind_d DNS lookups, but can also be the result of other factors as well.
Assuming the following: The issue self-remediates with no intervention from admins Delays hit 5 second increments of time Frequency on a per-node basis is ~15 minutes The quickest path to resolution and validation is to disable DNS caching for the groupnet in use: # isi network groupnets modify <groupnet> --dns-cache-enabled=false If it is preferred to keep DNS cache enabled, a mitigation option would be to restart the DNS caching service on all nodes: # isi_for_array 'killall -9 isi_cbind_d' After restarting the service, verify that it has been restarted on all nodes (based on timestamp when the process started): # isi_for_array 'ps auxwp `pgrep isi_cbind_d`' NOTE: This may need to be repeated if the issue re-occurs in the future, at which time manual intervention will be required to proactively restart or resort to disabling the cache until an upgrade can be administered (details below). If the above does not resolve your issue, there may be some other factors in play. Most prior issues with isi_cbind_d causing timeouts have been resolved in current code. It is recommended to engage Dell PowerScale Support for further assistance with investigating the issue. If there are any questions regarding this issue and related paths to resolution, or if assistance is required, please contact Dell PowerScale Support.
Click on a version to see all relevant bugs