Loading...
Loading...
This issue may affect OneFS 9.5.x or later, causing SupportAssist to disconnect from SCG or preventing telemetry data from being sent to AIOps. An event may be seen on the cluster similar to: SupportAssist has had too many failed tasks. Please contact Dell Support.
This mechanism maintains a lock that prevents any instance of isi_crispies_d on other nodes from acquiring the lock and initiating ESE. Also, it prevents the Active Control Program ( isi_mcp ) from launching another isi_crispies_d process on the affected node. The ESE (SupportAssist user) owns the isi_crispies_d process. A simple way to check if the SupportAssist connection is good is to run the command below and complete within 5 minutes: PowerScale-2# isi_gather_info --supportassist --nologs -f /ifs/data/Isilon_Support/last_full_gather Note: If it seems like it is stuck on the UPLOADING stage, cancel by doing Ctrl + C and stop the gather by running: PowerScale-2# isi diagnostics gather stop How a normal ESE service should look like: PowerScale-2# isi_for_array -s ps auwwx | grep -i ese PowerScale-2: ese 44944 0.0 0.0 0 0 - Z 09:50 0:00.01 <defunct> <<<<<< IgnorePowerScale-3: ese 62688 60.7 0.1 226468 109624 - Ss 09:51 0:04.24 /usr/local/ese/lib/ese/ese <<<<< Normal ese output How a normal output should look like: PowerScale-1# isi_for_array -s 'fstat' | grep ese PowerScale-1: ese ese 34759 text / 79619 -r-x------ 6287570 r <---- ese present in second column PowerScale-1: ese ese 34759 wd / 2 drwxr-xr-x 1024 r PowerScale-1: ese ese 34759 root / 2 drwxr-xr-x 1024 r PowerScale-1: ese ese 34759 0 /dev 28 crw-rw-rw- null rw PowerScale-1: ese ese 34759 1 /dev 28 crw-rw-rw- null rw PowerScale-1: ese ese 34759 2 /dev 28 crw-rw-rw- null rw PowerScale-1: ese ese 34759 3 /ifs 4322756906 -rw-r--r-- 0 w PowerScale-1: ese ese 34759 4 /ifs 4322820152 -rw-r--r-- 0 w PowerScale-1: ese ese 34759 5 /ifs 4322820105 -rw-r--r-- 6090752 rw Determine if there is a LOCK on isi_crispies_d: PowerScale-4# isi_for_array -s ps auwwx | grep -v grep| grep -i ese PowerScale-1: ese 74315 0.0 0.1 168164 63536 - I Sat04 0:00.01 /usr/libexec/isilon/isi_crispies_d /usr/bin/isi_crispies_d PowerScale-4# isi_for_array -s 'fstat' | grep esePowerScale-1: ese isi_crispies_d 74315 text / 75004 -r-xr-xr-x 17320 r PowerScale-1: ese isi_crispies_d 74315 wd / 2 drwxr-xr-x 1024 r PowerScale-1: ese isi_crispies_d 74315 root / 2 drwxr-xr-x 1024 r PowerScale-1: ese isi_crispies_d 74315 0 /dev 28 crw-rw-rw- null rw PowerScale-1: ese isi_crispies_d 74315 1 /dev 28 crw-rw-rw- null rw PowerScale-1: ese isi_crispies_d 74315 2 /dev 28 crw-rw-rw- null rw PowerScale-1: ese isi_crispies_d 74315 3 /var 36931 drwxr-xr-x 4608 r PowerScale-1: ese isi_crispies_d 74315 4 /var 37084 -rw------- 5 w PowerScale-1: ese isi_crispies_d 74315 5* local dgram fffff80389e59cc0 <-> fffff80389812a80 PowerScale-1: ese isi_crispies_d 74315 6 /dev 28 crw-rw-rw- null r PowerScale-1: ese isi_crispies_d 74315 7* local stream fffff80389846c00 <-> fffff80378e64900 PowerScale-1: ese isi_crispies_d 74315 8* pipe fffff80f61966000 <-> fffff80f61966168 0 rw PowerScale-1: ese isi_crispies_d 74315 9* pipe fffff80f61966168 <-> fffff80f61966000 0 rw PowerScale-1: ese isi_crispies_d 74315 10* local stream fffff810dfcb6540 <-> fffff80dbccabf00 PowerScale-1: ese isi_crispies_d 74315 11 /ifs 10255925248 -rwxr-xr-x 0 r <--- crispies_d lock on second column PowerScale-1: ese isi_crispies_d 74315 12* internet stream tcp fffffe900c0ef448 PowerScale-1: ese isi_crispies_d 74315 13* pipe fffff804c18202f8 <-> fffff804c1820460 0 rw <--- lock is present on node 1 PowerScale-1: ese isi_crispies_d 74315 14* pipe fffff80b6d274be0 <-> fffff80b6d274d48 0 rw PowerScale-1: ese isi_crispies_d 74315 15* pipe fffff80b6d274d48 <-> fffff80b6d274be0 0 rw PowerScale-1: ese isi_crispies_d 74315 16* pipe fffff8138a2d3168 <-> fffff8138a2d3000 0 rw PowerScale-1: ese isi_crispies_d 74315 17* pipe fffff815c331e2f8 <-> fffff815c331e460 0 rw PowerScale-1: ese isi_crispies_d 74315 19* pipe fffff8037865f8e8 <-> fffff8037865fa50 0 rw If a crispies lock has been identified, go to the steps in the Resolution section. If a crispies lock was not identified, DO NOT apply the resolution section and contact Dell PowerScale Support .
Note: Login as root before proceeding Connect to the node where the lock is present (in this example, Node 1), if not already connected using SSH. Run the following command to forcibly terminate any isi_crispies_d processes associated with ESE: PowerScale-1# for pid in $(ps auxww | grep /usr/bin/isi_crispies_d | grep ese | grep -v grep | awk '{print $2}'); do kill -9 $pid; done Note: This action must be run on the affected node only and should resolve the isi_crispies_d lock issue. Clearing this lock is essential to restore normal operation and to prevent disruptions to SupportAssist connectivity. After running the command, wait 5–10 minutes, then verify connectivity using the command below, ensuring it completes within 5 minutes: PowerScale-2# isi_gather_info --supportassist --nologs -f /ifs/data/Isilon_Support/last_full_gather If the command finishes, also check your SCG connection (if it is in use) and or CIQ - APEX AIOPs. If the connection remains unsuccessful after following this article, initiate a Full gather logset and contact Dell PowerScale Support . This issue is resolved in OneFS 9.10.1.4 and 9.12.0.x code levels. If the cluster cannot be upgraded to the code level listed here, a workaround is to add the following lines to /etc/mcp/override/crontab (using VI, create it if it's not existing): # Daily kill crispies lock process 10 * * * * root for pid in $(ps auxww | grep /usr/bin/isi_crispies_d | grep ese | grep -v grep | awk '{print $2}'); do kill -9 $pid; done Note: If SupportAssist or connectivity is still disconnected, reprovision it by generating a new access key and PIN, refer to KB215531 .
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.