Symptoms

Unity array is running operating system 5.3 with SupportAssist enabled.Unity Storage Processors (SPs) are rebooting frequently (every 2 or 3 hours) without generating a dump file.The start_c4.log shows that the SP reboots are because of ese (embedded service enabler) failure.The SP logs show frequent error messages for SupportAssist service has stopped working.The ese_startup.log shows ese container restarting frequently.Live Analysis: /EMC/C4Core/log/start_c4.logDC Analysis: \spx\EMC\C4Core\log\start_c4.log A 08/09/23 15:10:50 ha_policy.pl requested to reboot spa with hint because of ese failure B 08/09/23 16:22:04 ha_policy.pl requested to reboot spb with hint because of ese failure A 08/09/23 17:39:14 ha_policy.pl requested to reboot spa with hint because of ese failure B 08/09/23 18:55:40 ha_policy.pl requested to reboot spb with hint because of ese failure A 08/09/23 20:07:35 ha_policy.pl requested to reboot spa with hint because of ese failure B 08/09/23 22:20:21 ha_policy.pl requested to reboot spb with hint because of ese failure A 08/10/23 02:57:41 ha_policy.pl requested to reboot spa with hint because of ese failure B 08/10/23 04:09:59 ha_policy.pl requested to reboot spb with hint because of ese failure SP_LOG A 08/10/23 02:06:01.321 mlu 12d0004 [INFO] System: Operation Evacuate Slices: Completed 1, Failed 0 completed on 20000004b. [ALU 36360] -- A 08/10/23 02:39:41.283 mlu 12d0004 [INFO] System: Operation Evacuate Slices: Completed 59, Failed 0 completed on 200000054. [ALU 32903] A 08/10/23 02:39:51.306 EmcSupportSvcs 380057 [ERROR] User: SupportAssist service has stopped working. Repair it using svc_supportassist service command. A 08/10/23 02:41:13.581 mlu 12d0004 [INFO] System: Operation Evacuate Slices: Completed 1, Failed 0 completed on 200000054. [ALU 32903] -- B 08/10/23 03:12:40.818 CASAuth 560001 [INFO] Audit: Authentication successful.Username: p985_cb2153784@fspa.myntet.se ClientIP: 10.99.104.138. B 08/10/23 03:13:14.081 EmcSupportSvcs 380057 [ERROR] User: SupportAssist service has stopped working. Repair it using svc_supportassist service command. A 08/10/23 03:13:20.044 mlu 12d0004 [INFO] System: Operation freeze_file_system_ufs64 completed on 2800033134. -- A 08/10/23 03:33:07.710 mlu 12d0004 [INFO] System: Operation Evacuate Slices: Completed 1, Failed 0 completed on 200000043. [ALU 36228] B 08/10/23 03:34:21.402 EmcSupportSvcs 380057 [ERROR] User: SupportAssist service has stopped working. Repair it using svc_supportassist service command. A 08/10/23 03:34:24.984 mlu 12d0004 [INFO] System: Operation Truncate File completed on 9000effcb. -- A 08/10/23 04:08:33.303 mlu 16d0020 [INFO] System: Destroy of snapshot Destroying_20230810040736.870+00-000 completed. B 08/10/23 04:08:53.910 EmcSupportSvcs 380057 [ERROR] User: SupportAssist service has stopped working. Repair it using svc_supportassist service command. B 08/10/23 04:09:07.162 PEService 1660402 [INFO] System: Relocation is stopped for Storage Pool 0. -- A 08/10/23 05:39:40.278 mlu 12d0004 [INFO] System: Operation Evacuate Slices: Completed 1, Failed 0 completed on 200000046. [ALU 35864] A 08/10/23 05:42:16.903 EmcSupportSvcs 380057 [ERROR] User: SupportAssist service has stopped working. Repair it using svc_supportassist service command. A 08/10/23 05:42:39.223 MnsvcServer 7d8 [INFO] Authentication: Authentication session Session_61_1691640760: User p985_cb2153784 successfully authenticated in authority LDAP/fspa.myntet.se Live Analysis: /EMC/CEM/log/ese/ese_startup.logDC Analysis: SPA:/spa/EMC/CEM/log/ese/ ese_startup.log 251707:Thu Aug 10 04:10:35 2023 ready(22517): Container is not running 251771-Thu Aug 10 04:10:35 2023 start(22513): Running: /usr/bin/sudo /usr/bin/setfacl -m u:ecom:rwx /EMC/backend/CEM/ese 251885-Thu Aug 10 04:10:35 2023 start(22513): Command success 251940-Thu Aug 10 04:10:35 2023 start(22513): Mounting container host mount directory 252019-Thu Aug 10 04:10:35 2023 start(22513): Running: /EMC/Platform/bin/ese/ese_mount.sh --mount -- 254071-Thu Aug 10 04:10:37 2023 start(22513): Container has been successfully created 254150-Thu Aug 10 04:10:37 2023 start(22513): Running: /usr/bin/sudo /usr/bin/docker ps -f name=ese -f status=running --no-trunc 254272-Thu Aug 10 04:10:37 2023 start(22513): Result is: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 254393-(0) 254397:Thu Aug 10 04:10:37 2023 start(22513): Container is not running 254461-Thu Aug 10 04:10:37 2023 start(22513): Starting container 254519-Thu Aug 10 04:10:37 2023 start(22513): Running: /usr/bin/sudo /usr/bin/docker start ese 254607-Thu Aug 10 04:10:38 2023 start(22513): Command success: ese 254667- -- 292902-Thu Aug 10 05:44:39 2023 ready(13520): Running: /usr/bin/sudo /usr/bin/docker ps -f name=ese -f status=running --no-trunc 293024-Thu Aug 10 05:44:39 2023 start(13517): Running: /usr/bin/sudo /usr/bin/docker images dell-ese:latest 293125-Thu Aug 10 05:44:39 2023 ready(13520): Result is: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 293246-(0) 293250:Thu Aug 10 05:44:39 2023 ready(13520): Container is not running 293314-Thu Aug 10 05:44:39 2023 start(13517): Result is: REPOSITORY TAG IMAGE ID CREATED SIZE 293422-dell-ese latest 97771f418a09 7 months ago 249MB 293481-(0) 293485-Thu Aug 10 05:44:39 2023 start(13517): Image is loaded -- 295840-Thu Aug 10 05:44:40 2023 start(13517): Container has been successfully created 295919-Thu Aug 10 05:44:40 2023 start(13517): Running: /usr/bin/sudo /usr/bin/docker ps -f name=ese -f status=running --no-trunc 296041-Thu Aug 10 05:44:41 2023 start(13517): Result is: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 296162-(0) 296166:Thu Aug 10 05:44:41 2023 start(13517): Container is not running 296230-Thu Aug 10 05:44:41 2023 start(13517): Starting container 296288-Thu Aug 10 05:44:41 2023 start(13517): Running: /usr/bin/sudo /usr/bin/docker start ese 296376-Thu Aug 10 05:44:41 2023 Live Analysis: Live Analysis: /EMC/CEM/log/ese/ese_startup.logDC Analysis: SPB:/spb/EMC/CEM/log/ese/ ese_startup.log 949027:Thu Aug 10 03:34:14 2023 ready(14205): Container is not running 949091-Thu Aug 10 03:34:14 2023 start(14202): Command success 949146-Thu Aug 10 03:34:14 2023 start(14202): Mounting container host mount directory 949225-Thu Aug 10 03:34:14 2023 start(14202): Running: /EMC/Platform/bin/ese/ese_mount.sh --mount 949316-Thu Aug 10 03:34:14 2023 start(14202): Command success: Start to mount. -- 951277-Thu Aug 10 03:34:16 2023 start(14202): Container has been successfully created 951356-Thu Aug 10 03:34:16 2023 start(14202): Running: /usr/bin/sudo /usr/bin/docker ps -f name=ese -f status=running --no-trunc 951478-Thu Aug 10 03:34:16 2023 start(14202): Result is: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 951599-(0) 951603:Thu Aug 10 03:34:16 2023 start(14202): Container is not running 951667-Thu Aug 10 03:34:16 2023 start(14202): Starting container 951725-Thu Aug 10 03:34:16 2023 start(14202): Running: /usr/bin/sudo /usr/bin/docker start ese 951813-Thu Aug 10 03:34:16 2023 start(14202): Command success: ese 951873- -- 973168-Thu Aug 10 03:51:55 2023 start(3243): Image is loaded 973222-Thu Aug 10 03:51:55 2023 start(3243): Running: /usr/bin/sudo /usr/bin/setfacl -m u:ecom:rwx /EMC/backend/CEM/ese 973335-Thu Aug 10 03:51:55 2023 ready(3246): Result is: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 973455-(0) 973459:Thu Aug 10 03:51:55 2023 ready(3246): Container is not running 973522-Thu Aug 10 03:51:55 2023 start(3243): Command success 973576-Thu Aug 10 03:51:55 2023 start(3243): Mounting container host mount directory 973654-Thu Aug 10 03:51:55 2023 start(3243): Running: /EMC/Platform/bin/ese/ese_mount.sh --mount 973744-Thu Aug 10 03:51:55 2023 start(3243): Command success: Start to mount. -- 975689-Thu Aug 10 03:51:57 2023 start(3243): Container has been successfully created 975767-Thu Aug 10 03:51:57 2023 start(3243): Running: /usr/bin/sudo /usr/bin/docker ps -f name=ese -f status=running --no-trunc 975888-Thu Aug 10 03:51:57 2023 start(3243): Result is: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 976008-(0) 976012:Thu Aug 10 03:51:57 2023 start(3243): Container is not running 976075-Thu Aug 10 03:51:57 2023 start(3243): Starting container 976132-Thu Aug 10 03:51:57 2023 start(3243): Running: /usr/bin/sudo /usr/bin/docker start ese 976219-Thu Aug 10 03:51:57 2023 start(3243): Command success: ese 976278-

Cause

In rare instances, multiple ESE threads of different types are hitting a condition that causes them to become deadlocked, including the threads that listen to API requests. The deadlock condition eventually leads to API requests not being answered by ESE resulting in the Storage Processor (SP) reboots.

Resolution

Fix:This issue is fixed in the Unity operating system: 5.3.1.0.5.008Workarounds:There are two workarounds available for this issue. See the Additional Information section for more details.

Support Cases