Symptom

> Events backlog dates back to the day that MonetDB was in a hang state: root@Sourcefire3D:/Volume/home/admin# stats_unified.pl Current Time - Tue Feb 15 04:31:20 UTC 2022 ********************************************************************************** * FTD-01 - , version 6.6.1 ********************************************************************************** Channel Backlog Statistics (unified_event_backlog) Chan Last Time Bookmark Time Bytes Behind 0 2022-02-07 08:38:30 2022-02-06 16:38:47 0 1 2022-02-07 08:38:30 2022-02-07 08:37:58 496188076 > Coincidentally, the CPU utilization spiked to 100% at on all CPU cores at this time with mserver5 and monetdb consuming high CPU: top - 2022-02-07 04:55:05 up 178 days, 2:23, 2 users, load average: 9.92, 4.3 Tasks: 214 total, 4 running, 169 sleeping, 3 stopped, 0 zombie Cpu: 6.0%us, 93.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 32945264k total, 31037956k used, 1907308k free, 240144k buffers Swap: 6890136k total, 4011244k used, 2878892k free, 11357456k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3552 monetdb 20 0 9705m 6.6g 5328 R 298 20.9 50087:00 mserver5 77 root 20 0 0 0 0 I 18 0.0 3:10.55 kworker/3:1 75 root 20 0 0 0 0 R 17 0.0 71:28.14 kworker/1:1 4087 root 20 0 2834m 643m 11m S 17 2.0 17433:59 SFDataCorrelato 10767 root 20 0 0 0 0 R 16 0.0 3:02.94 kworker/2:2 28 root 20 0 0 0 0 I 16 0.0 19:59.38 kworker/0:1 11048 casuser 20 0 30.0g 1.5g 15m S 6 4.9 12868:57 dbsrv17 > MonetDB looks to be in a hung state since then and indicates a memory issue from the output of /var/log/monetdb/merovingian.log: 2022-02-07 04:52:57 ERR eventdb[3552]: #logmanager:!ERROR: GDKunlink(/Volume/lib/monetdb/dbfarm/eventdb/sql_logs/sql/log.1979888) 2022-02-07 04:52:57 ERR eventdb[3552]: #logmanager:!OS: Cannot allocate memory 2022-02-07 04:52:57 ERR eventdb[3552]: #logger_cleanup: failed to remove old WAL log.sql_logs/sql/log.bak-3594201 2022-02-07 04:52:57 ERR eventdb[3552]: #logmanager:!ERROR: GDKunlink(/Volume/lib/monetdb/dbfarm/eventdb/sql_logs/sql/log.1923566) 2022-02-07 04:52:57 ERR eventdb[3552]: #logmanager:!OS: Cannot allocate memory 2022-02-07 04:52:57 ERR eventdb[3552]: #logger_cleanup: failed to remove old WAL log.sql_logs/sql/log.bak-3594201 2022-02-07 04:52:57 ERR eventdb[3552]: #logmanager:!ERROR: GDKunlink(/Volume/lib/monetdb/dbfarm/eventdb/sql_logs/sql/log.1921571) 2022-02-07 04:52:57 ERR eventdb[3552]: #logmanager:!OS: Cannot allocate memory 2022-02-07 04:52:57 ERR eventdb[3552]: #logger_cleanup: failed to remove old WAL log.sql_logs/sql/log.bak-3594201 > There may be pending tasks that are running for a long time due to high CPU > No MonetDB core file was generated as well

Conditions

NA

Workaround

Restart MonetDB using the below command: pmtool restartbyid monetdb Now the CPU utilization will be back to normal, events will start populating

Further Problem Description