Symptoms
The system reports errors:
# perror 24
OS error code
24:
Too many open files
011d0004:3: Disk partition /var/asmdata1 has only 11% free
Disk partition /var/lib/mysql has only 11% free
There are also failures logged to /var/lib/mysql/mysqld.err when attempting to open files:
----------------------------------------------------------------------
170301
3:20:08 [ERROR] /usr/sbin/mysqld: Can't open file: './AVR/AVR_STAT_ASM_MEMORY_INFO_T.frm' (errno: 24)
170301
3:20:09 [ERROR] /usr/sbin/mysqld: Can't open file: './PRX/REQUEST_LOG.frm' (errno: 24)
Impact
MySQL ISAM tables becomes corrupted after MySQL exceeds the maximum number of allowed open file handles, and becomes nonfunctional.
Conditions
ASM and AVR are enabled.
Workaround
Run the following commands:
1. tmsh save sys ucs /var/tmp/last_good.ucs
2. /usr/share/ts/bin/recover_db.pl
