Symptoms
Networker 19.4 Isilon 8.2NDMP backup has error: "Too many open files".
Cause
Isilon 8.2 has open file limit in Isilon isi_ndmp_d.
Resolution
"Too many open files" is a known issue. It's recommended increase the open file limit of isi_ndmp_d.
Edit /etc/mcp/sys/services/ndmpd file and increase max_open_files
/bin/sleep 30/usr/bin/limits -B -P $(pgrep isi_ndmp_d) -n 32768
Edit the ndmpd on any node :
cd /etc/mcp/sys/servicescp ndmpd /ifs/data/Isilon_Supportvi /ifs/data/Isilon_Support/ndmpd
In the section , add the following lines:
/bin/sleep 30/usr/bin/limits -B -P $(pgrep isi_ndmp_d) -n 32768
Example:
enable
Network Data Management Protocol Daemon
<process name="isi_ndmp_d" pidfile="/var/run/isi_ndmp_d.pid"
startaction="start" stopaction="stop"/>
/usr/bin/isi_ndmp_d
/bin/sleep 30
/usr/bin/limits -B -P $(pgrep isi_ndmp_d) -n 32768
/usr/bin/killall isi_ndmp_d
Save the file (may have to use ! as the file is read only)Then distribute the file to all of the nodes:
isi_for_array 'cp /ifs/data/Isilon_Support/ndmpd /etc/mcp/sys/services'
Restart NDMP service:
isi services ndmpd disableisi services ndmpd enable
Check openfiles limit for isi_ndmp_d PIDs:
isi_for_array 'procstat -l $(pgrep isi_ndmp_d)|grep openfiles'
dklimki-mr1512e-4: 26547 isi_ndmp_d openfiles 16384 16384
dklimki-mr1512e-5: 77179 isi_ndmp_d openfiles 16384 16384
dklimki-mr1512e-3: 74909 isi_ndmp_d openfiles 16384 16384
dklimki-mr1512e-1: 27389 isi_ndmp_d openfiles 16384 16384
dklimki-mr1512e-2: 3725 isi_ndmp_d openfiles 16384 16384
Check again after 30 sec:
isi_for_array 'procstat -l $(pgrep isi_ndmp_d)|grep openfiles'
dklimki-mr1512e-3: 74909 isi_ndmp_d openfiles 32768 32768
dklimki-mr1512e-1: 27389 isi_ndmp_d openfiles 32768 32768
dklimki-mr1512e-2: 3725 isi_ndmp_d openfiles 32768 32768
dklimki-mr1512e-4: 26547 isi_ndmp_d openfiles 32768 32768
dklimki-mr1512e-5: 77179 isi_ndmp_d openfiles 32768 32768
