Symptoms
The elasticsearch service in VIDM runs out of file descriptors and then memory.In the /opt/vmware/elasticsearch/logs/horizon.log file, you see entries similar to:
"too many open files" and "OutOfMemoryException".
This can happen when the number of days of audit data is large (greater than 200 or so).
Cause
The elasticsearch service by default is configured with too few file descriptors
Resolution
To resolve the issue, follow below steps:
1. Add the following lines to /etc/security/limits.conf
elasticsearch hard nofile 65536elasticsearch soft nofile 65536
2. Add the following line to /etc/sysconfig/elasticsearch (create the file if necessary)
NFILES=65536
This must be done on all service nodes in the cluster.3.After making these changes, restart elasticsearch using:
service elasticsearch restart
To verify that the change has taken effect, get the pid of the elasticsearch process using “ps” and then run:
cat /proc/pid/limits
