Symptoms

While SourceOne is attempting to index incoming data, the Elastic Search Service (elasticsearch-service-x64.exe) may crash.The following error may be logged in the Application Event log on the Native Archive server performing the indexing: Log Name: ApplicationSource: Application ErrorDate: <DATE>Event ID: 1000Task Category: (100)Level: ErrorKeywords: ClassicUser: N/AComputer: <ServerName>Description:Faulting application name: elasticsearch-service-x64.exe, version: 1.1.0.0, time stamp: 0x5a022cf5Faulting module name: jvm.dll, version: 8.0.2210.11, time stamp: 0x5d1de5daException code: 0xc0000005Fault offset: 0x0000000000218048Faulting process id: 0x16fcFaulting application start time: 0x01d61cb64b81d3e2Faulting application path: <Drive>:\Program Files (x86)\EMC SourceOne\EXPBA\bin\Elastic\elasticsearch\bin\elasticsearch-service-x64.exeFaulting module path: <Drive>:\Program Files\Java\jre<Version>\bin\server\jvm.dllReport Id: b3fd4bd5-eb43-40ea-9750-0337fa3074b0 The following errors will be logged in the Elastic Service logs located in the <ElasticClusterName.log> on the Native Archive server under <SourceOne Installation Path>:\EMC SourceOne\ExPBA\Elastic\ElasticSearch\Logs: [2020-05-04T17:15:29,413][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [<Elastic Server Name>] fatal error in thread [Thread-4], exitingjava.lang.OutOfMemoryError: Java heap space When the Elastic Search Service crashes, the following errors will be logged in the ExAsElasticIdxObj.exe.log file: Failed to index document with sequence: <Num> , error: master_not_discovered_exceptionFailed to perform CURL operation, CURL ERROR [7] :: Couldn't connect to server (0x86042567) System call failed. (0x86040100) [ExAsElasticIdxObj.exe, ExElasticsearchClient.cpp(3547).CExElasticsearchClient::IngestBulkFileAsStrAndProcessResult] >System call failed. (0x86040100) [ExAsElasticIdxObj.exe, CExElasticIndexPipeline.cpp(95).CExElasticIndexPipelineImpl::IndexItem] >System call failed. (0x86040100) Unspecified error (0x80004005) [ExAsElasticIdxObj.exe, ExElasticsearchClient.cpp(3547).CExElasticsearchClient::IngestBulkFileAsStrAndProcessResult] >Failed to perform CURL operation, CURL ERROR [56] :: Failure when receiving data from the peer (0x86042567) [ExAsElasticIdxObj.exe, EXCurl.cpp(165).EXCurl::POST] >System call failed. (0x86040100) [ExAsElasticIdxObj.exe, ExElasticsearchClient.cpp(3547).CExElasticsearchClient::IngestBulkFileAsStrAndProcessResult] >System call failed. (0x86040100) [ExAsElasticIdxObj.exe, CExElasticIndexPipeline.cpp(95).CExElasticIndexPipelineImpl::IndexItem] >System call failed. (0x86040100) Unspecified error (0x80004005) [ExAsElasticIdxObj.exe, ExElasticsearchClient.cpp(3547).CExElasticsearchClient::IngestBulkFileAsStrAndProcessResult] >Failed to perform CURL operation, CURL ERROR [56] :: Failure when receiving data from the peer (0x86042567) [ExAsElasticIdxObj.exe, EXCurl.cpp(360).EXCurl::RunCurl] >System call failed. (0x86040100) [ExAsElasticIdxObj.exe, CExElasticIndexPipeline.cpp(95).CExElasticIndexPipelineImpl::IndexItem] >System call failed. (0x86040100) Unspecified error (0x80004005) [ExAsElasticIdxObj.exe, ExElasticsearchClient.cpp(3547).CExElasticsearchClient::IngestBulkFileAsStrAndProcessResult] >Failed to perform CURL operation, CURL ERROR [56] :: Failure when receiving data from the peer (0x86042567) [ExAsElasticIdxObj.exe, EXCurl.cpp(165).EXCurl::POST] >Failed to perform CURL operation, CURL ERROR [56] :: Failure when receiving data from the peer (0x86042567) [ExAsElasticIdxObj.exe, EXCurl.cpp(360).EXCurl::RunCurl] >Errors have occured when ingesting bulk data (0x86044815) [ExAsElasticIdxObj.exe, CExElasticIndexPipeline.cpp(95).CExElasticIndexPipelineImpl::IndexItem]

Cause

By default when the Elastic Search Service is installed, the MIN\MAX memory configuration for Java is set to 1024MB. When the Elastic Search Service reaches higher than 1024MB it crashes\stops. 

Resolution

To resolve the issue, increase the MIN\MAX memory configuration for Java to something higher than 1024MB.On the Native Archive server that has the Elastic Search Service installed perform the following steps: 1. Stop the SourceOne Index Service and the Elastic Search Service.2. Open a command prompt and change directory to the ElasticSearch folder (<sourceone installation path>\EMC SourceOne\ExPBA\bin\Elastic\elasticsearch\bin)3. Run the following command: elasticsearch-service.bat manager 4. If you get a popup from User Account Control, select YES5. On the manager, select the JAVA tab6. Set the "Initial Memory Pool" and "Maximum Memory Pool" values to something higher than 1024MB. Example:Initial Memory Pool: 2048Maximum Memory Pool: 2048 7. Select OK8. First Start the Elastic Search service and then the SourceOne Index Service. NOTE: If the service crashes again with the same errors, increase the values to 3072MB. **Make sure the server has enough physical memory available for the values you set.**

Support Cases