Symptoms

The NetWorker representational state transfer (REST) application program interface (API) is used to request information (HTTP GET) from a NetWorker Server in the environment. The REST API queries intermittently times out, and the restapi.log shows an internal server error message due to a Socket Timeout. YYYY-MM-DD HH:MM:SS INFO [https-jsse-nio-9090-exec-26] c.e.n.w.WebApiResponse - Response status Method: 'GET', URI:'v3/global/backups', Status '200' YYYY-MM-DD HH:MM:SS INFO [https-jsse-nio-9090-exec-26] c.e.n.w.WebApiResponse - Response status Method: 'GET', URI:'v3/global/backups', Status '500' ... YYYY-MM-DD HH:MM:SS ERROR [https-jsse-nio-9090-exec-11] c.e.n.w.WebApiExceptionMapper - Status 'Internal Server Error', msg: java.io.IOException: Broken pipe org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:333) ~[catalina.jar:9.0.10] A curl command to the NetWorker Server REST API shows: [root@NetworkerServer]# curl -X GET -H "Content-Type: application/json" "https://mynsr.mydomain.com:9090/nwrestapi/v3/global/backups" -u "administrator" -k -1 Enter host password for user 'administrator': curl: (28) Operation timed out after 300406 milliseconds with 0 out of 0 bytes received

Cause

The REST API calls "v3/global/backups" and "v3/global/jobs" alone is a resource-intensive query and can cause the request to timeout in larger environments. The NetWorker REST API services are hosted on an Apache Tomcat Java servlet container which is run on a Java Virtual Machine (JVM). The REST API uses JVM memory to process every request, and it has been shown that not all the memory is released after a resource-intensive query. The increase in JVM memory utilization may eventually lead to other less resource-intensive REST API requests to also timeout.

Resolution

Avoid using the resource intense requests to retrieve information from the NetWorker Server. For example, the "v3/global/backups" or "v3/global/clients/CLIENT_NUMBER/backups" with a defined time range can be used to retrieve the backups for each client instead of using the global "v3/global/backups" to retrieve all the backup in the NetWorker Data Zone. For example, we can restrict the results for saveTime between any specified time range. An example of a query for jobs which completed within the last 24 hours: ../global/backups example: https://NETWORKER_SERVER_ADDRESS:9090/nwrestapi/v3/global/backups?q=saveTime:["START" TO "END"] https://nsr.amer.lan:9090/nwrestapi/v3/global/backups?q=saveTime:["2024-07-24T00:00:01" TO "2024-07-24T23:59:59"] ../global/clients/client_resourceId_number/backups https://NETWORKER_SERVER_ADDRESS:9090/nwrestapi/v3/global/clients/client_resourceID_number/backups?q=saveTime:["START" TO "END"] https://nsr.amer.lan:9090/nwrestapi/v3/global/clients/87.0.90.20.0.0.0.0.196.80.99.102.192.168.9.150/backups?q=saveTime:["2024-07-24T00:00:01" TO "2024-07-24T23:59:59"] ../global/jobs: https://NETWORKER_SERVER_ADDRESS:9090/nwrestapi/v3/global/jobs?q=endTime:["START" TO "END"] https://nsr.amer.lan:9090/nwrestapi/v3/global/jobs?q=endTime:["2024-07-29T00:00:01" TO "2024-07-29T23:59:59"] NOTE: Modify the YYYY-MM-DDTHH:mm:SS in the START and END times accordingly. These examples are provided for demonstration purposes only. You may need to use additional REST API query specifications or filter functions to return the data wanted. You can find more information and examples in the REST API documentation: https://developer.dell.com/apis/2378/versions/v3/docs/GettingStarted.md If less resource-intensive REST API requests are timing out, then the NetWorker Server services may be restarted to release the existing JVM memory usage and start a new JVM session.

Support Cases