Symptoms

The collector will not connect to the gateway. The collector diagnostic logs (tomcat-Default directory – Catalina log) show the following errors: Nov 06, 2024 4:43:33 PM com.watch4net.apg.logging.util.LoggingUtils audit INFO: The ESRS EVE proxy has been retrieved Nov 06, 2024 4:43:37 PM com.emc.ciqc.api.ApiErrorHandler fatalError SEVERE: An error occured (9e480e22-db51-4381-890a-6d240b66d627) ! com.emc.ciqc.api.ApiErrorHandler$FatalError: Could not find the certificate. And Nov 06, 2024 4:43:47 PM com.watch4net.apg.logging.util.LoggingUtils audit INFO: The ESRS settings have been retrieved Nov 06, 2024 4:44:03 PM com.emc.ciqc.api.EsrsApiService lambda$getStatus$0 WARNING: ESRS manager test failed ! javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: Read timed out

Cause

Customer is unable to configure the APEX AIOps Observability Collector because they loaded a certificate on the gateway but did not load it on the collector. They would need to load the same certificate on the collector to allow it to connect to the gateway. The certificates must match.

Resolution

To allow an APEX AIOps Observability Collector to connect to Secure Connect Gateway (SCG) using a certificate, a certificate is created on the customers end by their security team then the same is loaded into the gateway and into the collector to allow the connection. The certificates must match for the connection to be valid. Here are the steps provided by the Networking team to create the certificate. #Before we can create a certificate, we have to create generate CA (Certificate Authority) 1. Generate RSA - this is the private key that you normally wouldn’t want to share with anyone openssl genrsa -aes256 -out ca-key.pem 4096 #note: I just put in Dell_123$ as my pass phrase as it is easy to remember, but whatever you enter, you have to remember it for later steps below 2. Generate a public CA Cert - generating public key (note: the info added here is just informational, like country/ou/etc) openssl req -new -x509 -sha256 -days 3650 -key ca-key.pem -out ca.pem #The information you enter above isn’t really important for testing purposes, for a real deployment it would matter. #Note: to view the ca.pem file in human readable format, you can use the following command – it is a long output, so I didn’t post a screenshot openssl x509 -in ca.pem -text #Now we have CA created above, we can generate certificate 1. Create an RSA key openssl genrsa -out cert-key.pem 4096 2. Create a Certificate Signing Request (CSR) openssl req -new -sha256 -subj "/CN=yourcn" -key cert-key.pem -out cert.csr 3. Create an extfile with all the alternative names echo "subjectAltName=DNS:your-dns.record,IP:10.49.71.166" >> extfile.cnf #Note: the IP should be correct here for your server. 4. Create the certificate openssl x509 -req -sha256 -days 365 -in cert.csr -CA ca.pem -CAkey ca-key.pem -out cert.pem -extfile extfile.cnf -CAcreateserial Now if we look at cert.pem, it is only the certificate without intermediate or root certificate included: #This is where most customers end and try to import, and we get invalid chain error in the SCG-VE logs. 5. Create a full chain To create a full chain, you need to append the root certificate (we don’t have intermediate certificate in example above): cat ca.pem >> cert.pem #Note in example above ca.pem is our root certificate and cert.pem is our server certificate. Now we have a full chain for cert.pem and we can apply that to SCG-VE without any exceptions. In a customer environment they will likely have a public CA and intermediate certificate involved, so they will normally have 3 paragraphs in the certificate they import into SCG-VE, could be even more depending on how many intermediates they have. 6. Rename the key file to satisfy SCG: Finally, we also need to change the name of the cert-key.pem file above to cert-key.key as SCG will not take another suffix for key file. You can export the cert-key.key & cert.pem using WinSCP so it is in a place where you can drag and drop into the UI for SCG-VE. There are other certificate formats, and you can always convert between different formats with openssl commands, easy enough to research on the public internet on how to do that. Easiest is to stick with pem format for testing purposes. From the gateway user guide - Secure Connect Gateway 5.x — Virtual Edition User's Guide Manage Certificates About this task A security certificate allows you to: ● Securely access the secure connect gateway user interface or perform any tasks in secure connect gateway through port 5700. ● Securely connect to the backend using RESTful protocol through port 9443. You can upload certificates in the .pem, .p12, or .pfx formats. If you upload a certificate in the .pem format, you must also upload the key in KEY format as a separate file. The .pem file must have the certificate chain that includes the certificate, intermediate, and root certificate authority. You can upload separate certificates for port 5700 and 9443 or a common certificate for both the ports. Steps 1. Go to Settings > Environment configuration > Gateway configuration > Certificate management. 2. Select the port number for which the certificate is applicable for and then upload the certificate and key file, if applicable. NOTE: If you upload a certificate in .pem format, you must also upload the key in KEY format as a separate file. 3. If you upload the certificate in .pem format, enter the passphrase. 4. Click Apply. 5. Click Restart now to restart secure connect gateway. ● You are logged out of the secure connect gateway user interface. ● The security certificates are applied to your instance. ● The certificate name and expiry information are displayed in the Certificate management section. Within SCG, the certificate would have to be set to use either port 9443 or both 5700 and 9443 as the collector uses port 9443. This same certificate used for your gateway will have to be loaded within the collector configuration screenshot noted below after being configured to use port 9443. Under APEX AIOps Observability you would drop the file in here and check off “Use this certificate to authenticate the gateway” then click on SAVE.

Support Cases