OPERATIONAL DEFECT DATABASE
...
...
Troubleshooting VMware Tunnel/MAG and API/AWCM Trust Errors.
Tunnel to API and AWCM As part of the network requirements the VMware Tunnel requires access to both the AirWatch API and AirWatch Cloud Messaging (AWCM) services. The Tunnel makes requests to both these endpoints to retrieve and update a list of devices that have permission to access internal resources. If the API or AWCM connections are failing, the Tunnel may be unable to get a list of approved devices and so will block devices attempting to reach internal resources. The Tunnel services that require communication to the API and AWCM endpoints are: Proxy services on both Windows and Linux*Per-App VPN service on Linux (VPND) Note: The Content-Gateway on both Windows and Linux does not currently make connections to API or AWCM, so this article is not relevant for troubleshooting Content-Gateway errors. The requests made by Tunnel to retrieve this information are done over a secure HTTPS connection; this requires that the Tunnel be able to trust the SSL certificates bound on those servers. If this trust fails we will encounter SSL Handshake errors in the log files. This article provides steps on how to resolve those errors. For the Tunnel Proxy (on both Windows and Linux) the API and AWCM requests will be made from the Relay server (if using a relay-endpoint configuration) unless the EnableOutboundCallsViaProxy option found in the config.xml file is set to true, in which case the requests are proxied to the Endpoint Tunnel server, and from there potentially to another outbound proxy. Starting in AirWatch version 9.1 with the release of Cascade mode for the Per-app VPN service, the API and AWCM requests will be made from both the Frontend and Backend Tunnel servers, and the below steps would need to be taken on both servers. If using a Basic configuration the steps need only be done on the one Frontend Tunnel server. Error in logs As the following examples show, failures to establish a secure connection to API and AWCM are easily identified in logs. Please note the letter (A-D) next to the error you are encountering to guide you through the resolution steps below: A: Error when connecting from API to Tunnel Proxy on Windows or Linux (from mag.log): WARN [main] [RelayMag:184] - Unable to get latest config from API Server! Reason: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification B: Error when connecting to AWCM from Tunnel Proxy on Windows or Linux (from mag.log): WARN [AwcmIdler] [AwcmIdler:130] - Communication to AWCM failed! Reason: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target C: Error when connecting to API from Tunnel Per-app VPN on Linux (from tunnel.log): INFO: TCPSocket::Connect Resolving address {API_HOSTNAME} ….(lines omitted for brevity)…. ERROR: SSLClient: Handshake with AWCM/API returns returns=-1 error=1 error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed D: Error when connecting to AWCM from Tunnel Per-app VPN on Linux (from tunnel.log): INFO: TCPSocket::Connect Resolving address {AWCM_HOSTNAME} ….(lines omitted for brevity)…. ERROR: SSLClient: Handshake with AWCM/API returns returns=-1 error=1 error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed Resolving the SSL Certificate Trust failure To resolve any of the four errors mentioned above we will import the root certificate (and in some cases also the intermediate certificate) from the API or AWCM server into the correct certificate trust store on the Tunnel server. If using a Relay-Endpoint Tunnel Proxy configuration these steps need only be done on the Relay server. If using the Cascade mode with Frontend and Backend Tunnel Per-app VPN, these steps may be required on both servers. The steps to accomplish this will vary based on platform and for which component is affected, and are outlined below. Before proceeding please note which error you are encountering, A-D, as shown in the above section. If the AirWatch API and AWCM services are on the same server they may both be using the same SSL certificate, but this is not always true as the API and AWCM services would create their own certificate bindings on their respective ports, typically tcp ports 443 and 2001. It is recommend to browse to your API or AWCM address from a desktop browser and verify which certificates are being used. If you notice that the certificate being used is invalid (expired or revoked certificate, address mismatch, etc.), the steps below will not resolve the issue and the certificate invalidity must be addressed first. Resolving Errors A & B: Tunnel Proxy to API or AWCM on Windows 1. Download a Base-64 encoded X.509 copy of the root and intermediate certificates from your API or AWCM site. This can be done by browsing to your API or AWCM URL from a desktop browser, viewing the certificate details, and then copying to a file. If you are unsure of what URL the Tunnel is using for API or AWCM you can find it here: Open the file at: C:\AirWatch\TunnelProxy\conf\config.xml and view the “ApiUrl” and “AwcmUrl” values 2. Copy the certificate file(s) to: C:\Program Files\Java\jre{version}\lib\security. This path will vary based on the JRE version installed. If you have multiple folders at the C:\Program Files\Java\ directory, we will first confirm on the specific version of Java being used by viewing the “wrapper.java.command” value found in the file at: C:\AirWatch\TunnelProxy\service\proxy-conf\proxyServiceWrapper.conf. Note that the path listed in this folder is a shortname and will appear slightly different than the path shown in the Windows Explorer window. 3. Open a Command Window into the same …\lib\security folder where we copied the certificate file(s) 4. In the command line, run the following command: keytool -importcert -keystore cacerts -trustcacerts -alias ALIAS_NAME -file FILE_NAME -storepass KEYSTORE_PASSWORD –noprompt …where ALIAS_NAME can be any name to help you identify this certificate within the Java Keystore at a later time…where FILE_NAME is the name of the root or intermediate .cer/.crt file that we copied into the folder in step 2…where KEYSTORE_PASSWORD is the Java Keystore password. By default the password is “changeit”. 5. If importing multiple certificates into the Java Keystore (such as both the root and intermediate certificates), complete step 4 for each of these certificates. 6. Restart the VMware Tunnel Proxy service and confirm if the error is still present in the logs. If the error is no longer occurring, you have completed this troubleshooting guide. Resolving Errors A & B: Tunnel Proxy to API or AWCM on Linux 1. Download a Base-64 encoded X.509 copy of the root and intermediate certificates from your API or AWCM site. This can be done by browsing to your API or AWCM URL from a desktop browser, viewing the certificate details, and then copying to a file. If you are unsure of what URL the Tunnel is using for API or AWCM you can find it here: Open the file at: /opt/vmware/tunnel/proxy/conf/config.xml and view the “ApiUrl” and “AwcmUrl” values 2. Copy the certificate file(s) to: /usr/java/default/jre/lib/security/ 3. Change your working directory to the same path as shown in step 2 and run the following command: keytool -import -keystore cacerts -trustcacerts -alias ALIAS_NAME - file FILE_NAME -storepass KEYSTORE_PASSWORD –noprompt …where ALIAS_NAME can be any name to help you identify this certificate within the Java Keystore at a later time…where FILE_NAME is the name of the root or intermediate .cer/.crt file that we copied into the folder in step 2…where KEYSTORE_PASSWORD is the Java Keystore password. By default the password is “changeit" 4. If importing multiple certificates into the Java Keystore (such as both the root and intermediate certificates), complete step 3 for each of these certificates. 5. Restart the Tunnel proxy service (the service name is "proxy") and confirm if the error is still present in the logs. If the error is no longer occurring, you have completed this troubleshooting guide. Resolving Errors C & D: Tunnel Per-App VPN to API or AWCM on Linux 1. To confirm that the error is due to a certificate missing from the trusted CA bundle, run the following command. If either command returns a certificate error proceed to step 2 to resolve. For API endpoint: curl https://{API_URL}/api/help --cacert /etc/pki/tls/certs/ca-bundle.crtThe expected response is a HTTP 401 unauthorized with no certificate errors For AWCM endpoint: curl https://{AWCM_URL}/awcm/status --cacert /etc/pki/tls/certs/ca-bundle.crtThe expected response is a HTTP 200 with “OK” returned and no certificate errors 2. Download a Base-64 encoded X.509 copy of the root and intermediate certificates from your AWCM site. This can be done by browsing to your API or AWCM URL from a desktop browser, viewing the certificate details, and then copying to a file. If you are unsure of what URL the Tunnel is using for AWCM or API you can find it here: Open the file at: /opt/vmware/tunnel/vpnd/server.conf and view the “api_address” and “awcm_address” values. For AWCM, please also note the “awcm_port” value. 3. Open the certificate file into a text editor and select and copy all of the text between and including the “-----BEGIN CERTIFICATE-----“ and “-----END CERTIFICATE-----“ lines. 4. Navigate to the path containing the ca-bundle used by Tunnel Per-App VPN. By default this is found at: /etc/pki/tls/certs/ca-bundle.crt. You can confirm or change the bundle by viewing the “api_ca” value in the server.conf file mentioned in step 2.a. 5. Paste the text copied in step 3 to the end of this file. If you are importing multiple certificates you can paste the text for those certificates in any order into the file. Save the file with these changes. 6. Restart the VPND service and confirm if the error is still present in the log. Note: The Proxy service on Windows Server was previously referred to as Mobile Access Gateway (MAG). Throughout this article we will refer both the Windows and Linux services as the VMware Tunnel but will make a distinction between Windows and Linux when relevant. Whether you are using the MAG on Windows or the Tunnel on Linux this article will still apply to your implementation.If using older versions of the Tunnel software, the filepaths mentioned above may vary from what is present on your installation. Other Languages: 日本語
For additional details, please refer to the following resources:Test connection for Tunnel fails for API/AWCM (50117926) MAG to API and MAG to AWCM fails with error "PKIX path building failed" (50101963)Gateway to AWCM and Gateway to API test connection fails after console upgrade with error "unable to find valid certification path to requested target" (50122003
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
