OPERATIONAL DEFECT DATABASE
...
...
Update: VMware will resume deploying the Windows OS updates in accordance with Microsoft best practices on May 14th, 2021Versions Identified All versions of Workspace ONE UEM. Symptoms Any workflow where Workspace ONE UEM needs to load or generate a certificate or private key is impacted. This can manifest as failures in the following workflows: Android device communication with Workspace ONE UEMApple device communication with Workspace ONE UEMWorkspace ONE (AirWatch) SDK applications' ability to receive configurations from and send information to Workspace ONE UEMEnterprise integration through AirWatch Cloud Connector (ACC) for components such as: Directory ServicesCertificate Authority (PKI)PowerShell for Email Management The observed impact may not immediately signify the underlying cause. For example, a user tries to log into Content or Web on an enrolled device; receives error 4 (which pertains to missing profile); investigation reveals that fetching SDK profile is failing with status code 500 because Secure Channel handshake between the app and Workspace ONE UEM Device Services encountered an error.This issue can typically be identified by the presence of the following error message: CryptographicException: Key does not exist. The error is seen in the log file for any of the Workspace ONE UEM services or websites residing on servers that host the following components: Web ConsoleDevice ServicesREST API A Device Services log snippet showing this error: Error WanderingWiFi.AirWatch.DeviceServices.Handlers.SecureChannelEndPointHandler+<ProcessRequestAsync>d__51.MoveNext *** EXCEPTION *** System.Security.Cryptography.CryptographicException: Key does not exist. at System.Security.Cryptography.Pkcs.SignedCms.ComputeSignature(CmsSigner signer, Boolean silent) at WanderingWiFi.AirWatch.BusinessImpl.Helpers.CmsHelper.SignMessage() at WanderingWiFi.AirWatch.DeviceServices.Handlers.SecureChannelEndPointHandler.EncryptResponse(ProcessSecureChannelPayloadResponse payloadResponse, X509Certificate2 serverCertificate, X509Certificate2 deviceCertificate, DeviceType platform, Int32 apiVersion, EscrowedSmimeCertificatesFeatureFlag featureFlag) at WanderingWiFi.AirWatch.DeviceServices.Handlers.SecureChannelEndPointHandler.<ProcessRequestAsync>d__51.MoveNext()Diagnostics Context ******************* PID: 3680 Process Name: w3wp Process Identity: Error obtaining Application Identity: Object reference not set to an instance of an object. HTTP Request Identity: HTTP Request URL: https://xxx.xx.xxx/deviceservices/securechannel.aws/v2?deviceid=1234asd-1asd-asdasd1234-123asdd&bundleid=com.air-watch.agent HTTP Request Headers: Cache-Control: no-cache Connection: keep-alive Content-Length: 444 Content-Type: application/x-www-form-urlencoded Accept: application/json; charset=utf-8 Accept-Encoding: gzip, deflate, br Accept-Language: en-us Host: xxx.xx.xxx User-Agent: Hub/31 CFNetwork/1209 Darwin/20.2.0 SecureChannelRequestType: settingsEndPoint *******************
This issue has been linked to an update to .NET Framework released on Nov 19 2020 and included in a cumulative update to Windows dated February 9, 2021.Following are the platform and patch updates (KB articles) that can cause this issue: PlatformArticleWindows Server 2008 for 32-bit Systems Service Pack 2KB4603005 KB4602961 Windows Server 2008 for x64-based Systems Service Pack 2KB4603005 KB4602961 Windows Server 2008 R2 for x64-based Systems Service Pack 1KB4603002 KB4602958 Windows Server 2008 R2 for x64-based Systems Service Pack 1 (Server Core installation)KB4603002 KB4602958 KB4603002Windows Server 2012KB4603003 KB4602959 KB4601887Windows Server 2012 R2KB4603004 KB4602960 KB4598502Windows Server 2016KB4601318 KB4601051 KB4597247Windows Server 2019KB4601887 KB4588962 KB4598499Windows Server, version 1909 (Server Core installation)KB4601056 Windows Server, version 2004 (Server Core installation)KB4601050 Windows Server, version 20H2 (Server Core Installation)KB4601050 Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites.
The above mentioned updates if applied to servers hosting Workspace ONE UEM Web Console, Device Services, or REST API, will adversely impact the outlined workflows across Workspace ONE UEM.
Our Product team is actively working to resolve this issue in a timely manner. Patches for supported versions of Workspace ONE UEM are being developed and will be listed here as they are made available. List of all supported releases of Workspace ONE UEM and their End Of General Support date can be found here: https://kb.vmware.com/s/article/2960922 Action required Customers with SaaS-hosted instances of Workspace ONE UEM VMware will patch environments on supported versions between March 29th, 2021 and April 15th, 2021This patch will have zero downtimeDedicated SaaS customers with environments on unsupported versions are recommended to schedule an upgrade to a supported version Customers with On-premise instances of Workspace ONE UEM Ensure your environment is on a supported version of Workspace ONE UEM; deploy Workspace ONE UEM patches as they are made available.After patch deployment (to any of the versions listed below), deploy Clean-MachineKeys tool to prevent machine key accumulationIf you are still running into issues take the following steps: Clear SCC from cert stores of all app nodes. Please follow the instruction on how to clear Secure Channel Certificate from all Workspace ONE UEM nodes and database over here .Run Certificate installer with CertificateInstaller -i on all app nodes to fix the corrupted keys. If the above steps do not resolve your issues please reach out to the support team for further analysis. NOTE: VMware has identified an issue (CRSVC-19488) with certain certificate authorities after deploying the fix that was developed to fix CRSVC-18259. More details can be found here: https://kb.vmware.com/s/article/83253.If your environment uses one of the identified certificate authorities which are impacted by CRSVC-19488, it will not be patched until a fix for both issues can be deployed. If you have already requested a specific date and time for your environment to be patched, it will be canceled and you will need to work with Workspace ONE Support to determine when it can be rescheduled.This new issue may cause the deployment of the fix for CRSVC-18259 to be delayed past the April 15th deadline. Fix Versions This issue is resolved in Workspace ONE UEM 2102. In addition to this, the fix is available as a patch for the following versions:** Before deploying the following patches please read: https://kb.vmware.com/s/article/83253 Workspace ONE UEM 1909 - Issue is resolved in 19.9.0.46 and above. Patch is available on the Resources PortalWorkspace ONE UEM 1910 - Issue is resolved in 19.10.0.21 and aboveWorkspace ONE UEM 1912 - Issue is resolved in 19.12.0.22 and aboveWorkspace ONE UEM 2001 - Issue is resolved in 20.1.0.30 and above. Patch is available on the Resources PortalWorkspace ONE UEM 2003 - Issue is resolved in 20.3.0.21 and above.Workspace ONE UEM 2004 - Issue is resolved in 20.4.0.19 and aboveWorkspace ONE UEM 2005 - Issue is resolved in 20.5.0.39 and above. Patch is available on the Resources PortalWorkspace ONE UEM 2006 - Issue is resolved in 20.6.0.17 and aboveWorkspace ONE UEM 2007 - Issue is resolved in 20.7.0.12 and aboveWorkspace ONE UEM 2008 - Issue is resolved in 20.8.0.24 and above. Patch is available on the Resources PortalWorkspace ONE UEM 2010 - Issue is resolved in 20.10.0.14 and above.Workspace ONE UEM 2011 - Issue is resolved in 20.11.0.18 and above. Patch is available on the Resources PortalWorkspace ONE UEM 2101 - Issue is resolved in 21.1.0.9 and above Please subscribe to this page to receive the updated information. NOTE: Update to resolution for Workspace ONE UEM versions 2008 and 2011 Patches released on Feb 24th to address this issue for Workspace ONE UEM 2008 (20.8.0.22) and Workspace ONE UEM 2011 (20.11.0.16) were found to not fully address this issue. Updated patches for these versions are now available (20.8.0.24 and 20.11.0.18) and linked above. Please note that deploying 20.8.0.22 or 20.11.0.16 does not introduce any new issues – it just do not completely address CRSVC-18259. If you have already deployed these versions, please consider installing the updated patches to resolve this issue. Update: If your Workspace ONE UEM environment is 21.9 and above, the machine key tool needs to be run until all the old keys are purged. If the below steps are followed Restart all WS1 services, then Clean-MachineKeys with an option just larger than the time that took, e.g., createdbefore 1_hour_ago, orReboot the server, then Clean-MachineKeys without any -createdbefore option, then the number of keys deleted should fall very, very quickly, in just 1 or 2 usages. (As long as the exclusion period is less than the time between each use of the tool.)
List of all supported releases of Workspace ONE UEM and their End Of General Support date can be found here: https://kb.vmware.com/s/article/2960922
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
