Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Workspace ONE UEM services and application pools may fail to start once stopped. This issue is typically observed alongside the following error message in the service's log: Error AirWatch.Security.CodeSigning.WriteValidationFailureToWindowsEventLog Unable to retrieve publisher or publisher not trusted. Assembly: System.ComponentModel.Annotations, Version=4.2.1.0, Culture=neutral, PublicKeyToken=ABCDE00F. Assembly Location: <DLL File Path> Method: AirWatch.Security.CodeSigning.WriteValidationFailureToWindowsEventLog
Root Cause Analysis (RCA): The Root Cause Analysis for this incident is now available - RCA-WS1UEM_SigningCertExpiry_08072023-SINST-176145
DLLs are expected to be signed and validated before loaded and executed. The Windows OS checks the validity of the signing certificate when loading DLLs to start a service. The internal CA and certificate used to sign DLLs shipped with Workspace ONE UEM, is set to expire at 1320 UTC on 8th Aug, 2023. Past expiration, app pools or services dependent on these DLLs will fail to start or function properly on/after 8th Aug, 2023. This will impact all instances of Workspace ONE UEM which have not been mitigated through the relevant patch or workaround (noted below).Impacted components include (but not limited to): Device ServicesWeb ConsoleSelf Service PortalDevicesGatewayWS1_APIDevice SchedulerDirectory Sync ServiceMEG Queue Service Based on current evaluation, the following services are NOT directly impacted: AirWatch Cloud Connector (ACC)AirWatch Cloud Messaging Service (AWCM)Workspace ONE Intelligence Connector (ETL Connector)Secure Email Gateway v2 (SEG v2)VMware TunnelEmail Notification Service v2 (ENS v2) Health of an app pool or endpoint can be verified by accessing the health check endpoints as noted in Workspace ONE UEM documentation .
A fix has been released for this issue for the following supported versions of Workspace ONE UEM available for On-premises consumption. NOTE: Customers on unsupported versions should upgrade to a supported version, listed below. The workaround may be applied if an upgrade is not an immediate option, however applying the workaround to an unsupported version has not been validated. Customers who experience issues while deploying the workaround must upgrade their environment to a supported version. Please note these patches differ from typical Workspace ONE UEM cumulative patches in a few ways: The patches take the form of a full UEM installer (akin to major version upgrade) and need to be executed on Console, Device Services, API, AWCM, and Database servers When the patch installer for a given version (2302, for example) is deployed, it will increment the patch version only (to 23.02.0.17, for example) Any future Workspace ONE UEM patches will need to be deployed on top of this fix Steps for deployment are noted at the end of this section. Workspace ONE UEM versionFix InstallerWorkspace ONE UEM 2302Workspace ONE UEM 23.2.0.17Workspace ONE UEM 2212Workspace ONE UEM 22.12.0.27Workspace ONE UEM 2209 *Workspace ONE UEM 22.9.0.35Workspace ONE UEM 2206Workspace ONE UEM 22.6.0.40Workspace ONE UEM 2203Workspace ONE UEM 22.3.0.50 Deployment Instructions Stop all Workspace ONE UEM services, and IIS on the application servers Backup the Workspace ONE UEM database, and take snapshots of the application servers Run the application installer (located in the Application folder within the .zip file) and wait till it prompts for the Database upgrade Run the Database installer (located in the Database folder within the .zip file) and complete deployment Complete application installer deployment Action required: Customers with Shared Hosting and Managed Hosting – Latest Mode: No further action required. Customers with Managed Hosting environments: VMware Cloud Operations will proactively patch such environments to ensure platform stability. The latest available patch will be applied so you may see a newer version applied compared with the minimum version which resolves the issue. The schedule for patching Managed Hosting environments will be published on September 20th, 2023 onwards. You can then view the date/time at which patching will be conducted for your environment, through the myWorkspaceONE portal. You can use the Upgrade Scheduler to reschedule patch deployment, if needed. For details on using the Upgrade Scheduler, please refer to KB 2960929 Customers with On-Premises environments: Please leverage the patches noted above. If a patch is not available for your version of UEM, please leverage the workaround noted below. NOTE: On-premises environments which have been updated with the patches noted above may experience a problem affecting administrator interaction with DDUI-based profiles. For more details on impact and resolution, please refer to KB 93911. * For on-premises UEM 2209 environments hosted on Windows Server 2012 R2, you would need to apply the custom patch from KB 90366 again. Please subscribe to this KB to be notified when updates are available.
As a workaround for On-premises environments a version agnostic UEM Digital signing utility tool is now available to re-sign affected DLLs to prevent the issue until a fix is available or deployed for your environment. Ensure full environment backups are taken prior to running the utility tool. Please expect a minor service outage of about 5-15 minutes as the tool will be performing stop and start actions of affected services.
Additional Information A list of currently supported version of Workspace ONE UEM and their availability for On-premises customers is note at KB 2960922 Frequently Asked Questions (FAQs) Q: Do I need to deploy the patch if I have already deployed the workaround utility? A: The utility re-signs affected DLLs with a valid certificate to mitigate the problem. After you've run the utility, it is advisable to patch your environment as soon as possible so your environment can easily undertake future maintenance activities Q: My environment is running a version of Workspace ONE UEM that is no longer supported. How do I mitigate impact? A: Please ensure your environment is upgraded to a supported version, using any of the installers linked above. These are full UEM installers and deploying them will upgrade UEM. If you are unable to upgrade immediately, you may leverage the workaround. Q: Does my server need internet access to deploy the utility? A: Please leverage the latest version of the utility (v6 and above) which do not require internet/external access for execution Q: Do I need special permissions in Powershell to run the utility? A: The Powershell session needs to be launched with elevated privileges (Run as Administrator). In the event that this is insufficient, bypass execution policy by following the steps noted in the README file. Q: I received an error message during the script stating that the service failed to start, what should I do? A: If the utility displays no other signing errors, attempt to start the service manually. If this does not work, stop all Workspace ONE UEM services, and IIS, manually, re-run the utility, and start services once it finishes execution. Change Log 8th Aug, 1130 PDT: Clarified Impact. Updated Resolution with patch details, deployment guidance, and guidance for On-premise and SaaS environments 8th Aug, 1430 PDT: Updated Impact details. Added FAQ. Updated version of Digital Signing Utility (v6) is now available 9th Aug, 0930 PDT: Updated resolution details. 9th Aug, 1530 PDT: Updated resolution with additional issue details. 11th Aug, 1730 PDT: Updated resolution details. 15th Aug, 1430 PDT: Added Root Cause Analysis 25th Aug, 1400 PDT: Updated additional notes for on-premises UEM 2209 21st Sept, 1500 PDT: Updated Action Required section to call out that the latest patch will be applied