Loading...
Loading...
Since a check was added in the Tokyo version (PRB1538125), a MID Server upgrading itself will not shut itself down until it has confirmation that the Upgrade process that takes over has started properly. It waits for a hard-coded 10 minutes, before giving up. The MID Server then stays up and deletes the temporary upgrade files. However, the upgrade Process may eventually start and carry on, killing the main MID Server service processes, and then deleting the files from the agent folder, that it had intended to replace. But the files it was going to replace them with, extracted from the ZIP file, are no longer in the temp folder. This leaves the MID Server down, and with files missing that prevent it being started again.
This is difficult to reproduce on demand without finding a way to slow down the bootstrapping of the upgrade process. The normal log line for the starting of the upgrade process is show: 2023-10-22T01:16:20.029-0400 INFO (AutoUpgrade.3600) [MIDUpgrader:501] Stopping MID server. Bootstrapping upgrade . 2023-10-22T01:16:20.029-0400 INFO (AutoUpgrade.3600) [MIDUpgrader:502] MIDUpgrader.startUpgradeRunner(), OperationalState=UPGRADING Some time later, the MID Server starts to log that it is still waiting for the upgrade process to start: 2023-10-22T01:35:05.525-0400 INFO (AutoUpgrade.3600) [MIDUpgrader:375] DistUpgrade file is not present . 2023-10-22T01:35:05.525-0400 INFO (AutoUpgrade.3600) [MIDUpgrader:323] Waiting for upgrade process started signal... The agent log will show this warning 10 minute later: 2023-10-22T01:45:06.952-0400 WARN (AutoUpgrade.3600) [MIDIssueLogger:83] Creating unique MID issue with message "Unable to receive upgrade process status" for source "Couldn't Launch Upgrade Process" The MID Server now assumes it can carry on as normal after logging the failed upgrade, and will clean up the temp files it extracted from the downloaded upgrade ZIP files. However, wrapper.log shows the upgrade process did eventually start after all, and signal to the MID Server service using the DistUpgrade file, but it is too late: INFO | jvm 1 | 2023/10/22 01:55:19.964 | Oct 22, 2023 1:55:19 AM com.snc.dist.mid_upgrade.DistUpgradeUtil writeStartedStatusToFile INFO | jvm 1 | 2023/10/22 01:55:19.964 | INFO: File path: C:\ServiceNow Mid Servers\....\agent\conf\ DistUpgrade It then goes on to kill the mid server service INFO | jvm 1 | 2023/10/22 02:06:15.453 | INFO: Killing process (PID = 7064). INFO | jvm 1 | 2023/10/22 02:06:29.104 | INFO: Killing process (PID = 8016). and delete files: INFO | jvm 1 | 2023/10/22 02:09:33.368 | Oct 22, 2023 2:09:33 AM com.snc.dist.mid_upgrade. UpgradeMain wipeDirs and fail to replace them: INFO | jvm 1 | 2023/10/22 02:09:33.369 | INFO: Copying files to MID server installation path. INFO | jvm 1 | 2023/10/22 02:09:33.369 | Oct 22, 2023 2:09:33 AM com.snc.dist.mid_upgrade.UpgradeMain run INFO | jvm 1 | 2023/10/22 02:09:33.369 | SEVERE: com.snc.dist.mid_upgrade.UpgradeException: java.io. FileNotFoundException : Source 'C:\ServiceNow Mid Servers\....\ agent\work\upgrade_temp \6339952203872892431\ agent' does not exist With missing binaries, the MID Server service can now not be manually started 2023/10/23 07:27:40 | Starting the ServiceNow MID Server_... service... 2023/10/23 07:27:41 | Launching a JVM... 2023/10/23 07:27:42 | Error: Could not find or load main class org.tanukisoftware.wrapper.WrapperStartStopApp 2023/10/23 07:27:42 | Caused by: java.lang.ClassNotFoundException: org.tanukisoftware.wrapper.WrapperStartStopApp
This problem is currently under review and targeted to be fixed in a future release. Subscribe to this Known Error article to receive notifications when more information will be available. The installation will now have missing files that will need to be replaced. This process can be used to repair it: KB0713557 How to manually restore or upgrade a MID Server after a failed auto-upgrade If you are attempting to prevent this issue from occurring in a future upgrade (rather than repairing a past upgrade), the following steps are another workaround: (1) From the navigator, go to MID Server > Properties (or open the ecc_agent_property table). (2) Click "New" to create a new property (3) Fill in the form with the following values: Name: mid.skip.dist.signal.for.shutdown Value: true MID Server: (leave blank) (4) Restart all the MID servers. Ultimately, the performance of the MID Server host should be investigated, because there is no valid reason for a simple java program to take more than 10 minutes to start up. The host should have a minimum of4 CPU cores, per MID Server install on that host. This is a documented minimum requirement for a MID Server install to be supported. For memory, add up the maximum heap allocation for each JVM, and then double it to account for the OS, child Powershell processes, any other apps and management/security agents running on the host, and additional upgrade process JVMs that will be running all at the same time, at the same time as the main mid server JVMs during an instance upgrade.
PRB1710037
Click on a version to see all relevant bugs
ServiceNow Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.