Loading...
Loading...
Upgrade fails during firmware staging process with error: Schedule firmware using VxRail Platform Service failed with timeout Lcm-web.log: 2022-03-15 14:57:04,936 ERROR [LCM] [lcm-node-0] c.v.l.c.b.BatchUpgrade [BatchUpgrade.java:1223] Attempt 2/3 of vSAN access exception. but failed with error: com.vce.lcm.exception.LCMInternalException: Schedule firmware using VxRail Platform Service failed with timeout. at com.vce.lcm.core.upgrade.esxi.EsxiFirmwareUpgrader.checkStatus(EsxiFirmwareUpgrader.java:315) at com.vce.lcm.core.upgrade.esxi.EsxiFirmwareUpgrader.scheduleBatchUpgradeStaging(EsxiFirmwareUpgrader.java:358) at com.vce.lcm.core.batch.BatchUpgrade.scheduleFirmwareStaging(BatchUpgrade.java:1809) ............................. ............................. 2022-03-15 14:57:05,152 ERROR [LCM] [lcm-node-0] c.e.m.m.u.s.LcmNodeUpgradeServiceImpl [LcmNodeUpgradeServiceImpl.java:83] failed to upgrade node with exception: com.vce.lcm.exception.LCMException: Schedule firmware using VxRail Platform Service failed with timeout. at com.vce.lcm.core.batch.BatchUpgrade.upgradeHost(BatchUpgrade.java:1573) at com.vce.lcm.core.batch.BatchUpgrade.performBatchUpgrade(BatchUpgrade.java:1023) at com.vce.lcm.core.batch.BatchUpgrade.performBatchUpgrade(BatchUpgrade.java:1205) ............................. ............................. Caused by: com.vce.lcm.exception.LCMInternalException: Schedule firmware using VxRail Platform Service failed with timeout. at com.vce.lcm.core.upgrade.esxi.EsxiFirmwareUpgrader.checkStatus(EsxiFirmwareUpgrader.java:315) at com.vce.lcm.core.upgrade.esxi.EsxiFirmwareUpgrader.scheduleBatchUpgradeStaging(EsxiFirmwareUpgrader.java:358) ............................. ............................. 2022-03-15 14:57:57,331 ERROR [LCM] [lcm-core-0] c.v.l.c.u.ComponentUpgradeService [ComponentUpgradeService.java:310] Component VMWARE_ESXI upgrade failed due to Schedule firmware using VxRail Platform Service failed with timeout. . 2022-03-15 14:57:57,399 ERROR [LCM] [lcm-core-0] c.v.l.a.LCMServiceImpl [LCMServiceImpl.java:1653] failed to upgrade: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip, with exception: com.vce.lcm.exception.LCMException: Schedule firmware using VxRail Platform Service failed with timeout. at com.vce.lcm.core.upgrade.ComponentUpgradeService.performComponentUpgrade(ComponentUpgradeService.java:329) at com.vce.lcm.api.LCMServiceImpl.performBatchUpgrade(LCMServiceImpl.java:1933) at com.vce.lcm.api.LCMServiceImpl.performCompositeUpgrade(LCMServiceImpl.java:1285) ............................. ............................. 2022-03-15 14:57:57,622 INFO [LCM] [lcm-core-0] c.e.m.m.u.r.BaseVirtualApplianceUpgradeProfilePool [BaseVirtualApplianceUpgradeProfilePool.java:798] add error message status, profile: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip, step: Upgrading, target: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip, action: Failure occurred while running an upgrade for bundle: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip., detail messages: Failure occurred while running an upgrade for bundle: VXRAIL_COMPOSITE-7.0.320-27377804_for_4.7.x.zip. The error message: Schedule firmware using VxRail Platform Service failed with timeout.. Checking the iDRAC web UI, there were no firmware scheduled jobs during the staging process. Checked the platform_svc.log and found the following errors: 2022-03-15T12:09:51Z platform_svc: [Thread-7] INFO - Updating in progress: 69 2022-03-15T12:10:14Z platform_svc: [Thread-2] INFO - Inventory update process is quiesced 2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - start to check the upgrade progress ... 2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - PTAgent response 200: 2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - { 2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - "inProgress": "true", 2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - "progressPercentage": "69" 2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - } 2022-03-15T12:10:51Z platform_svc: [Thread-7] INFO - Updating in progress: 69 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - start to check the upgrade progress ... 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - PTAgent response 200: 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - { 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - "inProgress": "true", 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - "progressPercentage": "69" 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - } 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - Updating in progress: 69 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - Upgrade timeout. PTAGENT seems alive. We have to restart it. 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - no software vib change in progress, do not wait 2022-03-15T12:11:51Z platform_svc: [Thread-7] INFO - Call ptagent start/restart command/etc/init.d/DellPTAgent restart 2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR - Error occurs during upgrading via PTAgent. Retrying ... 2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR - Traceback (most recent call last): 2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR - File "/opt/vxrail/bin/platforms/dell/plugin/__init__.py", line 1097, in do_pt_payload_fw_upgrade 2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR - idrac_fw_packages) 2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR - File "/opt/vxrail/bin/platforms/dell/plugin/__init__.py", line 1034, in _onetime_pt_payload_fw_upgrade 2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR - raise Error('PTAgent firmware upgrade timeout') 2022-03-15T12:13:08Z platform_svc: [Thread-7] ERROR - endpoints.error.Error: PTAgent firmware upgrade timeout These errors in platform_svc.log were found looped retrying until the upgrade failed in the vCenter GUI. Manually upgrading firmware through iDRAC Web UI, we can see one of the following errors: Message: An internal error occurred. Unable to complete the specified operation. Recommended Response Action: Retry the operation. If the issue persists, reset the iDRAC and retry the operation. Message: The requested job cannot be scheduled because Lifecycle Controller is not enabled. Recommended Response Action: Enable Lifecycle Controller and retry the operation. Lifecycle Controller can be enabled by using the F2 System Setup by pressing F2 during server startup, then select iDRAC Settings > Lifecycle Controlller and enable the Lifecycle Controller feature. Retry the operation after the server restarts.
The LifeCycle controller was found in Disabled status. With LCC disabled, iDRAC cannot upload and schedule the firmware to the job queue. Run the command racadm get LifeCycleController.LCAttributes to verify: racadm>>racadm get LifeCycleController.LCAttributes #AutoDiscovery=Off AutoUpdate=Disabled BIOSRTDRequested=False CollectSystemInventoryOnRestart=Enabled #DiscoveryFactoryDefaults=Off IgnoreCertWarning=On IPAddress= IPChangeNotifyPS=Off #Licensed=No LifecycleControllerState=Disabled PartConfigurationUpdate=Apply Always PartFirmwareUpdate=Match firmware of replaced part ProvisioningServer= #StorageHealthRollupStatus=2 #SystemID=0716 !!UserProxyPassword=******** (Write-Only) UserProxyPort=80 UserProxyServer= UserProxyType=HTTP UserProxyUserName=
To resolve this issue: 1. Connect to the iDRAC of the node that fails during firmware staging process through SSH. 2. Enable the LCC with command racadm set LifeCycleController.LCAttributes.LifecycleControllerState 1 racadm>> racadm set LifeCycleController.LCAttributes.LifecycleControllerState 1 Object value modified successfully 3. Validate the status of LCC again with command racadm get LifeCycleController.LCAttributes racadm>> racadm get LifeCycleController.LCAttributes #AutoDiscovery=Off AutoUpdate=Disabled BIOSRTDRequested=False CollectSystemInventoryOnRestart=Enabled #DiscoveryFactoryDefaults=Off IgnoreCertWarning=On IPAddress= IPChangeNotifyPS=Off #Licensed=No LifecycleControllerState= Enabled <---- (correct value) PartConfigurationUpdate=Apply Always PartFirmwareUpdate=Match firmware of replaced part ProvisioningServer= #StorageHealthRollupStatus=2 #SystemID=0716 !!UserProxyPassword=******** (Write-Only) UserProxyPort=80 UserProxyServer= UserProxyType=HTTP UserProxyUserName= 4. Retry the upgrade and verify in the iDRAC UI that the firmware are being uploaded and scheduled in the job queue.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.