Loading...
Loading...
vRealize Automation 7.0, 7.0.1, 7.1, and 7.2 Provisioning or data collection requests are stuck in "In progress" status.The load balancer monitor may or may not be displaying more than one MS nodes as active.Machine deployments get stuck in "Requested" stateThe logs indicate that more than one Manager Service is active at the same time. This could be verified by examining a passive Manager Service log. You may see messages similar to the following after the node has entered the passive state: [UTC:2017-09-19 21:00:35 Local:2017-09-19 17:00:35] [Info]: [sub-thread-Id="6" context="" token=""] Successfully marked current node as passive in the database....[UTC:2017-09-19 21:02:38 Local:2017-09-19 17:02:38] [Debug]: [sub-thread-Id="12" context="" token=""] DC: Created data collection item, WorkflowInstanceId 195158, Task state, EntityID <Entity_ID>, StatusID = <Status_ID> vRealize Automation 7.3 Machine deployments get stuck in "Requested" stateEvent broker subscriptions do not go past VMPSMasterWorkflow32.Requested phase PRE event stateIaaS Manager service recently was failed over
vRealize Automation 7.0, 7.0.1, 7.1, and 7.2 Multiple Manager Services running causes a race condition. This is generally caused by user error on installation or when a systems admin attempts to manually failover the services without properly stopping the problematic node.Note: Manager Service automatic failover is introduced in vRealize Automation 7.3. vRealize Automation 7.3 In vRealize Automation 7.3 upon IaaS Manager failover - the passive Manager Service is not able to stop one or more of the scheduled operations it manages (e.g., Data Collection) and the operations continue to be executed after the Manager Service node has entered the passive state. This results in multiple nodes racing for the same tasks, which leaves some of those tasks in an inconsistent state.
vRealize Automation 7.0, 7.0.1, 7.1, 7.2, and 7.3 The automatic manager service failover trigger for this issue is resolved in vRealize Automation 7.3.1 and above, available at VMware Downloads.Although one of the specific triggers for this issue is resolved in 7.3.1,7.4. Specifically the automatic manager service failover logic.The scenario of the IaaS workflow engine becoming overloaded can still occur due to other triggers, for example if the manager service is manually started on the secondary IaaS manager node when automatic failover is not enabled.In which case the same workaround below can still be followed.
vRealize Automation 7.0, 7.0.1, 7.1, and 7.2 To work around the issue: Stop the passive Manager Service and restart the active one.If swapping the manager service components does not resolve the stuck in requested issue, follow the SQL instructions located below for vRealize Automation 7.3. vRealize Automation 7.3 To work around the issue: Run the following SQL query against IAAS SQL Server database. The query lists all VM's stuck in "Requested" status in descending creation date order. These VM's need to be deleted from IAAS SQL database in order to free IAAS workflow engine. SELECT vm.VirtualMachineId, vm.VirtualMachineName, vm.VirtualMachineState, vm.RecCreationTime, vm.VMCreationDate FROM InstanceState ins JOIN VirtualMachine vm ON (ins.uidInstanceID = vm.VirtualMachineID) WHERE vm.VirtualMachineState = 'Requested' ORDER BY vm.VMCreationDate desc Stop all IAAS services (Manager service, DEM Orchestrator, DEM worker, and proxy agent etc.)BACKUP any related appliances and databases (both IAAS SQL Server database and VRA database) to allow a restore if there is any problem on running steps belowCompile attached usp_RemoveVMFromVRA.sql against IAAS SQL server database.Open the attached clear_requested_vm.sql in SQL query window and run the query to remove VMs stuck in "Requested" status.Restart all IAAS services. Install the hotfix attached to this article (Applicable to 7.3.0 GA only) Download hf-1988761.7zUnzip / extract the file contents to the Windows IaaS Manager Service component systems3 total files need replaced: DynamicOps.Core.Common.dll,DynamicOps.VMPS.CommonRuntime.dllDynamicOps.VRM.DataCollectionService.dll Create backups of the old files by either renaming the file or moving to a backup directory in the Windows file systemReplace all 3 files with the extracted files.Restart the Manager Service. Note: "DynamicOps.Core.Common.dll" is located in multiple folders within "<IaaS installation location>\VMware\vCAC".Although the change is related to the Manager Service only, ensure every occurence of "DynamicOps.Core.Common.dll" is updated. The IaaS workflow engine should pick up new requests now.
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.