Loading...
Loading...
Groups and Roles maintenance transactions may get canceled halfway due to hitting the UI Transaction Quota limit, which leads to incorrect role inherits. Adding or removing a roles to/from a group with high number of users does not add or remove the role to/from all the users in the group. After the default time of 298 seconds, the transaction is cancelled by the UI Transactions quota rule. This issue can leave the group in an inconsistent state where not all the users have same role in the group. When there are thousands of users in some groups with millions of records in the sys_user_has_role table, and when making changes to group roles and other related changes, the processing occurs in the the user's session and is subject to being cancelled due to the UI Transaction Quota Rule. Leaving the role assignment change in an incomplete state.
Make sure the two plugins "ContextualSecurity: Role Management" and "Contextual Security: Role Management V2" plugins are activated. Create a group. For more information, see the product documentation topic Create a user group. Add 5000 users to the group. Try to add a role to the group. For more information, see the product documentation topic Assign a role to a group. If the transaction completes before 298 seconds, add few hundred users to the group. Try adding another role to the group. Repeat adding users and adding roles until the UI transaction quota cancels the transaction. An instance can be determined for impact if the following is run below in scripts background: GlideTransaction.get().setAllowExcessiveLogging(true); new RoleManagementVerify().verifyInheritedRoles(); If the numbers are mismatched, it is an indicator of impact. If you were notified regarding this defect, it is because an instance was found to have a mismatch and can benefit from having the property workaround below enabled.
Have ServiceNow Technical Support run the Role Inheritance fix script, or request that the Role inheritance verify script is run to check for inconsistencies before running the fix script if any are found. Refer to KB0752104 for details. Add system property glide.ui.schedule_slushbucket_save_for_group_roles and set it to true. This will cause group role additions or removals to be performed in the background, thus will not be immediately visible on the group and will need a refresh to confirm completion. Make role changes on groups using the Edit button on the related list. this will cause the processing to be performed by a scheduled job and not subject to the UI Transaction quota rule. To see when that job is done, refresh the group role related list to when the role is added/removed from the group. When using "Edit" on the related list of a user group, you will see an information message at the top: The property will move the transaction from the front end to the backend to avoid the quota rule transaction from being initiated. This can happen for any instance, only that larger customers would be more common to run into this as the transactions would take longer which reach the quota limit. FAQs: 1) When I changed the property it did not show the message: "Job to add or remove role(s) form user(s) of group has been queued" This can be a few things: Confirm the system property was added correctly with the correct name and value. After changing the property did you refresh your page before adding/removing roles You are not on the fixed version and not all scenarios work, more information below: This system property was made available In Madrid and above. This problem was created as not all scenarios worked. So without being on a fixed release for this problem it will only work when: A role is added to or removed from a group A role is added to or removed from another role It will not work when A user is being added to or removed from a Group. This is what was fixed in this problem. So if you enable this in your instance before a fixed release, you can start utilizing two scenarios, then once you upgrade to a fixed release, you will then utilize all the scenarios. For the two situations where adding the system property is that the "Edit" button must be used to remove a role for a role or group. The slushbuckets appear. If using the check box on the related list, and then the "Actions on selected rows" 'Delete', no job is created and the transaction is subject to the UI Transaction Quota Rule. 2) What impact will I notice when applying this fix? Adding this property will move the transaction from the UI to a background job. The transaction will still take exactly the same time, it will still use the exact same code to run the role management, it just will use a background job instead of your UI transaction, now as its a background job it will run as system. This is why its beneficial to use this property as: i) The transaction will not time out via UI transaction quota rules. ii) Starting this transaction will not lock out your session till it is completed. Since the job runs as a background job there is no feedback in the UI when the job completes. To check for the background job : Navigate to sys_trigger_list.do Add a filter "Trigger class IS com.glide.ui.M2MSlushbucketSaveJob" If there are any records it would mean the job is still running, and the "sys_created_by" field indicates which user triggered the change. Since the job runs as a background job any sys_user_has_role record inserted or updates by the job will show created by "system" (however see (7) below for Tokyo changes) 3) Why are only some instances affected? This validation script was run on your instances and it detects if you have any mismatched roles in your instances. GlideTransaction.get().setAllowExcessiveLogging(true); new RoleManagementVerify().verifyInheritedRoles(); The issues with mismatched roles come from transaction timeouts, specifically the UI Quota rule. When you are adding/removing roles it uses a UI transaction, if that transaction hits the quota rule the transaction will cancel and the roles added/removed will not be complete. So unless your instance has hit this issue some of your instances may not be affected. but note that it still may happen so we recommend to change the property so it runs in the background and you will never hit this issue. 4) Can we determine and find the transactions that caused this issue? Finding the transaction will just show the cancelled transaction, which yes can prove it happened, but is not really useful to fix. Our recommendation is to run the detection script and check for any discrepancies. GlideTransaction.get().setAllowExcessiveLogging(true); new RoleManagementVerify().verifyInheritedRoles(); *** Script: Number of inherited-role records in sys_user_has role, current: 52184, after re-calculation: 52259 *** Script: Number of users with discrepancies for inherited roles: 6 5) How do I read the results from the detection script? So this script is looking for an inherited-roles records in sys_user_has_role and checking if some are missing or should be removed. The script will provide an update like this: *** Script: Number of inherited-role records in sys_user_has role, current: 52184, after re-calculation: 52259 *** Script: Number of users with discrepancies for inherited roles: 6 What this means is currently there are 52184 records in the table, but it should have 52259, this is because of 6 discrepancies: If there are discrepancies it will provide those findings like *** Script: User: <username>, inherited roles to be ADDED: <list of roles> For example: *** Script: User: TESTUSER1, inherited roles to be ADDED: approver_user *** Script: User: ITILUSER2, inherited roles to be ADDED: feature_user,service_editor,project_manager,skill_user,template_read_global,project_user,it_demand_user,project_portfolio_user,sla_manager,flow_operator,workspace_user,rate_model_user,demand_user,portfolio_editor,baseline_user,resource_user,fiscal_calendar_user,demand_manager,idea_manager,it_project_user,report_user,view_changer,report_group,pps_resource,financial_mgmt_user,cmdb_read,interaction_agent,it_project_portfolio_user,tm_test_manager,portfolio_viewer,timecard_approver,it_project_manager,sn_svc_workspace.service_workspace_user,service_viewer,planning_console_user,timecard_user,tm_tester,it_demand_manager *** Script: User: TESTUSER3, inherited roles to be DELETED: itil So we can see that 2 users should have those roles inherited and 1 user should have that role removed. If you have your instance affected you can. a) If it's a sub prod and your prod is NOT affected, if possible you can clone over your sub prod to fix this issue and then add the property to true so it will not be affected again. b) If this is a prod instance or one you cannot clone over you can raise a case to support and they will work with you to run the recalculation script to fix this issue on your instance. Also if you can provide the findings from the detection script when raising that case it will help speed the process along. 6) Why are the updated by and created by fields showing as system? With this property fix to move the transaction to the background, the job will now run as a system user. So therefore the updated by and created by fields will show as system. 7) Important Changes in the Tokyo and later releases If the property is not already present on an instance when it is upgraded to Tokyo or later then this property is added and set to true. In addition a job is scheduled to perform a role recalculation four hours after the upgrade has completed (PRB1538935). See (2) in this section for details of the job and how to track it. This job will only execute on instances which have the "Contextual Security: Role Management V2" plugin installed (id=com.glide.role_management.inh_count). The role recalculation takes place on a separate table which is populated based on the configuration in the sys_user, sys_user_role, sys_user_group, sys_user_role_contains, sys_group_has_role and sys_user_grmember tables. If changes are made concurrently to the rebuild to these tables it is possible that the resulting table may not contain the results from those concurrent changes. If you believe you have encountered this please contact ServiceNow Customer Support. Prior to Tokyo the background job would run as system which would cause an issue when scoped admin roles were involved, since system does not have authority to assign or revoke the scoped role. This was fixed in the Tokyo release (PRB1539950). Effectively the background job impersonates the user that made the changes in the slushbucket. Until Tokyo Patch 2 it was not possible to grant the security_admin role from the background job (PRB1596964).
PRB1480562
Click on a version to see all relevant bugs
ServiceNow Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.