Loading...
Loading...
The Key Management Framework (KMF) requires a connection via the sys_properties "glide.kmf.vault.token" record that is used to authenticate to the Vault PKI. It does this with an access token that is generated in the field "value", used for authentication. For best practice, these access tokens are short lived, with TTL being 24h. As soon as an attempt is performed connecting to the vault, if the token in "glide.kmf.vault.token" is expired (>24h), a new token is generated which involves an update to the "glide.kmf.vault.token" "value" field with the new generated token. The issue is that in San Diego and above, the update process for the "glide.kmf.vault.token" property is performed with ignore cache set as FALSE instead of TRUE. As a result, a full cache flush is performed on the instance, which can impact instance performance.
While cloning is stated as an example, is it one of the reproducing steps that regenerates the token. Further examples are in the FAQ. 1. Instance A - Ensure the system property - 'glide.kmf.vault.token' is expired. 2. Clone is performed from Instance A to B. 3. The log messages will look similar to the following below: 2022-07-07 03:54:01 (419) API_INT-thread-4 SYSTEM txid=66fa91251b64 WARNING *** WARNING *** Full cache flush triggered from java.base/java.lang.Thread.getStackTrace(Thread.java:1607) com.glide.sys.cache.CacheMan.flush(CacheMan.java:444) com.glide.sys.cache.CacheManager.flush(CacheManager.java:132) com.glide.script.GlideSystem.cacheFlush(GlideSystem.java:503) com.glide.script.GlideSystem.cacheFlush(GlideSystem.java:479) Result: Since the Access token- 'glide.kmf.vault.token' is expired, the cloning process resulted in regenerating the token which updates the sys_properties with the new token. This operation results in a full cache flush, which impacts the instance performance, since ignore_cache is set to FALSE on the property.
Updates to a sys_property will result in a full cache flush, meaning the system flushes this property's value from all other server-side caches. To prevent this, we can checkmark "Ignore cache" on the system property - 'glide.kmf.vault.token', which sets "Ignore cache" = TRUE. This will ensure the system will flush only the sys_property cache and will not affect the other caches. NOTE: This can only be updated by ServiceNow technical support as the property is "maint" role only. FAQ Q: I received a maintenance request to update the property, can I opt out of this? A: You can, but it's highly advised not to as it can only be updated by ServiceNow Technical Support since the property is a "maint" only property. You will need to raise a case separately to have the property updated if you decide to opt out of the maintenance. Opting out of the maintenance can be done on the COMM record form sent by clicking on "Opt out". Q: Can I run this maintenance myself? A: No. This property is a "maint" only property which is not accessible with "admin" role. Q: What will the maintenance do? A: It will update the 'glide.kmf.vault.token' sys_property as ignore cache to TRUE. It performs this as a customization so the property will persist on upgrade, otherwise it will reset back to ignore cache to FALSE until the instance is on a fixed version. Q: What is the 'glide.kmf.vault.token' property and is it necessary? A: The property is needed to authenticate to the Vault PKI. It is a necessary property used as part of authentication. Q: My instance has the KMF global plugin (com.glide.kmf.global) installed but I did not receive notification about another instance for this maintenance. Should I be concerned? A: The required instances needed for this maintenance was sent 2-Dec. Beyond this date if you did not receive a message for another instance, then the maintenance was not determined as necessary as the property is not populated. The property is needed once it connects to the vault, if it is not populated then the instance would later generate it when needed, but it generates it as ignore cache FALSE. The property can be created and pre-populated with ignore cache as TRUE, but it can only be performed through technical support. Please open a case to have them perform this action. Q: I noticed this is impacting San Diego and above but I'm receiving a notification when my instance is pre-San Diego. Should I still receive this maintenance? A: Yes, having the property updated is still beneficial to the instance to set ignore cache as TRUE to prevent a full cache flush later in the future when the instance is on San Diego (or above). Q: What versions are impacted by this issue? A: San Diego and Tokyo versions are impacted. Q: Are there other regeneration situations where “glide.kmf.vault.token” is regenerated on an expired property that doesn’t depend on clone? A: It happens irrespective of the instance type, once a connection occurs to the Vault, it will perform the check on the property. Q: The “glide.kmf.vault.token” expires every 24 hours, is this correct? A: Yes, that's correct. It expires every 24 hours. Q: The workaround mentioned states to checkmark "Ignore Cache" on "glide.kmf.vault.token". However in the "Steps To Reproduce" example, a clone is performed which regenerated the expired token. When should I update the "Ignore Cache" for a clone? A: The update to the system property to ignore cache should be performed pre-clone, however please take note of the exclusions and preservations for the clone to ensure it will not be overwritten on the clone. Q: What other scenarios exist that calls the regeneration token call to happen? Are there other KMF calls that we need to be aware of that checks into the expiration of the token property? If so, what are these functions? A: Every call to the vault checks for this token validity and can perform the token regeneration. It can occur in the following situations: Asymmetric key generation – If the instance needs asymmetric keys (such as public, private keys) to encrypt the data in transit when communicating with other instances It can occur during data transfers like key exchange - For example, instance data replication - IDR Q: If a clone happens and the target instance has an expired token – Does it also perform a cache flush on the source instance? Does it matter if the source token is expired or not? A: The source and target instance needs the other instance’s public key to encrypt the key exchanged. So, If the target instance has an expired token, only the target token will get updated, not the source.
PRB1589994
Click on a version to see all relevant bugs
ServiceNow Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.