Loading...
Loading...
ServiceNow identified an issue originating from the ECB (ww.ecb.europa.eu) endpoint associated with API timeout that impacts two out-of-box scheduled jobs, 'Retrieve System Rates' and 'ECB Exchange Rate Load', that run daily at 5:30 PM and 6:00 PM UTC. to fetch current exchange rates. As a result, these jobs may timeout due to a recent change with the OCSP endpoint for www.ecb.europa.eu, which prevents the instance from downloading the most current exchange rates. Outdated exchange rates can potentially impact financial transactions, reporting, and pricing calculations depending on business requirements. Logs indicate that the issue started on February 20th due to some updates at ECB endpoints that caused traffic contention. After updating the execution time, the two jobs are successfully fetching exchange rates (More details in Workaround and FAQ).
Login as admin Type sys_trigger.LIST on navigation filter. Search for "Retrieve System Rates" job and "ECB Exchange Rate Load" job. Check if currency rates have been loaded for the expected dates based on the job’s scheduled execution time. Review the localhost logs for any OCSP check failures while retrieving exchange rates from https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml.
The "Retrieve System Rates" and "ECB Exchange Rate Load" jobs run as scheduled and successfully update exchange rates in the respective tables. No errors appear in the logs related to OCSP validation or connection issues.
The jobs intermittently fail due to an OCSP check failure when attempting to retrieve data from the ECB endpoint. Logs show below errors: The localhost log will indicate failures related to OCSP check for outgoing REST API while attempting to retrieve content of https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml: OCSPCheck WARNING *** WARNING *** OCSPCheck HTTP status: 504, error message: Method failed: (/) with code: 504 GlideSSLProtocolSocketFactory WARNING *** WARNING *** OCSP revoke check IOException for www.ecb.europa.eu *** Script: ECB Exchange rate load failed: Connection/Open Error: unknown tag 28 encountered
To prevent data integrity issues, we applied emergency maintenance on March 8th and 9th on ServiceNow instances. As a part of the update, the execution run times of the two scheduled jobs were modified to reduce the traffic congestion and ensure jobs run successfully. Here are the steps to either execute the jobs manually or update the execution time if you need to: Step 1: Navigate to Scheduled Jobs Login as Admin. On the navigation filter type - sys_trigger.LIST Step 2: Run the Jobs Manually or reschedule them as needed Locate the "Retrieve System Rates" and "ECB Exchange Rate Load" jobs. Click on the job name to open the record. Click 'Execute Now' to run the job immediately. Alternatively, update the Runtime field to a random runtime and save the record. Step 3: Validate the Updated Rates After executing the jobs, verify that the rates have been successfully loaded in the relevant tables: fx_system_rate table (loaded via the "Retrieve System Rates" job) fx_rate table (loaded via the "ECB Exchange Rate Load" job) To check the rates are populated daily check the "Created" value on these records. If you encounter any issues, please reach out for further assistance. FAQ Q: Name of the scheduled jobs that are impacted? A: Retrieve System Rates and ECB Exchange Rate Load Q: What caused the issue with scheduled jobs? A: The issue originates from the ECB's endpoint and is unrelated to any changes on the ServiceNow side. Q: When are the impacted scheduled jobs set to run originally? A: Retrieve System Rates and ECB Exchange Rate Load are set to run at 5:30 pm UTC and 6:00 pm UTC respectively. Q: How might delaying this change affect exchange rate accuracy and system reliability? A: Delaying this change could have led to outdated exchange rates being used in critical financial transactions, potentially impacting reporting accuracy, pricing calculations, and overall system reliability. By making this adjustment now, we are mitigating risks and ensuring the continued accuracy of exchange rate updates. Q: If I have customized the run time for the job will maintenance override the customization? A: The maintenance will check whether the scheduled jobs have been customized. To identify if a job has been customized (and to prevent the write audit from modifying it), we will compare the following fields to their Out-of-the-Box (OOB) values: runtime and max_drift. Q: When will this maintenance happen? A: Maintenance was completed on Friday, March 7th for sub-production and Saturday, March 8th for Production instances, followed by COMM on March 10th. Q: How is an instance determined as potentially impacted or impacted? A: Both the impacted scheduled jobs ("Retrieve System Rates" and "ECB Exchange Rate Load") are by default present on all instances. So maintenance will run on all instances. Q: Is this maintenance still required if scheduled jobs are inactive or changed by the customer? A: Maintenance will not touch the impacted scheduled jobs (Retrieve System Rates" and "ECB Exchange Rate Load") if they are modified by customers. Q: I have modified both the impacted scheduled jobs so maintenance did not make any changes. What should I do? A: Perform the changes manually by following the steps provided on the workaround section on the KB. Q: What does the maintenance do? A: The two impacted scheduled jobs, “Retrieve System Rates” and “ECB Exchange Rate Load” were scheduled to run at 5:30 PM and 6:00 PM UTC respectively are now updated to run as follows: - Sub-production instances will now run the jobs between 8:00 PM and 10:00 PM UTC. - Production instances will now run between 3:30 PM and 5:30 PM UTC. Q: Why was it necessary to change the execution time of the scheduled jobs? A: We have staggered the execution times of the scheduled jobs to minimize traffic congestion and reduce the likelihood of failures. The change was made after identifying that the jobs were intermittently failing when executed simultaneously. Q: Was the issue resolved after the maintenance? A: After monitoring the scheduled jobs post-update, we are confident it reduced traffic congestion to ECB, resulting in successful jobs and ensuring the accuracy of exchange rate updates Q: Will this maintenance have any service impact? A: We do not anticipate that the maintenance script will cause service impact. Cache flush or restart is NOT required during the maintenance. Q: I am not using this feature and I am still notified for this maintenance. Is this required? A: Yes, this is a proactive maintenance for future. Q: How do I know if I am impacted with this problem? A: In the logs if you see below your instance is impacted *****Script: ECB Exchange rate load failed: Connection/Open Error: unknown tag 28 encountered Q: Can I opt out of this maintenance or reschedule? A: The maintenance was already performed. Rescheduling is not an option. Q: Can I run this modify the impacted scheduled jobs to run at different times after the maintenance? A: Yes, please follow the steps provided on the workaround section. Q: I have an instance that wasn't added for maintenance even though I believe it is affected. What do I do? A: Please follow the steps provided on the KB. Q: Can I make changes to the impacted scheduled jobs after the maintenance? A: Yes, make sure you are scheduling both the impacted jobs anytime excluding 5:30pm UTC and 6:00 pm UTC Q: Is this issues only in specific versions? A: The instances is affected irrespective of any version they are in.
PRB1861950
Click on a version to see all relevant bugs
ServiceNow Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.