POSTED: Jul 3, 2019 12:33:50 PM
Root Cause Analysis:
|
Description: |
Data Inconsistency in NA11, NA21, EU11, EU21 |
|
Timeframe: |
2019-07-03 12:00PM CDT to 2019-07-04 5:30PM CDT |
|
Impact: |
Customers may have experienced inability to generate new documents via the Document Generation feature. |
|
Cause: |
A change to distributed database replication settings caused some database replicas not to have the data they needed. |
|
Resolution: |
A hotfix was deployed to increase database query consistency levels so clients always get the most current data from the database, and data inconsistencies were repaired. |
|
Next Steps: |
We are searching out other queries which may benefit from increasing database query consistency levels, and we're developing a less disruptive procedure for modifying database replication settings. |
[UPDATED: Jul 8, 2019 8:11 AM] - In addition to the issues experienced on NA11 and NA21 on July 3, the EU11 and EU21 Production environments experienced the same service availability issues on July 4 from approximately 10:00 AM to 6:00 PM CST. The SpringCM Technology Team isolated and repaired the service availability issue in that time.
[RESOLVED: Jul 3, 2019 7:06 PM] The SpringCM Technology Team has isolated and repaired the service availability issue on the NA11 and NA21 instances. The problem began at approximately 12:20 PM CST and was resolved by 6:06 PMCST. We apologize for the inconvenience and will be providing a root cause analysis of the issue in a near future.
[UPDATE: Jul 3, 2019 3:52:00 PM] - The SpringCM Technology Team continues to work on isolating the service availability issue on the NA11 and NA21 instances. Please check the status of trust.springcm.com for updates regarding this issue.
