[RESOLVED] NA11 | NA21 | EU11 | EU21 - Document Generation Performance

 POSTED: Jul 3, 2019 12:33:50 PM        NA21, NA11, Document Creation

Root Cause Analysis:

Description:

Data Inconsistency in NA11, NA21, EU11, EU21

Timeframe:

2019-07-03 12:00PM CDT to 2019-07-04 5:30PM CDT

Impact:

Customers may have experienced inability to generate new documents via the Document Generation feature.

Cause:

A change to distributed database replication settings caused some database replicas not to have the data they needed.

Resolution:

A hotfix was deployed to increase database query consistency levels so clients always get the most current data from the database, and data inconsistencies were repaired.

Next Steps:

We are searching out other queries which may benefit from increasing database query consistency levels, and we're developing a less disruptive procedure for modifying database replication settings.

[UPDATED:  Jul 8, 2019 8:11 AM] -  In addition to the issues experienced on NA11 and NA21 on July 3, the EU11 and EU21 Production environments experienced the same service availability issues on July 4 from approximately 10:00 AM to 6:00 PM CST.  The SpringCM Technology Team isolated and repaired the service availability issue in that time.

[RESOLVED:  Jul 3, 2019 7:06 PM]  The SpringCM Technology Team has isolated and repaired the service availability issue on the NA11 and NA21 instances.  The problem began at approximately 12:20 PM CST and was resolved by 6:06 PMCST.  We apologize for the inconvenience and will be providing a root cause analysis of the issue in a near future.

[UPDATE: Jul 3, 2019 3:52:00 PM] - The SpringCM Technology Team continues to work on isolating the service availability issue on the NA11 and NA21 instances. Please check the status of trust.springcm.com for updates regarding this issue.

[POSTED: Jul 3, 2019 12:33:50 PM]The SpringCM Technology team is working to isolate a service availability issue, impacting the Document Generation application on the NA11/NA21 instances. During this time customers may experience inability to generate new documents via the Document Generation feature. Please check the status of trust.springcm.com frequently for updates regarding this issue.

Subscribe to Email Updates

Subscribe by email to get notified as soon as we post a status update.