SpringCM Production accounts will be unavailable Friday, November 1 from 9pm to Midnight CDT for the 19.8 Product release. We apologize for any inconvenience.
NA11 | NA21 | EU11 | EU21 Production Unavailable During 19.8 Product Release
POSTED: Oct 23, 2019 12:17:59 PM
Downtime,
NA21,
NA11,
Product Release,
EU11,
EU21
[RESOLVED] NA21 | Service Degradation
POSTED: Oct 14, 2019 11:25:43 AM
Production,
NA21,
Service Outage
[ROOT CAUSE ANALYSIS]
|
Description: |
NA21 Environment Performance Degredation |
|
Timeframe: |
2019-10-14 , 08:22AM CT to 08:54AM CT |
|
Impact: |
Customers experienced slowness or timeout while navigating SpringCM |
|
Cause: |
An unexpected database snapshot was generated for multiple publications which caused a degradation in database performance. |
|
Resolution: |
Once the snapshot generation finished performance returned to normal. We have a support case open with the hosting vendor for investigation of the cause of the snapshot generation. |
[RESOLVED] The SpringCM Operations team detected and resolved a service availability issue on the NA21 instance starting at 10:22AM - and ending at 10:54AM CT. During this time, customers may have experienced latency, an inability to access resources, or received an error when loading pages.
NA11 | NA 21 | EU 11 | EU 21 Production Unavailable During 19.8 Product Release
POSTED: Oct 7, 2019 3:29:42 PM
Downtime,
NA21,
NA11,
Product Release,
EU11,
EU21
Please be advised that maintenance has been scheduled for the NA11 and NA21 Production Environments for 9pm CDT on Friday October 11 to 12AM CDT on Monday October 14. Additionally, maintenance has been scheduled for the EU11 and EU21 Production Environments for 21:00 UTC on Friday, October 11 to 24:00 UTC on Monday October 13. During the maintenance window customers should not expect reports to fail but will likely return out of date/incomplete data.
[RESOLVED] NA21 | Service Degradation
POSTED: Aug 30, 2019 1:39:57 PM
NA21,
Service Degradation
[ROOT CAUSE ANALYSIS Sept, 24, 2019]
|
Description: |
Spring - UI Session Management Intermittent Connection Failures/Outage |
|
Timeframe: |
2019-08-30 , 13:00CT to 15:30CT |
|
Impact: |
About 10% of (SpringCM) users whose accounts are primary to the NA21 instance of SpringCM got errors trying to login and active sessions got redirected to the login page or received a gateway time out. |
|
Cause: |
During a recent release we changed session management platforms. There was an increase in requests to the new platform and unanticipated traffic patterns |
|
Resolution: |
Modified the maximum number of clients that can be handled simultaneously. Rolled back implementation to the previous version. |
[RESOLVED] The SpringCM Technology team identified and resolved the service availability issue on the NA21 instance on Friday, August 30 at approximately 15:30 CST. The outage lasted approximately from 13:00 CST to 15:30 CST (2.5 hours). We apologize for the inconvenience and will be providing a Root Cause Analysis in the near future.
NA11 | NA21 | EU11 | EU21 September 19.7 Product Release
POSTED: Aug 27, 2019 4:33:33 PM
NA21,
NA11,
Product Release,
EU11,
EU21,
Zero Downtime
[UPDATE] NA11 | NA21 | EU11 | EU21 - Production August 19.6 Product Release
POSTED: Jul 22, 2019 3:30:16 PM
NA21,
NA11,
Product Release,
EU11,
EU21
[UPDATE July 25, 2019] - The Production environment will be unavailable during this product release, from Friday, August 9, 9PM - Midnight US Central Time. We inadvertently reported in the original trust post that we did not anticipate any impact to availability. We apologize for any inconvenience.
[RESOLVED] NA11 | NA21 | EU11 | EU21 - Document Generation Performance
POSTED: Jul 3, 2019 12:33:50 PM
NA21,
NA11,
Document Creation
Root Cause Analysis:
|
Description: |
Data Inconsistency in NA11, NA21, EU11, EU21 |
|
Timeframe: |
2019-07-03 12:00PM CDT to 2019-07-04 5:30PM CDT |
|
Impact: |
Customers may have experienced inability to generate new documents via the Document Generation feature. |
|
Cause: |
A change to distributed database replication settings caused some database replicas not to have the data they needed. |
|
Resolution: |
A hotfix was deployed to increase database query consistency levels so clients always get the most current data from the database, and data inconsistencies were repaired. |
|
Next Steps: |
We are searching out other queries which may benefit from increasing database query consistency levels, and we're developing a less disruptive procedure for modifying database replication settings. |
[UPDATED: Jul 8, 2019 8:11 AM] - In addition to the issues experienced on NA11 and NA21 on July 3, the EU11 and EU21 Production environments experienced the same service availability issues on July 4 from approximately 10:00 AM to 6:00 PM CST. The SpringCM Technology Team isolated and repaired the service availability issue in that time.
[RESOLVED: Jul 3, 2019 7:06 PM] The SpringCM Technology Team has isolated and repaired the service availability issue on the NA11 and NA21 instances. The problem began at approximately 12:20 PM CST and was resolved by 6:06 PMCST. We apologize for the inconvenience and will be providing a root cause analysis of the issue in a near future.
[UPDATE: Jul 3, 2019 3:52:00 PM] - The SpringCM Technology Team continues to work on isolating the service availability issue on the NA11 and NA21 instances. Please check the status of trust.springcm.com for updates regarding this issue.
NA11 | NA21 | EU11 | EU21 June 19.5 Product Release
POSTED: Jun 6, 2019 11:25:07 AM
Production Environment,
NA21,
NA11,
Product Release,
EU11,
EU21
SpringCM's June 2019 Product Release will be deployed to the Production environments on Friday, June 28th 2019 between 9PM and Midnight, US Central Time.
[RESOLVED] NA 21 | Workflow Latency
POSTED: May 21, 2019 11:18:14 AM
NA21
Root Cause Analysis:
|
Description: |
NA21 Workflow Latency |
|
Timeframe: |
2019-05-21 09:50:00 CST to 2019-05-21 14:05:00 CST |
|
Impact: |
Some customers may have experienced some latency in workflow execution in the NA21 datacenter. All workflows completed successfully, however there was a degradation in execution time. |
|
Cause: |
A workflow condition was encountered that resulted in intermittent performance degradation. |
|
Resolution: |
The affected instances were aborted which resolved the immediate issue. Hotfix patches will be applied so that the same conditions will not cause blocking going forward. |
[RESOLVED: MAY 21, 2019 02:10 PM CST] The SpringCM Technology Team has resolved the performance degradation issue on the NA21 instance. The problem began at 09:50 AM and was resolved by 2:05 PM CST. We apologize for any inconvenience or impact to your teams. Given the close proximity of this issue with yesterday's similar issue on NA21, we'll be on conducting an in depth root analysis to prevent this kind of occurrence from happening again.
