[ROOT CAUSE ANALYSIS]
Timeframe: |
2021-01-07 01:24:42.129 CT |
Impact: |
PROD NA11 customers may have experienced workflow step failures. |
Cause: |
After performing routine maintenance, a service that processes workflow events was left disabled. As a result of the unprocessed workflow events, the workflow steps timeout was reached and the step failed. |
Resolution: |
The service processing workflow events was re-enabled. |
Next Steps: |
Investigation into why automated enablement of this service failed. The priority of these alerts will be raised significantly. |
[UPDATE: Jan 11, 2021 15:10 CT] The DocuSign / SpringCM Technology teams continue to investigate the Root Cause Analysis of this issue. Regrettably, we are unable to offer a full RCA at this time. We anticipate being able to offer a full RCA within the next 48 hours.
[RESOLVED: Jan 7, 2021 20:02 CT] The DocuSign / SpringCM Technology teams have resolved the performance degradation issue on the Production and UAT NA11 instances.
The team will begin identifying the cause of the issue and will post a Root Cause Analysis on this trust post within 48 hours. We apologize for the impact this issue has caused customers on the NA11 Production and UAT instances.
Please contact Technical Support if you have any questions or concerns regarding impacted workflows.
[UPDATED: Jan 7, 2021 18:00 CT] The DocuSign/SpringCM Technology teams have isolated the performance degradation issue on the Production and UAT NA11 instances. They are currently working to resolve the issue.
Please check the status of trust.springcm.com frequently for updates regarding this issue.
[POSTED: Jan 7, 2021 16:45 CT] The DocuSign/SpringCM Technology teams are working to fix a performance degradation issue on the application tier on the NA11 (Production and UAT) instances. During this time, customers may experience latency in certain workflow steps, some resulting in timeouts.