[RESOLVED] NA11 | Intermittent Latency PROD & UAT

 POSTED: Jun 15, 2021 3:03:08 PM        NA11, Latency, Intermittent

[ROOT CAUSE ANALYSIS]

Description:

Intermittent Failures Accessing CLM NA11 Services

Timeframe:

18:24 UTC 6/15/21 to 14:57 UTC 6/17/21

Impact:

Customers in CLM NA11 experienced intermittent failures accessing the system.

Cause:

Upstream ISP provider experienced multiple DDoS events, impacting network operation.

Resolution:

ISP provider implemented additional defenses to mitigate current and future DDoS events. ISP will continue to monitor for future DDoS events and mitigate them.

Next Steps:

CLM will re-evaluate their upstream provider’s services. Additionally, CLM will implement new services and systems that will improve reliability, stability, and security.


[RESOLVED June 23, 2021, 09:45 CDT] The DocuSign CLM Technology team has declared this incident resolved.  The last detected latency related to this incident occurred on 6/17/2021.  The DocuSign Technology team will provide an official Root Cause Analysis of this incident within 48 hours.  Thank you for your patience and we apologize for the inconvenience.

[UPDATED June 22, 2021, 09:45 CDT] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the day until resolution is declared.

[UPDATED June 21, 2021, 10:00 CDT] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the day until resolution is declared.

[UPDATED June 18, 2021, 17:20 CDT] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the weekend until resolution is declared.

[UPDATED June 18, 2021, 08:10 CDT] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the day until resolution is declared.

[UPDATED June 17, 2021, 17:00 CDT] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the evening until resolution is declared.

[UPDATED June 17, 2021, 08:30 CDT] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the day until resolution is declared.

[UPDATED June 16, 2021, 16:30 CDT ] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the evening until resolution is declared.

[UPDATED June 16, 2021 13:20 CDT] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the day until resolution is declared.

[UPDATED June 16, 2021 06:40 CDT] The DocuSign CLM Technology team has seen some improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the morning until resolution is declared.

Please check the status of trust.springcm.com for updates regarding this issue.

[UPDATED June 15, 2021 20:15 CDT] The DocuSign CLM Technology team has seen an improvement in the performance issue but cannot declare resolved until our 3rd party vendor has given clearance.  The team will continue to monitor the incident through the night until resolution is declared.

Please check the status of trust.springcm.com for updates regarding this issue.

[UPDATED June 15, 2021 16:30 CDT] The DocuSign CLM Technology team has isolated the performance issue and is working with our 3rd party vendor on which the incident it occurring.  During this time customers with accounts on our NA11 Production and UAT instances may experience intermittent latency issues as well as the potential for encountering page errors while navigating the site.  

Please check the status of trust.springcm.com frequently for updates regarding this issue.

[POSTED Jun 15, 2021 15:03 CDT] The DocuSign CLM Technology team is working to isolate a performance degradation issue, on the application tier of the NA11 instance.

Continue reading...

DocuSign CLM June 21.5 Product Release: PROD 06/25/2021

 POSTED: Jun 11, 2021 10:00:00 AM        NA21, NA11, Product Release, EU11, EU21, US11, US12

The DocuSign CLM and CLM.CM June 21.5 Product Release will be deployed to the EU11, EU21 Production environments on Friday, June 25, 2021 between 1:30 PM and 5:30 PM, U.S Central Time, and to the NA11, NA21, US11, US12 Production environments between 9:00 PM and Midnight, US Central Time. We do not anticipate any impact to platform availability or access during this time.

Continue reading...

DocuSign CLM May 21.4 Product Release: PROD 05/14/21

 POSTED: Apr 30, 2021 10:00:00 AM        Production, NA21, NA11, Product Release, EU11, EU21, Zero Downtime, US11, US12, 21.4

The DocuSign CLM and CLM.CM May 21.4 Product Release will be deployed to the EU11, EU21 Production environments on Friday, May 14, 2021, between 1:30 PM and 5:30 PM, U.S Central Time, and to the NA11, NA21, US11, US12 Production environments between 9:00 PM and Midnight, US Central Time. We do not anticipate any impact to platform availability or access during this time.

Continue reading...

DocuSign CLM February 21.2 Product Release: PROD - 2/26/2021

 POSTED: Feb 12, 2021 4:00:00 PM        Production Environment, NA21, NA11, Product Release, EU11, EU21, Zero Downtime, US11, US12, 21.2

The Docusign CLM and CLM.CM February 21.2 Product Release will be deployed to the EU11, EU21 Production environments on Friday, February 26, 2021 from 1:30PM - 5:30PM, U.S Central Time, and to the NA11, NA21, US11, US12 Production environments between 9:00 PM and Midnight, US Central Time.

Continue reading...

[RESOLVED] NA11 | UAT Workflow Step Timeouts

 POSTED: Jan 7, 2021 4:45:10 PM        Workflow, NA11, Latency

[ROOT CAUSE ANALYSIS]

Timeframe:

2021-01-07 01:24:42.129 CT
2021-01-07 17:44:54.074 CT

Impact:

PROD NA11 customers may have experienced workflow step failures.

Cause:

After performing routine maintenance, a service that processes workflow events was left disabled. As a result of the unprocessed workflow events, the workflow steps timeout was reached and the step failed.

Resolution:

The service processing workflow events was re-enabled.

Next Steps:

Investigation into why automated enablement of this service failed. The priority of these alerts will be raised significantly.

 

[UPDATE: Jan 11, 2021 15:10 CT] The DocuSign / SpringCM Technology teams continue to investigate the Root Cause Analysis of this issue.  Regrettably, we are unable to offer a full RCA at this time.  We anticipate being able to offer a full RCA within the next 48 hours.

[RESOLVED: Jan 7, 2021 20:02 CT] The DocuSign / SpringCM Technology teams have resolved the performance degradation issue on the Production and UAT NA11 instances.  

The team will begin identifying the cause of the issue and will post a Root Cause Analysis on this trust post within 48 hours.  We apologize for the impact this issue has caused customers on the NA11 Production and UAT instances. 

Please contact Technical Support if you have any questions or concerns regarding impacted workflows.

[UPDATED: Jan 7, 2021 18:00 CT] The DocuSign/SpringCM Technology teams have isolated the performance degradation issue on the Production and UAT NA11 instances.  They are currently working to resolve the issue.

Please check the status of trust.springcm.com frequently for updates regarding this issue.

[POSTED: Jan 7, 2021 16:45 CT] The DocuSign/SpringCM Technology teams are working to fix a performance degradation issue on the application tier on the NA11 (Production and UAT) instances. During this time, customers may experience latency in certain workflow steps, some resulting in timeouts.

Continue reading...

[RESOLVED] NA11 | NA21 | EU11 | EU21 - SSO Authentication Incident

 POSTED: Dec 15, 2020 9:29:32 AM        NA21, NA11, EU11, EU21, SSO Cert

[ROOT CAUSE ANALYSIS Dec 17 08:15 CT]

Timeframe:

Start time: 19:50 CST 14th December 2020

End Time: 19:15 CST 15th December 2020 

Impact:

Many PROD NA11, NA21, EU11 & EU21 customers would have encountered errors related specifically to SSO where SSO authentication was leveraged. Features reported to be impacted were SSO Authentication, API Authorization Calls, Desktop Applications and Office Online.

Cause:

While making an infrastructure change we did not adequately test on certificate attributes and their compatibility with various SAML providers. We did not immediately correlate the alarms that were firing to the change that was made because we receive similar alarms that are caused by vulnerability testing

Resolution:

New SSO Certificates were re-provisioned and retested.

Next Steps:

More targeted alerts specific to issues with SSO certificates are being instrumented.

The scope of testing will be expanded to include the full range of functionality that interfaces with SSO certificates.

 

[RESOLVED Dec 15 21:35 CT] The DocuSign CLM / SpringCM Technology teams have resolved the incident.  Please see the Trust Post: SSO Certificate Upgrade for details should you still be receiving SSO errors.

We apologize for the impact of this incident and its impact to customers today.

A complete root cause analysis with exact time frames will be provided via this Trust Post within 48 hours.  

[UPDATED Dec 15, 2020 11:50 AM CT]  The DocuSign CLM/SpringCM Technology teams have identified the source of the incident and are working to fully resolve this issue.  At present time issues with API are fully resolved, as are SSO Authentication and Desktop Applications, except in those circumstances where customers have not been able to trust the new SSO Certificate in accordance with the Planned SSO Certificate Upgrade. Unfortunately, issues with Office Online still persist and will not be resolved until the Technology Team is able to provide new SSO Certificates.

The CLM Technical Support team will make another Trust Post later today which will include  instructions regarding the timing and release of a second round of updated SSO Certs, which customers will need to handle similarly to the first.  We regret this inconvenience and apologize for the impact this will cause.  

A complete root cause analysis will be provided via this Trust Post within 48 hours.

[UPDATED Dec 15, 2020 10:55 AM CT] The DocuSign CLM/SpringCM Technology teams continue to work on resolving the service availability issue on Production instances.  At present time we no longer see issues with API errors, however, customers may still experience inability to authenticate via SSO and Desktop Applications (Edit, Office Online).  Please check the status of trust.springcm.com frequently for updates regarding this issue.

Continue reading...

NA11 Scheduled Maintenance Beginning Saturday, 12/12/2020

 POSTED: Dec 4, 2020 5:00:00 PM        Maintenance, NA11

The DocuSign CLM/SpringCM Technology Team will be conducting scheduled maintenance beginning Saturday, December 12th at 22:00 - 02:00 CT Dec 12 (04:00 - 08:00 GMT, Dec 12).

Continue reading...

NA11 Scheduled Maintenance Beginning Saturday, 11/21/2020

 POSTED: Nov 16, 2020 5:00:00 PM        Downtime, NA11, Scheduled Maintenance

The DocuSign CLM/SpringCM Technology Team will be conducting scheduled maintenance beginning Saturday, November 21st at 22:00 - 03:00 CT Nov 22 (04:00 - 09:00 GMT, Nov 22).

During this time customers on the NA11 instance may experience moments of brief inaccessibility to their account.

We apologize for any inconvenience during this time.

Please reach out to DocuSign CLM Technical Support for any questions or concerns and check back here for updates regarding the schedule.

Continue reading...

DocuSign CLM October 20.8 Product Release: PROD - 10/23/2020

 POSTED: Oct 14, 2020 5:00:00 PM        Production Environment, NA21, NA11, Product Release, EU11, EU21, 20.8, US11, US12

The DocuSign CLM and CLM.CM October 20.8 Product Release will be deployed to the NA11, NA21, EU11, EU21, US11, US12 Production environments on Friday, October 23, 2020 between 9:00 PM and Midnight, US Central Time.

We do not anticipate any impact to platform availability during this time.

Release Notes for this upcoming release can be found on the 
DocuSign Support Center.

Please contact 
Technical Support if you have any questions.

Continue reading...

[RESOLVED] NA11 | Performance Degradation Issue

 POSTED: May 21, 2020 4:52:58 PM        Performance, NA11, CLM.CM, CLM.DS

[RESOLVED] The  DocuSign CLM/SpringCM Technology Team has resolved a performance degradation issue on the NA11 instance. The problem began at 18:24 UTC and was resolved as of 19:44 UTC.  During this window customers on NA11 may have experienced delayed workflow starts, and latency in Doc Launcher and Document Preview.  

Continue reading...

Subscribe to Email Updates

Subscribe by email to get notified as soon as we post a status update.