It was necessary for the DocuSign CLM/SpringCM Technology teams to update the certificate used to sign outgoing SAML requests. This is scheduled to take place between 19:00 and 20:00 CT 12/15/2020. The short timeframe between this notice and the update was necessary given the incident documented in the Trust Post earlier today.
[RESOLVED] NA11 | NA21 | EU11 | EU21 - SSO Authentication Incident
POSTED: Dec 15, 2020 9:29:32 AM
NA21,
NA11,
EU11,
EU21,
SSO Cert
[ROOT CAUSE ANALYSIS Dec 17 08:15 CT]
|
Timeframe: |
Start time: 19:50 CST 14th December 2020 End Time: 19:15 CST 15th December 2020 |
|
Impact: |
Many PROD NA11, NA21, EU11 & EU21 customers would have encountered errors related specifically to SSO where SSO authentication was leveraged. Features reported to be impacted were SSO Authentication, API Authorization Calls, Desktop Applications and Office Online. |
|
Cause: |
While making an infrastructure change we did not adequately test on certificate attributes and their compatibility with various SAML providers. We did not immediately correlate the alarms that were firing to the change that was made because we receive similar alarms that are caused by vulnerability testing |
|
Resolution: |
New SSO Certificates were re-provisioned and retested. |
|
Next Steps: |
More targeted alerts specific to issues with SSO certificates are being instrumented. The scope of testing will be expanded to include the full range of functionality that interfaces with SSO certificates. |
[RESOLVED Dec 15 21:35 CT] The DocuSign CLM / SpringCM Technology teams have resolved the incident. Please see the Trust Post: SSO Certificate Upgrade for details should you still be receiving SSO errors.
We apologize for the impact of this incident and its impact to customers today.
A complete root cause analysis with exact time frames will be provided via this Trust Post within 48 hours.
[UPDATED Dec 15, 2020 11:50 AM CT] The DocuSign CLM/SpringCM Technology teams have identified the source of the incident and are working to fully resolve this issue. At present time issues with API are fully resolved, as are SSO Authentication and Desktop Applications, except in those circumstances where customers have not been able to trust the new SSO Certificate in accordance with the Planned SSO Certificate Upgrade. Unfortunately, issues with Office Online still persist and will not be resolved until the Technology Team is able to provide new SSO Certificates.
The CLM Technical Support team will make another Trust Post later today which will include instructions regarding the timing and release of a second round of updated SSO Certs, which customers will need to handle similarly to the first. We regret this inconvenience and apologize for the impact this will cause.
A complete root cause analysis will be provided via this Trust Post within 48 hours.
[UPDATED Dec 15, 2020 10:55 AM CT] The DocuSign CLM/SpringCM Technology teams continue to work on resolving the service availability issue on Production instances. At present time we no longer see issues with API errors, however, customers may still experience inability to authenticate via SSO and Desktop Applications (Edit, Office Online). Please check the status of trust.springcm.com frequently for updates regarding this issue.
