Post Mortem: ConnectWise Cloud Outage, 21 February, 2018

  By: Paul MacNeill   |   Wise-Sync   |   Feb 23, 2018

On 21 February 2018 at 8:35 am (AEDT), ConnectWise API experienced an outage. As a result of this outage, our partners experienced 23 hours 20 minutes of downtime of services as we worked with ConnectWise to restore full service. We would like apologize to our partners for the impact to your daily operations as a result of this outage. 

Here's what we know:

  • On 21 February, at 8:45 am (AEDT), the first error, "500 - Internal Server Error", was reported to Wise-Sync support, followed by multiple partners reporting the same error.
  • Tickets were immediately escalated to our Senior Product team, who were able to investigate and identify that the issue was only affecting ConnectWise Cloud users.
  • At 9:07 am (AEDT) we logged a ConnectWise Platform Ticket for the error, 'Increasing 500 Internal Server Errors'.
  • Two hours later, we still had not received an update from the ConnectWise Platform team. 
  • At 11:44am (AEDT) a further ticket was logged with ConnectWise Support requesting emergency escalation.
  • Throughout the day (evening and over night), the Wise-Sync team continued to follow up with ConnectWise requesting an update, for which we were only advised 'The Cloud On-Call Team are working on it'.  
  • On 22 February, at 6:27 am (AEDT)  ConnectWise confirmed that a fix was deployed to AU, EU and Staging Cloud platforms. NA Cloud was scheduled to be released at 1 pm (AEDT) which was 9pm ET.
  • An interim work-around was provided for NA partners where a specific Endpoint could be used, which was deployed to all NA cloud partners.
  • Shortly after, Wise-Sync resolved critical overrun of logging server affecting all user login as a result of log bloat due to extended outage.
  • At 7:00 am (AEDT) on 23 February, ConnectWise Support confirmed that the patches were now fully deployed and the issue now considered resolved.

A full detailed report can be viewed here.

We have raised our concerns with ConnectWise and are working with them to achieve better communication channels and monitoring of progress during such outages; we recognise the importance of services such as ours and how they can be impacted by upstream providers. Ensuing clarity of communication throughout outages is for us, paramount. At Wise-Sync we strive to provide the best possible service and will continue working diligently with our integration partners  to provide our partners with a stable and efficient syncing and payment tool.

To all those thank you for your patience and continued support of Wise-Sync and Wise-Pay. If you have any questions or need assistance, please reach out to our team on support@wise-sync.com.

 

SUBSCRIBE

Recent posts

Request a Demo
CRN Impact Award Winners Announced - Congratulations Wise-Sync Partners
  By:- Paul MacNeill   |   Wise-Sync   |   May 7, 2018

This year's CRN Impact Awards winners were announced at the recent CRN Impact Awards Gala Dinner on the 6th of April, 2018.  We just wanted to..

Wise-Sync and Wise-Pay Planned Outage: 5th March, 2018 from 05:00am - 08:00am (AEDT)
  By:- Paul MacNeill   |   Wise-Sync   |   Mar 3, 2018

On the 5th of March from 0500 - 0800am Wise-Sync and Wise-Pay will be offline for a planned data migration. During this time no Wise-Sync or..

Post Mortem: ConnectWise Cloud Outage, 21 February, 2018
  By:- Paul MacNeill   |   Wise-Sync   |   Feb 23, 2018

On 21 February 2018 at 8:35 am (AEDT), ConnectWise API experienced an outage. As a result of this outage, our partners experienced 23 hours 20..