October 2022 Outage Information
On Monday evening, October 24, 2022, Concord Broadband experienced an outage for 63 confirmed customers lasting in some cases up through 4:35pm on Tuesday, October 25, 2022. This is inconsistent with the service we strive to provide to our customers, and as such we have compiled the following comprehensive report that outlines what happened, what staff and contractors did to resolve the problem, and how we can improve afterward.
Because the report is long and highly technical, we have assembled the following summary, but please see the full report should you have questions after reading the summary.
On Thursday night, October 20th, a crew from a third-party vendor was on-site in Concord to replace of a cable damaged by rodents last year. When this cable was cut, we now know it triggered a software glitch in the equipment used to provide internet service to customers.
The glitch manifested in several different ways, affecting areas of the network not in the vicinity of the Laws Brook repair work. The impacted customers were returned to service quickly, and the manufacturer of the equipment began troubleshooting. This work continued all weekend by multiple Concord Broadband staff members.
By Monday, a plan was in place to attempt to remedy the situation when suddenly a number of customers lost connectivity after no intervention or actions on the part of staff or our vendor. Because this was not a disruption due to customer hardware failures or a break in the fiber, Concord Broadband was unable to determine the precise number of customers impacted. Based on total bandwidth in use, we knew the total did not exceed 10% of customers (~160).
Concord Broadband staff worked with our vendor non-stop to determine the cause and an appropriate response. The vendor was reluctant to point to the fiber repairs as the root cause, and absent a reason, they had to escalate the case until developers and engineers were involved.
By 4pm on Tuesday, there was enough data to suggest which physical device had suffered a software failure, and the decision was made to reboot it. There had been a reluctance to reboot it earlier without more data because an ill-advised reboot could have widened the outage.
After the reboot, the issue was immediately resolved. No configuration changes were necessary. Any temporary changes made for a small number of customers during the investigation phase were reversed. Concord Broadband staff worked with customers who had issues after the resolution, most of them requiring a reboot of their ONTs.
While Concord Broadband believes the response and communication were much improved from the last unplanned outage, communication – in particular once the helpdesk vendor implemented a recorded phone message – could have been better. A total of 3 News and Notices posts and 10 Tweets were issued during and after the outage, and all customers who provided an email address were kept up-to-date during restoration efforts.
Thank you to all the staff who spent countless hours in service of the team and our customers.
Should you have any questions after reading the full report, please reach out to us at firstname.lastname@example.org. Thank you for your continued support of this community resource.
No maintenance is scheduled at this time.