A disruption to 911 services across Southeast Nebraska in early September began when a third-party contractor installing a natural gas line at Windstream’s data center near Centennial Mall cut a city of Lincoln irrigation line.
The damage to the sprinkler apparently was not reported when it occurred sometime around Aug. 28, Brad Hedrick, Windstream’s regional president of operations for Nebraska, told the Public Service Commission on Thursday.
But the incident set off a chain of events that eventually led to 911 working either sporadically or not at all for several hours between Sept. 2 and 3, as 18 of 20 public safety answering points were affected, Hedrick told commissioners the incident.
That outage, as well as a shorter disruption to 911 services for Windstream customers that occurred Nov. 28, were part of an investigation and public hearing initiated by the commission this fall.
People are also reading…
According to Hedrick, the September incident began after water found its way into the concrete vault housing Windstream’s electrical systems at 1440 M St. and onto a copper busbar, causing a short.
“That’s what caused the fire and the explosion†that damaged both a Lincoln Electric Systems transformer as well as Windstream’s power equipment, Hedrick told commissioners during the three-hour hearing.
No one was injured in either the fire or the “small†explosion, which happened at about 11:13 p.m. on Sept. 1, and while Lincoln Fire and Rescue were responding, a backup generator kicked in to keep telecommunication services running without interruption.
The generator continued to provide enough electricity to keep three switches housed in the facility operating for about 13 hours until a loose thermocouple wire erroneously signaled the generator was overheating and triggered an automatic shutdown at about 12:23 p.m. on Sept. 2, Hedrick said.
According to Hedrick’s testimony, the generator was last inspected in 2018, but the loose thermocouple wouldn’t have been discovered during routine maintenance activities. The generator, which has operated for about 700 of its 10,000-hour life, has since been fixed, he said.
As Windstream sought to bring a portable generator capable of generating a megawatt of electricity to Lincoln in the meantime — the closest generator the company located was in Des Moines, Iowa — batteries continued to power the switches until they began to reach “critically low levels.â€
Hedrick said top officials within Windstream, including the senior vice president of operations and vice president for network engineering, made the decision to shut down one of the three switches in an effort to preserve electricity to the other two at about 6 p.m. on Sept. 2.
Shutting down one of the switches also helped avoid an equipment crash, which Hedrick said could have “resulted in a catastrophic" failure that would have extended the duration of the outage.
“I can’t fault their decision-making process,†he said.
Before taking the switch offline, Hedrick added, Windstream gave advance notice to the local public safety access points, the Public Service Commission, and Gov. Jim Pillen’s office that a disruption to 911 services was possible, adding it was unsure of how extensive the outage might be.
The switch shutdown was completed by about 10 p.m. that evening, Hedrick said. Power was restored to the building at about 1:40 a.m. the next morning, Sept. 3, with communication services starting to come online about an hour later.
The service interruption affecting 911, local and long-distance calling lasted about 4½ hours total, Hedrick said, and affected more than 13,000 customers, mostly those who had transitioned to a new switch without a redundancy in place as the state seeks to update its 911 systems.
A second outage Nov. 28 originated from instability in Windstream’s core network — what Hedrick described as “a momentary blip†— which created a “ripple effect†across its system.
Two switches at the Lincoln facility required a restart after the instability reached them, but a software bug prevented them from doing so automatically, he added, resulting in 911 calls and other traffic coming through the switch to fail.
The November disruption lasted 1 hour and 10 minutes, according to Windstream’s report.
An alarm on the system designed to notify the company of the outage also did not trigger, which led to information about the outage being unavailable until the problem was resolved several weeks later.
Hedrick told the commission Windstream has taken several steps to rectify both issues.
The Arkansas-based company has identified two portable generators in Lincoln capable of serving its needs should its on-site generator fail, he said.
Windstream is also planning to replace infrastructure that would allow a portable generator to hook into its electrical system with greater ease.
Hedrick also said Windstream is completing work to create a redundancy for public safety access points that did not have one previously during a transfer to the new system in order to avoid future outages.
To address the second disruption, Hedrick said the alarming system has been restored, and that Windstream was exploring options to replace the equipment with the bug that prevented the automatic restart from taking place.
While members of the commission were sympathetic to the series of events leading to the disruption of 911 service in September, they also said the decision to locate two switches inside a single facility created the environment for that outage to occur.
“It’s starting to come together for me clearer and clearer,†Commissioner Dan Watermeier of Syracuse said. “If those (switches) had been separated, it might have taken care of quite a bit of the redundancy issue there.â€
Hedrick said he agreed with Watermeier’s point that “geographic diversity would be better†in choosing where it located equipment meant to provide redundancies in its 911 system, but also signaled that "hindsight is 20/20."
Commissioner Kevin Stocker of Scottsbluff asked if there were plans to locate equipment above places where water could leak onto it, while Commissioner Eric Kamler of Geneva asked if Windstream could say “with any degree of confidence†whether the issue had been solved.
Hedrick said the work to waterproof the vault where the electric control system is housed has been completed, and said Windstream believed there was a “decreased probability of a recurrence†with the steps it had taken.
The commission took no action Thursday. Watermeier said the commission would accept written comments from the public through Jan. 12.
Chris Dunker's most memorable stories from 2023
Chronicling the biggest issue of the 2023 legislative session; the impact of a new scholarship program; and hunting for bullfrog tadpoles were among Chris Dunker's five favorite stories this year.
A new law prohibiting transgender youth from accessing certain medical care left Nola Rhea and her mother, Heather, to consider leaving Nebras…
The Nebraska Promise has helped 18,000 low-income students attend the University of Nebraska tuition-free since it began three years ago.
The final debate over a bill banning certain health care procedures for transgender youth and abortions after about 12 weeks encapsulated the …
A month after Rodney Bennett started at the University of Nebraska-Lincoln, the 21st chancellor embarked on a road trip to begin learning more…
In a pond downstream from AltEn, where pesticide-coated seeds were turned into ethanol, UNL students seek out tadpoles to learn how much damag…