Fourteen years ago, a software bug in the alarm system of a FirstEnergy Corp. control room led to the most widespread power outage in the United States. Over 55 million people in both the U.S. and Canada lost power for almost two days in what became known as the Northeast Blackout of 2003.

On Wednesday, a panel titled “Energy Security and Resilience,” discussed how the energy industry and U.S. government are working to prevent accidents such as the 2003 blackout. The panel was part of the 2017 UT Energy Week.

Participants included Jimmy Glotfelty, executive vice president of Clean Line Energy Partners, Lt. General Ken Eickmann, a retired member of the U.S. Air Force and Michael Goin, IT Manager at Austin Energy.

Glotfelty said when the 2003 blackout first occurred, the FBI and Secret Service investigated the incident to see if it was a terrorist attack. 

“But it wasn’t (a terrorist attack),” Glotfelty said. “There was no physical intrusion. This was our own dumb fault. We did not have the right components and tools to ensure that our system was reliable.”

According to Glotfelty, the outage, which under secure conditions could have been easily managed, cascaded across U.S. electric grids before the voltage system stopped it from reaching the South.  

“The greatest thing about our grid was the fact that we had a voltage system that stopped the cascading outage,” Glotfelty said. “So to prevent (similar incidents) in the future, we need to think ahead, to gold-plate the system.”

Glotfelty said the electric system needs higher voltage lines and extra capacity so it can absorb events when they happen, preventing power outages.

During the panel, Glotfelty also emphasized the importance of redundancy, or adding extra parts in case of failure.

“The term security does not mean physical security or cyber security; it means redundancy,” Glotfelter said. “How do we protect our system in the event that the single most important element of the system goes down? That is the right thing to be talking about.”

Glotfelty added that while redundancy is expensive, it’s the most important thing to work on.

Another aspect of the panel discussion focused on cybersecurity issues and virtualization, or the process of moving information between computer servers. 

“Virtualization has changed the landscape of how we do computers,” Goin said. “We’re able to move things from a server that’s having a problem to another server that’s ready to stand up in its place.”

Although virtualization has been very efficient, Goin said hackers have access to more servers because they are all connected.

Like Glotfelty, Goin said that redundancy is key to fighting cybersecurity.

“If you’ve got five networks running across the town, why not run them twice?” Goin said. “And for God’s sake, don’t run them through the same category. One shovel takes down the whole system and you’ve got some explaining to do.”

The panelists agreed that because of their nature, cyber and energy security are impossible to completely eliminate or protect against. 

“(Security systems) are changing every day, every hour, and the people who are trying to hack into our security don’t care if and when they get caught because they’re not subject to our rules or laws,” Glotfelty said. 

Glotfelty said hackers have little to lose by trying to hack into U.S. cyber and energy security, which is dangerous for the U.S.

“We have to be perfect every time, but they only have to be perfect once,” Glotfelty said.