Louisiana Superdome by night (Photo credit: Wikipedia)

by Bob Shively, Enerdynamics President and Lead Instructor

When I worked for Pacific Gas and Electric Company (PG&E) in the late 1980s, the San Francisco area hosted a classic World Series in which the San Francisco Giants played against their cross-bay rivals the Oakland A’s.  Going into the Series, PG&E spent significant time and energy ensuring reliable power to both stadiums.  My friend and co-worker Bill (whose last name shall remain anonymous), was the account representative for the Giants’ stadium, Candlestick Park. Just before the series began, he was quoted in the company newspaper saying it was virtually impossible for the stadium to lose power.

Sadly, the Loma Prieta Earthquake struck just prior to Game 3, and power at the stadium immediately went out due to loss of transmission into the area.  In the following days as PG&E struggled to restore power throughout the area, we were all reminded that power supply is never 100% reliable no matter how much time and money is spent on engineering and infrastructure.

Flash forward 23 years, and half of New Orleans’ Superdome lost power just after halftime of the Super Bowl. Players were forced to wait on the sidelines for 35 minutes before power could be restored. As the root-cause investigations go forward, we are beginning to learn what caused the outage. According the local utility Entergy’s press release:

Entergy New Orleans, Inc. announced today that it has traced the cause of Sunday’s outage to an electrical relay device.

The device was specifically installed to protect the Mercedes-Benz Superdome equipment in the event of a cable failure between the switchgear and the stadium.

While the relay functioned without issue during a number of high-profile events – including the New Orleans Bowl, the New Orleans Saints-Carolina Panthers game, and the Sugar Bowl – during Sunday’s game, the relay device triggered, signaling a switch to open when it should not have, causing the partial outage.

This device has since been removed from service and new replacement equipment is being evaluated.[1]

So what is a relay, and how could it cause an outage?  A relay is a device that automatically signals a breaker to open when the relay senses an unsafe power condition such as too high a current, too high a voltage, reverse power flow, too high a frequency, or too low a frequency.  Breakers interrupt the flow of power to prevent potentially harmful problems from occurring in electrical systems. Relays are used to protect equipment within a premise and/or to protect the utility lines serving a premise.  Modern protective relays such as the one at the Superdome are microprocessor-based digital relays, meaning that software within the processor on the relay is used to determine when to open the breaker.

This raises the question of whether the protective relay opened when it shouldn’t have due to a manufacturing defect, incorrectly programmed software, an external hack of the software or whether there was indeed a power condition that resulted in the relay properly opening the breaker.

The answer to this question has not yet been definitively announced. But the manufacturer of the relay, S&C Electric Company, stated on Friday that the relay opened the breaker as “a result of the electric load current exceeding the trip setting for the switchgear relay as set by the system operators. Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power.” [2]

Assuming the relay functioned properly, what could have caused the outage? One answer is simply that the relay was programmed with a wrong tolerance that was set too low. Loads in the stadium were apparently well below the capacity of the circuits since the circuits are designed to carry heavy air conditioning loads in the summer. The Beyonce halftime show was apparently powered primarily by standby generators within the stadium.  But at least one unofficial comment has suggested that some such outside power sources may have been connected into the stadium electrical system.  We don’t know if this is true, but if so, this certainly could have caused a power condition that propagated itself back to the relay, in which case the relay would have properly opened based on what appeared to be a fault or other problem with the internal system. So a second explanation could be power quality issues occurring inside the stadium.  And although there has been nothing to suggest that this occurred, microprocessors do have the potential of being hacked if they are connected to the internet.  This means that a third explanation could be that someone accessed the software on the relay remotely and maliciously changed the settings. This seems unlikely, but not impossible.

We’ll have to wait for more analysis to find out why the relay acted.  In the meantime, we are reminded again that no power supply can ever be expected to have 100% reliability.