Keeping the Lights On

When a small thunderstorm cell that formed in central Iowa in June 2012 grew into a monster storm called a derecho, 22 people lost their lives, and residents from the Midwest to the East Coast suffered extensive property loss and power outages. The derecho, which gets its name from the Spanish word for “straight,” is distinguished from its cousin, the tornado, (Spanish for “twist”) because its storms move in a straight line. But those storms can still do considerable harm.

In this particular instance, storm winds as high as 87 miles per hour were reported across the Baltimore-Washington metro area. Throughout the region, problems were exacerbated because 911 call centers went down, making it impossible for residents to summon first responders. About 1 million customers lost power in Virginia, resulting in the largest outage in the state’s history not related to a hurricane. Approximately 1.6 million customers lost power in Maryland, and water restrictions went into effect in the state after the water facilities in some heavily populated suburban counties lost power. Disruptions to credit card verification services meant that even businesses with power could not conduct sales unless customers had cash.

The magnitude of the disruptions caused by the storm led federal, state, and local governments to reassess the resiliency of the power grid and telecommunications systems. Two reports, one from the state of Maryland and one from the Federal Communications Commission (FCC), provide a look at the disruptions caused by the storm and how the region can better weather the storms of the future.

Grid Resiliency

In July 2012, Maryland Governor Martin O’Malley signed an executive order directing state officials to gather data and talk with experts on improving the resiliency of the state’s electric distribution system. O’Malley cited the derecho as the impetus for the task force, but he also noted that erratic weather patterns and unexpected storms must be considered the norm going forward. To prepare for such an environment, O’Malley asked the task force to evaluate the feasibility of placing electric supply and distribution lines underground and to present other options to improve the resiliency of the electric grid.

As a working definition, resiliency was defined as the “ability of the distribution system to absorb stresses without experiencing a sustained outage,” of more than five minutes. However, the task force stressed that resiliency extends beyond the distribution of energy to the preparedness of the populace and the emergency response system.

The task force held eight roundtable discussions over 60 days, hearing testimony from 50 experts. State agencies and utilities were asked to share information on outages from the derecho and recent hurricanes and blizzards as well as staffing levels at the time of the storm.

In its final report, Weathering the Storm: Report of the Grid Resiliency Task Force, the task force offers 11 recommendations, ranging from policy improvements to disaster drills. The recommendations are designed to be taken as a whole rather than as a set of independent suggestions. As the report explains: “While some may call attention to certain of the task force’s recommendations, it is critical to understand that they work best as a unified strategy. The coordinated implementation of the recommendations is as important as the recommendations themselves; if rolled out in an ‘a la carte’ manner, they may not produce the expected results.”

In its recommendations, the task force addresses policy concerns, steps utilities must take to harden the grid, and planning for future disasters.

Policy

Among the topics addressed in the policy recommendations were agency differences in data collection on outages and the need for improving information sharing among state agencies.

When comparing the outage information from the three major state utilities, the task force found that each company collected data differently and used different terminology. The resulting reports could not be compared, and some information was simply unavailable. For example some utilities failed to keep information on outages across different types of systems such as overhead or underground lines. And other companies made no distinction among the load densities experienced in different areas, meaning that outages in rural areas were not distinguished from outages in urban areas. To solve this problem and have better information going forward, the task force recommended standardized reporting criteria.

Another policy recommendation involved special-needs customers. The task force found that utilities maintain lists of medically vulnerable customers so that these customers can be informed of planned outages and impending weather issues. However, the utilities have been reluctant to share these lists with emergency management officials due to privacy concerns. Because emergency management agencies could benefit from knowing which homes house vulnerable citizens, the task force recommended getting permission from these citizens to share their information.

The report also recommended enhanced information sharing among utilities and state and local governments. For example, the task force noted that smart meters, which allow two-way communication between individual electric meters and the utility, could provide data. Used primarily for billing, the meters also provide accurate information on outages. The data these meters collect could be shared with government agencies, obviating the need for agencies to keep duplicate records.

Hardening the grid

Beyond policy changes, two major actions recommended by the task force were to place vulnerable lines underground and to improve tree-trimming programs.

The information gathered after the derecho showed that the underground lines lost power much less frequently than overhead lines. However, placing electric lines underground is expensive, costing from five to 10 times what an overhead line costs. Therefore, the task force recommended that only the highest-risk lines should be moved underground.

The highest-risk lines are those feeding electricity to the substations; these were more frequently damaged as compared to regular distribution lines that transmit electricity from the substations to homes and businesses. While there are relatively few of these supply lines, the task force found that they accounted for almost 20 percent of the system outages. The task force noted that “a utility can prevent more customer outages per circuit mile by undergrounding substation lines than any other type of line.”

This was borne out by statistics on those lines that were already underground before the storm hit. For example, 100 percent of the underground substation supply lines remained operational during the storm. For above ground lines, the success rate ranged from 7 percent to 64 percent depending on the utility in question.

Another recommendation involves tree trimming. Such “vegetation management” issues involve several state and local agencies as well as individual homeowners. The expense and who should bear it can be an issue, and the choice of what to trim and how extensively can be controversial. The task force calls on state officials to align state and local laws, regulations, and ordinances. It also recommends studying property and contract law to determine how to help utilities better address the issue.

The task force included recommendations for how regulators or legislators might provide better financial tools that can help utilities undertake the expenditures necessary to improve resiliency. For example, one recommendation was for the state to issue regulations to allow utilities to treat these expenditures differently through a “tracker cost recovery mechanism.” Such a mechanism, implemented through regulation, would allow utilities to recoup losses more quickly.

Another recommendation was to pass laws or implement regulations that would allow utilities to align utility rates with steps taken to improve resiliency—in other words, to let utilities raise rates if they could demonstrate that they had implemented resiliency measures. Such a system would be dependent on gathering reliable metrics, warned the task force, but could be used to reward utilities that take steps to improve resiliency and penalize those that are slow to act.

Planning

The task force also had several recommendations on planning to help mitigate the damage that could be caused by future disasters. One recommendation was for utilities to conduct annual disaster drills with state and local emergency management agencies.

This recommendation came directly from the requests of the task force participants. The report says: “Throughout the roundtable discussions, the parties consistently emphasized the need for collaboration, information sharing, and relationship building between various parties, with special emphasis on that between the local emergency management agency and its utility.”

Other recommendations from the task force called for reports on various topics that could help fuel future action. One such recommendation was for a study to determine the cost of utility outages to different customer classes—industrial, commercial, institutional, and residential. The task force also calls on utilities to formulate long-term plans for dealing with the “graying” work force. Retiring workers, noted the task force, could compromise future resiliency.

The task force also recommended that the governor establish a group to study and report on how advances in technology could help improve resiliency, such as in the areas of communications and renewable energy.

It’s this final recommendation that most interests Dr. Eric D. Wachsman, director of the University of Maryland Energy Research Center and William L. Crentz Centennial Chair in Energy Research at the University of Maryland. Wachsman spoke to the task force about expanding the state’s use of microgrids.

A microgrid uses fuel cells to generate enough power for a business, an office park, or even a small neighborhood. There are many types of fuel cells, but the most common are powered by the chemical reaction between hydrogen and oxygen. The microgrid, which is connected to the power grid, can then draw energy from, or sell energy back to, the utility as necessary. Because the microgrid is connected to the power grid, utilities do know when power is not flowing to the microgrid.

However, microgrids have proven resilient during widespread power outages. For example, during Hurricane Sandy, the Food and Drug Administration’s research facility in White Oak, Maryland, a microgrid powered all the buildings on the campus for more than two days. “Microgrids would improve the resilience of the grid because those people have power and the utility doesn’t have to respond during an outage,” says Wachsman. “It’s a way to bring resiliency to the entire grid, and it’s a way to provide security, because it’s easy to protect.”

While the microgrid did not make it into the final recommendations, the issue was given serious attention and was part of a “major discussion,” according to Wachsman. One of the main obstacles facing the widespread implementation of microgrids is that fuel cells cost two to three times more than purchasing energy through utilities. However, it might be worth the cost for some businesses. “This is important if you are in a business where maintaining electric power is critical. A microgrid should be a capital investment,” says Wachsman.

Implementation

While the report asks that all recommendations be taken as a whole, it is perhaps unavoidable that they will be implemented in pieces, because as a practical matter, that’s probably the only way anything will get done, especially given that some recommendations require regulatory or legislative action.

Some of the participating utilities are “moving forward aggressively with implementation of the new tree-trimming mandates as well as the deployment of numerous smart grid initiatives including advanced metering and distribution automation technologies,” says George Owens. Owens, who is president of Downes Associates, Inc., a consulting engineering company specializing in electrical utilities, served on the task force. He had previously served as the chief investigator and witness for the Maryland Attorney General’s Office in an investigation of outages in 2010 and 2011. He notes that it will take several years to determine whether the measures undertaken by all of the utilities will prove sufficient to measurably improve the resilience of the state’s utilities.

Telecommunications

The derecho also played havoc with telecommunications systems in the region. Specifically, 911 call centers in six states lost service. Seventeen centers across three states lost service completely, meaning that more than 2 million people were unable to contact 911 at all. In response, the FCC issued a report in early January offering recommendations for improving resiliency. FCC Chairman Julius Genachowski has announced that the agency will issue regulations based on the report.

In researching the issue, the FCC reviewed more than 500 outage reports from 22 service providers. The agency also interviewed representatives from communications providers, equipment manufacturers, 911 call center representatives, and state and local governments.

The FCC notes that the derecho provided a true test of the area’s 911 call centers. “Because the derecho was swift and unanticipated, it differed from hurricanes and superstorms like Isaac and Sandy, for which there typically is advance warning and an opportunity for communications providers to prepare by testing equipment, adjusting staff levels, and supplementing training over an extended period of time.”

In its report, Impact of the June 2012 Derecho on Communications Networks and Services, the FCC noted that the communications failures were due “in large part [to] avoidable planning and system failures, including the lack of functional backup power, notably in central offices.”

To address these issues, the FCC recommended maintaining sufficient backup power at central offices, and following best practices regarding maintenance and auditing of telecommunications systems.

Backup power. The most significant issue, according to the report, was the failure to have backup power at central 911 locations. For example, at the Arlington, Virginia, central office, the backup power requires two generators to run the office. However, one of the generators failed after air entered the fuel system. This caused the second generator to become overheated and shut down. Personnel attempted to bring in a mobile generator but could not obtain the unit until after power had been restored.

This power failure was especially crucial because the Arlington office was responsible for monitoring the status of equipment at 34 other sites. According to the report: “Thirty minutes after losing commercial and generator power in Arlington, and shortly after the rest of the Arlington office went on battery power, Verizon lost the ability to monitor its network in northern Virginia, crippling its repair efforts and ability to receive alarms that signal additional equipment failure.”

Because of this incident and others like it, the FCC recommended that communications providers “maintain robust, resilient backup power in central offices, supported by appropriate testing, maintenance, and records retention.”

Best practices

The FCC report notes that many 911 system failures would have been avoided had telecommunications companies simply adhered to existing industry best practices as spelled out in guidance issued in 2011 by the agency’s Communications Security, Reliability and Interoperability Council (CSRIC). These best practices were designed to ensure “optimal security and reliability of communications systems, including telecommunications, media, and public safety.”

The best practices give guidance on routine maintenance, staffing, auditing, and power redundancy.

New technology. Though the report dealt primarily with existing technology, it does concede that new IP-based systems have advantages that make them more resilient, including greater redundancy and reliability. The FCC report noted that, had these technologies been in place, “they likely could have significantly lessened the derecho’s impact on emergency communications.”

According to the FCC, wireless networks withstood the derecho better than wired systems. However, cell towers are still reliant on power, so they begin to fail in the hours or days after a storm when batteries begin to give out.

Glenn Gerstell of Milbank, Tweed, Hadley & McCloy in Washington, D.C., who participates in the federal National Infrastructure Advisory Council (NIAC) Working Group, says that U.S. companies must take lessons from other countries that are used to facing natural disasters. “In places that are prone to natural disasters, flooding, mudslides; there are telecom operators that have exceptional levels of resiliency, in part because they recognize their vulnerabilities,” says Gerstell. “The lessons are: have several levels of backup and a certain number of days of fuel on site and plan for what to do after the fuel runs out. Companies must not only have backup but have protocols in place to constantly monitor and check the status of systems and have available technicians and spare equipment [to] repair the backup system.”

While new technology will address some of the current problems, that technology will come with new problems. “Technology is fabulous, but you need to have simple backup systems in place no matter what. All companies need simple protocols to address the foreseeable,” says Gerstell.