Something in the Water
It was the first known infiltration of U.S. critical infrastructure by a nation-state hacking group. Between August 28 and September 18, 2013, an Iranian hacker with connections to the Islamic Revolutionary Guard gained unauthorized access to the supervisory control and data acquisition (SCADA) systems of the Bowman Avenue Dam in Rye Brook, New York.
With this access, the hacker obtained information about the status and operation of the dam, including the water levels, temperature, and status of the sluice gate that controls water levels and flow rates.
However, the hacker was prohibited from obtaining control of the gate because it had been manually disconnected for maintenance.
After a lengthy investigation, the U.S. Department of Justice indicted seven Iranians for their alleged roles in both the dam hacking and a broader series of distributed denial of service (DDoS) attacks on financial institutions in New York state.
"The infiltration of the Bowman Avenue Dam represents a frightening new frontier in cybercrime," said Preet Bharara, then U.S. attorney of the southern district of New York. "These were no ordinary crimes, but calculated attacks by groups with ties to Iran's Islamic Revolutionary Guard and designed specifically to harm America and its people."
Since the infiltration of the dam, the United States has made some progress in addressing cybersecurity threats to critical infrastructure. The U.S. Department of Homeland Security (DHS) designated critical infrastructure verticals, created information sharing and analysis centers for each vertical, and conducted extensive outreach to the private sector—which owns and operates most critical infrastructure in the United States.
DHS released a cybersecurity strategy for 2018 to 2022 over the past summer (see Security Management, August 2018, "Cyber Goals Past Due"). It also recently created the National Risk Management Center, which will focus on creating a cross-cutting approach to defending U.S. critical infrastructure.
The center "will employ a more strategic approach to risk management born out of the re-emergence of nation-state threats, our hyperconnected environment, and our survival and its need to effectively and continually collaborate within the private sector," said DHS Secretary Kirstjen Nielsen in a speech at the 2018 National Cybersecurity Summit.
A focal point of these efforts is addressing cybersecurity threats to the North American electric grid. However, some experts have expressed concerns about whether enough is being done to address vulnerabilities to water and wastewater systems.
"The power grid gets a lot of the news coverage because you can imagine what it's like to have no power," says Chris Grove, director of industrial security at Indegy. "But people don't think about water as much because they don't understand what a total outage looks like."
One recent example of this was in Washington, D.C., on July 12 when the District of Columbia Water and Sewer Authority issued a boil order for thousands of residents. The order was in response to the discovery that an open valve at a pumping station created a loss of pressure in parts of the district's distribution center for roughly one hour.
That loss of pressure could have allowed contaminants to enter the water system that 100,000 residents and visitors use to cook, clean, bathe, and drink.
The boil order was lifted 48 hours later after the district had conducted a thorough testing of its system to ensure that no contamination was present. But during this time, residents and businesses were forced to stock up on bottled water and take other measures to reduce the boil order's impact.
There was also a six-hour delay in issuing the boil order, which meant some individuals could have been exposed to contaminated water while the authority crafted its emergency alert. Officials later said the delay occurred because they were working to pinpoint the exact area affected.
The authority has since conducted a full audit of the incident and released an after-action report to improve its monitoring and alert process for future incidents, should they occur.
"This report makes good on our promise to be as open and transparent with our customers as we can," said D.C. Water CEO and General Manager David L. Gadis in a statement. "We can and will do better. Although I'm proud of how quickly our team restored water pressure, how infrequently these types of incidents occur at our facility, and the many ways we shared the information with our customers, I want us to constantly improve."
Included in the list of recommendations from the audit were valve restrictions, such as placing operational controls at pumping stations to prevent releases of pressure by requiring a supervisor to approve when divider valves are opened; a review of the authority's SCADA alarm protocols; and adding a second server to reduce the likelihood that the authority's website would be overwhelmed.
While the D.C. incident was the result of a physical system error, experts like Grove are concerned about how well water and wastewater systems are equipped to address cyber threats to their infrastructure.
Many systems operators perceive that they are protected from cyberattacks because they use air gaps—meaning there's no direct connection between the system that controls the operations of the water or wastewater system and the Internet.
But with advancements in technology, many systems are not as isolated from the Internet as their operators might perceive, Grove says.
"Maybe in the old days, but nowadays it's not," he explains. "They tend to need to update systems, or they want to get data from a remote plant to a central facility for accounting purposes or to find out how many gallons of water they treated. All those things have evaporated the air gap."
Another vulnerability is the inability to regularly update systems inside the treatment facilities themselves—sometimes because of the air gap or because the system always needs to be operational.
"These systems on the industrial side, they're meant to be running stagnant for 10, 20, or 30 years so they aren't updated," Grove says.
"They don't take them down and patch them, and the end result is once an attacker gets past that mythical air gap…there's nothing to stop them from moving laterally, embedding themselves to stay there, and doing the things they want to do," he explains.
For instance, many of these systems—along with treating wastewater—also produce fresh water. In a worst-case scenario, a plant could be compromised and forced to begin dumping wastewater into creeks and rivers. Then, customers would turn on their taps and no water would come out.
"After a few days, when all the bottled water runs out, most people believe they'd just go to the mountains and live off the earth," Grove explains. "But if the places where we get our fresh water have been polluted, now we're going to have a tough time meeting that demand. "
"And without water, everything stops. You can't run a factory without water. You can't even make gasoline without water."
The perception of an air gap can also play against security professionals who are seeking to get funding to create a more layered approach to enhance the security of the system.
According to the U.S. Environmental Protection Agency's (EPA's) Cybersecurity Guide for States, one of the main challenges for water and wastewater utilities is the lack of resources for information technology and security specialists to assist with creating a cybersecurity program.
The entire threat landscape is hard to grasp, especially if the system relies on physical security stop gaps to avoid cyberattacks.
"Utility personnel may believe that cyberattacks do not present a risk to their systems or feel that they lack the technical capability to improve their cybersecurity," according to the guide.
In his conversations with security teams at water treatment facilities, Grove says he often finds that executives believe the air gap is a failsafe system, so additional security measures aren't needed to protect it from attack.
"Then it works against them and ends up causing them to struggle getting what they need to make a true, layered security model," Grove adds.
For instance, this would include mapping to determine what assets are in the system and how they're vulnerable, along with a monitoring system to detect when an infiltration has occurred—such as in the Bowman Avenue Dam attack.
The EPA's Cybersecurity Guide also offers a worksheet for water and wastewater treatment facility operators to create an effective cybersecurity program. Action items include auditing IT systems and identifying vulnerabilities, implementing secure remote access practices, improving physical security for IT equipment, and conducting cybersecurity training for utility staff and contractors.
"Talk to your IT service providers and others who manage your IT systems about how to carry out these actions at your utility," the guide suggested.
The guide also includes a section offering seven suggested steps for responding to a suspected cyber incident at a water utility. These include actions such as disconnecting compromised computers from the network, rather than rebooting systems; isolating all affected systems; and alerting customers as needed.
And while some are slow to take this advice, Grove says he has seen perceptions changing over the last year as cyberattacks have become more mainstream and understanding of the vulnerabilities in critical infrastructure deepens.
"If something as simple as a drip of water can actually take our society down, then people become much more interested in addressing it," he adds.