Spies in the Supply Chain

Not all security incidents are created equal. They don’t all get the attention of the CEO. But one in the fall of 2020 did. Cybersecurity firm FireEye received a notification through its internal systems that an employee had registered a second device to access corporate networks.

It seemed odd. So, CEO Kevin Mandia was briefed and the security team followed up with the employee to ask him if he had registered an alternative device to access the work network. He said no, and FireEye launched an investigation—discovering that someone else had bypassed FireEye’s two-factor authentication system to register the device, gain access to FireEye’s systems, and make off with the company’s Red Team tools.

But how did the hacker get in? To find out, FireEye conducted a thorough analysis of its systems and identified that the point of earliest compromise occurred in spring 2020 from a system connected to Orion business software, a product it had purchased from the firm SolarWinds, Mandia said in an Aspen Institute briefing on the breach.

FireEye ultimately decided to reverse engineer SolarWinds’ software, and discovered that Orion itself had been compromised. Hackers had infiltrated the software supply chain, compromising the SolarWinds system to covertly gain access to its customers’ systems.

“After an initial dormant period of up to two weeks, [the attack method] retrieves and executes commands, called ‘Jobs,’ that include the ability to transfer files, execute files, profile the system, reboot the machine, and disable system services,” according to FireEye’s blog about the breach. “The malware masquerades its network traffic as the Orion Improvement Program (OIP) protocol and stores reconnaissance results within legitimate plugin configuration files, allowing it to blend in with legitimate SolarWinds activity. The backdoor uses multiple obfuscated blocklists to identify forensic and anti-virus tools running as processes, services, and drivers.”

And FireEye was not SolarWinds’ only high-profile customer. It also did business with numerous U.S. federal government departments and agencies, telecommunications firms, Fortune 500 companies, and many others.

FireEye’s decision to disclose then set off a mad dash among other SolarWinds customers to determine if they also had been compromised. The U.S. Cybersecurity and Infrastructure Security Agency (CISA), part of the U.S. Department of Homeland Security, issued an emergency directive requiring U.S. government agencies to take a variety of actions, including disconnecting or powering down SolarWinds Orion products on their networks.

“SolarWinds is so prevalent it’s almost like what Kleenex is to tissues,” said Jake Williams, an analyst and senior instructor at the SANS Institute, as well as founder of Rendition InfoSec, in a SANS webinar held shortly after the disclosure. “They are one of if not the de facto network management system with 300,000 plus customers.”

SolarWinds’ position as a network management system (NMS) made it a lucrative target for infiltrating other networks because it could communicate with devices it was managing or monitoring on customers’ networks, Williams explained.

The sophistication of the infiltration also made it nearly impossible for customers to detect and was the work of a threat actor with the “resources, patience, and expertise to gain access to and privileges over highly sensitive information if left unchecked,” CISA said in a statement.

The agency would later join the FBI, the Office of the Director of National Intelligence, and the National Security Agency (NSA) in a task force dubbed the Cyber Unified Coordination Group to investigate and remediate the incident. In a statement, the task force attributed the SolarWinds breach to Russia as part of an intelligence gathering effort affecting approximately 18,000 public and private sector SolarWinds customers, including multiple U.S. government agencies.

Russia has denied any involvement in the breach of SolarWinds and subsequent infiltration of government and corporate networks. In an interview with Russian news agency TASS, Kremlin spokesman Dmitry Peskov said, “any accusations of Russia’s involvement are absolutely baseless, they are more likely to be a continuation of blind Russophobia that is resorted to in case of any incident.”

While initial concerns pointed to the possibility that the hackers could use their access to disrupt their victims’ networks, many in the U.S. government have called it an act of espionage to further intelligence gathering efforts.

Speaking in an Aspen Institute panel in January 2021, U.S. Senator Mark Warner (D-VA), incoming chair of the U.S. Senate Intelligence Committee, said Americans need to be concerned about the ability of a nation-state actor to intrude into government and private sector networks.

Warner also added that the intrusion was spurring conversation about whether it was “within the bounds of acceptable espionage? Countries spy on each other, but the volume and level in terms of governmental entities and private sector enterprises…ought to be alarming to all of us.”

While the scope of the SolarWinds infiltration may be unique, the number of cyber-espionage attacks is on the rise, says John Grim, senior manager of investigative response at Verizon and lead author of Verizon’s inaugural Cyber-Espionage Report published in fall of 2020. The report analyzed data collected for Verizon’s annual Data Breach Investigations Report (DBIR) to assess the state of cyber-espionage across the globe and within public and private sectors.

The analysis found that generally the education, finance, information, manufacturing, mining and utilities, and public sectors were hardest hit by cyber-espionage. Threat actors—most (85 percent) associated with a nation-state—also managed to compromise their targets within seconds to days through a variety of techniques, such as backdoors (91 percent), phishing (90 percent), downloaders (89 percent), and more. And once inside, threat actors would linger—often for months, as seen in the SolarWinds compromise of FireEye—to exfiltrate data from their victims and risk detection.

“In the real world—by extension the cyber world—it’s a challenge to detect. These threat actors are after data that is sensitive and proprietary,” Grim says, adding that many successful cyber-espionage breaches are not reported because they may remain undetected or may not be required to be disclosed because they did not compromise personally identifiable information.

Threat actors who engage in espionage also work to fly under the radar or blend in by using the tools of the network environment, such as IT administrative rights, Grim explains.

To help address the increasing number—and potential severity—of cyber-espionage intrusions, Warner advocated for an accounting of incidents and an establishment of norms. He praised FireEye’s Mandia for his commitment to disclosing the breach and providing details to help security practitioners better protect their systems. But Warner cautioned that relying on the “goodwill and patriotism” of CEOs was not enough—rules and policies are needed to require disclosures.

Also at the Aspen Institute panel, Katie Moussouris, founder and CEO of Luta Security, added that while the idea of creating norms in cybersecurity for espionage and weapons is popular, those involved are hesitant to take options off the table.

“The idea of setting norms feels to me like we’re in the decline of the digital Roman Empire and we’re trying to tell people it’s not okay to use elephants to cross the Alps,” she says. “Meanwhile, [the adversary] is using elephants to cross the Alps and we will be overrun.”

Moussouris also said that instead of focusing on limiting the use of a specific technology or the development of a weapon, any regulations and norms should focus on behaviors and use case scenarios.

“It’s not the technology that needs to be under these norms, it’s the behavior we need to enact to preserve the world order in general,” she added.

In the meantime, there are actions that security practitioners can take to limit the threat and increase their ability to detect intruders in their systems. This begins, Grim says, with assessing the most valuable data, the safeguards surrounding that data, and the tools and people with access to that data.

Grim also recommends vetting third party entities and having written agreements in place about the security provisions related to such parties.

“Monitor their access into your environment and, at least annually, review your written agreements,” he says. “So, when we get more into the applications that may be provided from an outside entity, we’re making sure they are fulfilling their obligations.”

Williams made similar suggestions in the SANS webinar, adding that these types of intrusions are extremely difficult to detect, and sometimes the best course of action is to have a robust response plan.

For organizations compromised by the SolarWinds hack, “I’m willing to say that unless they were doing some nasty stuff in your environment this was not something that most of us were going to prevent,” Williams said. “If I ran SolarWinds in my environment, I would have been compromised as well.”