More than a month after patches were published for a major vulnerability in Java, users were making 10,000 downloads per hour using the unpatched version of the software—roughly 30 percent of all global Java downloads.

David Nalley, president of the Apache Software Foundation, provided those metrics in testimony in February 2022 before the U.S. Senate Committee on Homeland Security and Governmental Affairs.

“The reality is that humans write software, and as a result there will continue to be bugs, and despite best efforts some of those will include security vulnerabilities,” Nalley said.

Nalley was on Capitol Hill to answer questions from lawmakers about the Log4Shell vulnerability and its impact on the broader security ecosystem. Log4Shell is a vulnerability in the popular open-source software logging tool Log4j, which is used by developers around the world to support programs, products, and systems—potentially even millions of them.

When exploited, this crack in the open-source infrastructure of the Internet can force a system to use Log4j’s functions, including downloading and running malicious code, to inflict harm. A researcher at Alibaba Cloud reported the vulnerability to Apache in November 2021, which immediately set to work with its security team to create a patch to repair the Log4Shell vulnerability. They churned out a solution in almost record time—roughly two weeks.

But before the patch could be widely implemented, someone posted information about Log4Shell to GitHub on 9 December 2021, alerting the world to its existence and setting off a race to patch before malicious actors could exploit the vulnerability.

24

Percentage of the most popular open-source projects that contain at least one known security vulnerability.

---

The security community rapidly responded. Researchers put together lists of affected products, tools, and software; security staff took deep dives into their networks to determine where exposures might lie; and organizations launched resource pages devoted to understanding and assessing the vulnerability. Government agencies from around the world, including the Australian Cyber Security Centre, the Canadian Centre for Cyber Security, the FBI, New Zealand’s National Cyber Security Centre, the UK’s National Cyber Security Centre, and the U.S. Cybersecurity and Infrastructure Security Agency (CISA), also issued a joint advisory to provide mitigation guidance on Log4Shell.

CISA Director Jen Easterly called the vulnerability an “unacceptable risk to federal network security,” and instructed federal civilian agencies to take immediate action to protect their networks. She also urged all other organizations to do the same.

“If you are using a vulnerable product on your network, you should consider your door wide open to any number of threats,” Easterly said in a statement.

Open-source software tools are a challenge to secure because they are based on “software with source code that anyone can inspect, modify, and enhance,” according to Red Hat’s Opensource.com. “Source code is the part of the software that most computer users don’t ever see; it’s the code computer programmers can manipulate to change how a piece of software—a ‘program’ or ‘application’—works. Programmers who have access to a computer program’s source code can improve that program by adding features to it or fixing parts that don’t always work correctly.”

Open-source software is a critical part of the digital ecosystem and is used in millions of different ways, including in the form of the Linux operating system and Apache Web Server. A final product, for instance, might contain hundreds of open-source packages, according to Trey Herr, director of the Cyber Statecraft Initiative under the Scowcroft Center for Strategy and Security at the Atlantic Council.

In a risk assessment released by the U.S. Department of Commerce and the Department of Homeland Security (DHS), open-source software and device firmware were cited as the most vulnerable areas in the information and communications technology (ICT) supply chain.

“The ubiquitous use of open-source software can threaten the security of the software supply chain given its vulnerability to exploitation,” the assessment found. “Furthermore, the complexity of the ICT supply chain has led many Original Equipment Manufacturers (OEMs) to outsource firmware development to third party suppliers, which introduces risks related to the lack of transparency into suppliers’ programming and cybersecurity standards.”

And when compromised, addressing the vulnerability in an open-source software is multifaceted for security vendors, such as Cisco, that not only need to address their own organizational security but also the security of their products being used by clients, said Brad Arkin, senior vice president, chief security and trust officer, Cisco Systems, in his testimony on Capitol Hill.

Cisco had a few advantages in addressing Log4Shell, though. It had the resources—money and staff—to immediately address the problem. And it had prior experience in handling a major Internet infrastructure issue.

“In 2014, our industry faced a similarly widespread zero-day vulnerability called ‘Heartbleed,’” Arkin said. “At that time, it took Cisco 50 days to identify the full list of software that required updates to remediate the vulnerability and several additional weeks to apply the necessary software patches. With our Log4j response, Cisco was able to respond significantly faster and was able to identify the use of the Log4j library within its products and services and provide software patches for affected products within 10 days.”

Others, however, did not have those advantages and were unable—or unwilling—to address the issue. Speaking at the same hearing as Nalley and Arkin, Jen Miller-Osborn, deputy director of threat intelligence, Unit 42, for Palo Alto Networks, said there are numerous servers—on premise and in the cloud—that are not patched.

“As is typical for many high severity [remote code execution] vulnerabilities, adversaries have conducted massive scanning activity for Log4Shell with the intent of seeking out and exploiting unpatched systems,” she explained.

Some of those target systems could include public finance entities running Java-based software that incorporates Log4j, according to a Fitch Ratings wire report in January 2022.

Malicious actors could use Log4j, for instance, to disrupt services and extort public financial entities for ransom, the Fitch report said.

---

This was effectively a crack in the foundation of our software infrastructure.

---

“Ransomware attacks on public finance entities have increased in the past three years. Log4Shell makes the risk of attacks more acute due to the ubiquity of Java-based software, the prevalence of a patchwork of legacy systems across the sector, and the finite resources of IT staff,” the Fitch report explained. “Many Java applications are unsupported or proprietary and organizations that do not have robust asset inventories of active applications may not be able to quickly identify embedded Log4j code. Additionally, the large costs of updating existing equipment and software generally mean that institutions, particularly smaller entities, may rely on legacy systems for many years, even with outdated or unsupported software, leading to gaps in institutional security.”

Both Arkin and Nalley touched on this point in their testimony, explaining that both of their organizations developed patches to address the Log4Shell vulnerability and rolled them out.

“Once we got the info to our customers, they had to take it and apply it within their patch management windows,” Arkin said. “That’s not something we track.”

It’s also difficult to say what organizations were impacted by Log4Shell exploits because global uniform incident report requirements do not yet exist. For instance, the United States lacks a breach notification law or regulation—except for some instances where personally identifiable information was potentially compromised.

Nalley did acknowledge that there have been some positives from the disclosure of Log4­Shell, including an increase in security auditing and code validation of open-source software. But the incident points to a broader, older issue, said Herr, from the Scowcroft Center, in the hearing.

Rather than a flaw, “this was effectively a crack in the foundation of our software infrastructure,” Herr said of Log4Shell, adding that software supply chains remain vulnerable; there have been 140 separate attacks against open-source code projects since he and his team began cataloging them in 2010. Further research from the Atlantic Council has found that 29 percent of the most popular open-source projects contain at least one known security vulnerability.

“Our task is to ensure the long-term viability and security of open source as it enables these important and widely used technologies,” Herr said. “In working to improve the security of open source we should not seek to fix these communities, but to become a better partner to them to enable open-source developers, maintainers, and consumers to better secure each other.”

Some of that work is underway. In February 2022, the U.S. National Institute of Standards and Technology (NIST) released five documents containing guidance, frameworks, and recommendations for software security practices and software security labeling. Following the SolarWinds supply chain breach, U.S. President Joe Biden issued an executive order in May 2021 instructing agencies—including NIST—to enhance cybersecurity through initiatives related to the security and integrity of the software supply chain.

Herr, however, said that more needs to be done. He recommended that the U.S. Congress authorize the creation of an open-source security outreach and partnership program office within CISA’s Cybersecurity Division. Additionally, Herr said CISA should also assess the open-source risk across the technology ecosystem—not just the government’s footprint.

“The product of these assessments is urgently needed to guide investment decisions by both the public and private sectors intended to better secure open-source software supply chains and to prioritize federal actions to proactively manage risk rather than wait for future crises,” Herr explained.

Lastly, Herr said that investments need to be made to secure open-source software because it is part of the critical digital infrastructure. One proposal before the U.S. House of Representatives, as of Security Management’s press time, would create Critical Technology Security Centers (CTSC) inside of DHS, including one focused on open-source security.

“Adequately resourced, this CTSC for open source would provide a starting point for federal efforts to improve the health and long-term security of the open-source ecosystem,” Herr said.

By “funding the mundane,” Herr added, Congress and other stakeholders will be able to create structural improvements in the security of software supply chains across all developers and maintainers against future attacks.

“The cyberattack surface we’re forced to defend is being shaped as we speak by these vulnerabilities,” he said. “These aren’t just protections for us—but to make the management of the security landscape we face abroad easier.”