While the rest of the world was continuing its battle against the COVID-19 pandemic on 9 December 2021, a new struggle was about to take place on computer networks around the world that would lead to one of the most intensive cybersecurity community responses to ever occur.

That day, an Alibaba security engineer would inform the Apache Software Foundation (ASF) via email that a vulnerability he had previously reported to them was being discussed on WeChat. The vulnerability affected Log4j, an open-source Java-based logging framework that is used to collect and manage information about system activity in thousands of software packages globally.

The WeChat discussion included a redacted screenshot of a proof-of-concept exploit. ASF had already been working on addressing the Log4j vulnerability because the security engineer had reported it on 24 November 2021. Now, however, it was clear ASF’s response team and the security community at large needed to jump into overdrive to remediate the vulnerability before nefarious actors could exploit it.

ASF upgraded Log4j and assigned it a Common Vulnerabilities and Exposures (CVE) identifier that was made publicly visible. At the same time, government agencies like the U.S. Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (DHS CISA), corporations, and security practitioners set to work to assess their risk posture and mitigate it as quickly as possible.

Ashish Gupta, CEO of Bugcrowd, a crowdsourced security platform, was one of those individuals. While the full scope of the vulnerability was still emerging that day in December, he and his global team knew that they needed to jump into action right away.

“It was clear to us that this is something that’s going to be big, and we need to put all our resources into helping our customers understand where the vulnerability could be,” Gupta explains in a call with Security Management.

Bugcrowd leveraged its own security engineers, as well as feedback on its research platform, to triage the vulnerability and provide information to its clients. And while several months have passed since disclosure, that work remains ongoing.

“It’s a long-tail vulnerability that we need to keep an eye on,” Gupta says. “We had systems to respond to it, we had firewalls. It helped that DHS provided thoughts on how to outsmart potential adversaries. A lot of great building blocks were in place and deployed to help address a rather large, and very impactful vulnerability.”

Assessing the Damage

That initial assessment was correct, according to a new report out this week from the U.S. Cyber Safety Review Board (CSRB) that analyzed the events surrounding the disclosure of the Log4j vulnerability in December 2021 and its known impact. For its inaugural review, the CSRB engaged approximately 80 organizations and individuals to create a timeline of the vulnerability and understand its impact.

“At the time of writing, the board is not aware of any significant Log4j-based attacks on critical infrastructure systems,” according to the report. “Somewhat surprisingly, the Board also found that to date, generally speaking, exploitation of Log4j occurred at lower levels than many experts predicted, given the severity of the vulnerability.”

Through its analysis, the board found that five days after Log4j's disclosure Cloudflare observed 400 exploitation attempts of the vulnerability per second. Cisco Talos also reported observing the Internet of Things botnet Mirai exploiting Log4j; Checkpoint researchers reported observing nation-state threat actors exploiting the vulnerability; and other cybersecurity researchers reported similar levels of activity.

The board explained that it was difficult to arrive at ths conclusion of how the Log4j exploit was used because there is no authoritative source to study and understand “exploitation trends across geographies, industries, or ecosystems,” the report said. “Many organizations do not even collect information on specific Log4j exploitation, and reporting is still largely voluntary.”

What the board’s review did find, however, was that the vulnerability impacted nearly every networked organization and required swift action to address it. Like Cisco explained in testimony before Congress earlier this year and Bugcrowd’s CEO attested to, many enterprises and vendors embarked on a discovery period to determine where they used Log4j after it was disclosed.

“The pace, pressure, and publicity compounded the defensive challenges: security researchers quickly found additional vulnerabilities in Log4j, contributing to confusion and ‘patching fatigue;’ defenders struggled to distinguish vulnerability scanning by bona fide researchers from threat actors; and responders found it difficult to find authoritative sources of information on how to address the issues,” the report explained. “This culminated in one of the most intensive cybersecurity community responses in history.”

For instance, Gupta says that thousands of Bugcrowd researchers provided input to hundreds of customers on how Log4j impacted their networks. Sometimes these were unique findings, but other times researchers were discovering the same specific Log4Shell exploit running on the same asset.

The response to Log4j, however, was not equal. Organizations that understood prior to December 2021 how they used Log4j and had technical resources and processes to manage their risk and respond to incidents were able to take more effective action. But the board found that few fell into this category, meaning most organizations spent significantly more time to respond.

For instance, the board found that one U.S. federal government department dedicated 33,000 hours to responding to the Log4j vulnerability. Doing so meant that other mission-critical work was delayed. This high-level of urgent response also contributed to burnout amongst cybersecurity practitioners.

Disclosure Concerns

In a separate blog post, board member Katie Moussouris, founder and CEO of Luta Security, shared a list of lessons learned from the Log4j incident for all organizations. These included having an asset inventory, practicing rapid update rollouts, and building relationships with vendors so the organization is familiar with the process for updates. 

The Log4Shell vulnerability was originally discovered in China by a security engineer on the Alibaba Cloud security team on 24 November 2021. The engineer reported the vulnerability to the ASF. In its assessment of the incident, the board expressed concern that the Chinese government may require vulnerabilities be disclosed to it first before informing the affected company—giving China an offensive advantage.

"This is a disturbing prospect given the [People's Republic of China] government's known track record of intellectual property theft, intelligence collection, surveillance of human rights activists and dissidents, and military cyber operations," the board explained.

Moussouris explained in her blog post that while the board did not offer an opinion, she is personally concerned that the U.S. government and others might adopt a similar requirement.

“Only the organizations that are responsible for creating a fix should know about a vulnerability before a patch is available,” she wrote. “Adding government entities to the embargo during vulnerability coordination and disclosure will not meaningfully add to our safety, but it does meaningfully and dramatically increase the risk of a leak before a patch is ready. It would also create a new high-value target: a government-run treasure trove of unpatched vulnerabilities. Aggregating vulnerabilities from multiple software vendors in one place would raise the risk of a Pandora’s box event if that database of bugs was compromised.”

Moving Forward

To learn from the Log4j response and better prepare for future incidents, the board made 19 recommendations broken into four categories: addressing continued risks of Log4j, driving exiting best practices for security hygiene, building a better software ecosystem, and investing in the future.

“The board notes that the community should implement improvements to forestall the next Log4j-type event while also undertaking actions to intervene and remediate the present risks,” according to the report. “Therefore, our recommendations blend the need for continued vigilance with a drive toward medium-term adoption of existing security hygiene best practices and investments to improve cybersecurity processes, frameworks, and policies. Additionally, new investment and novel approaches are necessary to bring about transformations that scale to protect the entire ecosystem at an optimal level of capability and maturity.”

One of the recommendations, for instance, suggest that software developers be trained in secure software development. The U.S. federal government could lead this effort by investing in and engaging with higher education institutions and training programs. The board also said universities and community colleges should incorporate cybersecurity training and secure development practices for computer science programming degrees.

Another recommendation included creating a pilot for open-source software maintenance support for critical services.

“Open-source developers are usually not paid to maintain their software, especially not older versions that are often deployed deeply and widely across systems,” the board wrote. “Log4j was a particularly extreme example of this. The burden of maintenance then falls to organizations that use open-source software, and can be expensive, time-consuming, and unpredictable. Organizations should be able to project the maintenance costs of open source software, and obtain that information from a trusted source. Funding the maintenance of critical open-source software could drive a more sustainable model for security at scale and enable timely and effective distribution of updates and mitigations.”

Other recommendations were more forward looking, including examining the efficacy of a Cyber Safety Reporting System, exploring the feasibility of establishing a software security risk assessment center of excellence, and studying the incentive structures required to build secure software. These efforts could put the cybersecurity community in a better position to mitigate future risks from Log4j and other open-source software vulnerabilities that are widely used across networks.

“Log4j remains deeply embedded in systems, and even within the short period available for our review, community stakeholders have identified new compromises, new threat actors, and new learnings,” the board wrote. “We must remain vigilant against the risks associated with this vulnerability, and apply the best practices described in this review.”

Vigilance and scanning to ensure the community understands how Log4j and other open-source software vulnerabilities will change will be critical as we move forward, Gupta says.

“Tactics, techniques, and procedures evolve,” he adds. “We need to be on guard for malware, worms, and ransomware that are born through Log4j.”