For the first time, the Cyber Safety Review Board (CSRB) has released a review of the December 2021 Log4j security vulnerability. The report highlights lessons learned, and makes independent, strategic, and actionable recommendations to the Secretary of Homeland Security.
The report reviews the event surrounding the disclosure of the Log4j vulnerability, which impacted virtually every networked organization. While there is no comprehensive list of Log4j users, the open source Java-based logging framework is simple to use, free to download and effective, making it popular among Java developers that have embedded it and other additions into thousands of software packages.
Based on observations, CSRB says, “responders, spanning the public and private sectors, the open source community, and researchers globally, collaborated and communicated in a dedicated fashion, working through weekends and the December holidays.” The CSRB noted high levels of cooperation, extensive use of social media for rapid sharing of mitigation advice, innovative response actions from the Department of Homeland Security’s (DHS) Cybersecurity and Infrastructure Security Agency (CISA), and the creation of new shared community resources.
Organizations that responded most effectively to the security vulnerability understood their use of Log4j and had technical resources and mature processes to manage assets, assess risk, and mobilize their organization and key partners to action. Most modern security frameworks call out these capabilities as best practices.
However, few organizations were able to execute this kind of response, or the speed required during this incident, causing delays in both their assessment of the risk and in their management of it. When the Apache Software Foundation (ASF) made upgrades for Log4j available, deploying them was itself a risk decision, forcing a tradeoff between possible operational disruption and timeliness, completeness, and compensating controls.
According to CSRB, organizations spent significant resources as they struggled with the vulnerability. One federal cabinet department dedicated 33,000 hours to vulnerability response to protect the department’s networks. The costs — sustained over many weeks and months — delayed other mission-critical work, including incident response to other security vulnerabilities.
CSRB predicts tension between the collective need for crisis-driven risk management and the foundational investments that would support more timely incident response efforts to future incidents. Perhaps most significantly, the force exerted on the urgent response and the challenges in managing risk also contributed to professional “burnout” among defenders that may, compounded with the generally intense pace of many cybersecurity jobs, have a long-term impact on the availability of cybersecurity talent.
Significant risk still remains as Log4j vulnerability is not over. CSRB classifies Log4j as an endemic vulnerability,” and that vulnerable instances of Log4j will remain in systems for many years to come, and even decades or longer.
To reduce the recurrence of security vulnerabilities like Log4j, it is essential that public and private sector stakeholders create centralized resourcing and security assistance structures that can support the open source community going forward, the report found. The report provides 19 actionable security recommendations for the government and the security industry.
Below, security thought leaders react to the findings of the CSRB report.
Tim Mackey, Principal Security Strategist at Synopsys Cybersecurity Research Center:
Rarely do we get a comprehensive review of the impact and root causes of a cyber incident so quickly after the incident occurred, but that is precisely what we have from the CSRB in their report on Log4Shell and log4j.
Open source software is fundamentally managed differently than commercial software, but open source software plays a key role in the success of commercial software. The “long-tail” scenario outlined in the report is one we’ve seen with countless past vulnerabilities, and one that favors attackers since their success is based on having at least one victim who hasn’t patched their systems. Given management of open source software is different from commercial software, and open source powers commercial software, reliance on a commercial vendor to alert consumers of a problem presumes that the vendor is properly managing their usage of open source and that they are able to identify and alert all users of their impacted software — even if support for that software has ended.
With patch management being a challenge at the best of times, to mitigate the risk of unknown open source governance within vendors, software consumers should implement a trust-but-verify model to validate whether the software they’re given doesn’t contain unpatched vulnerabilities.
Michael Skelton (Codingo), Senior Director of Security Operations at Bugcrowd:
Dealing with Log4J is a marathon, one that will take years more to resolve. Java, and Log4j are prevalent everywhere, not only in core projects but in dependencies that other projects rely on, making detection and mitigation not as simple an exercise as it may be with other vulnerabilities. While the initial wave of Log4J findings has subsided, we do still see Log4J over bug bounty programs somewhat frequently as the crowd dives deeper into the vulnerability and looks into the dependencies of projects for its presence.
John Bambenek, Principal Threat Hunter at Netenrich:
The Log4j vulnerability was first widely known in December. This report is eight months after that. At this point, anyone still vulnerable is highly unlikely to read this report or in much of a position to do anything about it if they did. Most of the American economy is small to medium businesses that almost always never have a CISO and likely not even a CIO. Until we find ways to make the public without security budgets safe, no high-level list of best practices will move the ball significantly.
Casey Ellis, Founder and CTO at Bugcrowd:
This is an incredibly dense report, but one that I hope a lot of folks — both inside and external to government — read and digest. The vulnerabilities in Log4j prompted “one of the most intensive cybersecurity community responses in history,” and there are a great many lessons to be learned from it and hopefully applied back into software and F/OSS vulnerability management.
Of particular interest are some of the comments in the executive summary about the PRC’s potential actions against Alibaba for violating then-recent vulnerability disclosure laws and the potential for this to have a chilling effect on security research in good faith out of mainland China. Given the progress on deconflicting anti-hacking laws like the CFAA and the CMA in the West, seeing the PRC seemingly take steps in the opposite direction is a vulnerability international relations shift that will be worth keeping an eye on.
Matthew Warner, CTO and Co-Founder at Blumira:
The complexity of patching unknown log4j systems continues to add more difficulties for organizations. A purchased appliance may have a vulnerable version of Log4j without any knowledge of the organization. There continues to be exploitation of Log4j across internet-exposed VMWare Horizon servers that have not been patched, even within hours of CISA notifications of vulnerable hosts. In the grand scheme of cybersecurity, however, Log4j is not unprecedented; even three years after exposure, there continues to be exposed RDP that is vulnerable to BlueKeep.
Vulnerabilities that live within infrastructure have longevity and stickiness due to the complexity of networks and IT turnover that results in undocumented devices. It will take many years for the industry to remove and update all legacy Log4j solutions and support to identify impacted solutions, and getting this information to organizations will be necessary for privacy/public partnership success.
Mike Parkin, Senior Technical Engineer at Vulcan Cyber:
The CSRB report goes into quite a bit of detail and depth on the Log4J incident and comes out with a comprehensive list of recommendations going forward. Not surprisingly, many of the recommendations are things we’ve been saying in the industry for some time about following best practices and trying to improve hygiene.
Hopefully, this report will help push more organizations to follow best practices and collectively clean up our acts so that the next time a Log4J style vulnerability surfaces, and there will be, we’ll be in a better position to deal with the Next Big Thing vulnerability quickly and effectively. It could also, hopefully, lead to more active initiatives to drive the message home.
Brad Crompton, Intelligence Director at Intel 471:
It is highly recommended users upgrade Log4j to the latest version. Where upgrading Log4j is not feasible instantaneously, exploitation attempts still can be averted by removing the JndiLookup class from the classpath. Intel 471 has not seen a sustained effort to leverage the vulnerability in attacks, but that does not mean threat actors won't use the vulnerability in the future.
Chris Clements, Vice President of Solutions Architecture at Cerberus Sentinel:
The Log4j incident demonstrates just how at risk thousands of organizations are due to uncontrolled third-party software dependencies they may not even be aware that they are using. It’s a real wake-up call for both developers and end users that upstream changes from developers not under your control can pose a significant risk to your products and operations. Worse, in such situations, vulnerable organizations can be left with little recourse for remediation unless they have the development capabilities to correct the vulnerability themselves. In a sense, we are lucky that the Log4j issue had an almost perfect storm of remediation response. The developers themselves:
- Were still actively maintaining the project
- Quickly recognized the seriousness of the issue
- Dedicated themselves to urgently correcting the issue
Had any of these items not been the case, vulnerable products and organizations could have had little means of self-correcting the problem. The thing is, curative actions by the Log4j development team mostly happened out of the goodness of their hearts. As free and open-source software, they were likely under no obligation at all to fix the issue had they not wanted to. This type of uncontrolled dependency should be a major concern for product developers looking to avoid catastrophic situations in the future either from dependency developer mistakes or even directly malicious actions like purposefully introducing vulnerabilities or backdoors.
Roger Grimes, data-driven defense evangelist at KnowBe4:
This is an excellent report and it makes good recommendations across the board. There are no surprises and things that have not been said many times before by other groups, but it’s great to see them all collected into one report. However, the most welcome recommendations include improving the default security and health of the open source software ecosystem, including recommending that developers get trained in secure software development. We know for sure that one of the single best things we can do to improve software security is to make sure all developers have training in common security bugs and how to avoid them. Yet, very few universities and programming schools give developers any training in secure software development, much less make it a key part of their curriculum. Why do we still have an escalating number of software vulnerabilities...over 20,000 publicly identified bugs last year? Because we don’t teach developers how to avoid those bugs. We get what we sow. In this case, DHS is saying we need to fix that, in open source software as well.
For more CSRB findings, please visit www.dhs.gov.
