A critical remote code execution vulnerability (CVE-2021-44228) has been publicly disclosed in Log4j, an open source logging utility widely used in applications, including many by large enterprise organisations.
The vulnerability allows threat actors to exfiltrate information from and execute malicious code on systems running applications that use the library by manipulating log messages.
There are already reports of servers performing internet-wide scans in an attempt to locate vulnerable servers, and our threat intelligence teams are seeing attempts to exploit this vulnerability at alarming volumes. Log4j is embedded in many popular frameworks and many Java applications, making the impact widespread.
On December 9, 2021, a critical vulnerability involving unauthenticated remote code execution and data exfiltration (CVE-2021-44228) was reported in Log4j, which raised concerns about how often open source logging is used. This wide application, coupled with ease of use, makes its impact particularly high.
The vulnerability is being actively exploited and security teams around the world are working on research and elimination.
The compromised machine enables cybercriminals to exfiltrate data and remotely deliver software launched by Log4j. This gives the attacker the ability to run arbitrary commands on the server, reveal information and secrets, and also allows the server to be the starting point for additional attacks, potentially against machines secured deep within the network without direct access to the Internet.
Developers in the Java world widely use Log4j to facilitate error logging and debugging information. Therefore, vendors of products with Java-based software may be vulnerable. Even if an application does not directly use Log4j, many common tools and frameworks use Log4j internally and therefore can introduce this vulnerability into the application stack.
Severity of the Log4j vulnerability
Although Akamai first observed attempts to exploit the Log4j vulnerability on December 9, we are seeing a growing body of evidence to suggest that the vulnerability may have been exploited for months. Since the publication of the vulnerability, we have seen many variants of the exploit with a constant attack traffic of around 2 million attempts per hour.
The speed at which variants evolve is unprecedented.
While the most important action to take for any vulnerability is fixing infected systems, security researchers understand that this takes time. In many cases, organisations may not even know which systems are vulnerable. Thus, additional mitigation measures should be deployed to minimize the threat surface.
More than half (~ 57%) of the attacks seen so far come from attackers previously classified as malicious actors in Akamai’s threat intelligence database. We anticipate that due to the sheer volume of unpatched systems, we will continue to see exploit attempts for months to come.
Many servers, especially those in high-risk environments such as direct access to the Internet, may already have been compromised.
To identify indicators of compromise, the following command can show exploit attempts:
sudo egrep -i -r ‘\$\{jndi:(ldap[s]?|rmi|dns)://’ /var/log
While WAF defences can be very effective for web servers at this stage, organisations should also consider alternative attack paths that could lead to a compromise. To this end, we recommend using micro-segmentation to gain insight into possible impact and reduce risk and spread.
Detection of Log4j abuse using Enterprise Threat Protector
Akamai threat researchers have been reviewing customer DNS, proxy, and sinkhole data looking for anomalous behavior relating to the actors attempting to use Log4j vulnerability in the wild. In the DNS data, we were surprised to see DNS queries for domains containing the string “jndi:ldap”.
These are invalid DNS queries that contain invalid characters in the domain name and therefore will not resolve. Additionally, we saw traffic to known malicious domains that were redirected to sinkhole servers; these domains hosted malicious Java code that was used to abuse vulnerable servers. All of these are indications of potential abuse on customer networks.
Targets
Hackers target a variety of targets, including home users, service providers, source code developers, security researchers, and other developers.
Large companies like Amazon can patch their systems quickly, but smaller organisations may take longer. In the meantime, hackers are scanning the web looking for vulnerable servers and setting up machines to deliver malicious payloads. Then, they query these services to trigger log messages. 404 errors, for example, are commonly triggered by a maliciously crafted query. Then, Log4j processes the request as instructed.
Mitigation Measures
When updates are available, agencies must update software using Log4j to the newest version, which is the most effective and manageable long-term option. Where updating is not possible, the following mitigating measures can be considered as a temporary solution and apply to the entire solution stack.
“The first challenge is to find out where your code and applications might have the vulnerability,” wrote William Malik, Trend Micro’s VP of Infrastructure, in his latest blog about Log4j. Leverage a tool that can scan for Log4j in your source code libraries and identify down to the line of code where the malicious string is residing.
- Disable Log4j library. Disabling software using the Log4j library is an effective measure, favouring controlled downtime over adversary-caused issues. This option could cause operational impacts and limit visibility into other issues.
- Disable JNDI lookups or disable remote codebases. This option, while effective, may involve developer work and could impact functionality.
- Disconnect affected stacks. Solution stacks not connected to agency networks pose a dramatically lower risk from attack. Consider temporarily disconnecting the stack from agency networks.
- Isolate the system. Create a “vulnerable network” VLAN and segment the solution stack from the rest of the enterprise network.
- Deploy a properly configured Web Application Firewall (WAF) in front of the solution stack. Deploying a WAF is an important, but incomplete, solution. While threat actors will be able to bypass this mitigation, the reduction in alerting will allow an agency SOC to focus on a smaller set of alerts.
- Apply micropatch. There are several micropatches available. They are not a part of the official update but may limit agency risk.
Recommended Risk Management Approach
Both the likelihood of exploitation and the impact on mission critical functions are extremely high. Adversaries are actively exploiting this vulnerability in unpredictable ways that are increasingly able to penetrate affected solution stacks connected to agency networks. Mature exploit tools are freely available, allowing even unsophisticated adversaries to gain a foothold, maintain persistence, and inflict damage.
Adopting mitigations that factor timeliness and ease of execution, as well as completeness is paramount. While more complete mitigations are preferable, prompt, simple actions can buy time to develop and implement more complex and complete ones.
The CISA based in the United States expects that many solutions stacks will require multiple mitigating steps in the coming months to adequately address risk from this vulnerability.
DevOps role in remediation
If you are part of a DevOps / DevSecOps team, the distinction between who is responsible for security can seem blurry or even non-existent.
Simply put, yes, security is a shared responsibility where developers focus on building securely and security teams focus on supporting developers in selecting the right tools that automate security and perform well in pipelines. Existing CI / CD. All of this to say that Log4j dominates all aspects of your life, you need to focus on securing (potentially) vulnerable applications.
Insufficient Log4j mitigations
Since the first Log4j vulnerability was announced, several proposed weaknesses have proven ineffective and should no longer be considered a reliable cause.
Upgrading the Java version is not enough. The initial exploit didn’t work on Java versions newer than 6u212, 7u202, 8u192 or 11.0.2 because the default configuration in these versions prevent class loading via JNDI (Java Naming and Directory Interface) from remote servers. However, security researchers have since shown that attackers can build payloads that take advantage of classes in the application’s own classpath instead of remote ones, so this doesn’t prevent all attacks.
