F5 iControl REST Endpoint Authentication Bypass Technical Deep Dive

Introduction

F5 recently patched a critical vulnerability in their BIG-IP iControl REST endpoint CVE-2022-1388. This vulnerability is particularly worrisome for users because it is simple to exploit and provides an attacker with a method to execute arbitrary system commands.

POC

Let’s examine the inner workings of this vulnerability. The vulnerability is used below to execute the id command using a single HTTP request:

POST /mgmt/tm/util/bash HTTP/1.1
Host: 127.0.0.1
Authorization: Basic YWRtaW46aG9yaXpvbjM=
X-F5-Auth-Token: asdf
User-Agent: curl/7.82.0
Connection: X-F5-Auth-Token
Accept: */*
Content-Length: 39
{“command”:”run”,”utilCmdArgs”:”-c id”}

Deep Dive

The iControl REST service runs as a Java application listening on localhost port 8100. This application is exposed by an Apache web server acting as a reverse proxy. The vulnerability lies at the intersection of the authentication mechanisms used by iControl and Apache. The basic flow of a request from the client to the iControl REST service can be seen in the following diagram:

mod_auth_pam.so

F5 has a custom Apache module called mod_auth_pam.so that is responsible for some of the client authentication. This module registers a hook using ap_hook_check_authz and ap_hook_check_access_ex

My version of IDA is unable to decompile the hook function and Ghidra also struggles to work with this function, but we can continue with the disassembly.

This function does a lot of stuff, but let’s zoom in on the parts concerned with authentication.

Here we can see that the function checks the request headers for the presence the X-F5-Auth-Token. If the header is found it goes onto eventually allow the request to be sent to the iControl REST service. Otherwise, the Authorization header is processed and the request is rejected if the credentials are invalid.

iControl REST Service

Once the request arrives the iControl REST service, the X-F5-Auth-Token value is validated in EvaluatePermissions.java and the request is rejected if it is invalid:

However, notice what happens if the X-F5-Auth-Token header is not present, execution continues in completeEvaluatePermission without a token.

Here there is a conversion of “identity information” into basic auth information in setBasicAuthFromIdentity. After some research we find that the identity information gets populated in RestOperationIdentifier.java in setIdentityFromBasicAuth.

In this function we can see that if the X-Forwarded-Host header or contains 127.0.0.1 or localhost, the user name is extracted from the basic auth header and used to populate the identity data without checking the password. This means that if we set the Host header to localhost we can effectively bypass the permission checks in completeEvaluatePermission by using basic authorization with the credentials admin:.

This is putting the cart before the horse because we cannot bypass the mod_auth_pam.so basic authentication check without setting the X-F5-Auth-Token header and we cannot bypass the REST service’s verification of X-F5-Auth-Token without a valid token.

The Error

Most developers are probably be familiar with keep-alive and close variations of the Connection header, but the Connection header can be used by proxy clients to indicate to the proxy that certain headers should be removed before the request is passed on. (See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection for more explanation).

It turns out that the check for the X-F5-Auth-Token header in mod_auth_pam.so is done before the Connection header is processed. This means that we can add X-F5-Auth-Token as a value for the Connection header to skip the basic authorization checks by mod_auth_pam.so and have the X-F5-Auth-Token header removed from our request before it gets passed to the iControl REST service! We can use the /mgmt/tm/util/bash to execute arbitrary system commands as root.

The Patch

Let’s examine what F5 has done to remediate this vulnerability. It appears majority of the fix is implemented in mod_auth_pam.so to make sure that there aren’t any values in the Connection header that the application cannot handle. This check is implemented in their hook function seen below:

Additionally, X-F5-Auth-Token validation has been moved to this hook function and AuthTokenWorker.java in the iControl REST service has been updated to make sure that its cache is always written to disk, presumably so that mod_auth_pam.so is able to correctly validate the token.

Summary

To wrap things up here is an overview of the necessary conditions of a request for exploiting this vulnerability:

1. Connection header must include X-F5-Auth-Token
2. X-F5-Auth-Token header must be present
3. Host header must be localhost / 127.0.0.1 or the Connection header must include X-Forwarded-Host
4. Auth header must be set with the admin username and any password

IOCs can be found in the /var/log/audit log file. Unrecognized commands executed by the mgmt/tm/util/bash endpoint should be cause for concern. An attacker can use this vulnerability to do just about anything they want to on the vulnerable server. This includes making configuration changes, stealing sensitive information and moving laterally within the target network.