Automatically Resolve Errors During Analysis with Mayhem

Mayhem is a platform for developers, testers, and product analysts alike to come together and automatically perform behavior testing, a comprehensive methodology for detecting and recommending fixes for security exploits of a target application. Unlike static analysis techniques, behavior testing involves dynamic execution of the target application on a particular system. 

When executing the target in the context of a dynamic analysis, many things can go wrong. More often than not, the encountered errors are due to some misconfiguration in the target image. In other cases, the failure is a result of missing user configuration values in the Mayhemfile (this is the specification file of the analysis that Mayhem performs).

In order to overcome these failures and successfully analyze a target application, Mayhem employs mechanisms that first identify the root cause of a potential issue and then try to resolve it by intelligently providing different configuration values.

Tuning Timeouts

One frequently used configuration value in Mayhem is the command timeout. This value is the duration (in seconds) that Mayhem will wait for a single test to finish execution. Since Mayhem version 2.1, tests that result in a timeout—with respect to this value—are reported in the UI. The timeout configuration value is useful when there is a need to set an upper bound for the acceptable execution time of the target application.

However, the command timeout is often left unspecified when there is no interest in detecting high execution time. For Mayhem to successfully perform an analysis, the target must be able to read input from an input source. But what happens when the target application doesn’t get to read any input within the default timeout? This is clearly a problematic situation that requires some intervention to overcome. 

At a first glance, a reasonable action would be to increase the command timeout in order to give more time for the target to read some input. But how much should the increase be? Setting a high timeout value can sometimes result in poor analysis performance. 

To better illustrate this result, let us consider an example: The execution time for a particular application is less than 1 second for 99% of the inputs, but roughly 101 seconds for the remaining 1%. In other words, the P99 latency is 101 seconds. This will result in an average execution time of 2 seconds for each test. On the other hand, if all tests were executed in less than one second, the upper bound for the average execution time would be one second. 

It is evident that if a tiny portion of the tests have a high execution time due to a high timeout value, the overall tests that are run are fewer. Less tests run per second means that bugs are found slower.

For runs with an unspecified command timeout that result in an initial failure to read input (due to insufficient execution time), Mayhem can infer a custom timeout value that is high enough to enable the target application to read some input, but low enough to not degrade the analysis performance with high-latency tests. 

In addition to the command timeout, Mayhem applies the same automatic error handling for failures that arise similarly from an insufficient default value for the network timeout, which is the amount of time (in seconds) to wait for the target to accept network input.

Disabling Analysis Optimizations

Mayhem mainly operates under 3 compatibility levels. The compatibility level is set through the MFUZZ_COMPAT_LEVEL  environment variable. Each level disables a set of analysis optimizations, with MFUZZ_COMPAT_LEVEL=2 having all optimizations enabled and MFUZZ_COMPAT_LEVEL=2 having all optimizations disabled. 

Even though level 0 provides performance benefits, it is not compatible with some target applications. For example, a small subset of multi-threaded applications will fail to run on Mayhem with all analysis optimizations turned on. However, with a small penalty in performance, these applications can still be analyzed for security defects with optimizations turned off. 

Mayhem can automatically detect the appropriate compatibility level and the applicable optimizations for a given target when the MFUZZ_COMPAT_LEVEL is not explicitly specified. Consequently, a run that would otherwise fail with the default optimizations, can be saved and still produce useful findings.

Mayhem Automatically Sets Correct Network Configuration

Mayhem can ingest targets that receive their input via the network. The Mayhemfile dedicates a special section that describes how the input is received from the network. The network.client configuration value is an optional boolean parameter that specifies whether the target is a network client or a network server. 

This distinction is important, because the mechanism of providing input for the target changes drastically between the two settings. Mayhem can automatically detect if your application is a network client or a network server and set this value accordingly.

Another set of runtime errors for some network targets that are analyzed with Mayhem comes from targets that are listening for connections on low ports (also known as privileged ports). Normal users can not have processes listening on these ports. As a result, the target application terminates early with an error before reading any input and cannot be analyzed. 

Since Mayhem is running all applications in a containerized environment, the user can be changed to 0:0 when running the target command (equivalent to docker run --user 0:0 ... ). This modification of the default uid:gid pair can remedy the issues caused by the target listening on a low port. Consequently, a failed run is avoided by inferring useful information about the nature of the target application and then modifying configuration values.

Last but not least, Mayhem can also modify the container filesystem that hosts the target application to recover from mundane errors. At some point, all developers have been frustrated by the following error when trying to execute a program:

root@d179d8e69a2f:/# ./program
bash: ./program: Permission denied
root@d179d8e69a2f:/# chmod +x ./program

Setting the exec bit for the binary file is an easy solution, but sometimes it can be forgotten. Figuring out that a run failed because of this, fixing it, then starting a new run can be boring and, more importantly, time-consuming. That is why Mayhem takes care of automatically setting the exec bit, making the target executable so that the dynamic analysis can be successful.

Results

For a long time at ForAllSecure, we have been striving to enable users to see results as fast as possible. It became evident that users were stumbling upon some commonplace errors while trying to run their targets, so we identified a set of frequent misconfigurations and took action to remedy the situation. 

Since we introduced automatic recovery from errors in Mayhem, we have seen users ingest their targets faster and with more ease. It is interesting to see what the most common misconfigurations are. The diagram below shows the distribution of different misconfigurations over the failed runs that Mayhem was able to recover from:

Giving you a successful run on your first try is one of our top priorities when creating developer tools. We know that debugging packaging issues is frustrating, time consuming, and requires user intervention. 

With automatic error resolution, Mayhem detects and intelligently recovers from runtime errors, thus saving developers time and effort otherwise spent re-running their application with a slightly different configuration. 

Try Mayhem for free today to automatically resolve common errors like optimization level, command timeout, advance triage, exec-bit, and network misconfigurations during analysis.