How to Increase Test Coverage (And Confidence!) With Mayhem in 4 Easy Steps

As software development becomes increasingly complex, ensuring the quality of the software is essential. One critical aspect of quality assurance is test coverage, which refers to the percentage of the code covered by automated tests. The higher the test coverage, the more confidence we have in the software's functionality and reliability.

In this post, we will explore how to increase test coverage in your API with Mayhem in four easy steps.

What is Mayhem?

Mayhem is a product to help organizations build reliable APIs. It is designed to be easy to use, flexible, and scalable. Mayhem provides a set of tools to automatically create and execute test cases using a state of the art fuzzing engine. This engine can be integrated with continuous integration and delivery systems. 

‍

{{api-cta}}

Here are some tips on how to use Mayhem to increase test coverage:

1. Create an API specification

Creating a specification for your API is an essential step in the API development process. A specification helps to define the API's functionality, behavior, and data structures, making it easier for developers to use and integrate the API into their applications. It defines the purpose and scope of the API, the endpoints, methods and responses, the security requirements (authorization/authentication) and the versioning strategy.

If you don’t have a specification for your API yet, you can use network recording and Postman Collections to get you started.

Because of the benefits of having a good specification, formats like OpenAPI (formerly known as Swagger) and Postman Collections are ubiquitous. Mayhem can ingest this format.

2. Instrument your API

The exact steps for instrumenting the application will depend on the programming language and API being used. 

Generally, the process involves adding code to the application that tracks the execution of the code during the test. This code generates data on the parts of the application that were executed during the test, including which functions and lines of code were called. This data is then used to calculate a coverage percentage, which indicates how much of the application was tested.

For example, in Golang we can use the -cover flag while running our application, and for Python we could use the coverage module to do the same.

3. Run a fuzzing campaign

Now we have a specification and an API instrumented to measure coverage, the next step is to create test cases. It's essential to create test cases that cover both positive and negative scenarios. 

Although it is very important to write unit tests as part of the development work, elaborate system tests can require a lot of time and overlap with behavior already covered in unit tests. 

Often, there’s boiler plate code in an application that is tedious to unit test and offers low return on investment. You may still want to make sure that code works, or it may even be required of you in the industry you work in. 

Instead of spending a lot of development effort in doing that yourself, Mayhem’s fuzzing engine can create unique test cases that cover various scenarios: different inputs, outputs, and error conditions.

Mayhem creates and runs the test cases to learn to create new unique inputs based on the responses received from the API. Failing tests—tests that showcase the existence of an exploit such as SQL Injection, or cause an error on the server—are retained in a regression set. 

After the run, a report is provided as well, which provides detailed information about the test results, including the number of tests executed, how long test cases took, the number of tests passed, and the number of tests failed.

4. Analyze the results and coverage

The final step is to analyze the results of the test cases. The report generated by Mayhem provides true positives about issues that threaten the reliability of your API, which can be used to identify areas that require mending. By analyzing the results, you can identify defects and improve the overall quality of the API.

It is also essential to look at the coverage report that is generated as part of the fuzzing campaign. By comparing that to the coverage report from your unit tests, you can see which parts of the code Mayhem covers that may not be covered by your unit tests.

Coverage tools even allow you to combine reports so you can inspect if there’s any code that wasn’t covered by either your unit tests or the fuzzing campaign. At that point, you can decide to cover code that wasn’t hit by a unit test, remove the code because it’s unreachable, or schedule another, longer fuzzing job to try to hit that code.

In conclusion, Mayhem provides a powerful set of functionality for making sure your APIs are reliable. By following the tips outlined in this article, you can increase test coverage, reduce defects, and improve the overall quality of your software. With Mayhem, you can be confident that your API is thoroughly tested and meets the highest standards of quality.