The Mayhem for API Difference - A ZAP - Mayhem for API Scan Comparison

With the rapid development of modern web APIs, developers must balance quality, reliability and security with time to market. When approaching testing, developers will aim for covering the “happy path”, that is, testing that the API behaves correctly under a strict set of conditions. While happy path testing provides some degree of confidence, one can expect any number of issues once the API is exposed to consumers.

API developers can try to find these issues early with negative testing, but without sufficient planning, negative testing is often ignored. This means that there are any number of unknown issues of unknown severity hidden in every API that may lead to benign errors, or at worse, serious security holes.

There are many tools that can be used to to fill in this gap, by “shifting left” the testing of APIs, using inputs that are meant to elicit insecure or unexpected behavior. One of these tools is ForAllSecure’s Mayhem for API.

Mayhem for API automates testing REST APIs by bringing the full might of fuzzing methodology to API testing. With the guidance of an API specification, Mayhem for API provides accurate and informative test coverage tailored to any REST API.

Another tool in this space is OWASP ZAP. ZAP is an open-source web application security scanner that can be used by both those new to application security as well as professional penetration testers. ZAP API scan is a script packaged with ZAP Docker images tuned for performing active scans against APIs. It is tuned to APIs, so it doesn’t bother looking for things like XSS

Both tools share the same goal:  Autonomously scan APIs to find security and implementation issues

In order to know what tool is right for you, we have put together some examples to see how both work with an example vulnerable REST API. Let’s begin! 

Prerequisites

• Docker (with docker compose configured)
• A Mayhem for API account. Sign up for Free! 
• Mayhem for API cli, mapi, installed

The Target API

We have chosen VAmPI as our Target API.

VAmPI is a vulnerable API made with Flask and it includes vulnerabilities from the OWASP top 10 vulnerabilities for APIs.

It was created by GitHub user erev0s as a vulnerable API to evaluate the efficiency of tools used to detect security issues in APIs

Clone the VAmPI source from GitHub: https://github.com/erev0s/VAmPI

git clone https://github.com/erev0s/VAmPI.git 

docker build -t vampi_docker:latest -f VAmPI/Dockerfile VAmPI 

docker compose -f VAmPI/docker-compose.yaml up -d

curl --location --request POST "http://localhost:5002/users/v1/register" \
--header 'Content-Type: application/json' \
--data-raw '{
"email":"foo@example.com",
"username": "foo",
"password": "bar"
}'

curl --location --request POST "http://localhost:5002/users/v1/login" \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "foo",
"password": "bar"
}'

{
"auth_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NjIwNjkxNTIsImlhdCI6MTY2MjA2OTA5Miwic3ViIjoiZm9vIn0.7aB_94z7FmcGYNTaL67DW47Ht2WTBKlQ85eCbvmlBLM",
"message": "Successfully logged in.",
"status": "success"
}

export VAMPI_TOKEN="eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NjIwNjkxNTIsImlhdCI6MTY2MjA2OTA5Miwic3ViIjoiZm9vIn0.7aB_94z7FmcGYNTaL67DW47Ht2WTBKlQ85eCbvmlBLM"

mapi run vampi auto ./VAmPI/openapi_specs/openapi3.yml \
--ignore-endpoint "^GET /createdb$" \
--header-auth "Authorization: token ${VAMPI_TOKEN}" \
--url "http://localhost:5002" \
--html mapi-report.html

Note some of the options used:

•  --header-auth "Authorization: token eyJ0eX…

• This uses the API token from the login. When scanning an API, you will be able to uncover more issues if you can successfully authenticate with it.

• --ignore-endpoint "^GET /createdb$" \

• The /createdb  endpoint should really only be called at most once per run to pre-populate the database. If it is called by the scanner, the results will be flaky as it recreates the database on every call.

• --html mapi-report.html

• This will create an html report with the results of your run.

At the end of the scan, your results will appear:

-Issues-
RULE_ID SUMMARY METHOD PATH
sql-injection GET /users/v1/{username}
internal-server-error AttributeError GET /books/v1
internal-server-error sqlite3.IntegrityError POST /books/v1
internal-server-error AttributeError GET /books/v1/{book_title}
internal-server-error sqlite3.OperationalError GET /users/v1/{username} 

docker compose -f VAmPI/docker-compose.yaml down 

docker compose -f VAmPI/docker-compose.yaml up -d

curl --location --request POST "http://localhost:5002/users/v1/register" \
--header 'Content-Type: application/json' \
--data-raw '{
"email":"foo@example.com",
"username": "foo",
"password": "bar"
}'

curl --location --request POST "http://localhost:5002/users/v1/login" \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "foo",
"password": "bar"
}'

export VAMPI_TOKEN= &ltresult from previous request &gt

mkdir zap_working_dir

cp ./VAmPI/openapi_specs/openapi3.yml zap_working_dir

sed -i '' 's/localhost:5000/localhost:5002/g' zap_working_dir/openapi3.yml 

docker run --network=host -v $(pwd)/zap_working_dir:/zap/wrk \
-t owasp/zap2docker-weekly zap-api-scan.py \
-t "openapi3.yml" \
-f openapi -d \
-r zap-vampi.html \
-z "-config replacer.full_list\\(0\\).description=auth1
-config replacer.full_list\\(0\\).enabled=true
-config replacer.full_list\\(0\\).matchtype=REQ_HEADER
-config replacer.full_list\\(0\\).matchstr=Authorization
-config replacer.full_list\\(0\\).regex=false
-config replacer.full_list\\(0\\).replacement=token\\ ${VAMPI_TOKEN}
-config globalexcludeurl.url_list.url.regex='createdb$'"

Once the scan completes, you can find the results under ./zap_working_dir/zap-vampi.html.

Comparing the Tools

Results

We have collected the results from both tools and will dig into what was found by each. These results were obtained using the latest version of the software products as of September 1, 2022.

Severity

Issue Count

Description

ZAP

Mayhem for API

ZAP

Mayhem for API

High / Error

0

9

No high severity issues were found, including the SQL injection caught by Mayhem for API.

Mayhem for API discovered a SQL Injection and a number of internal server errors. The SQL Injection is a critical issue and should be resolved.

Internal Server Errors. While usually benign, as a result of an uncaught exception, these can lead to a poor experience for API consumers or indicate a deeper underlying issue.

Medium / Warning

2

9

Errors are triggered for missing CSP Header (which may be implemented at the load balancer rather than in the API) and Anti-CSRF token (which is more of an issue when cookies are involved – not with bearer token access).

Warnings range from issues with the API deviating from its OpenAPI specification, to timeouts on endpoints with performance that degrades after prolonged use. 

Low

5

N/A

Some low severity issues were found. These are mainly around  

web/application server is leaking version information via the "Server" HTTP response header and missing X-ContentType-Options Header.

Mayhem for API does not report issues below the Warning threshold.

Informational

2

N/A

This is mainly various 404 and 401 responses from the API

Execution

Results aren’t everything… both ZAP - API Scan and Mayhem for API are designed to be executed from the command line. Mayhem for API has been designed with an approachable command line user experience. Let’s compare how certain options, used in these examples, are expressed by each tool in the command line.  

Description

ZAP

Mayhem for API

Header Authentication

-z "-config replacer.full_list\\(0\\).description=auth1 -config replacer.full_list\\(0\\).enabled=true -config replacer.full_list\\(0\\).matchtype=REQ_HEADER -config replacer.full_list\\(0\\).matchstr=Authorization -config replacer.full_list\\(0\\).regex=false -config replacer.full_list\\(0\\).replacement=token\\ ${VAMPI_TOKEN}

The ‘-z’ option is used to pass configuration replacement values to the ZAP configuration. These arguments can be simplified by persisting a config file and passing it - at the cost of managing a config file. 

--header-auth "Authorization: token ${VAMPI_TOKEN}"

Header-based authentication is a first-class concept in Mayhem for API.

Specification Type

-f openapi

ZAP requires the specification type to be set up front

N/A

Mayhem for API will auto-detect your specification type, with support for swagger, OpenAPI, HAR and Postman Collections.

Specification Preprocessing

sed -i '' 's/localhost:5000/localhost:5002/g' zap_working_dir/openapi3.yml

ZAP requires that the openAPI spec server URL matches the target API. This means that the spec may need to be edited when running locally, against a different domain, or on a randomly assigned port. 

N/A

Mayhem for API will run against any URL provided by the ‘--url’ argument. If the specification is mostly valid, it will run!

Exclude/Include endpoints

-config globalexcludeurl.url_list.url.regex='createdb$'"

Config overrides can be used to exclude certain URLs from the specification with regular expressions. The specification itself can also be modified.

--ignore-endpoint "^GET /createdb$"

Mayhem for API makes it easy to configure exclude/include lists for endpoints and groups of endpoints (that use OpenAPI tags).

Conclusion

Both Mayhem for API and ZAP (with ZAP - API Scan) can be used to autonomously scan your REST APIs for quality and security issues. 

ZAP will help you to discover some inconsistencies in your API such as leaking server version information via HTTP Response Header, or returning Unexpected Content-Type. 

Mayhem for API will more consistently find high severity issues, such as SQL Injection - and helps you keep your specification honest with your implementation by identifying issues where the API is not behaving how it is defined in the spec. 

With command line options for setting header bearer token, endpoint exclusion/inclusion and no need to modify the OpenAPI spec URL prior to running - we believe that Mayhem for API is easier to get up and running against your APIs.

While we think that Mayhem for API is all you need for API scanning needs, we also understand the value provided by ZAP for organizations that are looking to cover all their bases. 

With Mayhem for API, you can include results from ZAP - API Scan in every run with the –zap command-line argument! 

mapi run … --zap

 mapi run … --zap_import_json_results <path-to-zap-results.json>

Try it yourself!

Don’t just take our word for it, try it yourself!

Test this VAmPI and your own APIs by heading over to Mayhem for API to get started for free and start fuzzing!