Mayhem for API vs ZAP: The Difference

Mayhem for API provides developers with security, verification, and performance data before code gets deployed to help them build quality APIs faster.

Current API testing does not go deep enough, if at all, in probing performance and reliability. Mayhem for API brings fuzzing automation technology to the realm of API testing, allowing developers to find those hard-to-expose defects that only fuzzers are built to find and other API testing tools are not equipped to provide.

Mayhem for API finds four times as many high-risk defects as OWASP ZAP, an open source product. Also, time is money. Mayhem for API can find more issues, with fewer false positives, in less time than ZAP.  This is essential when integrating into CI/CD and providing a compelling developer experience.

Security

Mayhem for API

ZAP

Why You Need This

Intelligent API Exploration

✅

🚫

Mayhem's advanced AI takes the work out of exploring your API, API parameters, and sequence of requests. Mayhem's AI techniques include hill climbing, generational search, and power scheduling algorithms.

Business logic checking

✅

🚫

Mayhem has an extensible plugin infrastructure so you can add custom business logic checks to test as part of every scan. Mayhem has an extensible plugin infrastructure so you can add custom business logic checks to test as part of every scan.

Built-in rich SARIF reporting

✅

🚫

Mayhem creates a single SARIF report over all results. SARIF is a common language for security tools geared towards creating a common reporting framework, including Github CodeQL, Visual Studio, and others. The SARIF report can be created locally, so you can archive the report with your own artifacts tools and infrastructure.

Accuracy

Best

Good

Accuracy means you can have confidence bugs reported are real, have the most extensive checking possible, and that you do not waste time on false positives.

OWASP Top 10

Best

Good

Mayhem has best-in-class API support for finding, prioritizing, and filtering false positives for the OWASP Top 10.

API Authentication

✅

✅

Mayhem supports Basic Auth, Bearer Tokens, Cookie Authentication, and custom authentication methods via an extensive plugin system.

REST API

✅

✅

Mayhem handles all REST verbs, and finds novel sequences of verbs to trigger previously unknown bugs.

GRPC Scanning

✅

✅

Mayhem supports your gRPC servers via a gRPC gateway.

Verification

Mayhem for API

ZAP

Why You Need This

Test Coverage Level (TCL)/Specification Contract Coverage

✅

🚫

Find out where your spec is wrong. Verify and validate that your API implementation meets your specification so that downstream consumers can program against it with confidence. Mayhem for API is the only tool of its kind to reach (TCL) Test Coverage Level 7.

Test coverage metrics

Positive and Negative

Neg only

Have confidence Mayhem results are meaningful to your application, and not just junk API requests. Mayhem verifies that your API is behaving in accordance with your API specification by testing valid, invalid and malicious requests. Mayhem does not require instrumentation, and learns how to elicit 2xx responses in addition to checking for issues.

JUnit reporting

✅

🚫

Mayhem hits every endpoint in your spec. In addition, the Mayhem coverage report breaks down the positive and negative API specification coverage. The JUnit report shows you the TCL/Contract coverage of how many API endpoints we could successfully interact with (receive 2xx response) so that you can tell no bugs in successful interactions from no bugs but no interaction was successful.

Performance

Mayhem for API

ZAP

Why You Need This

Endpoint P50/90/99 latency metrics

✅

🚫

Mayhem measures P50, P90, and P99 response latency to make sure your code meets your requirements, and code changes do not hurt performance. Almost all organizations expect a latency of less than 50 milliseconds (ms), while realtime APIs require a latency of 30ms or less. All latency measurements are available in our well-documented API for integration with your own tools, such as Excel, Tableau, and DevOps dashboards.

Incomplete responses

✅

🚫

Mayhem measures incomplete responses, which happen when the server closes the connection before the body is sent, e.g., the response length is 500 bytes but less than 500 bytes are sent. Incomplete responses trigger additional requests from clients which can slow down your servers.

Timeout measurements

✅

🚫

Mayhem finds endpoints that can cause user request timeouts that could cause user degredation or denial of service vulnerabilities.

Supported Specs and Third Party Tools

Mayhem for API

ZAP

Why You Need This

OpenAPI Support

All Current Versions, including v3.1, v3.0, v2.0

Only v3.0/2.0

Mayhem accepts any OpenAPI specification, including the latest v3.1 revision. 3.1 was released in February 2021. 3.0 was released in 2017. See https://OpenAPI.tools for more information.

Built-in Postman Collections support

✅

🚫

You can use your existing postman collection specs within Mayhem out-of-the-box.

Native HAR to OpenAPI conversion

✅

🚫

You can use HARs with Mayhem out-of-the-box so you do not need to run complicated scripts to convert your current HAR files to OpenAPI.

ZAP Integration and Noise Cancelation

✅

N/A

Mayhem will run ZAP in addition to Mayhem logic so you see what other pentesters will see. Mayhem even integrates all ZAP output into a single report, and filters out meaningless noise reports by default.

Developer Productivity

Mayhem for API

ZAP

Why You Need This

Noise cancelation and finding de-duplication

Mayhem bucketizes findings so you can focus on core issues and don't waste time with hundreds of reports for the same finding.

Configuration as Code

✅

🚫

Mayhem configuration is stored as a YAML file so you can check in and revision control it with the code you are testing.

Scriptable API

✅

🚫

Need to interact with results in a new way? The Mayhem server has it's own API for you to interface with.

Stand-alone native executable

✅

🚫

Mayhem runs as native code, is a single stand-alone executable, and does not add additional dependencies and risk such as requiring Java. Mayhem is written in Rust so it's type-safe with blazing-fast compiled code performance.

Stacktrace parsing

✅

🚫

Mayhem reports include stacktraces (when enabled on the server) so that your team more quickly identify what lines of code are at fault.

Local artifact support for HTML and SARIF reports

✅

🚫

You can store all results from Mayhem in a local artifact repository.

Web UI

✅

🚫

Share results with your organization and view results on any device with a web browser.

Reproduce with curl

✅

🚫

Mayhem tells you how to reproduce results with a CURL command so you can reproduce findings on your own.

Visual Studio Integration

✅

🚫

Enable developers to reproduce previously unseen findings on their local environment for a faster dev-loop/bug-fix experience. Integrate it with your favorite editor through SARIF/Junit reports and fix security issues without ever leaving Visual Studio. Integration built by Microsoft.

Performance

Best

Good

Mayhem is up to 5x faster by making more requests in a smaller amount of time than competitors. Mayhem is written in Rust so it's type-safe with blazing-fast compiled code performance.

Robust OpenAPI Specification Parsing

Best

Good

Mayhem supports loose parsing of OpenAPI specifications so that small errors in your spec do not prevent Mayhem from running.

Continuous Integration and Testing

✅

✅

Mayhem's github action allows you to trigger a MAPI on every build so you find issues as quickly during development as possible.MAPI is available as a dockerized app so that you can run it anywhere, and more securely.

Dockerize application

✅

✅

Mayhem is available as a dockerized app so that you can run it anywhere, and more securely.

Enterprise

Mayhem for API

ZAP

Why You Need This

Unlimited Scans (Free: 50 scans/month)

✅

🚫

Unlimited Users

✅

N/A

Single-Sign On Support

✅

🚫

Github, local signup, and integration with your enterprise SAML, OpenID, or OAuth provider. Other providers supported upon request

LDAP and Active Directory Authentication Support

✅

🚫

Enterprise customers can integrate Mayhem with their existing LDAP and AD integrations.

User and Organizations

✅

🚫

Mayhem allows anyone to create organizations and invite users to those organizations.

Team and Group Access Control and Authorization

✅

🚫

Reflect your organizational access control. Coming September 2022.

24x7 Professional Support

✅

🚫

Mayhem has online support and live discord discussion with the product engineering team.

Online community

✅

🚫

Robust community to find other like-minded organizations and users.

Regular releases with new features and bug fixes

✅

🚫

Mayhem is backed up a team of engineers so you don't have to.

Private Cloud

✅

N/A

Ask us! Requires internet connectivity.

Build Reliable APIs.

‍

Find out how ForAllSecure can ensure the quality of your APIs with autonomous fuzz testing.

Free Trial Learn More

Benchmarks 

Note: tests performed against VAmPI with default settings.

Mayhem for API

ZAP

Why You Need This

High Risk Unique Findings Count

4 unique, 100% TP, 0% FP

0 unique, 0% TP, 0% FP

Mayhem has a 100% trust positives and 0% false positive rate. ZAP had 0 findings (0% true positive rate), and StackHawk has a 0% true positive rate and 100% false positive rate.

Medium Risk Findings Count

0

0

Equal

Low Risk Findings Count

11 unique, 100% TP, 0% FP

2 unique, 100% TP, 0% FP

Mayhem finds more issues, and more relevant issues noting places the spec does not match the implementation. 

Info Risk Count

1 unique, 100% TP, 0% FP

1 unique, 100% TP, 0% FP

At ForAllSecure, we are committed to delivering innovative testing solutions that address the demands of modern development trends including CI/CD, DevSecOps, Agile, and more. We’d love to show you what our latest addition to Mayhem can do for you. To schedule personalized 1:1 time with our team of security experts, request a demo here. 

‍