A review of software security investments reveals that a majority of spending is in application testing solutions, such as static analysis, software composition analysis, and scanners. 

These conventional testing approaches, however, test only known or common attack patterns, only addressing CVEs or CWEs. But what about the unknown vulnerabilities -- the weaknesses attackers often exploit?

‍

Fuzz Testing Finds Unknown Vulnerabilities

Over the years, security testing has evolved from basic static analysis and manual penetration testing to dynamic application security testing methods like fuzz testing. Traditional methods focused mainly on addressing known vulnerabilities. However, as applications have become more complex, attackers are exploiting previously unknown vulnerabilities (zero-day threats), which cannot be detected by conventional methods. 

‍

What is fuzz testing?

Fuzz testing is a technique where malformed inputs are sent to an application in hopes of triggering anomalous behavior. Anomalous behavior is usually a sign of an underlying vulnerability -- typically a zero-day. Fuzzing is a proven technique that maximizes defect detection with the least amount of time and resources. As a result, it not only saves organizations time and money but also enables technical teams to focus on strategic, high-value initiatives rather than manual, repetitive tasks.

‍

Comparing The Operating Costs of 3 Common AppSec Testing Solutions: Is Fuzz Testing a Better ROI?

In this blog post, we’ll explore the operating costs of three common AppSec testing solutions. This framework can also help organizations predict which testing solution will deliver the most value based on their unique needs. 

While product licensing is a common consideration when choosing a solution, it’s a mistake to assume it’s the largest cost. Below, we’ll break down the cost structure for each solution.

‍

1. Manual Penetration Testing Operation Costs

Penetration testing has no direct product license or operational cost. However, we urge readers to consider how service costs can impact your organization’s budget.

Recurring service costs are considered an operational expense (OpEx), while annual product licenses are considered a capital expense (CapEx). Depending on your organization, acquiring an OpEx budget may be more challenging than acquiring a CapEx budget. 

The availability of OpEx budget is unpredictable, hinging on company performance or quarterly financial reporting timelines. This raises a critical question: is security testing a luxury or a necessity for your organization? 

‍

2. Protocol Fuzzing Operation Costs

Protocol fuzzers charge on a per protocol basis. Market research shows that most vendors offer around 32 protocols and file formats in a “standard” package suitable for mid-level fuzzing.

One key consideration when evaluating protocol fuzzers is ensuring your chosen tool supports the specific protocols or file formats your organization uses. Remember: Engineers from these vendors must manually build the library of test suites based on RFCs. Therefore, test suites for newer or uncommon protocols, such as 5G or Zigbee, are either unavailable or immature. 

Organizations that choose to build their own test suite may find it more costly and even impossible due to lack of technical expertise in the talent market.

‍

3. Bootstrapped Continuous Fuzzing Operation Costs

Bootstrapping fuzzing is an alluring alternative, because open-source fuzzers, such as AFL, are available free of charge. However, free solutions often come with hidden costs. Security engineers familiar with tools like ClusterFuzz and OSS-Fuzz have disclosed that while it is possible to bootstrap and operate these high-performance fuzzers in production, people often underestimate the complexity of maintaining these solutions in production environments. 

This comment echoes what we’ve observed in the market as well. Many of our customers have shared that one of their biggest missteps was underestimating the ongoing maintenance demands of a bootstrapped fuzzing solution. 

Several Mayhem customers previously bootstrapped their own continuous fuzzing solutions. Some were successful in developing a minimum viable product (MVP) that was deployed into their organization and gained internal buy-in and traction. 

However, they eventually transitioned to Mayhem because they realized that they had become a development organization for their bootstrapped fuzzing solutionMany customers have shared that one of their biggest missteps was underestimating the ongoing maintenance demands of a bootstrapped fuzzing solution—deploying bug fixes and building new features on an ongoing basis. Eventually, maintenance became a distraction from the larger application security vision for the department.

‍

Fuzz Testing With Mayhem

Mayhem is priced based on two factors: tier and number of cores. The appropriate tier for your organization depends on the features you require, while the number of cores dictates the scale and speed of your analysis. Simply put, more computing power enables faster, more comprehensive fuzzing runs.

‍

Mayhem’s ROI Goes Beyond Fuzz Testing

Mayhem offers advanced fuzz testing and goes beyond with a suite of intelligent testing technologies, such as generative AI, machine learning, symbolic execution, and behavioral analysis

• Intelligent Analysis: Mayhem creates and executes behavior-driven tests specifically designed for your applications and APIs to uncover hard-to-find vulnerabilities
• No false positives: Unlike other tools, Mayhem executes your code, ensuring that every vulnerability found is real.

• Maximize Test Coverage: self-learning behavior testing improves code coverage across your applications
• Accurate Supply Chain Risk: See which third-party components you actually use and what can be removed from your codebase.
• Automated Regression Testing: Mayhem builds and runs regression tests that verify application behavior as updates are shipped.
• Automated Triage: Every result comes with proof and reproductions, so you can skip triage and go right to remediation.

‍

This advanced approach maximizes ROI, optimizes resources, and accelerates time to market.

‍

‍