The Evolution of Security Testing

Security needs to be part of the development experience. This has given rise to the application security space. And, like the internet itself, it needs to evolve. AppSec 1.0 was built with waterfall methodologies in mind. These include static analysis software testing and penetration testing, which assume that security is binary. You are either secure or insecure, there is no grey area. Yet it is in the grey areas, the zero days, where the attackers lurk.

It then becomes a question of code coverage: Is your application security solution providing protection for your organization?

Likely not. A benchmarking study by the NSA Center for Assured Software found that the average SAST tool covers only 8 out of 13 weakness classes and finds only 22 percent of flaws in each weakness class. Based on these numbers, the average SAST tool is likely to find only 14 percent of the vulnerabilities in an application’s code.

Verification and validation testing enable organizations to address the grey area and find zero-days. Verification and validation testing is conducted in two different ways:

• Positive Testing, or functional testing, is a testing process where an application is sent a valid set of inputs. The purpose of positive testing is to ensure the application behaves as expected. While this type of testing is typically conducted by QA teams, modern collaborate closely with security or development teams.
• Negative Testing, or non-functional testing, is a testing process where an application is sent an invalid set of inputs. The purpose of negative testing is to ensure the application remains stable in unexpected use cases. While this type of testing is typically conducted by security teams, modern DevOps shops may collaborate closely with QA or development teams.

Positive testing is easier to conduct. There is a finite number of features and flows introduced per release. Therefore, there is a finite combination of valid inputs to test. Thus, there is a clear definition of what “done” means. Security is, however, a game of cat and mouse. The goal is to always trying to stay one step ahead of the attacker. And when you are trying to stay ahead of something, speed (and agility) is everything. So negative testing is necessary.

Instead of looking at code as static, as either secure or insecure, AppSec 2.0 security testing understands the developer and attacker mindsets by identifying coding mistakes early and often. This is where fuzz testing or AppSec 2.0 plays a vital role.

What is Fuzz Testing?

Fuzz testing, or fuzzing, is a dynamic Application Security Testing (DAST) technique for negative testing. Fuzzing aims to detect known, unknown, and zero-day vulnerabilities. Fuzzers send malformed inputs to targets. Their objective is to trigger bad behaviors, such as crashes, infinite loops, and/or memory leaks. These anomalous behaviors are vulnerabilities.

Fuzzing helps organizations verify that the application works as expected, even in unexpected situations. This is key as ecosystems get complex. It’s not just about people mistreating applications, it’s also about how an application will react if an integrated app misbehaves. In other words, if a system connected to your app acts up, can the app still function? Or will it crash? And if it crashes, does that allow for malicious code to run instead?

At a high level, fuzzing provides predictability. If testing is done continuously during the development cycle, this decreases time to market and should reduce the costs associated with the application over its lifetime. There will be a lower number of in-field issues when properly tested first. Fuzzing also contributes to productivity. It’s a security test solution that protects developer productivity with zero false positives. Zero. The crashes that are reported are indeed reproducible vulnerabilities, allowing developers to address them quickly.

As the issues are shared and fixed, regression testing remembers the previous test crashes and verifies the remediations.

Fuzz testing is a heavy-weight yet versatile DAST solution that is able to conduct multiple types of testing across the SDLC. It runs quietly, continuously, and synchronously in the background as a part of the build process. It’s also proven technology. Google, for example, identifies 80% of bugs with fuzz testing while the other remaining 20% is found through other means (SCA) or in production.

Continuous Testing with Advanced Fuzz Testing

There are different types of fuzzers available.

Random fuzzers are just that: random. It’s hard to tell how much code coverage is tested.

Template takes known good input and mutates it. Again, it’s hard to tell how much code coverage there is.

Guided fuzzing provides a roadmap. It’s more targeted. With guided fuzzing, the code coverage increases to autonomously map out the usable parts of the application for testing. When paired with symbolic execution, this technique becomes more powerful. Symbolic execution takes binaries and mathematically reasons through various logic and functions, so it can break into new areas of the program for further testing. We call the synthesis of these two techniques Advanced Fuzz Testing (AFT).

Advanced Fuzz Testing is particularly effective for continuous testing because it aims to protect developer productivity at all costs. How? With autonomous test case management. Advanced Fuzz Testing works by using symbolic execution to identify how it can break through functions. Then, it uses guided fuzzing to craft test cases to test each branch. It uses the application's behavioral feedback to inform what test cases to generate next. If it identifies minor anomalous behavior, it will continue to probe by generating alternative variations.

Download: Guide To Automated Continuous Security Testing

See how continuous testing enables security teams to keep pace with development and operations teams in modern development, and to deliver deep integration and automation of security tooling.

Download the Whitepaper More Resources

This means better and more thorough testing and coverage. This is a significant breakthrough from conducting coverage-guided fuzzing alone. Guided fuzzers can be powerful with guidance from a technical security expert to help inform the fuzzer how to traverse through the code. However, without manual intervention, it will continue blindly, guessing inputs to break through functions, which can result in breaking the code or the fuzzer being stuck for a while. 

The Value of AFT to the Developer

For individual developers, there’s component testing. As AFT identifies issues, developers are notified that there are issues. AFT has a zero false-positive rate and to prove it, it shares a test case also known as a witness to reproduce the issue. They also provide system level information that will help developers understand the consequence (from a technical standpoint) to the application. AFT runs deduplication. Often, a single defect can be triggered by multiple test cases. AFT would reduce these to one test case. The point of this is to increase testing efficiency.

The Value of AFT to the Development Team

For development teams, there’s regression testing. Generally, each developer owns a component of the larger application. Ideally, when the components are assembled, they’ll work together seamlessly. This is hardly the reality. Google cites that 45% of bugs that they identify are through regression testing. AFT compiles the test case from each component for regression testing. This is where the dedup effort at the component level pays off in dividends because the regression testing time will be reduced through efficiency.

The Value of AFT to the Security Team

Fuzzing provides a proactive approach to security testing. It is called negative or non-functional testing. It shows whether or not an application can withstand unexpected situations, and it helps uncover zero days. One way to think about (and justify) Advanced Fuzz Testing is that it is like penetration testing in machine form. Like pen testing, Advanced Fuzz Testing thinks outside the box. However, there are benefits to Advanced Fuzz Testing not found with pen testing. Unlike pen testing, Advanced Fuzz Testing is continuous, not just a point in time. It can be performed at machine scale, speed, and accuracy. As a result, this technique offers more coverage than what a human is capable of doing.

The Value of Advanced Fuzz Testing

• AFT provides software predictability

• AFT provides better and more thorough testing and coverage

• AFT provides a lower number of in-field issues when properly tested first

• AFT provides regression testing, so you know that something is fixed

• AFT demonstrates an application can withstand unexpected situations

• AFT uncovers more zero days than other AST

Learn more by downloading our Guide To Automated Continuous Security Testing