Fuzz Testing ROI Framework Executive Summary Fuzz testing maximizes application security testing with a fraction of time, cost, and resources. Get insight into the type of value you can expect from your fuzzer. A review of software security investments reveals that a majority of spending is in application testing solutions, such as static analysis, software composition analysis, and scanners. These conventional testing approaches, however, test known or common attack patterns, only addressing CVEs or CWEs. But what about the unknown vulnerabilities -- the weaknesses attackers often exploit? Fuzz testing is a technique where malformed inputs are sent to an application in hopes of triggering anomalous behavior. Anomalous behavior is usually a sign of an underlying vulnerability -- typically a zero-day. Fuzzing is a proven technique that maximizes defect detection with the least amount of time and resources. As a result, it not only buys organizations time and money, it also frees scarce technical resources from manual, mundane tasks and allows them to focus on strategic initiatives that require true expertise. This framework is a model for framing the way you evaluate the economic return of investing in fuzz testing or other comparable solutions. Organizations can also use this framework to help predict which fuzz testing solutions will offer the most value based on organizational needs. Solutions In this framework, we will evaluate four • Bootstrapped Continuous Fuzzing comparable security testing techniques. Bootstrapped continuous fuzzing refers to the practice of internally developing your own continuous fuzzer • Manual Penetration Testing utilizing open-source fuzzers such as AFL. AFL is a Penetration testing, also known as pentesting or coverage-guided generational fuzzer (also known as ethical hacking, refers to the practice of a person or guided fuzzing) and, unlike protocol fuzzers, it has persons simulating attacks against a software to almost AI-like capabilities built into its engine. When it identify weaknesses that could lead to exploits. sends a test case to a target, it is able to monitor the This is offered a service where application security target’s reaction. It, then, takes in the feedback to experts leverage both manual and dynamic testing influence the next set of test cases it generates solutions to conduct testing for a determined on-the-fly. Overtime, the test cases become amount of time. increasingly closer to probing at the application's weaknesses. Additionally, this means all test cases it • Protocol Fuzzers crafts are custom generated for your applications. Protocol fuzzing is a dynamic application security testing solution for negative testing. Think of • ForAllSecure Mayhem protocol fuzzers and all other fuzzers as a team of Mayhem is an advanced fuzz testing technique that penetration testers in machine form. Unlike a combines guided fuzzing with symbolic execution, a service, you can scale up and down the number of patented technology from 10 years of academic fuzzers based on demand. Protocol fuzzers rely on research at Carnegie Mellon University. You get all the a pre-defined library of grammar or file formats test capabilities of a coverage-guided generational fuzzer, cases. These pre-built test suites are crafted by a plus more! Like AFL, Mayhem is able to intelligently team of engineers tailored for specific applications generate custom test cases on-the-fly. Unlike AFL, in specific environments. Mayhem is able to overcome technical inefficiencies of guided fuzzing thanks to the ingenuity of symbolic execution. While guided fuzzers are intelligent, they are not perfect. Guided fuzzers rely on heuristics for Want to learn more? generating its input. While it is sufficient for the Download the What is Advanced short-term, it can impact your ROI in the long run. This is where symbolic execution comes in, tracing logical Fuzz Testing whitepaper or pathways through the executable code and therefore Mayhem solution brief. offering far greater greater code coverage. Mayhem produces a win-win situation to its users. ©2020 ForAllSecure 3 Product Operations We’ll start our analysis by addressing Remember: Engineers from these vendors must a contentious and multilayered topic: manually build the library of test suites based on RFCs. product licensing. Product licensing is Therefore, test suites for newer or uncommon protocols, an obvious cost, but it is a common such as 5G or Zigbee, are either unavailable or immature. misconception that it is the largest cost. Organizations that choose to build their own test suite, Below is a detailed walk through of may find it more costly and even impossible due to lack product cost for each solution. of technical expertise in the talent market. Manual Penetration Testing Bootstrapped Continuous Fuzzing Operation Costs Bootstrapping fuzzing is an alluring alternative, because Penetration testing has no direct product license open-source fuzzers, such as AFL, are available free of or operational cost. However, we urge readers charge. However, free is never free. Security engineers to consider how service costs can impact your with ClusterFuzz and OSS-Fuzz have disclosed that organization’s budget. while it is possible to bootstrap and operate these high-performance fuzzers in production, people often Recurring service costs are considered an operational underestimate the complexity of upstanding such expense (OpEx), while annual product licenses are solutions. Their comment echoes what we’ve observed considered a capital expense (CapEx). Depending in the market as well. Customers have cited to us that on your organization, acquiring OpEx budget may be one of the biggest oversights they made was not thinking more challenging than acquiring CapEx budget. The ahead to the ongoing maintenance cost of such a availability of OpEx budget is unpredictable, hinging complicated product. on company performance or quarterly financial reporting timelines. As a result, you will have to reflect Several brave ForAllSecure customers have attempted on whether security testing is something you would to bootstrapped their own continuous fuzzing solutions. consider a luxury or necessity. Some were successful in developing a minimum viable product (MVP) that was deployed into their Protocol Fuzzing Operation Costs organization. It even gained internal buy-in and traction. Ultimately, they eventually transitioned to ForAllSecure Protocol fuzzers charge on a per protocol basis. Our Mayhem because they realized that they had become a market research revealed that vendors offer roughly development organization for their bootstrapped fuzzing 32 protocols and files in a “standard” offering for solution -- deploying bug fixes and building new features decent, mid-level fuzzing. on an ongoing basis. Eventually, maintenance became a distraction from the larger application security vision for A critical consideration for those evaluating protocol the department. fuzzers is whether your tool of choice supports your desired protocol or file format. ©2020 ForAllSecure 4 ForAllSecure Mayhem ForAllSecure Mayhem is priced based on two factors: tier and number of cores. The appropriate tier for you will be determined by the features that you seek. The number of cores you use will be determined by the scale and speed you’d like out of your analysis engine. In short, the more computing power you place behind the fuzzing engine, the more effective your analysis runs will be. Vulnerability Assessment Now that you’ve made a purchase decision, what value can you expect during the vulnerability assessment process? Vulnerability management is described as the “cyclical practice of identifying, classifying, remediating, and mitigating vulnerabilities” in software. There are eight procedural steps for a single vulnerability assessment. The sections below walk through the effort involved in conducting each step with each solution. Vulnerability Management Cycle PREWORK ASSESS PRIORITIZE ACT • Determine scope • Report • Assign value • Remediate of program • Scan • Gauge exposure • Mitigate • Define roles and responsiilities • Identify assets • Add threat context • Accept risk • Select vulnerability assessment tools • Create and refine IMPROVE RE-ASSESS policy and SLAs • Eliminate underlying issues • Rescan • Identify asset context sources • Evolve process and SLA • Validate • Evaluate metrics ©2020 ForAllSecure 5 MANUAL PENETRATION PROTOCOL BOOTSTRAPPED FORALLSECURE TESTING FUZZING CONTINUOUS FUZZING MAYHEM STEP 1 When testing applications for the first time, attack surface analysis is the first and most critical task. Attack Surface Analysis Different from in-house tools, While the labor involved in attack surface analysis can be astonishing, it is a one-time, up-front cost per services require attack application. Automated solutions leverage existing configurations and attack surface analysis for surface analysis in every future assessments. service engagement -- meaning you will be paying the cost of doing attack surface analysis each time, regardless of whether the app has been tested before. Vulnerability analysis is the process of reducing security risks in applications. The purpose is to STEP 2 uncover security flaws within a target. The output of a vulnerability assessment is a list of defects Vulnerability Analysis suspected to have caused the software-under-test (SUT) to behaveunexpectedly. The vulnerability analysis step primarily focuses on configuration and running of the fuzzing solution(s).