Fending Off Cyber Attacks – Hardening ECUs by Fuzz Testing In designing vehicle communication networks, security test procedures play an important role in the development process. Fuzz testing, which originated in the traditional IT field, offers automatable tests which are used to efficiently check an ECU’s security-related capabilities. But how can test cases be generated using familiar exchange formats? Today’s motor vehicles are increasingly being integrated effects of a successful attack. Although development pro- into the Internet of Things (IoT). Vehicles are connecting to cesses for vehicle security have become better and are devices, other vehicles, infrastructure and cloud services via using increasingly more detailed threat models, there is still new interfaces. On the one hand, this enables new func- a gap in the development tools regarding the identification tions and business models. However, this added connectiv- and prevention of security vulnerabilities in existing soft- ity also opens more potential areas of attack related to ware. cyber security threats. The rapid introduction of new and In ECU tests (Figure 1), the focus is on validating functional complex functions harbors the risk that development requirements and error states within a defined scope – teams will hardly be able to keep up with the new attack which is limited by the boundaries of the selected parame- vectors that result and safeguard against them. ters. As soon as all functional requirements have been General guidelines for developing secure vehicle systems tested sufficiently, it is also necessary to work in the securi- are described in the Cybersecurity Guidebook for ty area with system inputs which are not part of the Cyber-Physical Vehicle Systems (J3061). It contains strate- “expected” parameters. Vulnerabilities lurk in this comple- gies and techniques for preventing and detecting security mentary set of inputs, and they are just waiting to be breaches, and it lists measures to mitigate the undesirable exploited. Due to time constraints, it is not possible to test 01 Technical Article / August 2018 all possible input values, especially when they depend on Compared with other security test methods such as pene- different states and conditions. Therefore, the goal of fuzz tration tests, this approach offers a good cost-benefit ratio. testing is not to test all the input parameters of a system Fuzzing engines can be categorized according to three completely, but instead to programmatically generate criteria: security tests that are more likely to identify implementa- > Generative engines and mutation engines: Generative tion errors. engines generate test data programmatically, while mutation engines generate test data by mutating What is Fuzz Testing and What Benefits Does It Offer? known valid data. The purpose of fuzz testing is to detect unknown vulnera- > “Simple” and “intelligent” fuzzing engines: In contrast to bilities by stressing a system with queries which are in part “simple” engines, intelligent engines are familiar with the random and quite inappropriate. This concept is not only structure of the data. However, one engine is not neces- applied in security tests, but in robustness tests in general. sarily better than the other. An entirely “simple” engine, Fuzz tests go far beyond the typical tests developers per- for example, can send fully random data to a system form to study all reasonably predictable manipulations. without requiring any knowledge of the system states or They involve generating additional test cases which ideally system parameters, like a data checksum in a message. come up with fully unexpected data. Randomly generated An entirely “intelligent” engine would follow the protocol input data that is applied over many test runs attains a and system specifications 100%, so it could potentially level of test coverage that is hardly attainable by manually never detect errors. A good middle path is to use a created tests. system that intelligently chooses which system parame- Since fuzz testing is the technique preferred by hackers for ters and logical approaches to subject to fuzz testing. revealing vulnerabilities, developers need an easy-to-use > The third classification criterion is knowledge about the extension for test tools which they can use to meet the system states and program structure. Here, the fuzz challenges of attackers head-on. Ideally, fuzz testing will tests check either a black box, white box or gray box support a method of test case generation that is largely system. For some system properties, a certain level of automated and can indicate problems “relatively” quickly. knowledge of the system state is essential. This applies Asset Definition Security Validation Threat and Risk Assessment Penetration Testing Derivation of Security Goals Fuzz Testing Security Architecture Design & Analysis Functional Security Testing Security Mechanisms Design & Analysis Secure Implementation of Nominal Functions and Security Mechanisms Figure 1: Fuzz Testing as part of the V-model for developing security-related ECUs 02 Technical Article / August 2018 to testing of diagnostic messages, for example. In UDS > A successful fuzz test solution must keep records of the (Unified Diagnostic Services), many functions and sub- test cases that are executed along with all relevant functions are protected by the 0x27 service, and possibly parameters. This lets testing engineers replicate the even by the knowledge of which session the diagnostic tests to compare results based on clearly defined test server is in. Without knowledge of this state machine, a cases and parameters. fuzzing engine would be unable to penetrate deeply into > The test tool should deliver reports that lend themselves the ECU’s code base. to implementing corrective actions as well as relevant summaries that present additional insights to test engi- Many fuzz test engines are designed for black box tests. neers. The Vector approach, on the other hand, aims to use proj- ect configuration data supplied by the tool chain so that it Currently, there are some promising smaller vehicle-specific can offer white box tests with knowledge of the network prototypes and IT-specific fuzzing solutions available. protocol and the specified device behavior. Furthermore, However, there is no standard automotive fuzzer for vehi- when software changes are made, developers can scale the cle tests which fulfills all the mentioned requirements. The generated test cases to adapt them to the scope of the signals in vehicle networks pose a real challenge in the changes. This significantly reduces the time required for development of fuzz tests. In generating inputs that can be tests in an incremental, continuous integration process. processed by ECUs, fuzzing tools which are popular in the IT sector cannot always fulfill requirements such as float- Requirements for Successful Fuzz Testing Solutions ing-point numbers, signed and unsigned N-bit integers, Based on own experiences and discussions with developers signal multiplexing and changing byte orders in the signal. of vehicle tests and security service providers, Vector has defined key requirements for successful fuzz testing solu- Concept for Automatically Generating and Executing tions in the automotive industry. Fuzz Tests > Automotive OEMs and suppliers generally have full How can the described requirements be fulfilled? The core access to databases in which the communication of the idea is to use vehicle communication databases to auto- entire vehicle network or of a subsystem is described. matically generate and execute the fuzz tests (Figure 2). Defined in these databases are regular messages, The overall concept consists of a system under test (SUT), expected value ranges, lengths, timing information and a communication database, a vehicle test framework and data types. These descriptions exist in standard data an engine for creating fuzz tests. The SUT might be a exchange formats such as DBC, LDF, FIBEX and AUTO- remaining bus simulation, an individual ECU, a complete SAR System Extract. The descriptions are an excellent subsystem of multiple ECUs or a complete vehicle system. source of information for configuring fuzz test genera- tors, which automatically generate test cases. Automotive Testing Framework Monitoring Network and Data Monitoring Communication Communication Visualization Database System under Test Logging Signal Selection Fuzz Testing Reporting Config. Generation Configuration Sending / Receiving Fuzz Testing Engine Fuzzed Messages Generate Fuzzed Signal Data Communication Channel Figure 2: Concept for automatically generated fuzz tests 03 Technical Article / August 2018 CANoe vTESTstudio Monitoring Network and Data Monitoring Communication Automotive Fuzz Test Configuration Visualization Communication System Database under Test Logging Signal Selection (SUT) Fuzz Test Generator Reporting Sending/Receiving Fuzz Test Unit Fuzz Tests Fuzzed Signals Communication Channel Figure 3: Fuzz testing with the standard CANoe tool A combination of simulated and real elements is also possi- prepares as a message that it sends to the SUT over the ble. The SUT is connected to the vehicle test framework via vehicle bus. Fuzz test execution is logged in such a way that one or more communication channels. all the test messages can be examined. Because the fuzz Vector uses the communication database as the central tests are integrated in CANoe, that tool’s internal logging supplier of input data for the vehicle