Towards Semi-automated Detection of Trigger-based Behavior for Software Security Assurance Dorottya Papp Levente Buttyán Zhendong Ma CrySyS Lab, Dept. of Networked CrySyS Lab, Dept. of Networked Center of Digital Safety and Security, Systems and Services, BME Systems and Services, BME Austrian Institute of Technology Magyar tudósok krt 2. Magyar tudósok krt 2. Donau-City-Straβe 1. Budapest 1117, Hungary Budapest 1117, Hungary Vienna, Austria Center of Digital Safety and Security, [email protected] [email protected] Austrian Institute of Technology Donau-City-Straβe 1. Vienna, Austria [email protected] ABSTRACT 1 INTRODUCTION A program exhibits trigger-based behavior if it performs undocu- Trigger-based behavior in software refers to execution of code that mented, often malicious, functions when the environmental condi- is activated on specific input values (the so called trigger inputs) tions and/or specific input values match some pre-specified criteria. and that performs some undocumented functions. Often, the users Checking whether such hidden functions exist in the program is are not aware of the existence of those undocumented functions important for increasing trustworthiness of software. In this paper, in the software. It is clear that using an application that has such we propose a framework to effectively detect trigger-based behav- hidden functions, which can be triggered by non-legitimate par- ior at the source code level. Our approach is semi-automated: We ties, is dangerous, as those functions can potentially implement use automated source code instrumentation and mixed concrete malicious actions. Indeed, the best examples for trigger-based be- and symbolic execution to generate potentially suspicious test cases havior include backdoors and logic bombs hidden in applications that may trigger hidden, potentially malicious functions. The test by malicious parties [25], although there are benign examples as cases must be investigated by a human analyst manually to decide well, such as easter eggs. Malware can also exhibit trigger-based which of them are real triggers. While our approach is not fully behavior [9, 17, 19]; however, in this paper, we are more interested automated, it greatly reduces manual work by allowing analysts to in trigger-based behavior in legitimate applications. focus on a few test cases found by our automated tools. The adversary model of trigger-based behavior assumes that the attacker has access to the software and is able to modify the CCS CONCEPTS implementation to implant the hidden, malicious behavior, which • Security and privacy → Malware and its mitigation; Soft- will be triggered during the execution of the software. In practice, ware security engineering; • Software and its engineering → the attacker may be a compromised developer or a compromised Operational analysis; contractor in the supply-chain trying to insert malicious code in the software, which can be exploited later in a targeted attack against KEYWORDS the user [6]. Or, the attacker may be a disgruntled employee at a company with access to the software used by that company [14]. Mixed Concrete and Symbolic Execution, Trigger-based Behavior, From a software security assurance perspective, the detection Static Analysis, Source Code Analysis, Software Security of trigger-based behavior in software is of paramount importance. ACM Reference format: Unfortunately, it is a difficult task. One approach is to try detecting Dorottya Papp, Levente Buttyán, and Zhendong Ma. 2017. Towards Semi- hidden functions by software testing. However, current software automated Detection of Trigger-based Behavior for Software Security Assur- testing approaches have limitations. Specification-based (or black- ance. In Proceedings of ARES ’17, Reggio Calabria, Italy, August 29-September box) testing techniques have no knowledge of the internal structure 01, 2017, 6 pages. of the software, therefore, they may only discover the hidden be- https://doi.org/10.1145/3098954.3105821 havior if the correct trigger inputs are specified. This limitation reduces their effectiveness because of the assumption that no one, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed but the attacker has knowledge of the trigger inputs. Structural for profit or commercial advantage and that copies bear this notice and the full citation (or white-box) testing, on the other hand, takes into account the on the first page. Copyrights for components of this work owned by others than the internal structure of the software which the hidden behavior is author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission part of. As such, white-box testing approaches can generate better and/or a fee. Request permissions from [email protected]. results. ARES ’17, August 29-September 01, 2017, Reggio Calabria, Italy White-box analysis can be performed at the binary level or at the © 2017 Copyright held by the owner/author(s). Publication rights licensed to Associa- tion for Computing Machinery. source code level. However, analyzing the binary can be difficult ACM ISBN 978-1-4503-5257-4/17/08...$15.00 due to the lack of semantic abstractions in low level code. And https://doi.org/10.1145/3098954.3105821 ARES ’17, August 29-September 01, 2017, Reggio Calabria, Italy D. Papp, L. Buttyán and Z. Ma even if one has access to the source code, detecting trigger-based correctly guess them. Considering the threat model, it is unreason- behavior by manual inspection can be tedious and error prone, able to assume that the tester knows the trigger inputs. In addition, as the human analyst has to interpret a potentially large amount the probability of correctly guessing the trigger inputs is low. of code written by someone else. Yet, today, manual source code White-box testing approaches, on the other hand, take into ac- analysis seems to be the approach that is used by practitioners count the internal structure of software. However, current auto- [1]. Although some researchers tried to automate the detection mated vulnerability-finding tools tend to focus on typical program- of trigger-based behavior, most of those works focus on binary ming vulnerabilities that are exploited by well-known and under- analysis of malware (see e.g., [2]), in which we are not interested stood attacks (e.g. buffer overflow). To assist analysts in creating test in this work. cases that cover trigger-based behavior as well, the SQA Tool [1] In this paper, we apply open-source and commercially available highlights code segments based on how much test-coverage would tools to effectively detect trigger-based behavior at the source code improve with their execution. This approach certainly improves level. We propose a framework which utilizes existing white-box the performance of human analysts, but the detection process is not analysis tools. Our approach is based on automated source code automated enough. Test cases must still be written by the analysts, instrumentation that makes the source code amenable to analysis which requires interpretation of the source code, as well as correctly by tools. Detection of trigger-based behavior is achieved by mixed determining of trigger inputs based on the source code. concrete and symbolic execution, which automatically outputs nu- Taking into account the internal structure of software is useful merous test cases for uncovered execution paths in the analyzed for the detection of trigger-based behavior, as shown in [8]. The au- software. Among those test cases, a small subset of potentially sus- thors modeled malicious behavior observed during execution, and picious test cases are automatically highlighted by our tools. Those were able to detect similar code segments in new malware samples. highlighted test cases must be investigated by a human analyst to However, this approach relies on already observed behavior and decide which of them could be a trigger input. Thus, our approach cannot detect new types of trigger-based behavior. is not fully automated, but it greatly reduces manual work by al- The ideal approach for detecting trigger-based behavior should lowing analysts to focus on the few highlighted test cases found by be able to: our automated tools. More specifically, our contributions are the following: (1) interpret how input values are handled, which is analogous to how input values modify the behavior of the analyzed (1) we present a framework for semi-automated detection of software, trigger-based behavior based on existing tools. To the best (2) automatically generate input values based on the interpre- of our knowledge, we are the first to show that existing tation, mixed concrete and symbolic execution tools can be suc- (3) identify conditions required to reach any part of the code, cessfully deployed for detecting trigger-based behavior; (4) detect suspicious conditions and decide whether the exe- (2) we demonstrate a proof-of-concept implementation of the cution path is malicious or benign framework for programs written in C using the LLVM toolchain; The first three requirements can be satisfied with symbolic exe- (3) we evaluate the results on real-world, open-source soft- cution [5], a technique that is able to calculate the constraints on ware samples. Our first result suggests that if mixed con- inputs such that execution takes a certain path (also known as the crete and symbolic execution succeeds, the hidden, mali- path condition). The technique assigns symbolic values to variables, cious behavior can be detected. that describe and track how the value of the variables depend on The paper is structured as follows. The challenges of detecting the input. At branches in the software, symbolic execution splits trigger-based behavior are discussed in Section ??, together with into two instances: in one instance, the symbolic variables are con- existing approaches to overcome those challenges.