CrashLocator: Locating Crashing Faults Based on Crash Stacks Rongxin Wu §, Hongyu Zhang †, Shing-Chi Cheung §, and Sunghun Kim § § Department of Computer Science and Engineering The Hong Kong University of Science and Technology, Hong Kong, China {wurongxin, scc, hunkim}@cse.ust.hk † Microsoft Research Beijing 100080, China [email protected] ABSTRACT crashed modules) at the time of crash, cluster similar crash reports Software crash is common. When a crash occurs, software devel- that are likely caused by the same fault into buckets (categories), opers can receive a report upon user permission. A crash report and present the crash information to developers for debugging. typically includes a call stack at the time of crash. An important Existing crash reporting systems [2, 14, 25] mostly focus on col- step of debugging a crash is to identify faulty functions, which is lecting and bucketing crash reports effectively. Although the col- often a tedious and labor-intensive task. In this paper, we propose lected crash information is useful for debugging, these systems do CrashLocator, a method to locate faulty functions using the crash not support automatic localization of crashing faults. As a result, stack information in crash reports. It deduces possible crash traces debugging for crashes requires non-trivial manual efforts. (the failing execution traces that lead to crash) by expanding the crash stack with functions in static call graph. It then calculates Over the years, various fault localization techniques (e.g., [1, 18, the suspiciousness of each function in the approximate crash trac- 21, 22]) have been proposed to help developers locate faults. es. The functions are then ranked by their suspiciousness scores These techniques suggest a list of suspicious program entities by and are recommended to developers for further investigation. We statistically analyzing both the passing and failing execution trac- evaluate our approach using real-world Mozilla crash data. The es of test cases. Developers can then examine the ranked list to results show that our approach is effective: we can locate 50.6%, locate faults. However, these fault localization techniques assume 63.7% and 67.5% of crashing faults by examining top 1, 5 and 10 the availability of complete information of failing and passing functions recommended by CrashLocator, respectively. Our ap- execution traces, while crash reports typically contain only the proach outperforms the conventional stack-only methods signifi- crash stacks dumped at the time of crashes. cantly. In this paper, we propose CrashLocator, a novel technique for Categories and Subject Descriptors locating crashing faults based on crash stacks and static analysis techniques. Our technique is targeting at locating faulty functions D.2.5 [ Software Engineering ]: Testing and Debugging- as functions are commonly used in unit testing and helpful for Debugging Aids. crash reproducing [5, 16]. For a widely-used system, one crashing General Terms fault could trigger a large number of crash reports. Therefore, a Measurement, Reliability sufficient number of crash stacks can be used by CrashLocator for crashing fault localization. Keywords CrashLocator expands crash stacks into approximate crash traces Crashing fault localization, crash stack, software crash, statistical (the failing execution traces that lead to crash) using static analy- debugging, software analytics. sis including call graph analysis, control flow analysis, and back- ward slicing. For effective fault localization, CrashLocator applies 1. INTRODUCTION the concept of term-weighting [24]: locating crashing faults is Software crashes are severe manifestations of software faults. treated as the problem of term weighting, i.e., calculating the im- Crashes are often required to be fixed with a high priority. Recent- portance of a function (term) for a bucket of crash traces (docu- ly, many crash reporting systems such as Windows Error Report- ments). CrashLocator considers several factors to weigh a func- ing [14], Apple Crash Reporter [2], and Mozilla Crash Reporter tion: the frequency of a function appearing in a bucket of crash [25] have been proposed and deployed. These systems automati- traces, the frequency of a function appearing in crash traces of cally collect relevant information (such as the crash stack and different buckets, the distance between a function and the crash point, and the size of a function. Using these factors, CrashLoca- tor calculates the suspiciousness score of each function in the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed approximate crash traces. Finally, it returns a ranked list of suspi- for profit or commercial advantage and that copies bear this notice and the full citation cious faulty functions to developers. The developers can examine on the first page. Copyrights for components of this work owned by others than ACM the top N returned functions to locate crashing faults. must be honored. Abstracting with credit is permitted. To copy otherwise, or republis h, to post on servers or to redistribute to lists, requires prior specific permission and/or a We evaluate CrashLocator using real crash data from three differ- fee. Request permissions from [email protected]. ent Mozilla products, Firefox, Thunderbird and SeaMonkey. The ISSTA'14, July 21 –25, 2014, San Jose, CA, USA . evaluation results are promising: using CrashLocator developers Copyright 2014 ACM 978-1-4503-2645-2/14/07... $15.00. can locate 50.6%, 63.7% and 67.5% of crashing faults by examin- Figure 2 is an example of crash stack in a Firefox crash report ing top 1, 5 and 10 functions in the returned ranked list, respec- (crash ID: f2f55573-e2cd-4ce9-92be-b16e72130904 ). The pro- tively. In addition, the evaluation results show that our approach gram crashed at Frame 0. Each frame contains a full-qualified outperforms the conventional stack-only approaches. function name and the source file position (source file name and line number). The main contributions of this paper are as follows: In large-scale systems such as Microsoft Windows and Mozilla- · We propose a novel technique for locating crashing faults. Firefox, developers receive a large number of crash reports sent Our framework is based on crash stacks only and does not by users at deployment sites. These crash reports are automatical- require additional information from deployment sites or ly grouped into different buckets, according to the crashed method program instrumentation. To our best knowledge, this is the signatures [25]. The grouping is based on the assumptions that a first time such a technique is proposed. crashing fault results in the same crashed method or similar · We implement our technique and evaluate our approach stacks. Ideally, each bucket should correspond to a unique crash- using Mozilla products, which are real and popular projects. ing fault. In this paper, our research goal is to locate the faulty function given a bucket of crash reports. The remainder of this paper is organized as follows: We introduce Frame 0 background information in Section 2. Section 3 introduces some Crash Signature nsWindow::ClearCachedResources() widget/windows/nsWindow.cpp:7821 nsWindow::ClearResourcesCallback(HWND__ *,long) observations based on our empirical study and formulates the Frame 1 crashing fault localization problem. Section 4 describes our ap- widget/windows/nsWindow.cpp:7812 proach. Section 5 presents our experimental design and Section 6 … Frame 63 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) shows the experimental results. We discuss issues involved in our ipc/glue/MessagePump.cpp:117 MessageLoop::RunHandler() approach in Section 7 and threats to validity in Section 8. Section Frame 64 9 surveys related work followed by the conclusion in Section 10. ipc/chromium/src/base/message_loop.cc:201 Frame 65 xul.dll@0x16d800 2. BACKGROUND Figure 2. A crash stack example. Despite immense efforts spent on software quality assurance, released software products are often shipped with faults. Some 3. CRASH LOCALIZATION PROBLEM faults manifest as crashes after software deployment. The crash information from deployment sites is useful for debugging crash- 3.1 Challenges for Locating Crashing Faults es. A crash report typically contains limited information such as crashed method signature and crash stacks. The execution context, including structural coverage, under which the fault was triggered, is unavailable. Localization of faulty functions based on such limited information is challenging. More specifically, we have identified the following two main challenges: incomplete crash trace and uncertain fault location. Crash Reporting System Incomplete crash trace Crash Information Crash stacks only contain partial crash (failing) traces and do not Software Crash with Crash Stack contain information of complete passing execution traces. There- fore, many conventional fault localization techniques cannot be applied. These fault localization techniques allow programmers to locate faults by examining a small portion of code [1, 18, 21, 22]. They contrast the passing execution traces with the failing ones, compute fault suspiciousness of individual program elements (such as statements and predicates), and present a list of program Developers Bug Reports Crash Buckets elements ranked by their fault suspiciousness. These techniques are not directly