Control-Flow Recovery from Partial Failure Reports Peter Ohmann Alexander Brooks Loris D’Antoni Ben Liblit University of Wisconsin–Madison, USA {ohmann,albrooks,loris,liblit}@cs.wisc.edu Abstract which necessarily results in an incomplete picture of any Debugging is difficult. When software fails in production, de- failing execution. bugging is even harder, as failure reports usually provide only Nevertheless, obtaining answers to questions about the an incomplete picture of the failing execution. We present a failing run is a key part of debugging. LaToza and Myers [22] system that answers control-flow queries posed by developers find that developers commonly ask reachability questions, as formal languages, indicating whether the query expresses often based entirely on the program’s control flow. For control flow that is possible or impossible for a given failure example, developers asked about possible transitive function report. We consider three separate approaches that trade off callers/callees and possible calling contexts. Additional precision, expressiveness for failure constraints, and scalabil- cases might arise when debugging deployed applications. ity. We also introduce a new subclass of regular languages, If developers suspect missed initialization, they might ask, “Is the unreliable trace languages, which are particularly suited it possible that init() did not run?” or, “Could x have been used to answering control-flow queries in polynomial time. Our here before being initialized there?” Developers might also system answers queries remarkably efficiently when we en- ask whether an application has set the appropriate privilege code failure constraints and user queries entirely as unreliable level, locked appropriate resources, or opened appropriate trace languages. streams. Each of these queries may consider operations across all of execution or only in specific stack contexts. Some of CCS Concepts • Software and its engineering ! Soft- these questions may be interactive (e.g., during an active ware testing and debugging; Software post-development debugging session), and require very fast answers. Others issues; Software performance; • Theory of computation ! may be non-interactive (e.g., a batch analysis job that runs Formal languages and automata theory; Regular languages overnight), and allow time for more precise answers. In this paper, we formalize and evaluate analysis techniques Keywords Deployed software, failure reporting that cope with incomplete data from post-deployment failure reports. Our system allows developers to ask yes/no control- 1. Introduction flow questions (e.g., may a particular statement have executed When programs crash at user sites, debugging is difficult on the failing run?), and provides answers based on failure and detailed execution information is very valuable. However, report data. We consider three different underlying solvers deployed applications usually cannot gather full traces of run- that present trade-offs in expressiveness for failure constraints, time information, and instead leave behind failure reports with precision in analysis results, and scalability. varying detail. For example, nearly all failure reports contain a This work substantially extends that of Ohmann et al. [30], stack trace from the failing execution, but most do not contain who extracted best-effort program coverage data given a trace of all executed statements. Developers must weigh possibly-incomplete failure report data of a very limited class. run-time overhead (including time, memory, and disk space) We show that the failure data considered in that work actually against the benefits of traced data for postmortem failure forms a special subclass of regular languages (which we title analysis. Prior approaches to post-deployment monitoring [4, the unreliable trace languages; see section 4), and we present 9, 17, 28, 33, 34, 41] routinely give up perfect information, an algorithm that can answer the queries posed by Ohmann et al. much more efficiently. We also define encodings for a much wider range of failure constraints, and allow more Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and precise analysis by ensuring calling-context sensitivity (via the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, our solver based on visibly-pushdown languages [2]). requires prior specific permission and/or a fee. Request permissions from [email protected]. After providing a motivating example and describing our PLDI’17 June 18–23, 2017, Barcelona, Spain © 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM. goal of encoding failure reports to answer user queries at a ISBN 978-1-4503-4988-8/17/06. $15.00 high level (section 2), we discuss our primary contributions: DOI: http://dx.doi.org/10.1145/3062341.3062368 Reprinted from PLDI’17, Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation, June 18–23, 2017, Barcelona, Spain, pp. 390–405. 390 Program Failure Report main() { foo() { if (/∗ ..A.. ∗/){ if (/∗ ..H.. ∗/){ Stack Trace Message Log A H N at bar:27 2016-11-15 03:41 "Processed" /∗ ..B.. ∗/ /∗ ..I.. ∗/ Cfoo } else { } else { J at foo:18 /∗ ..C.. ∗/ /∗ ..J.. ∗/ B C I J C at main:5 J foo(); bar(); bar /∗ ..D.. ∗/ /∗ ..K.. ∗/ D K C Intersection Emptiness Test } } foo /∗ ..E.. ∗/ /∗ ..L.. ∗/ E L /∗ ..F.. ∗/ } Query bar(); F M Answer Fbar /∗ ..G.. ∗/ bar() { Possible or Impossible May statement E J bar } /∗ ..M.. ∗/ G N have executed? Fbar /∗ ..N.. ∗/ } Figure 1. Overall system architecture and concrete example. Example inputs consisting of a program, failure report, and query flow into one of our main analysis algorithms , yielding an answer consisting of a single possible/impossible judgment . 1. We formally define the problem of answering control-flow The example failure report contains two elements: a failing queries from failure reports, and present a new system for stack trace (the program crashed at statement N in function answering these queries based on language intersection bar, which was called from function foo at statement J, etc.), emptiness (section 3). and a single log message from the failing run. For the purposes 2. We introduce a new class of subregular languages, the of this example, assume that we have statically determined unreliable trace languages, which, though unable to that the message “Processed” could only be printed by either precisely express some failure report elements, are ideally statement B or C. We discuss how to encode these and other suited to answering queries in polynomial time (section 4). common failure report elements in section 5. The lower right corner of fig. 1 shows a user query as prose. 3. We describe encodings for various common features of We formally define our class of queries in section 3.3. In brief, failure reports, including stack traces, program coverage we allow any control-flow query that can be expressed as a data, and call traces (section 5). visibly-pushdown language [2]. Our system answers the query 4. We perform an extensive evaluation of our techniques on with respect to the provided CFG and failure report. That is, a wide variety of failing applications (section 6). We can queries implicitly begin with: “On at least one path through answer queries remarkably efficiently when we encode the CFG consistent with the failure report, . .” failure constraints entirely as unreliable trace languages. In each of our approaches, we first encode the CFG, each failure report element, and the query as formal languages Section 7 positions our approaches with respect to related whose member strings correspond to sequences of statements work, and section 8 concludes. and call/return edge labels from the CFG. If the intersection 2. Problem and Motivating Example of all of these languages is non-empty, then there exists such a sequence that (1) could occur in the CFG, (2) is consistent This section describes the structure of our system using the with the entire failure report, and (3) satisfies the query. Thus, illustrative example given in fig. 1. Section 3 defines each we respond that the query is Possible. Otherwise, we answer shown element more formally. Impossible. An imprecise solver may always answer Possible The left side of fig. 1 shows a source code skeleton and to a query. In fig. 1, a precise solver should answer Impossible, its corresponding control-flow graph (CFG). Here, each node as there is no path through the CFG that passes through node refers to a statement in the program, solid lines refer to E before resulting in the final crashing stack. The example intraprocedural control flow, dotted lines indicate calls, and contains neither loops nor recursion, and statement E must dashed lines denote returns. Call and return edges are labeled always follow the return from the in-progress call to foo seen with the call site and the called procedure. The CFG includes in the stack trace. Thus, a precise solver will ensure that the three procedures: main consists of nodes fA; B;C; D; E; F;Gg, language of the CFG respects matching labels on call and foo consists of fH; I; J; K; Lg, and bar consists of fM; Ng. return edges (simulating a program stack). Upon receiving a failure report like the one in the top right Our three approaches vary in the machinery used to of fig. 1, we allow a developer to pose queries like the one implement this high-level strategy. Specifically, the languages shown in the bottom right of fig. 1. 391 for each element are encoded differently by each technique, Informally, a valid context-sensitive path only allows return and, therefore, the methods of computing intersections and edges to be triggered for the last unmatched function call.