GoTcha: An Interactive Debugger for GoT-Based Distributed Systems Rohan Achar Pritha Dawn Cristina V. Lopes Donald Bren School of ICS Donald Bren School of ICS Donald Bren School of ICS University of California, Irvine University of California, Irvine University of California, Irvine Irvine, CA, USA Irvine, CA, USA Irvine, CA, USA [email protected] [email protected] [email protected] Abstract Debugging distributed systems is hard. Most of the tech- niques that have been developed for debugging such systems use either extensive model checking, or postmortem analysis of logs and traces. Interactive debugging is typically a tool that is only effective in single threaded and single process applications, and is rarely applied to distributed systems. Figure 1. Information propagation in single-thread systems. While the live observation of state changes using interactive debuggers is effective, it comes with a host of problems indis- tributed scenarios. In this paper, we discuss the requirements an interactive debugger for distributed systems should meet, the role the underlying distributed model plays in facilitat- ing the debugger, and the implementation of our interactive debugger: GoTcha. GoTcha is a browser based interactive debugger for dis- tributed systems built on the Global Object Tracker (GoT) programming model. We show how the GoT model facili- tates the debugger, and the features that the debugger can offer. We also demonstrate a typical debugging workflow. Keywords Debugging Distributed Systems, Interactive De- bugging Figure 2. Information propagation in distributed systems. 1 Introduction None of these tools are interactive debuggers. Interactive Debugging faults and errors in distributed systems is hard. debuggers in a single threaded application context are pow- There are many reasons for the inherent difficulty, and these erful tools that can be used to pause the application at a have been extensively discussed in the literature [Beschast- predetermined predicate (often a line of code in the form of nikh et al. 2016; Schütz 1994; Valadares et al. 2016]. In partic- a breakpoint) and observe the state of the variables and the ular, the non determinism of concurrent computation and system at that point. They can observe the errors as they arXiv:1909.03167v1 [cs.DC] 7 Sep 2019 communication makes it hard to reliably observe error con- happen, and can be quite effective in determining the cause. ditions and failures that expose the bugs in the code. The Controls are often provided to execute each line of code in- act of debugging, itself, may change the conditions in the teractively and observe the change of state after each step. system to hide real errors or expose unrealistic ones. Many modern breakpoint debuggers provide the ability to To mitigate the effects of these difficulties, several ap- roll back the execution to a line that was already executed. proaches have been developed. These approaches range from This, along with the ability to mutate the state of the sys- writing simple but usually inadequate test cases [Ulrich and tem from the debugger, can be used to execute the same set König 1999], to rigorous but expensive model checking [Yang of lines over again and observe the state changes without et al. 2009] and theorem proving [Wilcox et al. 2015]. More having to restart the application. recently, techniques such as record and replay [Geels et al. Traditional interactive debuggers, however, become inad- 2006], tracing [Mace et al. 2018], log analysis [Fu et al. 2009], equate when used in parallel or distributed systems. Tech- and visualizations [Beschastnikh et al. 2016] have been used niques used in single threaded applications do not translate to locate bugs by performing postmortem analysis on the ex- well to a parallel or distributed context because information ecution of distributed systems after errors are encountered. creation and consumption is not sequential. To create an Rohan Achar, Pritha Dawn, and Cristina V. Lopes interactive debugger for a distributed context, information to visualize the information being exchanged. The D3S [Liu flow must be modeled differently. et al. 2008] tool allows programmers to define predicates that, This paper dives into the topic of interactive debuggers when matched during execution, parse the execution trace for distributed systems and presents an approach to the prob- to determine the source of the state changes. In interactive lem based on a specific model of distributed programming debugging these predicates are known as breakpoints and recently proposed, GoT [Achar and Lopes 2019]. The paper are the fundamental concept in interactive debugging. is organized as follows. First, we discuss the problem and Recently, a graphical, interactive debugger for distributed existing approaches to debugging distributed systems in Sec- systems called Oddity [Woos et al. 2018] was presented. Us- tion 2. In Section 3, we analyze the requirements to design an ing Oddity, programmers can observe communication in effective interactive debugger for distributed systems. This distributed systems. They can perturb these communica- includes both the features that the debugger should have, tions, exploring conditions of failure, reordered messages, and how we would go about designing those features. The duplicate messages etc. Oddity also supports the ability to underlying distributed computing model plays a dominant explore several branching executions without restarting the role in determining the feasibility of interactive debugging application. Oddity highlights communication. Events are and so, in Section 4, we determine the essential features that communicated, and consumed at the control of the program- a distributed computing model must have in order to create mer via a central Oddity component. However, the tool does an effective interactive debugger. We put our approach to not seem to capture the change of state within the node, the test with an implementation of a distributed debugger it only captures the communication. Without exposing the called GoTcha. GoTcha is build on top of the GoT distributed change of state within the node due to these communications, programming model which is discussed in Section 5. The im- the picture is incomplete. With this tool, we can observe if plementation of GoTcha using the example of a distributed the wrong messages are being sent, but we cannot observe if word frequency counting application, is discussed in Sec- a correct message is processed at a node in the wrong way. tion 6. We show how GoTcha meets the requirements of an interactive debugger in Section 7 and discuss what might be 3 Fundamental Requirements of in the future for interactive debugging in distributed com- Interactive Debuggers puting in Section 8. Whether for a single threaded application or a distributed application, there are two fundamental requirements that in- 2 Debugging Distributed Systems teractive debuggers must provide. First, the debugger should Interactive debugging of parallel and distributed systems has be able to observe, step by step, the change of state of the been discussed as early as 1981 [Schiffenbauer 1981], but the application. Second, the debugger must give the user con- idea has never been fully realized, mainly because it is very trol over the flow of execution, so that alternative chains of hard to do. However, there are many non interactive tools events across the distributed application components can aid developers in debugging distributed applications. A com- be observed. In this section, we discuss the design choices prehensive survey of the types of methods available can be available to us when trying to achieve both goals. found in Beschastnikh et al. [Beschastnikh et al. 2016]. In this survey, existing methods are grouped into seven categories: 3.1 Requirement 1: Observing State Changes testing, model checking, theorem proving, record and replay, The most important goal of an interactive debugger is to tracing, log analysis, and visualization. Each of these types of observe, step by step, the change of state of the computing tools offers different insights for the developer to find bugsin system. Therefore, it is important to understand how change the application. Tools for record and replay [Geels et al. 2007, of state can occur. A change of state can be abstracted to 2006; Liu et al. 2008], tracing [Falcone et al. 2018; Mace et al. the consumption and creation of information. For example, 2018], log analysis [Fu et al. 2009], and visualizers [Beschast- over the execution of a line of code in a single threaded nikh et al. 2016], try to parse the artifacts of execution such application, the current state of the variables in the applica- as logs, execution stack traces, and data traces to understand tion’s heap is consumed, and a new state is created. In such the change of state in a run of the distributed system. an application, we only have one dimension over which in- Many of these tools share features with interactive de- formation is created and consumed: time – not necessarily buggers, as they share the common goal of exposing er- world time, but time modeled as the sequence of operations, rors in the system to the developers. For example, tools like or causal time. Figure 1 shows this progression. The only task ShiViz [Beschastnikh et al. 2016] provide developers a way to that an interactive debugger designed for single threaded observe the information exchanged in a distributed system by applications has to do is to