Runtime Verification Carl Martin Rosenberg INF5140 June 2015 Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 1 / 27 Plan In broad terms, I hope to answer the following questions: What is Runtime Verification? Why is it useful? How does Runtime Verification compare to traditional Model Checking? How are the things we have learned in INF5140 used in Runtime Verification? After that, I will do a short live demo on how to use a tool called DTrace for Runtime Verification. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 2 / 27 What is Runtime Verification? Runtime Verification is an emerging research field with strong ties to Model Checking. Has had its own workshop series since 20011. As a first approximation, it is concerned with checking software based on data from actual runs of the software. Borrows many formalisms and methods from Model Checking, notably LTL. 1See http://www.runtime-verification.org/ Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 3 / 27 What is Runtime Verification? More precisely, let us follow[6][p. 36] and use the following definition: Runtime verification is the discipline of computer science that deals with the study, development and application of those verification techniques that allow checking whether a run of a system under scrutiny . satisfies or violates a given correctness property Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 4 / 27 What is Runtime Verification? More precisely, let us follow[6][p. 36] and use the following definition: Runtime verification is the discipline of computer science that deals with the study, development and application of those verification techniques that allow checking whether a run of a system under scrutiny . satisfies or violates a given correctness property Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 5 / 27 Unpacking the definition 1 We can understand verification as \[A]ll techniques suitable for showing that a system satisfies its specification”[7][p. 294] 2 What about a run? Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 6 / 27 The concept of a run In Runtime Verification, a run is typically represented by some log or trace. The trace can either represent a sequence of program states, or a series of events representing the program behavior (I/O, syscalls). Let's consider these two options more closely. [7][p. 294] Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 7 / 27 Traces as a sequence of program states One notion of a trace is that of a sequence of program states. This notion of a run is similar to the notion of a computation in traditional Model Checking[4][p. 13] If we had a graph that modeled the program, where each node represented a program state (ie. set of variable assignments), a run is a (possibly looping) path in this graph. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 8 / 27 Traces as a sequence of program behaviors Another notion of a trace is that of a behavior history. In this variety, we treat the program as a black box[7][p. 295], and conduct the verification based only on what we can see \from the outside", by analyzing the program I/O or inpecting the interaction between the program and the operating system. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 9 / 27 Two important differences from Model Checking The preceeding discussion of traces highlight two important differences between traditional Model Checking and Runtime Verification: 1 In Runtime Verification, we check if a property holds for a single run of the system. Model Checking typically checks all possible runs over the program model. [7][p. 294] 2 While Model Checking can operate with infinite runs, Runtime Verification is restricted to finite runs. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 10 / 27 All traces in Runtime Verification are finite Traditional Model Checking often want to analyze programs that ideally should run forever (ie. servers, core infrastructure). Therefore traditional Model Checking allows for (and often presumes) infinite runs. Runtime Verification, however, bases itself on real-world runs of systems, and these are by necessity finite. This has some implications for how we use LTL in Runtime Verification, as we shall see. [7][p. 294-295] Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 11 / 27 From Runs to Monitors Having humans conduct runtime verification by inspecting traces would be error-prone and infeasible. We want to make the computer analyze the traces in a systematic manner. This brings up the concept of a Monitor. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 12 / 27 Monitor A monitor is a device that reads a finite trace and yields a certain verdict[7][p. 294] A verdict is some value of some truth domain, ie. ftrue; falseg. The monitor is essentially a decider for the property in question. Figuring out how to create a monitor for a given specification is a central theme in Runtime Verification. Martin Leucker calls it \the distinguishing research effort"[6][p. 36] of Runtime Verirication. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 13 / 27 The basic picture Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 14 / 27 Offline versus online monitoring If the monitor consumes the trace while the program still runs, we call this online monitoring. On the other hand, if the trace is consumed after the execution has finished, we call it offline monitoring. [7][p. 295] Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 15 / 27 Online monitoring Online monitoring is especially interesting because it opens up the possibility for not only observing the program, but reacting if something bad happens [7]. However, online monitoring also requires us to be extra careful: We should not let the act of monitoring the program interfere excessively with the program execution. Hence, online monitoring makes it even more important to create monitors that 1 Use as little memory as possible 2 Use as little CPU time as possible Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 16 / 27 Criteria for good monitors: Impartiality and Anticipation Online monitoring requires monitors that are time and space–efficient. Also, since Runtime Verification works on finite traces, we need to ensure that 1 No monitor gives a verdict based on incomplete information. 2 As soon as sufficient information is obtained, the monitor should give a verdict. Martin Leucker[7][p. 295] calls these impartiality and anticipation, respectively. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 17 / 27 Achiving impartiality: Going beyond true and false If our monitor can only output true or false, we might get misleading results [7][p. 295]. Consider the LTL formula :p, which reads "no state satisfying p should occur". If a p is observed, the monitor should yield "false". However, as long as p is not observed, one should not say "true", because p might occur sometime in the future. In this case, it would be better for the monitor to report inconclusive [7][p. 297]. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 18 / 27 Achiving impartiality: Going beyond true and false Another example: Consider the LTL formula ♦p - which reads "eventually a p is observed". In this case, we should only report "true" when we find a p. Otherwise, we the monitor should say "inconclusive". [7][p. 297]. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 19 / 27 Achiving impartiality: Going beyond true and false In [7], Martin Leucker presents a variety of LTL that incorporates "inconclusive" into the semantics of the logic. He calls the resulting logic LTL3. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 20 / 27 LTL3 Leucker defines the semantics of LTL3 is defined as follows: if there is no continuation of u satisfying φ (considered as an LTL formula), the value of φ is false; if every continuation of u satisifes φ (considered as an LTL formula), it is true; otherwise, the value is inconclusive since the observations so far are inconclusive, and neither true or false can be determined. [7][p. 297] Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 21 / 27 Limitations of standard LTL There are many things we wish to verify, that cannot easily be verified in LTL. Suppose we want to verify that every opened file is eventually closed. As a first approximation, we could say (open ! Fclose). This gets close, but only says that "there should be a close for every open". Typically, we want a way to say that if file x is opened, file x gets closed. [7][p. 298] Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 22 / 27 The file descriptor example Volker and a former student of his, Eric Bodden, developed a way to enhance LTL to express this property2 In addition to the formalism for expressing the property, they developed a way to extract information from the running program using Aspect-oriented programming on the Java Virtual Machine. I will end by demonstrating how something similar can be achieved using DTrace. 2The paper cited in [7][p. 298] is [8]. See also [1]. Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 23 / 27 DTrace in one slide DTrace is an operating system technology originally developed for troubleshooting performance issues [3], [2], [5]. It can be used to harvest all kinds of data: 1 Information about the inner workings of a user program 2 Information about the syscalls a program executes 3 Detailed information about what goes on in the kernel 4 Much, much more. In my master thesis, I explore how DTrace can be used for Runtime Verification. Let's see it in action! Carl Martin Rosenberg (INF5140) Runtime Verification June 2015 24 / 27 BibliographyI Eric Bodden.