A Randomized Dynamic Program Analysis Technique for Detecting Real Deadlocks Pallavi Joshi Chang-Seo Park Mayur Naik Koushik Sen Intel Research, Berkeley, USA EECS Department, UC Berkeley, USA [email protected] {pallavi,parkcs,ksen}@cs.berkeley.edu Abstract large multi-threaded programs due to the exponential increase in We present a novel dynamic analysis technique that finds real dead- the number of thread schedules with execution length. locks in multi-threaded programs. Our technique runs in two stages. Several program analysis techniques, both static [19, 10, 2, 9, In the first stage, we use an imprecise dynamic analysis technique 27, 29, 21] and dynamic [12, 13, 4, 1], have been developed to de- to find potential deadlocks in a multi-threaded program by observ- tect and predict deadlocks in multi-threaded programs. Static tech- ing an execution of the program. In the second stage, we control niques often give no false negatives, but they often report many a random thread scheduler to create the potential deadlocks with false positives. For example, the static deadlock detector developed by Williams et al. [29] reports 100,000 deadlocks in Sun’s JDK high probability. Unlike other dynamic analysis techniques, our ap- 1 proach has the advantage that it does not give any false warnings. 1.4 , out of which only 7 are real deadlocks. Type and annotation We have implemented the technique in a prototype tool for Java, based techniques [5, 10] help to avoid deadlocks during coding, and have experimented on a number of large multi-threaded Java but they impose the burden of annotation on programmers. Predic- programs. We report a number of previously known and unknown tive dynamic techniques such as Goodlock [13] and its improve- real deadlocks that were found in these benchmarks. ments [4, 1] give both false negatives and false positives. For exam- ple, in our experiments we have found that an improved Goodlock Categories and Subject Descriptors D.2.5 [Software Engineer- can report as many as 254 false positives for our Jigsaw web server. ing]: Testing and Debugging; D.2.4 [Software Engineering]: Soft- Being imprecise in nature, most of these tools require manual in- ware/Program Verification spection to see if a deadlock is real or not. Nevertheless, these tech- General Terms Languages, Reliability, Verification niques are effective in finding deadlocks because they can predict deadlocks that could potentially happen during a real execution— Keywords deadlock detection, dynamic program analysis, con- for such a prediction, static analyses do not need to see an actual ex- currency ecution and dynamic analyses need to see only one multi-threaded execution. 1. Introduction Dynamic analysis based deadlock detection can be made precise by taking the happens-before relation [18] into account. However, A common cause for unresponsiveness in software systems is a it has several problems. First, it reduces the predictive power of deadlock situation. In shared-memory multi-threaded systems, a dynamic techniques—it fails to report deadlocks that could happen deadlock is a liveness failure that happens when a set of threads in a significantly different thread schedule. Second, it can perturb blocks forever because each thread in the set is waiting to ac- an execution significantly and can fail to report a deadlock that can quire a lock held by another thread in the set. Deadlock is a happen when no dynamic analysis is performed. common form of bug in today’s software—Sun’s bug database at We propose a new dynamic technique for detecting real dead- http://bugs.sun.com/ shows that 6,500 bug reports out of locks in multi-threaded programs, called DEADLOCKFUZZER, 198,000 contain the keyword ‘deadlock’. There are a few reasons which combines an imprecise dynamic deadlock detection tech- for the existence of deadlock bugs in multi-threaded programs. nique with a randomized thread scheduler to create real dead- First, software systems are often written by many programmers; locks with high probability. The technique works in two phases. therefore, it becomes difficult to follow a lock order discipline that In the first phase, we use an informative and a simple variant could avoid deadlock. Second, programmers often introduce dead- of the Goodlock algorithm, called informative Goodlock, or sim- locks when they fix race conditions by adding new locks. Third, ply iGoodlock, to discover potential deadlock cycles in a multi- software systems can allow incorporation of third-party software threaded program. For example, iGoodlock could report a cycle (e.g. plugins); third-party software may not follow the locking dis- ′ ′ of the form (t1,l1,l2, [c1, c2])(t2,l2,l1, [c , c ]), which says that cipline followed by the parent software and this sometimes results 1 2 there could be a deadlock if thread t1 tries to acquire lock l2 at in deadlock bugs [17]. program location c2 after acquiring lock l1 at program location c1 Deadlocks are often difficult to find during the testing phase ′ and thread t2 tries to acquire lock l1 at program location c2 af- because they happen under very specific thread schedules. Coming ′ ter acquiring lock l2 at program location c . In the second phase, up with these subtle thread schedules through stress testing or 1 DEADLOCKFUZZER executes the program with a random sched- random testing is often difficult. Model checking [15, 11, 7, 14, 6] ule in order to create a real deadlock corresponding to a cycle removes these limitations of testing by systematically exploring reported in the previous phase. For example, consider the cycle all thread schedules. However, model checking fails to scale for 1 They reduce the number of reports to 70 after applying various unsound Permission to make digital or hard copies of all or part of this work for personal or heuristics classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. PLDI’09, June 15–20, 2009, Dublin, Ireland. Copyright c 2009 ACM 978-1-60558-392-1/09/06. $5.00 ′ ′ (t1,l1,l2, [c1, c2])(t2,l2,l1, [c1, c2]) again. At each program state, bugging. This relieves the manual inspection burden associated the random scheduler picks a thread and executes its next statement with other imprecise techniques such as Goodlock. with the following exception. If t1 is about to acquire lock l2 at lo- • We propose two kinds of object abstraction techniques that help cation c2 after acquiring lock l1 at location c1, then the random us correlate thread and lock objects between iGoodlock and the scheduler pauses the execution of thread t1. Similarly, the random randomized scheduling algorithm. scheduler pauses the execution of thread t2 if it is about to acquire ′ ′ • lock l1 at location c2 after acquiring lock l2 at location c1. In this We have implemented DEADLOCKFUZZER in a tool for Java biased random schedule, it is very likely that both the threads will and have discovered subtle previously known and unknown reach a state where t1 is trying to acquire l2 while holding l1 and t2 deadlocks in large applications. To the best of our knowledge, is trying to acquire l1 while holding l2. This results in a real dead- DEADLOCKFUZZER is the first precise dynamic deadlock anal- lock. In summary, DEADLOCKFUZZER actively controls a random- ysis tool for Java that has been applied to large Java applica- ized thread scheduler based on a potential deadlock cycle reported tions. by an imprecise deadlock detection technique. The above technique poses the following key challenge. Phase 2. Algorithm II assumes that Phase I can provide it with precise knowledge about the thread and lock objects involved in the deadlock cycle. Unfor- The DEADLOCKFUZZER algorithm consists of two phases. In the tunately, since thread and lock objects are created dynamically at first phase, we execute a multi-threaded program and find poten- runtime, their addresses cannot be used to identify them across ex- tial deadlocks that could happen in some execution of the program. This phase uses a modified Goodlock algorithm, called informative ecutions, i.e. in the above example, addresses of t1,t2,l1,l2 do not remain the same between Phase I and Phase II executions. There- Goodlock, or simply iGoodlock, which identifies potential dead- fore, we need some mechanism to identify the same objects across locks even if the observed execution does not deadlock. We call executions. Specifically, we need a form of object abstraction such the modified algorithm informative because we provide suitable that if two dynamic objects in different executions are the same, debugging information to identify the cause of the deadlock—this they must have the same abstraction. For example, the label of a debugging information is used by the second phase to create real statement at which an object is created can be used as its abstrac- deadlocks with high probability. A limitation of iGoodlock is that tion. Such an abstraction of an object does not change across execu- it can give false positives because it does not consider the happens- tions. However, an abstraction could be the same for several objects before relation between the transitions in an execution. As a result the user is required to manually inspect such potential deadlocks. (e.g. if both l1 and l2 in the above example are created by the same statement). In this paper, we propose two techniques for comput- The second phase removes this burden from the user. In this phase, ing the abstraction of an object that helps us to distinguish between a random thread scheduler is biased to generate an execution that different objects more precisely—the first technique is motivated creates a real deadlock reported in the previous phase with high by the notion of k-object-sensitivity in static analysis [20] and the probability.