MIT OpenCourseWare http://ocw.mit.edu 6.005 Elements of Software Construction Fall 2008 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. 10/15/2008 Today’s Topics how to avoid debugging ¾assertions ¾code reviews how to do it when you have to ¾reducing test cases ¾hypothesis-driven debugging ¾binary search Debugging very hard bugs Rob Miller ¾Heisenbugs Fall 2008 © Robert Miller 2008 © Robert Miller 2008 Defensive Programming First Defense: Impossible By Design first defense against bugs is to make them impossible in the language ¾Java makes buffer overflow bugs impossible ¾automatic array bounds checking make buffer overflow bugs impossible second defense against bugs is to not make them ¾static typing eliminates many runtime type errors ¾correctness: get things riihght first time in the protocols/libraries/modules third defense is to make bugs easy to find ¾TCP/IP guarantees that data is not reordered ¾local visibility of errors: if things fail, we'd rather they fail loudly and ¾BigInteger guarantees that there will be no overflow immediately – e.g. with assertions in self-imposed conventions fourth defense is extensive testing ¾immutable objects can be passed around and shared without fear ¾uncover as many bugs as possible ¾caution: you have to keep the discipline last resort is dbidebugging • get the language to hel p you as much as possible , e.g. with pritivate and final ¾needed when effect of bug is distant from cause © Robert Miller 2008 © Robert Miller 2008 1 10/15/2008 Second Defense: Correctness Third Defense: Immediate Visibility get things right the first time if we can't prevent bugs, we can try to localize them to ¾don’t code before you think! Think before you code. a small part of the program • do your thinking in design; use a pattern to map that design to code ¾fail fast: the earlier a problem is observed, the easier it is to fix especially true when debugging is going to be hard ¾assertions: catch bugs early, before failure has a chance to contaminate (and be obscured by) further computation ¾concurrency • in Java: assert boolean-expression simplicity is key • note that you must enable assertions with -ea ¾modularity ¾unit testing: when you test a module in isolation, you can be confident that • divide program into chunks that are easy to understand any bug you find is in that unit (or in the test driver) • use abstract data types with well-defined interfaces ¾regression testing: run tests as often as possible when changing code. • avoid rep exposure • if a test fails, the bug is probably in the code you just changed ¾specification when localized to a single method or small module, • write specs for all modules, so that an explicit, well-defined contract bugs can be found simply by studying the program text exists between each module and its client © Robert Miller 2008 © Robert Miller 2008 Example: Assertions Code Review /* other eyes looking at the code can find bugs * Returns n!, the number of permutations of n objects. * n must be nonnegative. */ code review public static int fact(int n) { where would ¾carefu l, systematic study of source code by others (not origi nal author) if (n == 0) return 1; assertions be ¾analogous to proofreading an English paper else return n * fact(n-1); usefully added } to this code? ¾look for bugs, poor style, design problems, etc. ¾formal inspection: several people read code separately, then meet to /* discuss it * Returns (n choose k), the number of distinct subsets ¾lightweight methods: over-the-shoulder walkthrough, or by email * of size k in a set of size n. * Requires 0 <= k <= n. ¾manyygpq dev groups require a code review before commit */ code review complements other techniques public static int combinations(int n, int k) { ¾code reviews can find many bugs cheaply return fact(n) / (fact(k) * fact(n-k)); } ¾also test the understandability and maintainability of the code ¾three proven techniques for reducing bugs: reasoning, code reviews, testing © Robert Miller 2008 © Robert Miller 2008 2 10/15/2008 Let’s Review Some Code How to Debug public class PigLatin { 1) reproduce the bug with a small test case static String[] words; ¾find a small, repeatable test case that produces the failure (may take effort, public static String toPigLatin(String s) { but helps clarify the bug, and also gives you something for regression) words = s.split(" "); ¾don't move on to next step until you have a repeatable test String result = ""; 2) find the cause for (int i = 0; i <= words.length; ++i) { piggify(i); ¾narrow down location and proximate cause result += words[i]; ¾study the data / hypothesize / experiment / repeat } ¾may change code to get more information return result; } ¾don't move on to next step until you understand the cause public static void piggify(int i) { 3) fix the bug if (words[i].startsWith("a") || words[i].startsWith("e") || ...) { ¾is it a simple typo, or is it a design flaw? does it occur elsewhere? words[i] += "yay"; } else { 4) add test case to regression tests words[i] = words[i].substring(1); ¾then run regression tests to ensure that the bug appears to be fixed, and words[i] += words[i].charAt(0) + "ay"; no new bugs have been introduced by the fix } } © Robert Miller 2008 © Robert Miller 2008 } Reducing to a Simple Test Case Example find simplest input that will provoke bug /** * Returns true if and only if s contains t as a substring, ¾usually not the input that originally revealed existence of the bug * e.g. contains("hello world", "world") == true. ¾start with data that revealed bug */ ¾keep paring it down (binary search can help) public static boolean contains(String s, String t) { ... } ¾often leads directly to an understanding of the cause same idea is useful at many levels of a system ¾a user discovers that ¾method arguments contains("Life is wonderful! I am so very very happy all the time“, "very happy") ¾input files incorrectly returns false ¾keystrokes and mouse clicks in a GUI wronggpp approach: ¾try to trace the execution of contains() for this test case right approach: ¾first try to reduce the size of the test case ¾even better: bracket the bug with a test case that fails and similar test cases that succeed © Robert Miller 2008 © Robert Miller 2008 3 10/15/2008 Code for contains() Finding the Cause /** exploit modularity * Returns true if and only if s contains t as a substring, * e.g. contains("hello world", "world") == true. ¾start with everything, take away pieces until bug goes */ ¾start with nothing, add pieces back in until bug appears public static boolean contains(String s, String t) { take advantage of modular reasoning search: for (int i = 0; i < s.length(); ++i) { ¾trace through program, viewing intermediate results for (int j = 0; j < t.length(); ++j, ++i) { ¾insert assertions targeted at the bug if (s.charAt(i) != t.charAt(j)) continue search; ¾design all data structures to be printable (i.e., implement toString()) } ¾println is a surprisingly useful and universal tool return true; } • in large systems, use a logging infrastructure instead of println return false; use binary search to speed things up } ¾bug happens somewhere between first and last statement ¾so do binary search on the ordered set of statements © Robert Miller 2008 © Robert Miller 2008 Example: Finding a Sudoku Bug Regression Testing suppose a Sudoku solver produces the wrong answer whenever you find and fix a bug ¾store the input that elicited the bug 3.... 7.... Sudoku ¾store the correct output ...... ...... solver ¾add it to your test suite ...... ...... why regression tests help ¾helps to populate test suite with good test cases • remember that a test is good if it elicits a bug – and every regression parse make SAT SAT interpret print test did in one version of your code input formula solver assignment output ¾protects against reversions that reintroduce bug ¾the buggy may be an easy error to make ( since it happ ened once already) test-first debugging ¾when a bug arises, immediately write a test case for it that elicits it Note that this isn’t a state machine diagram or a module dependence diagram; it ¾once you find and fix the bug, the test case will pass, and you’ll be done shows data flow, which is often useful for thinking about bugs. © Robert Miller 2008 © Robert Miller 2008 4 10/15/2008 The Ugliest Bugs Example of a heisenbug we’ve had it easy so far public class Bank { int balance; ¾sequential, deterministic programs have repeatable bugs but the real world is not that nice… public Bank(int balance) { ¾tiiiming depen denc ies this.balance = balance; } ¾unpredictable network delays ¾varying processor loads public void deposit(int amount) { ¾concurrency balance += amount; heisenbugs } ¾nondeterministic, hard to reproduce public void withdraw(int amount) { ¾may even disappear when you try to loo k at it with printl n or deb ugger! balance -= amount; one approach } ¾build a lightweight event log (circular buffer) public int getBalance() { ¾log events during execution of program as it runs at speed return balance; ¾when you detect the error, stop program and examine logs } } © Robert Miller 2008 © Robert Miller 2008 Example of a heisenbug Summary // our bank account starts with $100 avoid debugging final Bank account = new Bank(100); // start a bunch of threads ¾it’s not fun and not productive List<Thread> threads = new ArrayList<Thread>(); ¾many of the techniques of this class are designed to save you from bugs for (int i = 0; i < 10; ++i) { approach it systematically Thread t = new Thread(new Runnable() { public void run() { ¾simplify test cases // each thread does a bunch of bank transactions ¾find cause before trying to fix for (int i = 0; i < 10000; ++i) { account.deposit(1); // put a dollar in account.withdraw(1); // take it back out }}}); t.