Debugging Table of Contents Debugging—The Nine Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems...................................................................................................................1 Chapter 1: Introduction.....................................................................................................................4 Overview..................................................................................................................................4 How Can That Work?...............................................................................................................4 Isn't It Obvious?.......................................................................................................................4 Anyone Can Use It...................................................................................................................5 It'll Debug Anything..................................................................................................................5 But It Won't Prevent, Certify, or Triage Anything.....................................................................6 More Than Just Troubleshooting.............................................................................................6 A Word About War Stories.......................................................................................................7 Stay Tuned...............................................................................................................................7 Chapter 2: The Rules−Suitable for Framing....................................................................................9 Chapter 3: Understand the System................................................................................................10 Overview................................................................................................................................10 Read the Manual....................................................................................................................11 Read Everything, Cover to Cover..........................................................................................12 Know What's Reasonable......................................................................................................13 Know the Road Map..............................................................................................................14 Know Your Tools....................................................................................................................14 Look It Up...............................................................................................................................15 Remember.............................................................................................................................16 Understand the System....................................................................................................16 Chapter 4: Make it Fail.....................................................................................................................17 Overview................................................................................................................................17 Do It Again.............................................................................................................................19 Start at the Beginning............................................................................................................19 Stimulate the Failure..............................................................................................................19 Don't Simulate the Failure......................................................................................................20 What If It's Intermittent?.........................................................................................................21 What If I've Tried Everything and It's Still Intermittent?..........................................................22 A Hard Look at Bad Luck.................................................................................................22 Lies, Damn Lies, and Statistics........................................................................................23 Did You Fix It, or Did You Get Lucky?.............................................................................23 "But That Can't Happen"........................................................................................................24 Never Throw Away a Debugging Tool...................................................................................25 Remember.............................................................................................................................26 Make It Fail.......................................................................................................................26 Chapter 5: Quit Thinking and Look................................................................................................27 Overview................................................................................................................................27 See the Failure.......................................................................................................................29 See the Details.......................................................................................................................31 Now You See It, Now You Don't............................................................................................33 Instrument the System...........................................................................................................33 Design Instrumentation In................................................................................................33 i Table of Contents Chapter 5: Quit Thinking and Look Build Instrumentation In Later..........................................................................................35 Don't Be Afraid to Dive In.................................................................................................36 Add Instrumentation On...................................................................................................36 Instrumentation in Daily Life.............................................................................................37 The Heisenberg Uncertainty Principle...................................................................................37 Guess Only to Focus the Search...........................................................................................38 Remember.............................................................................................................................38 Quit Thinking and Look....................................................................................................38 Chapter 6: Divide and Conquer......................................................................................................40 Overview................................................................................................................................40 Narrow the Search.................................................................................................................42 In the Ballpark..................................................................................................................43 Which Side Are You On?.................................................................................................44 Inject Easy−to−Spot Patterns................................................................................................44 Start with the Bad...................................................................................................................45 Fix the Bugs You Know About...............................................................................................46 Fix the Noise First..................................................................................................................46 Remember.............................................................................................................................47 Divide and Conquer.........................................................................................................47 Chapter 7: Change One Thing at a Time.......................................................................................48 Overview................................................................................................................................48 Use a Rifle, Not a Shotgun....................................................................................................49 Grab the Brass Bar with Both Hands.....................................................................................50 Change One Test at a Time...................................................................................................51 Compare with a Good One....................................................................................................51 What Did You Change Since the Last Time It Worked?........................................................52 Remember.............................................................................................................................54 Change One Thing at a Time...........................................................................................54 Chapter 8: Keep an Audit Trail.......................................................................................................55