The Fine Art of Writing a Bug Report

The Fine Art of Writing a Good Bug Report

Stable fly photo courtesy of

Rex Black Phone: +1 (830) 438-4830 RBCS, Inc. HTTP: //www.rexblackconsulting.com 31520 Beck Road E-mail: [email protected] Bulverde, TX 78163 What Is a Bug Report and Why Do We Write Them? • Bug report is a technical document – Describes failure mode in system under test (SUT) – The only tangible “product” of testing • Not a management problem escalation tool – “Build not delivered on time” is not a bug report summary – “Build 781 fails to install” is a bug report summary • Written to increase product quality – Documents a specific quality problem quality of SUT – Communicates to developers

Quality Week 2000 Copyright (c) 2000, Rex Black, All 2 Rights Reserved Are Bad Bug Reports a Problem? • Anecdotally, developers return many bug reports as unreproducible, leading to: – Wasted time writing the report – Frustration for tester and developer alike – No increase in product quality • Bug reports can be unreproducible due to: – Intermittence – Inconsistent test/development environments – Disputes over “correct” behavior • But many unreproducible bug reports are poorly conceived and poorly written

Quality Week 2000 Copyright (c) 2000, Rex Black, All 3 Rights Reserved Ten Tips for a Good Bug Report 1 Structure: test carefully 2 Reproduce: test it again 3 Isolate: test it differently 4 Generalize: test it elsewhere 5 Compare: review results of similar tests 6 Summarize: relate test to customers 7 Condense: trim unnecessary information 8 Disambiguate: use clear words 9 Neutralize: express problem impartially 10Review: be sure êRemember, writing is creative: two good bug reports on one problem can differ in style and content (but not substance)

• Let’s use a case study to illustrate these techniques • SpeedyWriter • A Java-based word processor • Supports the usual word processing functions • Differences highlighted in italics in case study

Quality Week 2000 Copyright (c) 2000, Rex Black, All 5 Rights Reserved Structure • Structured testing foundational to good bug reports – Use deliberate, careful approach to testing – Follow written test cases or run automated ones per written or standardized process – Take careful notes • Bug reporting begins when expected and observed results differ • Sloppy testing results in sloppy bug reports

Good software testing should be more like a carefully designed laboratory experiment than a random walk in an electrical storm--or worse. Testers are engineers, after all.

Ben Franklin standing in a M. Maria Sklodowska-Curie thunderstorm hoping to get hit performing a carefully by lightning designed experiment Quality Week 2000 Copyright (c) 2000, Rex Black, All 7 Rights Reserved Reproduce • Always check reproducibility of the failure as part of writing bug report üThree times is a good rule of thumb • Document a crisp sequence of actions that will reproduce the failure • Report intermittent, hard-to-repeat failures – Note failure incidence rate (i.e., 1 in 3 tries) ×Clean steps to reproduce addresses issue of unreproducibility head-on

Steps to Reproduce Nasty bug trashed contents of new file that I created 1. I started the SpeedyWriter editor, then I created a by formatting some text in Arial font, wasting my new file. time. 2. I then typed in four lines of text, repeating “The quick fox jumps over the lazy brown dog” each time, using different effects each time, bold, italic, strikethrough, and underline. 3. I highlighted the text, then pulled down the font menu, and selected Arial. 4. This nasty bug trashed all the text into meaningless garbage, wasting the user’s time. 5. I was able to reproduce this problem three out of three tries.

Quality Week 2000 Copyright (c) 2000, Rex Black, All 9 Rights Reserved Isolate • Change variables that may alter symptom – Make changes one by one – Requires thought and understanding of SUT – May not be immediately obvious • Can be extensive – Match amount of effort to severity of problem – Avoid getting into debugging activities ×Good isolation shows due diligence and gives developers head start on debugging

Steps to Reproduce Steps to Reproduce 1. I started the SpeedyWriter editor, then I created a 1. I started the SpeedyWriter editor, then I created a new file. new file. 2. I then typed in four lines of text, repeating “The 2. I then typed in four lines of text, repeating “The quick fox jumps over the lazy brown dog” each quick fox jumps over the lazy brown dog” each time, using different effects each time, bold, italic, time, using different effects each time, bold, italic, strikethrough, and underline. strikethrough, and underline. 3. I highlighted the text, then pulled down the font 3. I highlighted the text, then pulled down the font menu, and selected Arial. menu, and selected Arial. 4. This nasty bug trashed all the text into 4. This nasty bug trashed all the text into meaningless garbage, wasting the user’s time. meaningless garbage, wasting the user’s time. 5. I was able to reproduce this problem three out of 5. I was able to reproduce this problem three out of three tries. three tries. Isolation Isolation On the vague suspicion that this was just a Doesn’t happen on Solaris. formatting problem, I saved the file, closed SpeedyWriter and reopened the file. The garbage remained. If you save the file before Arializing the contents, the bug does not occur. The bug does not occur with existing files. This only happens under Windows 98.

Quality Week 2000 Copyright (c) 2000, Rex Black, All 11 Rights Reserved Generalize • Look for related failures in SUT – Does the same failure occur in other modules or locations? – Are there more severe occurrences of the same fault? • Avoid confusing unrelated problems – Same symptom can arise from different bugs ×Generalizing reduces duplicate bug reports and refines understanding of failure

Steps to Reproduce Steps to Reproduce 1. I started the SpeedyWriter editor, then I created a 1. I started the SpeedyWriter editor, then I created a new file. new file. 2. I then typed in four lines of text, repeating “The 2. I then typed in four lines of text, repeating “The quick fox jumps over the lazy brown dog” each quick fox jumps over the lazy brown dog” each time. time, using different effects each time, bold, italic, 3. I highlighted the text, then pulled down the font strikethrough, and underline. menu, and selected Arial. 3. I highlighted the text, then pulled down the font 4. This nasty bug trashed all the text into menu, and selected Arial. meaningless garbage, wasting the user’s time. 4. This nasty bug trashed all the text into 5. I was able to reproduce this problem three out of meaningless garbage, wasting the user’s time. three tries. 5. I was able to reproduce this problem three out of Isolation three tries. Also happens with Wingdings and Symbol fonts. Isolation On the vague suspicion that this was just a On the vague suspicion that this was just a formatting problem, I saved the file, closed formatting problem, I saved the file, closed SpeedyWriter and reopened the file. The garbage SpeedyWriter and reopened the file. The garbage remained. remained. If you save the file before changing the font of the If you save the file before Arializing the contents, contents, the bug does not occur. the bug does not occur. The bug does not occur with existing files. The bug does not occur with existing files. This only happens under Windows 98. This only happens under Windows 98.

Quality Week 2000 Copyright (c) 2000, Rex Black, All 13 Rights Reserved Compare • Examine results for similar tests – Same test run against earlier versions – Similar conditions, other tests, same version • Is failure a regression? – Change introduces defect not in earlier versions – Usually found when previously passed tests fail • Not always possible – Test previous blocked, reinstall impractical – Tested feature unavailable in earlier versions

Steps to Reproduce Steps to Reproduce 1. I started the SpeedyWriter editor, then I created a 1. I started the SpeedyWriter editor, then I created a new file. new file. 2. I then typed in four lines of text, repeating “The 2. I then typed in four lines of text, repeating “The quick fox jumps over the lazy brown dog” each quick fox jumps over the lazy brown dog” each time. time. 3. I highlighted the text, then pulled down the font 3. I highlighted the text, then pulled down the font menu, and selected Arial. menu, and selected Arial. 4. This nasty bug trashed all the text into 4. This nasty bug trashed all the text into meaningless garbage, wasting the user’s time. meaningless garbage, wasting the user’s time. 5. I was able to reproduce this problem three out of 5. I was able to reproduce this problem three out of three tries. three tries. Isolation Isolation New to build 1.1.018; same test case passed against Also happens with Wingdings and Symbol fonts. builds 1.1.007 (System Test entry) through 1.1.017. On the vague suspicion that this was just a Also happens with Wingdings and Symbol fonts. formatting problem, I saved the file, closed On the vague suspicion that this was just a SpeedyWriter and reopened the file. The garbage formatting problem, I saved the file, closed remained. SpeedyWriter and reopened the file. The garbage If you save the file before changing the font of the remained. contents, the bug does not occur. If you save the file before changing the font of the The bug does not occur with existing files. contents, the bug does not occur. This only happens under Windows 98. The bug does not occur with existing files. This only happens under Windows 98.

Quality Week 2000 Copyright (c) 2000, Rex Black, All 15 Rights Reserved Summarize • Put a short “tag line” on each report – Capture failure and impact on customer 7 Analogy: Bumper sticker • Harder than it seems – Testers must spend time thinking about this ÖAdvantages of good summaries – Get management attention – Name bug report for developers – Help set priorities

Summary Steps to Reproduce Arial, Wingdings, and Symbol fonts corrupt new files. 1. I started the SpeedyWriter editor, then I created a Steps to Reproduce new file. 1. I started the SpeedyWriter editor, then I created a 2. I then typed in four lines of text, repeating “The new file. quick fox jumps over the lazy brown dog” each time. 2. I then typed in four lines of text, repeating “The 3. I highlighted the text, then pulled down the font quick fox jumps over the lazy brown dog” each time. menu, and selected Arial. 3. I highlighted the text, then pulled down the font 4. This nasty bug trashed all the text into meaningless menu, and selected Arial. garbage, wasting the user’s time. 4. This nasty bug trashed all the text into meaningless 5. I was able to reproduce this problem three out of garbage, wasting the user’s time. three tries. 5. I was able to reproduce this problem three out of Isolation three tries. New to build 1.1.018; same test case passed against Isolation builds 1.1.007 (System Test entry) through 1.1.017. New to build 1.1.018; same test case passed against Also happens with Wingdings and Symbol fonts. builds 1.1.007 (System Test entry) through 1.1.017. On the vague suspicion that this was just a formatting Also happens with Wingdings and Symbol fonts. problem, I saved the file, closed SpeedyWriter and On the vague suspicion that this was just a formatting reopened the file. The garbage remained. problem, I saved the file, closed SpeedyWriter and If you save the file before changing the font of the reopened the file. The garbage remained. contents, the bug does not occur. If you save the file before changing the font of the The bug does not occur with existing files. contents, the bug does not occur. This only happens under Windows 98. The bug does not occur with existing files. This only happens under Windows 98. Quality Week 2000 Copyright (c) 2000, Rex Black, All 17 Rights Reserved Condense

• Eliminate extraneous words or steps – Reread report carefully – Avoid both cryptic commentary and droning on • Are any details or actions irrelevant? • Everyone’s time is precious, so don’t waste any of it on unnecessary verbiage, but don’t cut any meat, either

Summary Summary Arial, Wingdings, and Symbol fonts corrupt new files. Arial, Wingdings, and Symbol fonts corrupt new files. Steps to Reproduce Steps to Reproduce 1. Started SpeedyWriter editor, then created new file. 1. I started the SpeedyWriter editor, then I created a 2. Typed four lines of text, repeating “The quick fox new file. jumps over the lazy brown dog” each time. 2. I then typed in four lines of text, repeating “The 3. Highlighted text, then pulled down the font menu, quick fox jumps over the lazy brown dog” each time. and selected Arial. 3. I highlighted the text, then pulled down the font 4. This nasty bug trashed all text into meaningless menu, and selected Arial. garbage, wasting the user’s time. 4. This nasty bug trashed all the text into meaningless 5. Reproduced three out of three tries. garbage, wasting the user’s time. Isolation 5. I was able to reproduce this problem three out of New to build 1.1.018; same test case passed against three tries. builds 1.1.007 (System Test entry) through 1.1.017. Isolation Also happens with Wingdings and Symbol fonts. New to build 1.1.018; same test case passed against On vague suspicion this was a formatting problem, builds 1.1.007 (System Test entry) through 1.1.017. saved file, closed SpeedyWriter and reopened file. Also happens with Wingdings and Symbol fonts. Garbage remained. On the vague suspicion that this was just a formatting Saving file before changing font prevents bug. problem, I saved the file, closed SpeedyWriter and Bug does not occur with existing files. reopened the file. The garbage remained. Only happens under Windows 98. If you save the file before changing the font of the contents, the bug does not occur. The bug does not occur with existing files. This only happens under Windows 98.

• Remove, rephrase, or expand vague, misleading, or subjective words and statements • Make sure report is not subject to misinterpretation ÖGoal: Clear, indisputable statements of fact \Lead developer by the hand to bug

Summary Summary Arial, Wingdings, and Symbol fonts corrupt new files. Arial, Wingdings, and Symbol fonts corrupt new files. Steps to Reproduce Steps to Reproduce 1. Started SpeedyWriter editor, then created new file. 1. Started SpeedyWriter editor, then created new file. 2. Typed four lines of text, repeating “The quick fox 2. Typed four lines of text, repeating “The quick fox jumps over the lazy brown dog” each time. jumps over the lazy brown dog” each time. 3. Highlighted all four lines of text, then pulled down 3. Highlighted text, then pulled down the font menu, the font menu, and selected Arial. and selected Arial. 4. This nasty bug trashed all text into meaningless 4. This nasty bug trashed all text into meaningless garbage, including control characters, numbers, and garbage, wasting the user’s time. other binary junk, wasting the user’s time. 5. Reproduced three out of three tries. 5. Reproduced three out of three tries. Isolation Isolation New to build 1.1.018; same test case passed against New to build 1.1.018; same test case passed against builds 1.1.007 (System Test entry) through 1.1.017. builds 1.1.007 (System Test entry) through 1.1.017. Also happens with Wingdings and Symbol fonts. Reproduced with same steps using Wingdings and On vague suspicion this was a formatting problem, Symbol fonts. saved file, closed SpeedyWriter and reopened file. On vague suspicion this was a formatting problem, Garbage remained. saved file, closed SpeedyWriter and reopened file. Saving file before changing font prevents bug. Garbage remained. Bug does not occur with existing files. Saving file before changing font prevents bug. Only happens under Windows 98. Bug does not occur with existing files. Only happens under Windows 98, not Solaris, Mac, or other Windows flavors.

• Deliver bad news gently • Be fair-minded in wording and implications • Avoid: – Attacking developers – Criticizing the underlying error – Attempting humor or using sarcasm • Confine bug reports to statements of fact êYou never know who’ll read the reports

Summary Summary Arial, Wingdings, and Symbol fonts corrupt new files. Arial, Wingdings, and Symbol fonts corrupt new files. Steps to Reproduce Steps to Reproduce 1. Started SpeedyWriter editor, then created new file. 1. Started SpeedyWriter editor, then created new file. 2. Typed four lines of text, repeating “The quick fox 2. Typed four lines of text, repeating “The quick fox jumps over the lazy brown dog” each time. jumps over the lazy brown dog” each time. 3. Highlighted all four lines of text, then pulled down 3. Highlighted all four lines of text, then pulled down the font menu, and selected Arial. the font menu, and selected Arial. 4. All text converted to control characters, numbers, 4. This nasty bug trashed all text into meaningless and other apparently random binary data. garbage, including control characters, numbers, and 5. Reproduced three out of three tries. other binary junk, wasting the user’s time. Isolation 5. Reproduced three out of three tries. New to build 1.1.018; same test case passed against Isolation builds 1.1.007 (System Test entry) through 1.1.017. New to build 1.1.018; same test case passed against Reproduced with same steps using Wingdings and builds 1.1.007 (System Test entry) through 1.1.017. Symbol fonts. Reproduced with same steps using Wingdings and On vague suspicion this was a formatting problem, Symbol fonts. saved file, closed SpeedyWriter and reopened file. On vague suspicion this was a formatting problem, Garbage remained. saved file, closed SpeedyWriter and reopened file. Saving file before changing font prevents bug. Garbage remained. Bug does not occur with existing files. Saving file before changing font prevents bug. Only happens under Windows 98, not Solaris, Mac, or Bug does not occur with existing files. other Windows flavors. Only happens under Windows 98, not Solaris, Mac, or other Windows flavors.

Quality Week 2000 Copyright (c) 2000, Rex Black, All 23 Rights Reserved Review • Each tester should submit each bug report to one or more test peers for a review • Reviewing peers should: – Make suggestions to improve report – Ask clarifying questions – Even challenge “bugginess” if appropriate êTest team should only submit best possible bug reports, given time constraints appropriate to priority of bug

Quality Week 2000 Copyright (c) 2000, Rex Black, All 24 Rights Reserved Conclusions • A bug report is a technical document – Accurate – Concise – Well-conceived – High-quality • Instill tester focus on writing good reports • Quality indicators for good bug reports – Clear to management (especially the summary) – Actionable by development – Move quickly from opened to closed Quality Week 2000 Copyright (c) 2000, Rex Black, All 25 Rights Reserved ROI for Good Bug Reporting • Improved Test team communication to senior and peer management – Can enhance credibility, standing, and resources • Better tester/developer relationships – Helps developers be effective, reduces arguments • Faster, more efficient bug lifecycles – Quicker fixes and fewer reopens ÞSupport increased product quality