MANANGEMENT

TWO Current software projects spend about Software 40 to 50 percent of their effort on avoid- able rework. Such rework consists of effort spent fixing software difficulties that could Defect Reduction have been discovered earlier and fixed less expensively or avoided altogether. By implication, then, some effort must con- sist of “unavoidable rework,” an obser- Top 10 List vation that has gained increasing credibility with the growing realization Barry Boehm, University of Southern California that better user-interactive systems result from emergent processes. In such Victor R. Basili, University of Maryland processes, the requirements emerge from prototyping and other multistakeholder- shared learning activities, a departure from traditional reductionist processes that stipulate requirements in advance, ecently, a National Science insight has been a major driver in focus- then reduce them to practice via design Foundation grant enabled us to ing industrial software practice on thor- and coding. Emergent processes indicate establish the Center for Empiri- ough requirements analysis and design, that changes to a system’s definition that R cally Based - on early verification and validation, and it cost-effective should not be ing. CeBASE seeks to transform on up-front prototyping and simulation discouraged by classifying them as avoid- as much as pos- to avoid costly downstream fixes.” able defects. sible from a fad-based practice to an engineering-based practice through deri- vation, organization, and dissemination Software’s complexity and of empirical data on software develop- accelerated development ment and evolution phenomenology. The phrase “as much as possible” reflects the schedules make avoiding defects fact that software development must remain a people-intensive and continu- difficult. These 10 techniques can ally changing field. We have found, how- help reduce the flaws in your code. ever, that researchers have established objective and quantitative data, relation- ships, and predictive models that help For this updated list, we have added Reducing avoidable rework can pro- software developers avoid predictable pit- the word “often” to reflect additional vide significant improvements in software falls and improve their ability to predict insights about this observation. One productivity. In our behavioral analysis and control efficient software projects. insight shows the cost-escalation factor of how software cost drivers affected Here we describe developments in this for small, noncritical software systems to effort for the Cocomo II model (B. Boehm area that have taken place since the publi- be more like 5:1 than 100:1. This ratio et al., Software Cost Estimation with cation of “Industrial Metrics Top 10 List” reveals that we can develop such systems Cocomo II, Prentice Hall, 2000), we in 1987 (B. Boehm, IEEE Software, Sept. more efficiently in a less formal, continu- found that most of the effort savings gen- 1987, pp. 84-85). Given that CeBASE ous prototype mode that still emphasizes erated by improving software process places a high priority on software defect re- getting things right early rather than late. maturity, software architectures, and soft- duction, we think it is fitting to update that Another insight reveals that good ware came from reduc- earlier article by providing the following architectural practices can significantly tions in avoidable rework. Software Defect Reduction Top 10 List. reduce the cost-escalation factor even for large critical systems. Such practices THREE ONE reduce the cost of most fixes by confining About 80 percent of avoidable rework Finding and fixing a software problem them to small, well-encapsulated mod- comes from 20 percent of the defects. after delivery is often 100 times more ules. A good example is the million-line That 80 percent value may be lower expensive than finding and fixing it dur- CCPDS-R system described in Walker for smaller systems and higher for very ing the requirements and design phase. Royce’s book, Software Project Manage- large ones. Two major sources of avoid- As Boehm observed in 1987, “This ment (Addison-Wesley, 1998). able rework involve hastily specified

January 2001 135 Software Management

requirements and nominal-case design them later, we seek techniques that find tions. Improvements in fault detection and development, in which late accom- defects as early as possible. Numerous rates vary from 15 to 50 percent. Further, modation of off-nominal requirements studies confirm that peer review provides focused reading techniques facilitate train- causes major architecture, design, and an effective technique that catches from ing of inexperienced personnel, improve code breakage. A tracking system for 31 to 93 percent of the defects, with a communication about the process, and software-problem reports that records median of around 60 percent. Thus, the foster continuous improvement. the effort to fix each defect lets you ana- 60 percent value cited in the 1987 col- lyze the data fairly easily to determine umn remains a reasonable estimate. EIGHT and address additional major sources of Disciplined personal practices can rework. reduce defect introduction rates by up to 75 percent. FOUR Peer reviews, Several disciplined personal processes About 80 percent of the defects come analysis tools, have been introduced into practice. from 20 percent of the modules, and and testing catch These include Harlan Mills’s Cleanroom about half the modules are defect free. different classes of software development process and Watts Studies from different environments defects different Humphrey’s Personal Software Process over many years have shown, with amaz- points in the (PSP). ing consistency, that between 60 and 90 development cycle. Data from the use of Cleanroom at percent of the defects arise from 20 per- NASA have shown 25 to 75 percent reduc- cent of the modules, with a median of tions in failure rates during testing. Use of about 80 percent. With equal consis- Cleanroom also showed a reduction in tency, nearly all defects cluster in about Factors affecting the percentage of rework effort so that only 5 percent of the half the modules produced. defects caught include the number and fixes took more than an hour, whereas the Obviously, then, identifying the char- of peer reviews performed, the size standard process caused more than 60 per- acteristics of error-prone modules in a and complexity of the system, and the cent of the fixes to take that long. particular environment can prove worth- frequency of defects better caught by exe- PSP’s strong focus on root-cause analy- while. A variety of context-dependent cution, such as concurrency and algo- sis of an individual’s software defects and factors contribute to error-proneness. rithm defects. Our studies have provided overruns, and on developing personal Some factors usually contribute to error- evidence that peer reviews, analysis tools, checklists and practices to avoid future proneness regardless of context, however, and testing catch different classes of recurrence, has significantly reduced per- including the level of data coupling and defects at different points in the devel- sonal defect rates. Developers frequently cohesion, size, complexity, and the opment cycle. We need further empirical enjoy defect reductions of 10:1 between amount of change to reused code. research to help choose the best mixed exercises 1 and 10 in the PSP training strategy for defect-reduction investments. course. FIVE Effects at the project level are more scat- About 90 percent of the downtime SEVEN tered. They depend on factors such as the comes from, at most, 10 percent of the Perspective-based reviews catch 35 organization’s existing software maturity defects. percent more defects than nondirected level and the staff’s and organization’s Some defects disproportionately affect reviews. willingness to operate within a highly a system’s downtime and reliability. For A scenario-based reading technique structured software culture. When you example, an analysis of the software fail- (V.R. Basili, “Evolving and Packaging couple PSP with the strongly compatible ure history of nine large IBM software Reading Technologies,” J. Systems and Team Software Process (TSP), defect products revealed that about 0.3 percent Software, vol. 38, no. 1, 1997, pp. 3-12) reduction rates can soar to factors of 10 or of the defects accounted for about 90 offers a set of formal procedures for higher for an organization that operates percent of the downtime. Thus, risk- defect detection based on varying per- at a modest maturity level. Results tend to based testing—including understanding spectives. The union of several perspec- be less spectacular if the organization a system’s operational profiles and tives into a single inspection offers broad already employs highly mature processes. emphasizing testing of high-risk scenar- yet focused coverage of the document The June 2000 special issue of ios—is clearly cost-effective. being reviewed. This approach seeks to CrossTalk, “Keeping with PSP and generate focused techniques aimed at TSP,” offers a good set of relevant dis- SIX specific defect-detection goals by taking cussions, including experience showing Peer reviews catch 60 percent of the advantage of an organization’s existing that adding PSP and TSP to a CMM defects. defect history. Level 5 organization reduced acceptance Given that finding and fixing most Scenario-based reading techniques have defects by about 50 percent overall, defects earlier in the project development been applied in requirements, object-ori- and reduced high-priority defects by cycle is more cost-effective than finding ented design, and user interface inspec- about 75 percent.

136 Computer NINE A 1987 study in this area (P.S. Brown software engineering research challenge All other things being equal, it costs 50 and J.D. Gould, “An Experimental Study is one of several identified by a National percent more per source instruction to of People Creating Spreadsheets,” ACM Science Foundation study, “Gaining develop high-dependability software Trans. Office Info. Sys., July 1987, pp. Intellectual Control of Software Develop- products than to develop low-depend- 258-272) found that 44 percent of 27 ment,” which we recently summarized in ability software products. However, the spreadsheet programs produced by expe- Computer (May 2000, pp. 27-33). investment is more than worth it if the rienced spreadsheet developers contained project involves significant operations nontrivial defects—mostly errors in and maintenance costs. spreadsheet formulas. Yet the developers urely, our list can benefit from The analysis of 161 project data points felt confident that they had produced refinement and further empirical for the Cocomo II model resulted in an accurate spreadsheets. S research on defect reduction. Much added cost of 53 percent for its “required of the data we have reported, for exam- reliability” factor, while normalizing for ple, fails to account for the interaction the effects of 22 other factors. Does this between many of the variables that, if mean that Philip Crosby’s landmark The creators of known, could provide answers to ques- book, Quality Is Free (Mentor, 1980), Web-programming tions like: had it all wrong? Maybe for some low- facilities face • criticality, short-lifetime software, but the challenge of If I invest in peer reviewing, not for the most important cases. Cleanroom, and PSP, am I paying First, in the Cocomo II maintenance providing their tools for the same defects to be removed model, low-dependability software costs with the equivalent of three times? about 50 percent per instruction more to seat belts and • How much testing would this maintain than to develop, whereas high- air bags, along with investment enable me to avoid? dependability software costs about 15 safe-driving aids percent less to maintain than to develop. and rules of the road. We hope to involve the software com- For a typical life-cycle cost distribution munity in expanding the Software Defect of 30 percent development and 70 per- Reduction Top 10 List and other cur- cent maintenance, low-dependability rently available data into a continually software becomes about the same in cost Subsequent laboratory experiments evolving, open source, Web-accessible per instruction as high-dependability have reported defective spreadsheet rates handbook of empirical results on soft- software—again, assuming all other fac- between 35 and 90 percent. The analy- ware-defect reduction strategies. We also tors are equal. sis of operational spreadsheets reveals plan to initiate counterpart handbooks Second, in the Cocomo II-related qual- defect rates between 21 and 26 percent; for commercial off-the-shelf systems and ity model, high-dependability software the lower rates probably stem from cor- other emerging software areas. We wel- removes about four times as many rections already made during operation. come your participation in this effort and defects as average-dependability soft- Now, and increasingly in the future, urge you to visit the CeBASE Web site at ware, which in turn removes about four user programs will escalate from spread- http://www.cebase.org for further infor- times as many defects as low-depend- sheets to Web-scripting languages capa- mation. You can also find an expanded ability software. For example, consider ble of sending agents into cyberspace to version of this column at http://www. an average-dependability system such as make deals for you. The ranks of “sor- cebase.org/defectreduction/ top10. ✸ a commercial billing system, in which the cerer’s apprentice” user-programmers operational cost of software defects— will also swell rapidly, giving many due to lost worker time, lost sales, added have little training or expertise in how to customer service costs, litigation costs, avoid or detect high-risk defects tremen- loss of repeat business, and so on— dous power to create high-risk defects. Barry Boehm is director of the Univer- roughly equals life-cycle software devel- One study for the Cocomo II book esti- sity of Southern California Center for opment and maintenance costs. For such mated that the US will have 55 million Software Engineering. Contact him at a system, the increased defect rate of user-programmers by 2005. If we clas- [email protected]. using low-dependability software would sify active Web-page developers as user- make its ownership costs roughly three programmers, this prediction appears to times higher than the ownership costs of be on track. high-dependability software. Thus, the creators of Web-program- Victor R. Basili is a professor in the Insti- ming facilities face the challenge of pro- tute for Advanced Computer Studies and TEN viding their tools with the equivalent of the Department at the About 40 to 50 percent of user pro- seat belts and air bags, along with safe- University of Maryland. Contact him at grams contain nontrivial defects. driving aids and rules of the road. This [email protected].

January 2001 137