The Software Quality Challenge

Software Quality The Software Quality Challenge Watts S. Humphrey The Software Engineering Institute Many aspects of our lives are governed by large, complex systems with increasingly complex software, and the safety, security, and reliability of these systems has become a major concern. As the software in today’s systems grows larger, it has more defects, and these defects adversely affect the safety, security, and reliability of the systems. This article explains why the com- mon test-and-fix software quality strategy is no longer adequate, and characterizes the properties of the quality strategy we must pursue to solve the software quality problem in the future. oday, many of the systems on which defects per page while even poor-quality different ways they can be used, and the our lives and livelihoods depend are software has much less than one defect more ways users can use them, the harder runT by software. Whether we fly in air- per listing page. This means that the qual- it is to test all of these conditions in planes, file taxes, or wear pacemakers, our ity level of even poor-quality software is advance. This was the logic behind the safety and well being depend on software. higher than that obtained for other kinds beta-testing strategy started at IBM with With each system enhancement, the size of human written text. Programming is an the OS/360 system more than 40 years and complexity of these systems increase, exacting business, and these professionals ago. Early copies of the new system releas- as does the likelihood of serious prob- are doing extraordinarily high quality es were sent to a small number of trusted lems. Defects in video games, reservations work. The only problem is that based on users and IBM then fixed the problems systems, or accounting programs may be historical trends, future systems will be they found before releasing the public ver- inconvenient, but software defects in air- much larger and more complex than sion. This strategy was so successful that it craft, automobiles, air traffic control sys- today, meaning that just to maintain has become widely used by almost all ven- tems, nuclear power plants, and weapons today’s defect levels, we must do much dors of commercial software. systems can be dangerous. higher quality work in the future. Unfortunately, however, the beta-test- Everyone depends on transportation To appreciate the challenge of achiev- ing strategy is not suitable for life-critical networks, hospitals, medical devices, pub- ing 10 or fewer defects per million lines of systems. The V-22 Osprey helicopter, for lic utilities, and the international financial code, consider what the source listing for example, uses a tilting wing and rotor sys- infrastructure. These systems are all run such a program would look like. The list- tem in order to fly like an airplane and by increasingly complex and potentially ing for a 1,000-line program would fill 40 land like a helicopter. In one test flight, the defective software systems. Regardless of text pages; a million-line program would hydraulic system failed just as the pilot was whether these large life-critical systems are take 40,000 pages. Clearly, finding all but tilting the wing to land. While the aircraft newly developed or composed from mod- 10 defects in 40,000 pages of material is had a built-in back-up system to handle ified legacy systems, to be safe or secure, humanly impossible. However, we now such failures, the aircraft had not been they must have quality levels of very few have complex life-critical systems of this tested under those precise conditions, and defects per million parts. scale and will have much larger ones in the the defect in the back-up system’s soft- Modern, large-scale systems typically relatively near future. So we must do ware had not been found. The defect have enormous requirements documents, something, but what? That is the question caused the V-22 to become unstable and large and complex designs, and millions of addressed in this article. crash, killing all aboard. lines of software code. Uncorrected errors The problem is that as systems in any aspect of the design and develop- Why Defective Systems Work become more complex, the number of ment process generally result in defects in To understand the software quality prob- possible ways to use these systems grows the operational systems. The defect levels lem, the first question we must answer is If exponentially. The testing problem is fur- of such operational systems are typically today’s software is so defective, why aren’t there ther complicated by the fact that the way measured in defects per thousand lines of more software quality disasters? The answer is such systems are configured and the envi- code. A one million line-of-code system that software is an amazing technology. ronments in which they are used also with the typical quality level of one defect Once you test it and fix all of the prob- affect the way the software is executed. per 1,000 lines would have 1,000 undis- lems found, that software will always work Table 1 lists some of the variations that covered defects, while any reasonably safe under the conditions for which it was test- must be considered in testing complex system of this scale must have only a very ed. It will not wear out, rust, rot, or get systems. An examination of the number few defects, certainly less than 10. tired. The reason there are not more soft- of possibilities for even relatively simple systems shows why it is impractical to test The Need for Quality ware disasters is that testers have been able to exercise these systems in just about all possibilities for any complex system. So Software all of the ways they are typically used. So, why is complex software so defective? Before condemning programmers for to solve the software quality problem, all doing sloppy work, it is appropriate to we must do is keep testing these systems Some Facts consider the quality levels of other types in all of the ways they will be used. So Software is and must remain a human- of printed media. A quick scan of most what is the problem? produced product. While tools and tech- books, magazines, and newspapers will The problem is complexity. The more niques have been devised to automate the reveal at least one and generally more complex these systems become, the more production of code once the requirements 4 CROSSTALK The Journal of Defense Software Engineering June 2008 A The Software Quality Challenge and design are known, the requirements 1. Data rates Switche s Paths and design must be produced by people. 2. Data value s 1 2 Further, as systems become increasingly 3. Data err ors 4 6 complex, their requirements and design 4. Con figu ration variation s 9 20 grow increasingly complex. This complex- 5. Numbe r, type , and timing of 16 70 ity then leads to errors, and these errors simultaneou s process es 36 92 4 result in defects in the requirements, 6. Hardware f ail ures 49 3,43 2 design, and the operational code itself. 7. Network f ail ures 64 12 ,87 0 Thus, even if the code could be automati- 8. Ope rator err ors 81 48 ,62 0 cally generated from the defective require- 9. Version change s 10 0 184 ,75 6 ments and design, that code would reflect 10 . Power variation s 40 0 1.38 E+1 1 these requirements and design defects and, thus, still be defective. Table 1: Some of the Possible Testing Table 2: Possible Paths Through a Network When people design things, they make Variations defects. For example, a complex design mistakes. The larger and more complex likely to be exercised when such systems defect that produced a confusing operator their designs, the more mistakes they are are30 0 subjected to the stresses of high trans- message could pose no danger whileDes iagn/Code likely to make. From course data on thou- action volume, accidents, failures, or mili- trivial typographical mistake that changed sands of experienced engineers learning SM SM tary250 combat. a no to a yes could be very dangerous. the Personal Software Process (PSP ), it e d Since there is no way to tell in advance has been found that developers typically o The Defect Removal Problem c which defects would be damaging, we / f inject about 100 defects into every 1,000s 200 t o A defect is an incorrect or faulty con- must try to find them all. Then, after find- c s lines of the code they write [1]. The distri-e f struction in a product. For software, ing them, we must fix at least all of the e e bution for the total defects injected by 810 n Design Review Time i 150 d defects generally result from mistakes that l ones that would be damaging. experienced developers at the beginning ofl d a the designers or developers make as they t n PSP training is shown by the total bars ino a produce their products. Examples are The Testing Problem T 100 s Figure 1. While there is considerable varia- u oversights, misunderstandings, and typos. Since defects could be anywhereTotal in a large tion and some engineers do higher-quality o h t Furthermore, since defects result from software system, the only way testing 50 Test work, just about everybody injects defects. mistakes, they are not logical. As a conse- could find them all would be to complete- Developers use various kinds of tools quence, there is no logical or deductive ly test every segment of code in the entire to generate program code from their 0 process that1.

Load more