University of Tennessee, Knoxville TRACE: Tennessee Research and Creative Exchange The Harlan D. Mills Collection Science Alliance 9-1987 Cleanroom Software Engineering Harlan D. Mills M. Dyer R. C. Linger Follow this and additional works at: https://trace.tennessee.edu/utk_harlan Part of the Software Engineering Commons Recommended Citation Mills, Harlan D.; Dyer, M.; and Linger, R. C., "Cleanroom Software Engineering" (1987). The Harlan D. Mills Collection. https://trace.tennessee.edu/utk_harlan/18 This Article is brought to you for free and open access by the Science Alliance at TRACE: Tennessee Research and Creative Exchange. It has been accepted for inclusion in The Harlan D. Mills Collection by an authorized administrator of TRACE: Tennessee Research and Creative Exchange. For more information, please contact [email protected]. <4ag : Q U A L I T Y A S S U R A N C E Cleanroom Software Engineering Harlan D. Mills, Information Systems Institute Michael Dyer and Richard C. Linger, IBM Federal Systems Division Software quality can be R ecent experience demonstrates ate units oftime (real or processor time) of that software can be engineered the delivered product. The certification engineered under R under statistical quality control takes into account the growth of reliabil- statistical quality control and that certified reliability statistics can ity achieved during system testing before withbetter be provided with delivered software. delivery. anddelivered IBM's Cleanroom process' has uncovered To gain the benefits of quality control quality. The Cleanroom a surprising synergy between mathemati- during development, Cleanroom software process gives manage- cal verification and statistical testing of engineering requires a development cycle an software, as well as a major difference of concurrent fabrication and certification ment engineering between mathematical fallibility and of product increments that accumulate approach to release debugging fallibility in people. into the system to be delivered. This lets reliable products. With the Cleanroom process, you can the fabrication process be altered on the engineer software under statistical quality basis of early certification results to control. As with cleanroom hardware achieve the quality desired. development, the process's first priority is defect prevention rather than defect Cleanroom experience removal (of course, any defects not Typical of our experience with the prevented should be removed). This first Cleanroom process were three projects: an priority is achieved by using human IBM language product (40,000 lines of mathematical verification in place of pro- code), an Air Force contract helicopter gram debugging to prepare software for flight program (35,000 lines), and a NASA system test. contract space-transportation planning system (45,000 lines). A major finding in statistical certification of the software' these cases was that human verification, quality through representative-user testing even though fallible, could replace debug- atthesystem level. The measure of qual- ging in software development - even ityTis the mean time to failure in appropri- informal human verification can produce September 1987 0740-7459/87/0900/0019/$01.00 © 1987 IEEE 19 software sufficiently robust to go to sys- processing (1000 to 2000 lines), indicates The Cleanroom process permits a tem test without debugging. better productivity and quality with the sharper structuring of development work Typical program increments were 5000 Cleanroom process than with interactive between specification, design, and testing, to 15,000 lines of code. With experience debugging and integration - even the first with clearer accountabilities for each part and confidence, such increments can be time you use it.3 of the process. This structuring increases expected to increase in size significantly. management's ability to monitor work in All three projects showed productivity Management progress. Inexperienced software equal to or better than expected for ordi- perspective managers often fail to recognize and nary software development: Human At first glance, statistical quality control expose early software problems (like hard- verification need take no more time than and software development seem incom- ware or specification instability, inex- debugging (although it takes place earlier patible. Statistical quality control seems to perienced personnel, and incomplete in the cycle). apply to manufacturing, especially man- design solutions) and mistakenly think The combination of formal design ufacturing of multiple copies of a previ- they can resolve and manage these prob- methods and mathematics-based verifica- ously specified and designed part or lems over time. The Cleanroom process tion had a positive development effect: product. Software development seems to forces these early problems into the open, More than 90 percent of total product be a one-of-a-kind logical process with no giving all levels of management an oppor- defects were found before first execution. statistical properties at all. After all, ifthe tunity to resolve them. This is in marked contrast to the more cus- software ever fails under certain condi- The Cleanroom process requires stable tomary experience of finding 60 percent of tions, it will always fail under those con- specifications as its basis. Because speci- product defects before first execution. ditions. fications are often not fully known or veri- This effect is probably directly related to fied during initial development, it might the added care and attention given to appear at first glance that the Cleanroom design in lieu ofrushing into code and rely- Developing stable process does not apply. But, in fact, the ing on testing to achieve product quality. specifications early discipline of the Cleanroom process is A second encouraging trend is the drop most useful in forcing specification defi- in total defect count (by as much as half), establishes clear ciencies into the open and giving manage- which highlights the Cleanroom focus on accountability. ment control ofthe specification process. error prevention as opposed to error detec- As long as development is treated as a tion. industry averages at 50 to 60 trial-and-error process, the incompleteness With However, by rethinking the process of errors per 1000 lines ofcode, halving these of specification can be accommodated as statistical quality control itself from a just one more source oftrial and error. The numbers is significant. management perspective, we can find a The IBM language product result is diluted accountability between way to put software development under specifiers and developers. A better way is (Cobol/SF2) experience is especially statistical quality control with significant instructive-. This advanced technology to develop software to early, stable speci- management benefits. fications that remain stable in each itera- product, comparable in complexity to a But where do the statistics come from, compiler, was formally specified and then tion. This establishes a clear accountability when neither software nor its development between specification and development, designed in a process-design language. have any statistical properties at all? The Specification text exceeded design text by keeping management in control of speci- statistics come from the usage ofthe soft- fication changes. about four to one. Every control structure ware, not from its intrinsic properties. in the design text was verified in formal, Engineering software under statistical mathematics-based group inspection, so quality control requires that we not only Statistical quality the product proved very robust. A first specify the functional behavior of the soft- control phase of development (20,000 lines) had ware but also its statistical usage. Statistical quality control begins with an just 53 errors found during testing. Cleanroom software engineering is a agreement between a producer and Correctness verification was the corner- practical process to place software devel- receiver. A critical part of this agreement, stone ofthe project; many programs were onM*'rUt Inder- St-2Ati%'t;zd ~julicontro. explicit or implicit, is how to measure qual- redesigned to permit simpler verification The significance of a process under statisti- ity, particularlystatistical quality. For sim- arguments. Productivity averaged more cal quality control is well-illustrated by ple products with straightforward than 400 lines of code per man-month, modern manufacturing techniques where physical, electrical, or other measure- largely as a result of sharply reduced test- the sampling of output is directly fed back ments, the agreement may be simply stated ing time and effort compared to conven- into the process to control quality. Once - for example, 99 percent of certain fila- tional developments. the discipline of statistical quality control ments are to exhibit an electrical resistance A controlled experiment at the Univer- is in place, management can see the devel- within 10 percent of a fixed value. How- sity of Maryland, with student teams opment process and can control process ever, software is complex enough to developing a common project in message changes to control product quality. require a new understanding on how 20 IEEE Software statistical quality can be measured. and performance. So a certification of having no gotos without acquiring the fun- For even the simplest of products, there software quality is a business measure, damental