focusguest editors’ introduction

Inspection’s Role in Quality Assurance

David L. Parnas, Mark Lawford, McMaster University

espite more than 30 years’ effort to improve , companies still release programs containing numerous errors. Many major products have thousands of bugs. It’s not for lack of D trying; all major software developers stress software quality as- surance and try to remove bugs before release. The problem is the code’s complexity. It’s easy to review code but fail to notice significant errors.

Researchers have responded to these prob- lems by studying methods of formal correct- ness verification for programs. In theory, we now know how to prove programs correct with the same degree of rigor that we apply to mathematical theorems. In reality, this is rarely practical and even more rarely done. Most research papers on verification make simplifying assumptions (for example, a 1:1 correspondence between variables and vari- able names) that aren’t valid for real pro- grams. Proofs of realistic programs involve long, complex expressions and require pa- tience, time, and diligence that developers don’t think they have. (Interestingly enough, they never have time to verify a program be- fore release, but they must take time to re- spond to complaints after release.) Inspection methods can be more effective than informal reviews and require less effort than formal

16 IEEE SOFTWARE Published by the IEEE Computer Society 0740-7459/03/$17.00 © 2003 IEEE proofs, but their success depends on having a The TSE papers do communicate the re- sound, systematic procedure. Tools that sup- search results. We considered a paper to port this procedure are also important. make a valid contribution if it showed Connecting The Workshop on Inspection in Software how to apply known results to the prob- Engineering (WISE), a satellite event of the lem of inspecting software for suitability theory with 2001 Computer Aided Verification Confer- (fitness for use). The papers are intended practice is the ence (CAV 01), brought together researchers, for people who are willing to read de- essence of any practitioners, and regulators in the hope of tailed, careful research papers. finding new, more effective software inspec- type of tion approaches. Submissions described how We intend that each TSE paper and its corre- engineering. practitioners and researchers were performing sponding Software article have little overlap. inspections, discussed inspections’ relevance, One reports results; the other explains how to provided evidence of how refinement of the use those results and perhaps what research is inspection process and computer-aided tool still needed. support can improve inspections, and ex- We hope that future guest editors will emu- plained how careful could late and improve the linked-special-issues ap- make inspections more effective. The best proach for other topics important to both ideas from the workshop have been distilled practitioners and researchers. After all, con- into pairs of articles appearing in linked spe- necting theory with practice is the essence of cial issues of IEEE Software and IEEE Trans- any type of engineering. actions on . What we mean by inspection Why two linked special issues? By inspection we mean a systematic ap- As guest editors, we had a specific goal proach to examining a program in detail. Such when we proposed the joint special issues to an examination’s goal is to assess the quality the publications’ editorial boards. We had ob- of the software in question, not the quality of served that the practitioners habitually neglect the software development process. the kind of research found in TSE on the In general, inspection means examining a (sometimes correct) assumption that it’s irrele- product by following a prescribed, systematic vant to them. On the other hand, researchers process that aims to determine whether the tend to write for each other and to lose con- product is fit for its intended use. For exam- tact with the realities that practitioners must ple, many jurisdictions require vehicle safety face. The linked issues try to narrow this gap. inspections. (Some advocates of specific ap- Some contributors to WISE were practitioners proaches to software inspection assume that who explained what they’re doing and what their method defines “inspection.” In fact, the problems they encounter. Others were re- word was well defined much earlier.) They leg- searchers trying to discover and verify (either islate a list of parts that must be examined, mathematically or experimentally) some fun- measurements that must be made, and so on, damental facts. We thought that these re- and criteria for passing the inspection. The searchers should write articles that explained word “inspection” usually implies that the to practitioners why the problems they were process is described in documents (for exam- studying were relevant to practice. We also ple, checklists and printed forms) that explain thought that the practitioners could communi- exactly what the inspectors must do. These cate to researchers what inspecting a program documents try to ensure that each inspection is actually like. In addition, we asked that the is so careful and so complete that an inspec- purpose of each paper in one publication be tion’s failure to reveal any defects justifies hav- explained to the readers of the other. ing great confidence that the product will per- To summarize: form as required. An inspection process, while systematic The Software articles aim to make practi- and tightly prescribed, isn’t mechanical; the tioners aware of research ideas that they process description guides the inspectors but might be able to apply. These articles don’t isn’t so prescriptive that a machine could per- communicate the research results as com- form inspections without human involvement. pletely as a normal research paper would. Success depends on the inspectors understand-

July/August 2003 IEEE SOFTWARE 17 ing the product and the underlying technolo- cation helps determine mathematical correct- gies, knowing how to use the appropriate ness, but you can have error-free (mathemati- Early inspection tools, and having considerable experience do- cally correct) code that’s hard to understand ing related work. and difficult to maintain. In addition to find- of a document Because the inspectors, like all of us, have ing errors in code and related software docu- that states limits on their ability to handle details, the key ments, inspection can also help determine system to inspection of any complex product is a pol- whether coding style guidelines are followed, icy of divide and conquer—that is, examining comments in the code are relevant and of ap- requirements small parts of the product in isolation, while propriate length, naming conventions are clear can help insure ensuring that and consistent, the code is easy to maintain, and so on. Although these issues are irrelevant that the correct Nothing is overlooked to theorem provers, model checkers, and au- system is built. The inspected components’ correctness tomated testing tools, they are crucial to the implies the whole product’s correctness cost of building and maintaining large soft- ware products. The inspection’s organization as a set of You don’t need to wait until code is com- discrete steps must assure that each step is plete to reap inspection’s benefits. Early in- simple enough to carry out reliably and that spection of a document that states system re- one step can be carried out without detailed quirements can help insure that the correct knowledge of the others. Inspection can be system is built. In our experience, even when a time consuming. Moreover, no inspection product’s formal verification uses mathemati- process is perfect. Inspectors might take short- cal requirements, they might not accurately cuts, might inadequately understand what capture the designer’s or customer’s intent. In- they are doing, and might identify a product spection of a requirements document helps as- as acceptable when it isn’t. Nonetheless, a sure that the requirements are capturing the well-designed inspection process can find er- right thing. rors that other methods would miss and can engender great trust. Inspection’s future Although many companies now do inspec- Inspection’s benefits tion, they can do better. The Software articles Industry widely employs testing and the re- in this issue provide insights into how soft- search community widely advocates formal ware practitioners can improve their inspec- verification as methods for improving soft- tions’ effectiveness and applicability today. ware quality. Inspection falls somewhere be- The TSE research papers provide the theoreti- tween the two. Formal verification has yet to cal underpinnings of these suggested improve- catch on with software practitioners, while in- ments and offer insights into how we could spection in one form or another has been further improve inspections in the future. The widely adopted by industry and advocated by TSE articles are evidence that inspection con- leading software practitioners (for example, tinues to be an active area of academic research. see Stuart Ball’s Embedded Microprocessor Systems: Real World Design, Newnes, 2000, Refining software inspection pp. 16–24). This difference in acceptance has One way that researchers and practitioners three main causes: are addressing current inspection techniques’ limitations is by refining inspection methods You can inspect the code itself, not just to make them more appropriate for a particu- abstract models of it. lar setting. Such refinement will help inspec- Inspection doesn’t require as substantial a tors focus on the most important problems in training investment as verification. the sea of details. Inspection doesn’t require the time and the In “Practical Code Inspection Techniques for formula manipulation ability that verifica- Object-Oriented Systems: An Experimental tion of typical programs does. Comparison,” Alastair Dunsmore, Marc Roper, and Murray Wood propose three techniques for Inspection seeks to complement testing. inspecting OO systems and provide preliminary Testing helps detect errors, and formal verifi- data on their relative effectiveness.

18 IEEE SOFTWARE http://computer.org/software Reading can be a rudimentary form of in- prove inspection. Many opportunities exist for spection. In “Prioritized Use Cases as a Vehi- tools to improve inspection efficiency and ac- cle for Software Inspections,” Thomas Thelin, curacy, ranging from tools that support the in- Per Runeson, and Claes Wohlin combine use spection process’s workflow and bookkeep- cases and operational-profile testing to assess ing, to integrated computer-aided verification software from a user’s viewpoint. They then techniques that let inspectors ask the impor- use an experimental evaluation to compare tant questions and delegate some of the me- their method’s effectiveness to that of another chanical details to model checkers, theorem well-established reading technique. provers, and other tools. In “Tool Support for Although these papers differ in their con- Fine-Grained Software Inspection,” Paul An- clusions, they illustrate how customizing the derson, Thomas Reps, Tim Teitelbaum, and inspection process to the task at hand can pro- Mark Zarins explain how to use their vide benefits. CodeSurfer tool to make inspection of com- plex software systems more manageable. Systems with real-time requirements and concurrent activities Software systems that must deal with a va- riety of ongoing activities (for example, device management, user interactions, and external- n May of this year, a Soyuz TMA-1 event monitoring) are generally less trustwor- spaceship carrying a Russian cosmonaut thy than purely sequential programs. Concur- I and two American astronauts landed rency introduces a form of nondeterminism nearly 500 km off course after the craft unex- into the system—external events, which hap- pectedly switched to a ballistic reentry trajec- pen at unpredictable times, determine the in- tory. Preliminary indications are that the prob- ternal events’ order. (We consider a system to lem was caused by software in the guidance be nondeterministic when the information computer in the new, modified version of the available to the observer or assessor is insuffi- spaceship. That same week, Microsoft’s Pass- cient to determine the system behavior. In such cases, a system should be designed to handle all possible behaviors.) When nondeterminism is present, an assessor’s inability to remain Further Reading aware of all possible sequences makes inspec- tion more difficult. Nondeterminism also A. Aurum, H. Petersson, and C. Wohlin, ”State-of-the-Art: Software In- makes testing more difficult because a test se- spections after 25 Years,” Verification Reliability, vol. quence might cause an error in one case but 12, no. 3, Sept. 2002, pp. 133–154. not in another. M.E. Fagan, “Design and Code Inspections to Reduce Errors in Program One potentially worthwhile approach to Development,” IBM Systems J., vol. 15, no. 3, 1976, pp. 182–211. quality assessment of systems with real-time requirements in the presence of concurrency is T. Gilb and D. Graham, Software Inspection, Addison-Wesley, 1993. to restrict the design to place it in a class that’s P. Johnson, The WWW Formal Technical Review Archive, 1999, http:// easier to analyze. In what’s likely the special www2.ics.hawaii.edu/~johnson/FTR. issues’ most controversial article, “Making D.L. Parnas and D.M. Weiss, “Active Design Reviews: Principles and Software Timing Properties Easier to Inspect Practices,” J. Systems and Software, vol. 7, 1987, pp. 259–265. Also and Verify,” Jia Xu advocates handling con- published in Software Fundamentals: Collected Papers by David L. current real-time systems through a prerun- Parnas, D.M. Hoffman and D.M. Weiss, eds., Addison-Wesley, 2001, time scheduling approach. He asks designers Chapter 17. to accept strong restrictions on their work to make the inspector’s job easier. D.L. Parnas, “Inspection of Safety Critical Software Using Function Tables,” Proc. IFIP 13th World Computer Congress, vol. 3, North- Tool-supported software inspection Holland, 1994, pp. 270–277. Also published in Software Fundamen- We organized WISE as a satellite event of tals: Collected Papers by David L. Parnas, D.M. Hoffman and D.M. CAV 01 partly because we believe that com- Weiss, eds., Addison-Wesley, 2001, Chapter 19. puter-aided inspection and formal verification techniques have the greatest potential to im-

July/August 2003 IEEE SOFTWARE 19 port online information repository system was lic’s confidence that software can be used to found to have a major security flaw that let an build safe, secure systems. In response, both attacker access a user’s personal information software practitioners and software re- simply by knowing the user’s email address searchers must improve software’s reputation; and constructing an appropriate URL. the only way to do that is to improve software These are just the latest examples of soft- quality. Inspection is one way to do this. Still, ware quality lapses that are shaking the pub- we need further research to find more practi- cal, effective inspection approaches and to measure their effectiveness. We hope that these special issues of Software and TSE motivate About the Authors others to take up this increasingly important challenge. David L. Parnas is the director of the Software Quality Research Laboratory, a Science Foundation Ireland Fellow, and a professor of software engineering at the University of Limer- ick. He is also on leave from McMaster University. He is interested in most aspects of computer system design. He received his PhD in from Carnegie Mellon University and is licensed as a professional engineer in Ontario, Canada. He is a fellow of the Royal Soci- Acknowledgments ety of Canada and of the ACM, a senior member of the IEEE, and a member of the IEEE Com- We thank all the 2001 Workshop on Inspection in puter Society. Contact him at the Dept. of and Information Systems, Univ. of Software Engineering participants, particularly those Limerick, Limerick, Ireland; [email protected]. who submitted articles for consideration in the spe- cial issues. The special issues wouldn’t have been pos- Mark Lawford is an assistant professor in McMaster University’s Department of Comput- sible without the reviewers’ understanding and invalu- ing and Software, where he is helping to develop and teach the software engineering programs. able feedback to the editors and authors. Finally, we His research interests include discrete-event systems and the practical application of formal thank John Knight, Steve McConnell, and Warren methods to real-time systems. He received his PhD in electrical and from Harrison for encouraging the linked-special-issues the University of Toronto. He is a member of the IEEE Control Systems Society and the IEEE idea, and the IEEE Software and IEEE Transactions Computer Society. Contact him at the Software Quality Research Laboratory, Dept. of Computing on Software Engineering editorial staffs for their and Software, McMaster Univ., 1280 Main St. West, Hamilton, ON L8S 4K1, Canada. support and understanding.

20 IEEE SOFTWARE http://computer.org/software