FEATURE: SOFTWARE REVIEWS We examined more than 100,000 Contemporary peer reviews in OSS case studies of the Apache httpd server, Subversion, Linux, FreeBSD, KDE, and Gnome and found an efficient fit between Peer Review OSS developers’ needs and the mini- malist structures of their peer review processes.3 Specifically, the projects in Action: broadcast changes asynchronously to the development team—usually on a Lessons from Open Source mailing list—and reviewers self-select changes they’re interested in and com- Development petent to review. Changes failing to capture a reviewer’s interest remain unreviewed. Developers manage what can be an overwhelming broadcast Peter C. Rigby, Concordia University, Montreal, Canada of information by relying on simple email filters, descriptive email sub- Brendan Cleary, University of Victoria, Canada jects, and detailed change logs. The change logs represent the OSS proj- Frederic Painchaud, Department of National Defence, Canada ect’s heart beat, through which de- velopers maintain a conceptual un- Margaret-Anne Storey and Daniel M. German, University of Victoria, derstanding of the whole system and Canada participate in the threaded email dis- cussions and reviews for which they have the required expertise. // Open source development uses a rigorous but The OSS process evolved naturally to fit the development team and con- agile review process that software companies trasts with enforced inspections based can adapt and supplement as needed by on best practices that are easily mis- popular tools for lightweight collaboration and applied and end in false quality assur- ances, frustrated developers, and lon- nonintrusive quality assurance metrics. // ger development cycles. As Michael Fagan, the father of formal inspection, lamented about the process he devel- oped, “Even 30 years after its creation, it is often not well understood and more often, poorly executed.”1 In this article, we contrast OSS peer review with a traditional inspection process that’s widely acknowledged in the literature—namely, inspections SOFTWARE INSPECTION IS a form even though developers acknowledge performed on large, completed soft- of formal peer review that has long the value of formal peer review, many ware artifacts at specific checkpoints. been recognized as a software engi- also avoid it, and the adoption rates The inspectors are often unfamiliar neering “best practice.” However, the for traditional inspection practices are with the artifact under inspection, prospect of reviewing a large, unfa- relatively low.1,2 so they must prepare individually be- miliar software artifact over a period On the other hand, peer review fore the formal review by thoroughly of weeks is almost universally dreaded is a prevalent practice on successful studying the portion of code to be re- by both its authors and reviewers. So, open source software (OSS) projects. viewed. Defects are recorded subse- 56 IEEE SOFTWARE | PUBLisHED BY THE IEEE COMPUTER SOCieTY 0740-7459/12/$31.00 © 2012 IEEE quently at the formal review meeting, but the task of fixing a recorded defect Inspection software Asynchronous, Open source falls to the author after the meeting. reviews tool-supported reviews software reviews Some intrinsic differences between open source and proprietary develop- ment projects, such as self-selected Formal but Measureable Minimalist but cumbersome but lightweight lacks traceability versus assigned participation, suggest inspection processes at opposite ends of a continuum (see Figure 1). How- FIGURE 1. The spectrum of peer review techniques, from formal inspection to minimal- ever, neither formality nor aversion is process OSS review. Tool-supported, lightweight review provides a flexible but traceable fundamental to peer review. The core middle ground. idea is simply to get an expert to ex- amine your work to find problems you can’t see. Success in identifying defects search settings. This led to tools and inspection processes means that large, depends less on the process than on practices that let developers interact in completed artifacts are inspected at the expertise of the people involved.4 an asynchronous, distributed manner. specific checkpoints that might occur We present five lessons from OSS Furthermore, the hard time constraints many months apart. The calendar time projects that we think are transferable imposed by colocated meetings, the to inspect these completed artifacts is to proprietary projects. We also pres- rigid goal of finding defects, and the on the order of weeks. ent three recommendations for adapt- sole metric of defects found per line In contrast, most OSS peer re- ing these practices to make them more of source code encouraged a mentality views begin within hours of complet- traceable and appropriate for propri- of “Raise issues, don’t resolve them.”2 ing a change, and the full review dis- etary organizations, while still keep- This mentality limits a group’s ability cussion—which involves multiple ing them lightweight and nonintrusive to collectively solve problems and men- exchanges—usually takes one to two for developers. tor developers. days. Indeed, the feedback cycle is so By conducting asynchronous re- fast, we consider it a form of continu- Lesson 1: views and eliminating rigid inspection ous review, which often has more simi- Asynchronous Reviews constraints, OSS encourages synergy larities with pair programming than Asynchronous reviews support team between code authors, reviewers, and with inspection.6 discussions of defect solutions and find other stakeholders as they discuss the To illustrate, we quote Rob Har- the same number of defects as colo- best solution, not the existence of de- till, a former core developer of the cated meetings in less time. They also fects. The distinction between author Apache project and a founding devel- enable developers and passive listeners and reviewer can blur such that a re- oper of the Internet Movie Database: to learn from the discussion. viewer rewrites the code and an author “I think the people doing the bulk of Managers tend to believe that de- learns from and becomes a reviewer of the committing appear very aware of fect detection and other project benefits the new code. what the others are committing. I’ve will arise from colocated, synchronous seen enough cases of hard-to-spot ty- meetings. However, in 1993, Lawrence Lesson 2: pos being pointed out within hours of Votta found that reviewers could dis- Frequent Reviews a commit.” cover almost all defects during their in- The earlier a defect is found, the bet- dividual preparations for an inspection ter. OSS developers conduct all-but- Lesson 3: meetings, when they study the portion continuous, asynchronous reviews that Incremental Review of code to be reviewed.5 Not only did function as a form of asynchronous Reviews should be of changes that are the meetings generate few additional pair programming. small, independent, and complete. defects, but the scheduling for them ac- The longer a defect remains in an ar- The development of large software counted for 20 percent of the inspec- tifact, the more embedded it becomes artifacts by individuals or relatively iso- tion interval, lengthening the develop- and the more it will cost to fix. This lated developer groups means that the ment cycle. rationale is at the core of the 35-year- artifacts are unfamiliar to the review- Subsequent studies have replicated old Fagan inspection technique.1 How- ers tasked with inspecting them. Da- this finding in both industrial and re- ever, the term “frequent” in traditional vid Parnas and David Weiss first noted NOVEMBER/DeCEMBER 2012 | IEEE SOFTWARE 57 FEATURE: SOFTWARE REVIEWS that the resulting inspections are done Lesson 4: Invested, Cleary, solved this problem in his com- poorly by unhappy, unfocused, over- Experienced Reviewers pany with what he called a “reviewer whelmed inspectors.7 Invested experts and codevelopers as bug fixer” strategy, in which he pe- To facilitate early and frequent should conduct reviews because they riodically assigned one developer to feedback, OSS projects tend to review already understand the context in fix a bug in another developer’s code. smaller changes than proprietary proj- which a change is being made. As a bug fixer, the developer becomes ects,8 ranging from 11 to 32 lines in Without detailed knowledge of the a codeveloper as he or she reads, ques- the median case.3 The small size lets module or subsystem, reviewers can’t tions, understands, and reviews the reviewers focus on the entire change, reasonably be expected to understand bug-related code. This technique com- and the incrementality reduces review- a large, complex artifact they’ve never bines peer review with the primary task ers’ preparation time and lets them seen before. Checklists and reading of fixing bugs. It also helps manage maintain an overall picture of how the techniques might force inspectors to turnover risk by giving all developers change fits into the system. focus during an inspection,7 but they a broader understanding of the system. Equally important is the OSS di- won’t turn a novice or incompetent in- In Table 1, we use the literature and vide-and-conquer review style that spector into an expert. our research findings to compare five keeps each change logically and func- The developers involved in OSS re- reviewer types. tionally independent. For example, view tend to have at least one to two a change that combines refactoring years’ experience with the project; many Lesson 5: Empower a method with fixing a bug in the reviewers have more than four years, Expert Reviewers refactored method won’t be reviewed and a few have been with the project Let expert developers self-select until it’s divided into two changes. since its inception.3 In the OSS projects changes they’re interested in and com- Developers can either submit these we studied, we also found that main- petent to review. Assign reviews that independent changes as a sequence of tainers of a particular code section pro- nobody selects.