Feature: Software Reviews

We examined more than 100,000 Contemporary peer reviews in OSS case studies of the Apache httpd server, Subversion, Linux, FreeBSD, KDE, and Gnome and found an efficient fit between Peer Review OSS developers’ needs and the mini- malist structures of their peer review processes.3 Specifically, the projects in Action: broadcast changes asynchronously to the development team—usually on a Lessons from Open Source mailing list—and reviewers self-select changes they’re interested in and com- Development petent to review. Changes failing to capture a reviewer’s interest remain unreviewed. Developers manage what can be an overwhelming broadcast Peter C. Rigby, Concordia University, Montreal, Canada of information by relying on simple email filters, descriptive email sub- Brendan Cleary, University of Victoria, Canada jects, and detailed change logs. The change logs represent the OSS proj- Frederic Painchaud, Department of National Defence, Canada ect’s heart beat, through which de- velopers maintain a conceptual un- Margaret-Anne Storey and Daniel M. German, University of Victoria, derstanding of the whole system and Canada participate in the threaded email dis- cussions and reviews for which they have the required expertise. // Open source development uses a rigorous but The OSS process evolved naturally to fit the development team and con- agile review process that software companies trasts with enforced inspections based can adapt and supplement as needed by on best practices that are easily mis- popular tools for lightweight collaboration and applied and end in false quality assur- ances, frustrated developers, and lon- nonintrusive quality assurance metrics. // ger development cycles. As Michael Fagan, the father of formal inspection, lamented about the process he devel- oped, “Even 30 years after its creation, it is often not well understood and more often, poorly executed.”1 In this article, we contrast OSS peer review with a traditional inspection process that’s widely acknowledged in the literature—namely, inspections Software inspection is a form even though developers acknowledge performed on large, completed soft- of formal peer review that has long the value of formal peer review, many ware artifacts at specific checkpoints. been recognized as a software engi- also avoid it, and the adoption rates The inspectors are often unfamiliar neering “best practice.” However, the for traditional inspection practices are with the artifact under inspection, prospect of reviewing a large, unfa- relatively low.1,2 so they must prepare individually be- miliar software artifact over a period On the other hand, peer review fore the formal review by thoroughly of weeks is almost universally dreaded is a prevalent practice on successful studying the portion of code to be re- by both its authors and reviewers. So, open source software (OSS) projects. viewed. Defects are recorded subse-

56 IEEE Software | published by the IEEE computer society 0740-7459/12/$31.00 © 2012 IEEE quently at the formal review meeting, but the task of fixing a recorded defect Inspection software Asynchronous, Open source falls to the author after the meeting. reviews tool-supported reviews software reviews Some intrinsic differences between open source and proprietary develop- ment projects, such as self-selected Formal but Measureable Minimalist but cumbersome but lightweight lacks traceability versus assigned participation, suggest inspection processes at opposite ends of a continuum (see Figure 1). How- Figure 1. The spectrum of peer review techniques, from formal inspection to minimal- ever, neither formality nor aversion is process OSS review. Tool-supported, lightweight review provides a flexible but traceable fundamental to peer review. The core middle ground. idea is simply to get an expert to ex- amine your work to find problems you can’t see. Success in identifying defects search settings. This led to tools and inspection processes means that large, depends less on the process than on practices that let developers interact in completed artifacts are inspected at the expertise of the people involved.4 an asynchronous, distributed manner. specific checkpoints that might occur We present five lessons from OSS Furthermore, the hard time constraints many months apart. The calendar time projects that we think are transferable imposed by colocated meetings, the to inspect these completed artifacts is to proprietary projects. We also pres- rigid goal of finding defects, and the on the order of weeks. ent three recommendations for adapt- sole metric of defects found per line In contrast, most OSS peer re- ing these practices to make them more of source code encouraged a mentality views begin within hours of complet- traceable and appropriate for propri- of “Raise issues, don’t resolve them.”2 ing a change, and the full review dis- etary organizations, while still keep- This mentality limits a group’s ability cussion—which involves multiple ing them lightweight and nonintrusive to collectively solve problems and men- exchanges—usually takes one to two for developers. tor developers. days. Indeed, the feedback cycle is so By conducting asynchronous re- fast, we consider it a form of continu- Lesson 1: views and eliminating rigid inspection ous review, which often has more simi- Asynchronous Reviews constraints, OSS encourages synergy larities with pair programming than Asynchronous reviews support team between code authors, reviewers, and with inspection.6 discussions of defect solutions and find other stakeholders as they discuss the To illustrate, we quote Rob Har- the same number of defects as colo- best solution, not the existence of de- till, a former core developer of the cated meetings in less time. They also fects. The distinction between author Apache project and a founding devel- enable developers and passive listeners and reviewer can blur such that a re- oper of the Internet Movie Database: to learn from the discussion. viewer rewrites the code and an author “I think the people doing the bulk of Managers tend to believe that de- learns from and becomes a reviewer of the committing appear very aware of fect detection and other project benefits the new code. what the others are committing. I’ve will arise from colocated, synchronous seen enough cases of hard-to-spot ty- meetings. However, in 1993, Lawrence Lesson 2: pos being pointed out within hours of Votta found that reviewers could dis- Frequent Reviews a commit.” cover almost all defects during their in- The earlier a defect is found, the bet- dividual preparations for an inspection ter. OSS developers conduct all-but- Lesson 3: meetings, when they study the portion continuous, asynchronous reviews that Incremental Review of code to be reviewed.5 Not only did function as a form of asynchronous Reviews should be of changes that are the meetings generate few additional pair programming. small, independent, and complete. defects, but the scheduling for them ac- The longer a defect remains in an ar- The development of large software counted for 20 percent of the inspec- tifact, the more embedded it becomes artifacts by individuals or relatively iso- tion interval, lengthening the develop- and the more it will cost to fix. This lated developer groups means that the ment cycle. rationale is at the core of the 35-year- artifacts are unfamiliar to the review- Subsequent studies have replicated old Fagan inspection technique.1 How- ers tasked with inspecting them. Da- this finding in both industrial and re- ever, the term “frequent” in traditional vid Parnas and David Weiss first noted

November/December 2012 | IEEE Software 57 Feature: Software Reviews

that the resulting inspections are done Lesson 4: Invested, Cleary, solved this problem in his com- poorly by unhappy, unfocused, over- Experienced Reviewers pany with what he called a “reviewer whelmed inspectors.7 Invested experts and codevelopers as bug fixer” strategy, in which he pe- To facilitate early and frequent should conduct reviews because they riodically assigned one developer to feedback, OSS projects tend to review already understand the context in fix a bug in another developer’s code. smaller changes than proprietary proj- which a change is being made. As a bug fixer, the developer becomes ects,8 ranging from 11 to 32 lines in Without detailed knowledge of the a codeveloper as he or she reads, ques- the median case.3 The small size lets module or subsystem, reviewers can’t tions, understands, and reviews the reviewers focus on the entire change, reasonably be expected to understand bug-related code. This technique com- and the incrementality reduces review- a large, complex artifact they’ve never bines peer review with the primary task ers’ preparation time and lets them seen before. Checklists and reading of fixing bugs. It also helps manage maintain an overall picture of how the techniques might force inspectors to turnover risk by giving all developers change fits into the system. focus during an inspection,7 but they a broader understanding of the system. Equally important is the OSS di- won’t turn a novice or incompetent in- In Table 1, we use the literature and vide-and-conquer review style that spector into an expert. our research findings to compare five keeps each change logically and func- The developers involved in OSS re- reviewer types. tionally independent. For example, view tend to have at least one to two a change that combines refactoring years’ experience with the project; many Lesson 5: Empower a method with fixing a bug in the reviewers have more than four years, Expert Reviewers refactored method won’t be reviewed and a few have been with the project Let expert developers self-select until it’s divided into two changes. since its inception.3 In the OSS projects changes they’re interested in and com- Developers can either submit these we studied, we also found that main- petent to review. Assign reviews that independent changes as a sequence of tainers of a particular code section pro- nobody selects. conceptually related changes or com- vided detailed reviews when another de- Poorly implemented, prescriptive, bine them on a single topic or fea- veloper made a change. The maintainer heavyweight processes can give the il- ture branch. Although one developer often had to interact with, maintain, or lusion of following a best practice while might have all the required expertise evolve the changed code. Because code- realizing none of the advertised ben- to perform the review, it’s also pos- velopers depend on each other, they efits. Just as checklists can’t turn nov- sible that one person will have the have a vested interest in ensuring that ices into experts, a formal process can’t required systemwide expertise to un- the quality of changes is high. Further- make up for a lack of expertise. Adam derstand the refactoring and another more, because codevelopers are already Porter and his colleagues reported that will have detailed knowledge of a par- experts in part of the system under re- the most important predictor of the ticular algorithm that contains the view, they take less time to understand number of defects detected during re- bug fix. Intelligently splitting changes how a small change affects the system. view is reviewer expertise; the process lets stakeholders with different exper- Although codevelopers have the has minimal impact.4 tise independently review aspects of a highest level of investment, many or- In a development environment larger change, which reduces commu- ganizations can’t afford to keep more where the artifact author or manager nication and other bottlenecks. than one developer working on the assigns reviews, it can be difficult to Finally, changes must be complete. same part of a software system. A sim- know who should perform a review Discussing each solution step in a small ple alternative is to assign regular re- and how many reviewers to involve. group can be very effective, but it can viewers to particular subsystems. The The candidates’ expertise must be bal- be also be tiring. Furthermore, certain reviewers aren’t responsible for mak- anced with their workloads and other problems can be more effectively solved ing changes, but they follow and review factors. A rule-of-thumb in the inspec- by a single focused developer. Pair pro- changes incrementally. This technique tion literature is that two reviewers gramming involves two people in each also spreads the knowledge across the find an optimal number of defects— solution step, but with frequent asyn- development team, mitigating the risk the cost of adding more reviewers isn’t chronous reviews, reviewers only see of “getting hit by a bus.” justified by the number of additional incremental changes that the author In a small start-up organization, any defects detected.9 In OSS, the median feels are small, independent, and com- review costs can be prohibitive. One is two reviewers per review. These plete solutions. of the authors of this article, Brendan reviewers aren’t assigned; instead,

58 IEEE Software | www.computer.org/software Reviewer types and their costs, investment level in the code, review quality, and amount of knowledge transfer and community development that occurs during the review.

Reviewer type Cost Investment Quality Team building

able 1 able Independent reviewer Very high Low Medium Low T Pair programming Very high Very high High High

Codeveloper reviewer High High High High

Regular incremental reviewer Medium Medium Medium Medium

Reviewer as bug fixer Low Medium Low Medium

broadcasting and self-selection lead to are techniques to manage this email • metrics to gauge review efficiency natural load balancing across the de- barrage, but it’s difficult to track the and effectiveness. velopment team. review process for reporting and qual- Dictating a constant number of re- ity assurance, and it’s easy to inadver- Table 2 compares some popular peer viewers for each change ignores the dif- tently ignore reviews. Furthermore, review tools. ference between a simple change that the frequency of small changes can one reviewer can rubber stamp and a lead to fragmentation, which makes it Recommendation 2: complex one that might require a dis- difficult to find and review a feature Nonintrusive Metrics cussion with the whole development that consists of multiple changes. Mine the information trail left by team. The advantage of self-selection is Tools can help structure reviews and asynchronous reviews to extract light- that it’s up to the developers, who have integrate them with other development weight metrics that don’t disrupt devel- the most detailed knowledge of the sys- systems. Typically, they provide oper workflow. tem, to decide on the level of review Metric collection is an integral part given to each change. • side-by-side highlighted changes to of controlling, understanding, and di- On the other hand, self-selection files (diffs); recting a software project. However, can end in some changes being ignored. • inline discussion threads that are metric collection can disrupt develop- Managers can use tools to automati- linked to a line or file; ers’ workflows and get in the way of cally assign unreviewed changes to re- • capability to hide or show addi- their primary task to produce software. viewers. However, unselected changes tional lines of context and to view a For example, formally recording a might indicate areas of the code base diff in the context of the whole file; defect is a cognitively expensive task, that pose a problem, such as areas that • capability to update the code under sidetracking developers who are dis- only a single developer understands. review with the latest revision in cussing a change and forcing them to the version control system; formally agree on and record a defect. Recommendation 1: • a central place to collect all arti- Tool support doesn’t fix this problem. Lightweight Review Tools facts and discussions relating to a At AMD, Julian Ratcliffe found that Tools can increase traceability for man- review; defects were underreported despite agers and help integrate reviews with • a dashboard to show pending re- the simple CodeCollabotor reporting the existing development environment. views and alert code authors and mechanism: “A closer look at the re- OSS developers rely on information reviewers who haven’t responded to view archive shows that reviewers were broadcast and use minimalistic tool assignments; mostly engaged in discussion, using the support. For example, the Linux Ker- • integration with email and develop- comment threads to fix issues instead nel Mailing List has a median of 343 ment tools; of logging defects.”11 messages per day, and the OSS devel- • notification and assignment of re- Is the defect or the discussion more opers we interviewed received thou- views to individuals and groups of important? In the Linux commu- sands of messages per day.10 There developers; and nity, the amount of discussion on a

November/December 2012 | IEEE Software 59 Feature: Software Reviews

Comparison of some popular peer review tools.

Tool Main advantages Main disadvantages

CodeCollaborator Supports instant messaging-style discussion of LOC, metric Commercial license fee reporting, and tight integration with multiple development able 2 able environments, such as Eclipse and Visual Studio T

Crucible Integrates with the Jira bug tracker and other Atlassian Commercial license fee products

ReviewBoard Has a free, full-featured Web interface for review Requires setup and maintenance on an in-house server

Rietveld Runs on top of Google App Engine, so it’s quick and easy to Requires public hosting on Google Code or setting up the start reviewing; supports Subversion development (Gerrit is review system on an in-house server a git-specific implementation of Rietveld)

CodeStriker Has a Web interface that supports traditional inspection An older tool that lacks good support for lightweight review techniques

particular change is an indicator of that are looking for a way to run large, Software developers in most de- code quality. Indeed, Linus Torvalds, distributed software projects. velopment companies are accustomed who maintains the current release on OSS has much in common with to communicating in person, so they the Linux operating system, has re- agile development and the Agile might not welcome this practice. How- jected code, not because it’s incorrect, Manifesto:13,14 ever, peer review has proved more ef- but because not enough people have fective in an asynchronous environment tried it and discussed it on the mailing • a preference for working software than in a synchronous, colocated one. list. To Torvalds, the potential system over documentation and for em- Companies with large, distributed de- benefit of accepting code that hasn’t powering individuals over imposing velopment teams might consider using been discussed by a group of experts a rigid process; frequent, asynchronous reviews involv- doesn’t outweigh the risks. • handling changes by working in ing codeveloper discussions of small, Turning the amount of discussion small increments rather than fol- functionally independent changes as a during a review into a metric is trivial lowing a rigid plan; and substitute for pair programming. if a tool records discussions and asso- • working closely with the customer ciates them with file changes. On this rather than negotiating contracts. basis, a manager might ask developers ractitioners from both the OSS whether they think the group has ade- The most striking difference be- community and software com- quately discussed a part of the system tween the development methodolo- P panies have driven the devel- before its release. In our work, we’ve gies is that agile supports small, co- opment of lightweight peer review and demonstrated the extraction of many located developer teams, while OSS supporting tools. OSS practices have nonintrusive, proxy metrics from re- projects can scale to large, distributed evolved to maintain code quality ef- view archives.12 teams that rarely, if ever, meet in a co- ficiently within a distributed develop- located setting. OSS projects broad- ment group, and many companies are Recommendation 3: cast all communication—discussions, already adopting a lightweight, tool- Implementing code changes, and reviews—to the en- supported review approach, including a Review Process tire community. The need for the entire AMD11 and Cisco.15 We’re currently Large, formal organizations might community to see all communication working with the Canadian defense benefit from more frequent reviews is so strong that when a company pays department to develop an agile review and more overlap in developers’ work colocated developers to work on an style that fits its development teams. to produce invested reviewers. How- OSS project, it often requires them to We’re also actively seeking collabora- ever, this style of review will likely be summarize and broadcast all in-person tions with developers and companies more amenable to agile organizations discussion to the community. who use a lightweight peer review. Our

60 IEEE Software | www.computer.org/software goal is provide a systematic and prac- tical understanding of contemporary s peer review. Peter C. Rigby is an assistant professor of software engineering at Concordia University in Montreal, Canada. His research interests focus on understanding how developers collaborate to produce successful software systems. Rigby received his PhD in computer science at the University of Victoria, and the lessons and recommendations reported

References uthor in this article are largely based on his dissertation. Contact him at 1. M. Fagan, “A History of Software Inspec- [email protected]. tions,” Software Pioneers: Contributions to Software Engineering, Springer, 2002, pp. 562–573. 2. P.M. Johnson, “Reengineering Inspection,” Comm. ACM, vol. 41, no. 2, 1998, pp. 49–52. Brendan Cleary is a research fellow at the University of Victoria. 3. P.C. Rigby, “Understanding Open Source His research interests focus on managing commercial and research Software Peer Review: Review Processes, projects, and he’s the founder of a university spin-out company. Cleary Parameters and Statistical Models, and Un- has a PhD in computer science from the University of Limerick, Ireland. derlying Behaviours and Mechanisms,” 2011; Contact him at bcleary@ uvic.ca. http://thechiselgroup.org/rigby-dissertation. pdf. bout the A the A bout 4. A. Porter et al., “Understanding the Sources of Variation in Software Inspections,” ACM Trans. Software Eng. Methodology, vol. 7, no. 1, 1988, pp. 41–79. 5. L.G. Votta, “Does Every Inspection Need a Frederic Painchaud is a defence scientist at Defence Research Meeting?” SIGSOFT Software Eng. Notes, vol. and Development Canada and a part-time PhD student in computer 18, no. 5, 1993, pp. 107–114. science at Université Laval. His research interests include software 6. L. Williams, “Integrating Pair Programming architectural risk analysis, static and dynamic code analysis, and into a Software Development Process,” Proc. lightweight peer review. Painchaud has a master’s degree in computer 14th Conf. Software Eng. Education and Training, science from Université Laval. Contact him at frederic.painchaud@ IEEE, 2001, pp. 27–36. drdc-rddc.gc.ca. 7. D.L. Parnas and D.M. Weiss, “Active Design Reviews: Principles and Practices,” Proc. 8th Int’l Conf. Software Eng. (ICSE 85), IEEE CS, 1985, pp. 132–136. Margaret-Anne Storey is a professor of computer science and 8. A. Mockus, R.T. Fielding, and J. Herbsleb, “Two Case Studies of Open Source Software a Canada research chair in human-computer interaction for soft- Development: Apache and Mozilla,” ACM ware engineering at the University of Victoria, Canada. Her research Trans. Software Eng. and Methodology, vol. 11, interests center on technology to help people explore, understand, no. 3, 2002, pp. 1–38. and share complex information and knowledge. Storey received her 9. C. Sauer et al., “The Effectiveness of Software PhD in computer science from Simon Fraser University. Contact her at Development Technical Reviews: A Behavior- [email protected]. ally Motivated Program of Research,” IEEE Trans. Software Eng., vol. 26, no. 1, 2000, pp. 1–14. 10. P.C. Rigby and M.-A. Storey, “Understand- Daniel M. German is an associate professor of computer science ing Broadcast Based Peer Review on Open at the University of Victoria, Canada. His research areas are open Source Software Projects,” Proc. 33rd Int’l Conf. Software Eng. (ICSE 11), ACM, 2011, pp. source software engineering and the impact of copyright in software 541–550. development. German received his PhD in computer science from 11. J. Ratcliffe, “Moving Software Quality the University of Waterloo, Canada. Contact him at [email protected] or Upstream: The Positive Impact of Lightweight through his website at turingmachine.org. Peer Code Review,” Proc. Pacific NW Software Quality Conf. (PNSQC 09), 2009, pp. 171–180; www.pnsqc.org/past-conferences/2009-con- ference. 12. P.C. Rigby, D.M. German, and M.-A. Storey, “Open Source Software Peer Review Practices: A Case Study of the Apache Server,” Proc. 30th Int’l Conf. Software Eng. (ICSE08), IEEE CS, 2008, pp. 541–550. 13. K. Beck et al., The Agile Manifesto, 2001; J. Eckstein and H. Baumeister, eds., Springer http://agilemanifesto.org. 2004, pp. 85–93. 14. S. Koch, “Agile Principles and Open Source 15. J. Cohen, Best Kept Secrets of Peer Code Review, Selected CS articles and columns Software Development: A Theoretical and Em- white paper, Smart Bear, 2006; http:// are also available for free at smartbear.com/solutions/white-papers/ pirical Discussion,” Extreme Programming and http://ComputingNow.computer.org. Agile Processes in Software Eng., LNCS 3092, best-kept-secrets-of-peer-code-review

November/December 2012 | IEEE Software 61