Code Review Quality: How Developers See It Oleksii Kononenko Olga Baysal Michael W. Godfrey School of Computer Science School of Computer Science School of Computer Science University of Waterloo Carleton University University of Waterloo Waterloo, ON, Canada Ottawa, ON, Canada Waterloo, ON, Canada [email protected] [email protected] [email protected] ABSTRACT Due to volume of submitted contributions and the need to In a large, long-lived project, an effective code review pro- handle them in a timely manner, many code review pro- cess is key to ensuring the long-term quality of the code base. cesses have become more lightweight and less formal in na- In this work, we study code review practices of a large, open ture [4, 25]. This evolution of review process increases the source project, and we investigate how the developers them- risks of letting bugs slip into the version control repository, selves perceive code review quality. We present a qualita- as reviewers are unable to detect all of the bugs. In our tive study that summarizes the results from a survey of 88 recent study [18], we explored the topic of code review qual- Mozilla core developers. The results provide developer in- ity by conducting a quantitative investigation of what fac- sights into how they define review quality, what factors con- tors may influence the quality of evaluating code contribu- tribute to how they evaluate submitted code, and what chal- tions. The study was of quantitative nature as it employed lenges they face when performing review tasks. We found data mining and analysis of project's repositories. While we that the review quality is primarily associated with the thor- found that both technical and personal attributes are asso- oughness of the feedback, the reviewer's familiarity with the ciated with the review quality, many other factors such as code, and the perceived quality of the code itself. Also, we organization, its culture and structure, development cycles, found that while different factors are perceived to contribute time pressures, etc., can potentially influence how reviewers to the review quality, reviewers often find it difficult to keep assess code changes. Since these \hidden" factors are diffi- their technical skills up-to-date, manage personal priorities, cult to take into account in a quantitative analysis because and mitigate context switching. such data is not available, easily accessible, or extractable from the available artifacts, we decided to employ qualita- CCS Concepts tive research methods to fill the gap in the knowledge we •Software and its engineering ! Maintaining soft- had about the developer perception and attitude towards ware; Collaboration in software development; the code review quality. Our qualitative study is organized around an exploratory Keywords survey that we design based on the state-of-the-art quali- Code review, review quality, survey, developer perception tative research [8, 12, 28] and our own observations of the Mozilla code review process and interactions with Mozilla 1. INTRODUCTION developers during our previous research project [5]. We In a large, long-lived project, an effective code review pro- conducted an exploratory survey with 88 Mozilla core de- cess is key to ensuring the long-term quality of the code base. velopers. Our qualitative analysis of the survey data aims Code review is considered to be one of the most effective at addressing the following research questions: QA practices in software development. While it is relatively • RQ1: How do Mozilla developers conduct code review? expensive in terms of time and effort, it delivers benefits Existing literature offers several case studies of how of identifying defects in code modifications before they are code review processes are employed by various software committed into the project's code base [9]. Reviewers play development projects and organizations [4,6,11,21,26, a vital role in the code review process not only by shaping 27]. and evaluating individual contributions but also by ensuring • RQ2: What factors do developers consider to be influ- the high quality of the project's master code repository. ential to review time and decision? Code review explicitly addresses the quality of contribu- tions before they are integrated into project's code base. Code review is a complex process that involves people, their skills and social dynamics, as well as development Permission to make digital or hard copies of all or part of this work for personal or artifacts and environments; thus, it can be affected classroom use is granted without fee provided that copies are not made or distributed by both technical [15,21,25,27] and non-technical fac- for profit or commercial advantage and that copies bear this notice and the full cita- tors [7, 12, 19, 29]. tion on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- • RQ3: What factors do developers use to assess code publish, to post on servers or to redistribute to lists, requires prior specific permission review quality? and/or a fee. Request permissions from [email protected]. ICSE ’16, May 14-22, 2016, Austin, TX, USA While the quality assessment of code contributions is c 2016 ACM. ISBN 978-1-4503-3900-1/16/05. $15.00 an active research area, the topic of code review qual- DOI: http://dx.doi.org/10.1145/2884781.2884840 ity remains largely unexplored. To better understand developer perception and attitudes towards the quality plored the way stakeholders interact with one another during of the process that evaluates code changes, we further the code review process. Their findings provided insights to refine this research question into the following subques- developers about how to effectively manage large quantities tions: of reviews. Additionally, their investigation reveals that the identification of defects is not the sole motivation for modern { RQ3.1: How do reviewers assess the quality of a code review. Resolving non-technical issues, such as scope or patch? process issues, was among other motivations of code review. { RQ3.2: How do developers define a well-done code Baysal et al. [7] empirically studied the WebKit project review? and showed that review positivity, i.e., the proportion of { RQ3.3: What factors are perceived to contribute accepted patches, is also affected by non-technical factors. to the review quality? The authors found that factors from both personal and orga- • RQ4: What challenges do developers face when per- nizational dimensions affect code review response time and forming review tasks? the outcome of a review. Jiang et al. [15] studied, through We believe that it is important to understand what on- an analysis of the Linux Kernel, the relation between patch going problems developers deal with to provide them characteristics and the probability of patch acceptance as with better tools to support their daily tasks and ac- well as the time taken for patches to be integrated into the tivities. code base. The results of their study indicated that devel- oper experience and patch maturity impact the patch ac- Our main findings reveal that the review quality is pri- ceptance, while the number of suggested reviewers, affected marily associated with the thoroughness of the feedback, the subsystems, developer experience, and the time of submis- reviewer's familiarity with the code, and the perceived qual- sion affect the duration of a review. Bacchelli and Bird [4] ity of the code itself. As expected, we found that different examined the motivations, challenges, and outcomes of tool- factors including technical, personal and social signals, are based code reviews. They found that while the identification perceived to contribute to the review quality. Also, we found of defects is the main purpose of code review, other motiva- that reviewers often find it difficult to keep their technical tions such as team awareness and knowledge sharing exist skills up-to-date, manage personal priorities, and mitigate as well. context switching. Kemerer et al. [16] investigated the effect of review rate The paper makes the following contributions: (i.e., lines of code reviewed per hour) on the effectiveness of • A qualitative study with the professional OSS devel- defect removal and the quality of software products. Their opers who participated in our survey on the topic of study showed that allowing sufficient preparation time for re- code review quality. views and inspections can improve their quality. Hatton [13] • A thorough survey analysis that offers insights into found that defect detection capabilities differ among code re- the developer perception of the code review quality viewers | the \worst" reviewer was ten times less effective and factors affecting it, as well as identifies of the than the best reviewer. Moreover, he found almost 50% im- main challenges developers face when conducting re- provement in defects detection between settings where the view tasks. source code is inspected by two developers together (76% of • A publicly available dataset of 88 anonymized survey faults found) and where the source code is inspected by two responses. developers separately (53% of faults found). The rest of the paper organized as follows. Section 2 sum- Pull-based development is a new model for distributed marizes prior related work. Section 3 describes our method- software development; in this model proposed changes sub- ology including the survey design, participants, and data mitted as pull requests. In a quantitative study, Gousios et analysis. Section 4 presents the results of the qualitative al. [11] explored what factors affect the decision to accept a study. Section 5 discusses implications of our work and sug- pull request, as well as the time needed to make such a deci- gests possible future research directions. Section 6 addresses sion.