Masaryk University Faculty of Informatics Optimal Recommendations for Source Code Reviews Master’s Thesis Jakub Lipčák Brno, Spring 2017 Masaryk University Faculty of Informatics Optimal Recommendations for Source Code Reviews Master’s Thesis Jakub Lipčák Brno, Spring 2017 Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Jakub Lipčák Advisor: Bruno Rossi, PhD i Acknowledgement I would like to thank my advisor, Bruno Rossi, PhD., for all his help, support and guidance, which was very valuable and professional. I would also like to express my gratitude to my family for support- ing me in my studies. iii Abstract Software code reviews are an important part of the development process, leading to better software quality and reduced overall costs. However, finding appropriate code reviewers is a complex and time- consuming task. In this thesis we analyze several Code Reviewer Recom- mendation Algorithms designed to find appropriate code reviewers for opened pull requests. We re-implemented two of the major approaches (ReviewBot and RevFinder) and we proposed several modifications which improved the Top-5 accuracy of the RevFinder algorithm by 12.90% on average. Furthermore, we propose a novel code reviewer recommendation approach based on the Naive Bayes technique that takes into consideration the most important features of previously reviewed pull requests. Experiments using 35 239 pull requests from three open source projects show that our approach provides 88.59% accuracy on average considering a top 10 recommendation, which is by 12.35% better than the average accuracy provided by the RevFinder algorithm. Finally, we analyze the reproducibility problems of existing research in this area observed during the work on this thesis and we propose some recommendations for future research. iv Keywords Source Code Review, Code Reviewer Recommendation System, Pull Request, Distributed Software Development, Machine Learning, Gerrit v Contents 1 Introduction 1 1.1 Problem Statement ......................2 1.2 Goals .............................2 1.3 Thesis Structure ........................3 2 Code Reviews 5 2.1 What Do We Know about Code Reviews? ..........5 2.1.1 Code Review Process . .6 2.1.2 Objectives of Code Reviews . .7 2.1.3 Factors Influencing Code Reviews . .8 2.2 Code Review Tools ......................9 2.2.1 Gerrit . .9 2.2.2 GitHub . 10 2.2.3 Crucible . 11 2.2.4 Collaborator . 11 3 Code Reviewer Recommendation Algorithms 13 3.1 Definition ........................... 13 3.2 Evaluation Metrics ...................... 13 3.2.1 Top-k Accuracy . 14 3.2.2 Mean Reciprocal Rank . 14 3.2.3 Metrics from the Information Retrieval Domain 15 3.3 Existing Recommendation Algorithms ............ 15 3.3.1 Traditional Approaches . 16 3.3.2 Cross-project and Technology Experience . 18 3.3.3 Machine Learning . 19 3.3.4 Social Relations . 21 3.3.5 Features Summary . 22 3.4 Proposed Recommendation Algorithm ............ 22 3.4.1 Naive Bayes Classification . 23 3.4.2 Feature Extraction . 25 3.4.3 Reviewers Recommendation . 26 4 Design and Implementation 29 4.1 Implemented Algorithms ................... 29 4.1.1 ReviewBot Implementation . 29 vii 4.1.2 RevFinder Implementation . 30 4.1.3 Naive Bayes Reviewer Recommendation . 31 4.2 Implementation Details .................... 32 4.2.1 Design and Technologies . 32 4.2.2 Communication Interface . 32 5 Empirical Evaluation 35 5.1 Datasets ............................ 35 5.1.1 Description of Data Collection . 35 5.1.2 Structure of Data Collection . 36 5.2 Experimental Setup ...................... 37 5.3 Results ............................ 37 5.3.1 Baseline Approach Accuracy . 38 5.3.2 Proposed Approach Accuracy . 38 5.3.3 Comparison of Solutions . 39 5.3.4 Discussion . 40 5.3.5 Threats to Validity . 42 6 Reproducibility of Reviewer Recommendation Algorithms 45 6.1 Reproducible Research in Software Engineering ....... 46 6.2 Reproducibility Problems ................... 46 6.2.1 Source Code . 46 6.2.2 Data Sets . 47 6.2.3 Metrics . 47 6.2.4 Reproducibility Summary . 48 7 Conclusion 49 Bibliography 51 A Google Chrome Extension 55 B GitHub Repository 57 C Data Model and Datasets. 59 D Configuration and Deployment 61 D.1 Configuration ......................... 61 D.2 Deployment .......................... 62 viii 1 Introduction As distributed collaboration in the development of open source soft- ware projects gains more and more popularity, there is a need for tools able to support this kind of development process [1]. The pull-based model has been widely adopted across several open source projects in recent years as a method allowing external contributors to propose changes into the code base of the projects. Software code reviews are a crucial part of this model. They are considered as one of the most effective ways of improving the overall quality of the source code [2]. The outcomes of code reviews are highly dependent on the code re- viewers and it has turned out that finding appropriate code reviewers for pull requests in the distributed environments is often a non-trivial problem [3]. It has been found beneficial to have some effective tools which are able to automate the process of recommendation of software code reviewers for newly created pull requests [1]. It is a necessary step before reviewing the code changes and its automation can speed up the whole process of integrating new functionality into the main de- velopment branch as well as increase overall software quality [4]. A lot of research has been done in this area already and there are sev- eral existing approaches dealing with this task [4, 5, 6]. However, it is strongly believed that there are still areas worthy of further research in order to increase the efficiency and relevance of the results provided by the actual Code Reviewer Recommendation Algorithms [1]. Software code reviews are expensive because the good under- standing of large code changes by code reviewers is a time-consuming process and finding appropriate code reviewers can also be very labor- intensive for developers. The outcomes of Code Reviewer Recommenda- tion Algorithms should reduce the costs of this process. The automated recommendation of code reviewers would reduce the efforts neces- sary to find appropriate code reviewers and relevant results have the potential to reduce time spent by code reviewers on the understanding of large code changes [1]. 1 1. Introduction 1.1 Problem Statement Patanamon et al. [4] examined comments from more than 1 400 repre- sentative review samples of four open source projects. They discovered that 4% - 30% of reviews have code reviewer assignment problem. It takes approximately 12 days longer to approve a code change with a code reviewer assignment problem. This fact decreases the effectiveness of the Pull-based model. Code reviewers could be assigned to the pull requests immediately after their creation in an ideal scenario, but it is not a common case at the moment. The Pull-based model has a potential to be more effective and enhancements in this area are the main motivation for this thesis. 1.2 Goals The problem stated in Section 1.1 can be solved by better automated recommendation tools. We have identified two goals worth research- ing in this area. The main goal of this thesis is to evaluate several methods dealing with the recommendation of the most appropriate code reviewers for software code changes. It should provide new approaches and ideas capable of outperforming some of the existing algorithms. The imple- mented prototype will be able to recommend relevant software code reviewers and it will be evaluated on a set of projects. Furthermore, the implemented prototype should be usable in a real environment. The second goal of this thesis is related to reproducibility and replicability problems of past research about Code Reviewer Recom- mendation Algorithms. This thesis should report about such problems in existing research in this area and should provide some recommen- dations to avoid these problems in the future. It should also follow the reproducible research guidelines. 2 1. Introduction 1.3 Thesis Structure This thesis is divided into the following chapters: Chapter 2: Code Reviews gives an overview about the code review process, its objectives, factors influencing this process and also about existing code review tools. Chapter 3: Code Reviewer Recommendation Algorithms describes the most relevant algorithms in this area and common metrics for their evaluation. It also describes our own proposal based on the Naive Bayes technique, as we believed that we can achieve better results than some other algorithms using this approach. Chapter 4: Design and Implementation provides an overview of the implementation describing implemented algorithms and technologies used for the implementation. Chapter 5: Empirical Evaluation presents the results and discussion about our empirical evaluation. It also describes datasets used for testing the algorithms and provides answers to the research questions. Chapter 6: Reproducibility of Reviewer Recommendation Algorithms deals with reproducibility problems in Software Engineering and es- pecially in the area of Code Reviewer Recommendation Algorithms. Chapter 7: Conclusion summarizes the thesis, its contribution, fulfill- ment of the goals and ideas for the future work. 3 2 Code Reviews This chapter describes the code review process, factors influencing this process and tools for its support. The code review process was formalized in 1976 by M. E. Fagan. It was a highly structured process based on line-by-line group reviews in the form of Inspections [7]. The code review practices have rapidly changed in recent years [8]. Modern code review process has become informal (in contrast to Fagan’s definition), tool-based [9], more light- weight, continuous and asynchronous [10]. 2.1 What Do We Know about Code Reviews? Code reviews are understood as a process executed in order to improve the overall quality of software.