CONTEMPORARY PEER CODE REVIEW PRACTICES AND ASSOCIATED BENEFITS by AMIANGSHU BOSU DR. JEFFREY C. CARVER, COMMITTEE CHAIR DR. JEFF GRAY DR. MUNAWAR HAFIZ DR. RANDY SMITH DR. MARCUS BROWN A DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science in the Graduate School of The University of Alabama TUSCALOOSA, ALABAMA 2015 Copyright Amiangshu Bosu 2015 ALL RIGHTS RESERVED ABSTRACT Prior research indicates that peer code review is an effective method for reducing the num- ber of defects and improving the quality of code. Besides maintaining the integrity of the code, code review spreads knowledge, expertise, and development techniques among the review par- ticipants. In the recent years, many Open Source Software (OSS) communities and commercial organizations have adopted ’contemporary’ or ’modern’ code review, an informal, regular, tool- based process. Because both OSS and commercial developers spend a significant amount of effort performing code reviews, the primary goal of this dissertation is to better understand contemporary code review practices, its benefits, and factors that influence its outcomes. To address this goal, this dissertation describes empirical studies using surveys, software repository mining, and social network analysis. The first study is a survey of OSS developers to understand their collaboration and the process by which they form impressions of each other. The results suggest that coding-related factors influence impression formation the most among OSS developers. Therefore, the types of interactions where participants can judge a peers code or creativity (e.g., code review) should be crucial for peer impression formation. The results of this study motivated the selection of peer code review as the focus of this dissertation. The second study describes a survey of developers from 36 popular OSS projects and from Microsoft about: 1) the code review process in their projects, 2) their expectations from code review, and 3) how code review impacts impressions about their peers. The results suggest that the primary perceived benefit of code review is knowledge sharing, relationship building, better designs, and ensuring maintainable code, as opposed to the expected result of defect detection. Code reviews help build impressions between code review participants. Those impressions not only impact code review process but also future collaborations among developers. Due to the rarity of face-to-face interactions, OSS developers rely more on the reputation of and relationship with the author during code reviews. Conversely, Microsoft developers focus more on expertise and anticipated efforts. Finally, the third study aims to find the impact of developers’ reputation on the outcome of his/her code review requests. The results suggest that developers’ reputations help them re- ceive quicker feedback on their review requests, complete reviews in shorter time, and get their ii code changes accepted. Newcomers to OSS projects suffer the most due to delayed feedback, which may discourage their future participation. A reviewer recommendation system to triage incoming code review requests can be useful to reduce delayed feedback for newcomers. Based on the results from these studies, this dissertation makes recommendations for practitioners to adopt and improve code review practices. iii DEDICATION To my parents To my wife Mousumi To rest of my family iv ACKNOWLEDGMENTS Foremost, I would like to express my sincere gratitude and appreciation to my professor, mentor, and adviser, Dr. Jeffrey Carver. Without his guidance, support, and assistance it would not be possible for me to finish my dissertation. Dr. Carver always helped in identifying, formu- lating, and solving research problems, and patiently revised and re-revised the publications and presentations to improve the quality of my research results. I really appreciate his assistance during my job search, patiently pointing out my mistakes, and teaching me how to commu- nicate better. His guidance helped me become a better researcher, a better professional, and overall a better person. Thank you, Dr. Carver, for motivating me and always being available to help me through this process. To my other committee members, I am grateful for your willingness to take part in the eval- uation of my dissertation research. Dr. Jeff Gray - I appreciate you for spending time to review my faculty application materials and sending recommendation letters. Dr. Munawar Hafiz - I appreciate you for providing important guidance in the security project, answering my phone calls / emails, and soliciting valuable career advice. Dr. Marcus Brown - I appreciate your sug- gestions and your efforts to point out the grammatical mistakes and typos in this dissertation. Dr. Randy Smith - I appreciate your suggestions in improving my dissertation. I would like to add a special thanks to Dr. Nicholas Kraft for advising me whenever I seek directions and providing me valuable support during my job search. My sincerest gratitude and appreciation goes to my lovely wife Mousumi Das for constantly supporting me and going through hardships during my PhD years. I would also like to mention my mom, dad, and brother. They always provided me great support and motivation to pursue a PhD. I am also privileged to be associated with the current software engineering group at UA and like to thank my fellow labmates: Debarshi Chatterji, Ferosh Jacob, Aziz Naanthaamornphong, Christoper Corley, Jonathan Corley, Brian Eddy, Dustin Heaton, Elizabeth Williams, Amber Krug, Wenhua Hu, and Ahmed Al-Zubidy. Without your great and enjoyable company, it was not possible for me to pass the long journey here. To my friends in Tuscaloosa, Ashfakul Islam, Kazi Zunnurhain, Sufal Biswas, and Mohammad Asadul Hoque, and many others. Thank you v everyone, for those memorable moments, and great support. Finally, I would like to thank the financial support provided by the Department of Computer Science at UA, as well as the funding support from the National Science Foundation (Grant: 1322276) and the National Security Agency. vi CONTENTS ABSTRACT . ii DEDICATION . iv ACKNOWLEDGMENTS . v LIST OF TABLES . xiii LIST OF FIGURES . xv 1 INTRODUCTION . 1 1.1 Related Work . 6 1.2 Research Objective . 9 1.2.1 Importance of code reviews . 9 1.2.2 Code review process . 9 1.2.3 Differences between code review practices in OSS and commercial projects . 9 1.2.4 Characteristics of code review social networks . 10 1.2.5 Non-technical benefits of code reviews . 10 1.2.6 Impact of human factors on code reviews . 10 1.3 Organization . 10 1.4 Key Findings and Contributions . 13 1.5 Outline of the Dissertation . 14 2 PEER IMPRESSIONS FORMATION IN OSS ORGANIZATIONS: MOTIVATIONAL STUDY . 15 2.1 Research Questions and Hypotheses . 17 2.1.1 Experiences working with other participants . 17 2.1.2 Perceived Expertise . 17 vii 2.1.3 Trust . 18 2.1.4 Losing Mutual Trust . 18 2.1.5 Meeting in Person . 18 2.1.6 Belief about Peers’ Impressions . 19 2.1.7 Judging Peer Productivity . 20 2.1.8 Judging a Peer as Easy or Difficult to Work With . 20 2.2 Survey Design . 20 2.3 Data Analysis . 21 2.4 Respondent Demographics . 22 2.4.1 OSS Projects Represented . 22 2.4.2 Project Role(s) of the Respondents . 22 2.4.3 Voluntary vs. Paid-Participation . 23 2.4.4 Distribution of Participants across Organizations . 24 2.5 Results . 24 2.5.1 RQ1: Experiences working with other participants . 24 2.5.2 H1: Impact of Perceived Expertise on Peer Interactions . 27 2.5.3 H2: Impact of Trust on Peer Interaction . 28 2.5.4 RQ2: Losing trust in a peer . 29 2.5.5 H3: Effect of Meeting in Person on Peer Impression . 30 2.5.6 H4: Accuracy of Peer Impression . 32 2.5.7 H5: Effect of Project Contributions on Opinions about Productivity and Competency . 33 2.5.8 RQ3: Factors affecting Ease or Difficulty of Working Together . 35 2.6 Threats to Validity . 37 2.6.1 Internal validity . 37 2.6.2 Construct validity . 37 2.6.3 External validity . 38 2.7 Conclusion . 38 2.8 References . 39 viii Appendices . 44 2.A Survey questions . 44 3 INSIGHTS INTO THE PROCESS ASPECTS AND SOCIAL DYNAMICS OF CON- TEMPORARY CODE REVIEW . 48 3.1 Background . 50 3.1.1 Contemporary Code Review Process . 50 3.1.2 Overview of Contemporary Code Review Research . 50 3.2 Research Questions . 53 3.2.1 Importance of Code Review . 53 3.2.2 Code Review Process . 53 3.2.3 Impact of Code Review on Peer Impressions . 54 3.3 Research Method . 55 3.3.1 Survey Design . 55 3.3.2 Participant Selection . 56 3.3.3 Pilot Tests . 57 3.3.4 Data Collection . 57 3.3.5 Data Processing and Analysis . 58 3.4 Demographics . 59 3.4.1 Projects represented . 59 3.4.2 Respondent demographics . 60 3.5 Results . 60 3.5.1 RQ1: Importance of code reviews . 61 3.5.2 RQ2: Time spent in code reviews . 66 3.5.3 RQ3: Accepting review request . 67 3.5.4 RQ4: Characteristics of low quality code . 70 3.5.5 RQ5: Assistance to improve low quality code . 72 3.5.6 RQ6: Impact of high quality code . 76 3.5.7 RQ7: Impact of low quality code . 79 3.5.8 RQ8: Effect on peer impressions . 82 ix 3.6 Discussion . 84 3.6.1 Benefits of Code Review . 84 3.6.2 Peer Impression Formation . 84 3.6.3 Effects on Future Collaborations . 85 3.6.4 Effects of Distributed vs. Co-located Teams . 85 3.6.5 Differences Between OSS and Microsoft . 86 3.6.6 Effects of.