Exploration of the code re- view process: Can a code review checklist improve the process? Maiko Goudriaan Technische Universiteit Delft EXPLORATION OF THE CODE REVIEW PROCESS: CAN A CODE REVIEW CHECKLIST IMPROVE THE PROCESS? by Maiko Goudriaan in partial fulfillment of the requirements for the degree of Master of Science in Computer Science at the Delft University of Technology, to be defended publicly on Tuesday May 18, 2021 at 9:00 AM. Supervisor: Prof. dr. A. E. Zaidman Thesis committee: Prof. dr. A. E. Zaidman, TU Delft Dr. U. K. Gadiraju, TU Delft M. Fens MSc, Greenchoice An electronic version of this thesis is available at http://repository.tudelft.nl/. ABSTRACT In the software engineering field the code review process has been widely adopted as a quality control, in the current day code reviews have also multiple additional advantages. In this research we explored how the code review process is used at Greenchoice, a Dutch energy supplier. Subsequently, we created a code review checklist which was used by the developers in order to improve the process. In the first step of the exploration we used a questionnaire to retrieve Greenchoice’s motivation for the code review process. In the results we show that ensuring quality is the main motivation followed by knowledge sharing and functionality check. Secondly, we held 13 semi-structured interviews, performed a data analysis on the metadata of 8458 pull requests and performed a data analysis of 5400 code review comments to explore Greenchoice’s code review process further. In the results of the interviews we show that the reviewers tend to focus on understanding the change and finding obvious mistakes. They also mentioned code reviews could be performed stricter improving the maintainability and evolvability aspects. However, it can be hard to know where to check on for these aspects. In the results of the metadata analysis we show that the code reviews are performed quick, frequent and small changes are made, similar to the modern code reviews processes described in the literature. In the results of the code review comments analysis we show that most comments are related to explaining or understanding the change. Next, we created together with a scrum team a code review checklist which was was used by three scrum teams for two periods of two weeks. In order to measure the effectiveness of the code review checklist we used a pre questionnaire to gauge the current situation and the expectations of the usage. A post questionnaire was used to gauge the new situation and whether the expectations were met. After comparing the results we show in the results that the checklist is less appropriate to guide the reviewer step by step in a code review. Furthermore, we show that the items in the checklists should be applicable to the different changes in the pull requests. The strength of the checklist lies in addressing the items which are often forgotten in the development process and the code reviews. i PREFACE The completion of this thesis was not an easy task and sacrifices have been made in the creation of this mas- ter thesis, though no animals were hurt in this process. Lets get back to the start of my master thesis. It was December 2018 I found my first subject for my master thesis, unfortunately due to circumstances the subject did not go through. I decided to first complete the required courses and in the mean time look for a new sub- ject. In a chat with a teammate from sport I told that I was looking for a new subject and he proposed to talk to his team leader at work to see whether they had a subject. No sooner said than done I started in July 2019 my master thesis at Greenchoice. The period at Greenchoice went well, at least in my opinion, I enjoyed the stay and the lessons that I learned working in an industrial environment compared to the university projects. It was the end of January 2020 when my time at Greenchoice was over and just had to finish up the report. Early March the coronavirus has also reached the Netherlands for some days and the country went into a smart lockdown. At that time I spent most of my time for a music venue which had to be closed due to the smart lockdown. Next, was the preparation and opening of the music venue in the new 1.5 meters society. Hold up, I am forgetting something... I should also finish my master thesis. Motivation was hard to be found and since June 2020 several attempts have been made attempting to fin- ish the report. Small steps were made towards finishing the report, however, it was till January 2021 that I got the first impression that I was actually going to finish the report. The next months I spent finishing up the report and processing the (abundant) received feedback. After a long bumpy ride I present to you my final master thesis report. First off, I would like to thank Roelant for introducing me to Greenchoice and the support throughout my stay at Greenchoice. Furthermore, I would like to thank Greenchoice as a company and the employees, espe- cially the ones who participated in my research, for the fantastic time I had and helping me in my research. In special I would like to thank Mick and Jeroen for their supervision, guidance and support from the side of Greenchoice and most importantly their patience with me to complete my report after a slight delay. Further- more, I would like to thank Andy for the supervision, guidance and support from the side of the TU Delft and also his patience with me. Subsequently, I would like to thank Ujwal for taking place in the thesis committee. In addition, I would like to thank everyone else for the support through the sometimes hard corona times with the lockdowns. Lastly, I would like to thank my parents for keeping me fed throughout the past period. Maiko Goudriaan Delft, May 2021 ii CONTENTS 1 Introduction 1 1.1 Research design..........................................2 1.2 Introduction of Greenchoice...................................3 1.2.1 Greenchoice’s development process............................4 1.2.2 Greenchoice’s coding guidelines..............................4 1.2.3 Greenchoice’s old code review checklist..........................4 1.3 Research questions........................................5 1.4 Structure of thesis.........................................6 2 Related work 7 2.1 Formal software inspection....................................7 2.2 Informal software inspection...................................8 2.3 Modern code reviews.......................................9 2.3.1 Challenges in open source software development.....................9 2.3.2 Challenges in industrial setting.............................. 10 2.3.3 Resolving the size challenge................................ 10 2.3.4 Motivation for code reviews................................ 10 2.3.5 Actual defects discovered................................. 10 2.3.6 Usefulness and understanding code reviews........................ 10 2.3.7 Identify defect prone areas of code............................. 11 2.3.8 Checklists......................................... 11 3 Methodology 13 3.1 Initial code review questionnaire................................. 13 3.1.1 Questionnaire introduction................................ 14 3.1.2 Discovering the motivation for code reviews........................ 14 3.1.3 Comparing the different categories............................ 16 3.1.4 Discovering the initial thoughts with statements...................... 17 3.1.5 Error prevention in practice................................ 17 3.2 Exploring Greenchoice’s current code review process....................... 18 3.2.1 Interviewing the developers................................ 18 3.2.2 Meta data analysis..................................... 18 3.2.3 Comment data analysis.................................. 20 3.3 Creating a code review checklist................................. 23 3.3.1 Expectations and intentions regarding the checklist.................... 23 3.3.2 Joining a scrum team................................... 23 3.4 Using and refining the checklist.................................. 24 3.4.1 Explaining the general set-up............................... 24 3.4.2 Explaining the pre questionnaire.............................. 24 3.4.3 Explaining the post questionnaire............................. 26 4 Motivation for code reviews 27 4.1 Demographic data........................................ 27 4.1.1 Function.......................................... 27 4.1.2 Experience and familiarization............................... 28 4.1.3 Frequency and roles.................................... 28 4.2 Motivation for code reviews.................................... 29 4.2.1 Comparing the motivations................................ 29 4.2.2 Global categories...................................... 31 iii CONTENTS iv 4.3 Code review statements...................................... 33 4.4 Practice example......................................... 34 4.5 Discussion............................................ 34 4.5.1 Demographic data..................................... 34 4.5.2 Motivations........................................ 35 4.5.3 Code review statements.................................. 36 5 Current code review process 39 5.1 Interviews............................................. 39 5.1.1 Preparation for a pull request............................... 39 5.1.2 Performing a code review................................. 40 5.1.3 Strengths and weaknesses................................. 41 5.1.4 Potential improvements.................................