Bug Assignment: Insights on Methods, Data and Evaluation
Total Page:16
File Type:pdf, Size:1020Kb
Bug Assignment: Insights on Methods, Data and Evaluation by Ali Sajedi Badashian A thesis submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Computing Science University of Alberta © Ali Sajedi Badashian, 2018 Abstract The bug-assignment problem is prevalently defined as ranking developers based on their competence to fix a given bug. Previous methods in the area used machine-learning or information-retrieval techniques and considered textual elements of bug reports as evidence of expertise of developers to give each of the developers a score and sort the developers for the given bug report. Despite the importance of the subject and the substantial attention it has received from researchers during last 15 years, still it is a challenging, time-consuming task in large software projects. Even there is still no unanimity on how to validate and comparatively evaluate bug-assignment methods and, often times, methods reported in the literature are not repro- ducible. In this thesis, we make the following contributions. 1) We investigate the effect of three important experimental-design parameters in the previous research; the evaluation metric(s) they report, their definition of who the real assignee is, and the community of developers they consider as candidate assignees. Supported by our experiment on a comprehensive data set of bugs we collected from Github, we propose a systematic framework for evaluation of bug-assignment research. Addressing those aspects supports better evaluation, enables replication of the study and promotes its usage in other research or industrial applications. 2) We propose a new bug-assignment approach relying on the set of Stack Overflow tags as the thesaurus of programming keywords. Our approach, called Thesaurus and Time based Bug Assignment (TTBA), weights the relevance of a developer’s expertise based on how recently they have fixed a bug with keywords similar to the bug at hand. In spite of its ii simplicity, our method predicts the assignee with high accuracy, outperforming state-of-the- art methods. 3) We extend TTBA to consider a broader record of the developer’s expertise, consid- ering multiple sources of evidence of expertise. Then we investigate the information value of these information sources, considering various technical contributions to the project and contribution to social software platforms. We show that in addition to bug-fixing contribu- tions, other technical and even social contributions of the developers within version control system are useful for bug-assignment. We also show that extending the sources of expertise can improve the accuracy of assignee recommendations. 4) We study the impact and usefulness of the above contributions using a comprehensive data set of bugs we collected from 13 long-term open-source projects in Github. In addition to the technical work by developers, this data set includes social contributions of developers in the version control system. This is one of the biggest data sets made available online for further studies and research. iii Preface This thesis is an original contribution by Ali Sajedi-Badashian. Parts of this thesis has been published before or submitted for publication. Chapter 2 of this thesis has been submitted for publication: • Ali Sajedi-Badashian and Eleni Stroulia, “A Systematic Framework for Evaluating Bug-assignment Research”, ACM Computing Surveys (CSUR), 2018 Sajedi-Badashian and Stroulia 2018a. Chapter 3 of this thesis has been submitted for publication: • Ali Sajedi-Badashian and Eleni Stroulia, “TTBA: Thesaurus and Time Based Bug- Assignment”, Journal of Systems and Software, 2018 Sajedi-Badashian and Stroulia 2018c. Chapter 4 of this thesis has been submitted for publication: • Ali Sajedi-Badashian and Eleni Stroulia, “The Information Value of Different Sources of Evidence of Developers’ Expertise for Bug Assignment”, International Conference on Computer Science and Software Engineering (CASCON), 2018 Sajedi-Badashian and Stroulia 2018b. Chapter 5 of this thesis has been published in: • Ali Sajedi-Badashian, Abram Hindle and Eleni Stroulia, “Crowdsourced Bug Triag- ing”, International Conference on Software Maintenance and Evolution (ICSME), 2015 Sajedi-Badashian and Stroulia 2018b. • Ali Sajedi-Badashian, Abram Hindle and Eleni Stroulia, “Crowdsourced Bug Triaging: Leveraging Q&A Platforms for Bug Assignment”, Fundamental Aspects of Software Engineering (FASE), 2016 Sajedi-Badashian and Stroulia 2018b. There are other research that performed during the development of this thesis but are not discussed in this manuscript: iv • Ali Sajedi-Badashian, Afsaneh Esteki, Ameneh Gholipour, Abram Hindle and Eleni Stroulia, “Involvement, Contribution and Influence in GitHub and Stack Overflow”, International Conference on Computer Science and Software Engineering (CASCON), 2014 Sajedi-Badashian et al. 2014. • Ali Sajedi-Badashian and Eleni Stroulia, “Measuring user influence in github: the mil- lion follower fallacy”, CrowdSourcing in Software Engineering (CSI-SE), 2016 Sajedi- Badashian and Stroulia 2016. • Ali Sajedi-Badashian, Vraj Shah and Eleni Stroulia, “GitHub’s big data adaptor: an eclipse plugin”, International Conference on Computer Science and Software Engineer- ing, 2015 Sajedi-Badashian, Shah, et al. 2015. • Ali Sajedi-Badashian and Eleni Stroulia, “Realistic bug triaging”, Software Engi- neering Companion (ICSE-C), IEEE/ACM International Conference on, 2016 Sajedi- Badashian 2016. v To my mother For teaching me “life”, To my father, who is not between us anymore, For inspiring me “honesty”, To my wife, Zahra, and my daughter, Elham, For their love, patience and dedication in helping me during my studies. vi Acknowledgements I hereby thank Professor Eleni Stroulia for her supervision, advice and support during my studies. I also thank the financial supporters. Parts of this work were supported by the following grants: • Queen Elizabeth II Graduate Scholarship1 funded by Faculty of Graduate Studies and Research (FGSR)2 at University of Alberta. • Graduate Student Scholarship3 funded by Alberta Innovates - Technology Futures (AITF)4 • Natural Sciences and Engineering Research Council of Canada (NSERC) and the GRAND NCE. • European Joint Conferences on Theory and Practice of Software (ETAPS) 2016 student scholarship (travel award) • Faculty of Graduate Studies and Research (FGSR) Travel Award • Two GSA Academic Travel Awards • GRAND Postgraduate Scholar Award 1 https://www.ualberta.ca/graduate-studies/awards-and-funding/scholarships/queen-elizabeth-ii 2https://www.ualberta.ca/graduate-studies 3https://fund.albertainnovates.ca/Fund/BasicResearch/GraduateStudentScholarships.aspx 4https://innotechalberta.ca vii Contents 1 Introduction 1 1.1 Problemstatement ................................ 2 1.2 Contributions ................................... 4 1.3 ThesisOutline................................... 6 2 A Systematic Framework for Evaluating Bug Assignment Research 7 2.1 Introduction.................................... 10 2.2 AsurveyonBug-assignmentResearch . .... 13 2.2.1 Surveyprotocolsandprocess . 13 2.2.2 Bug-assignmentTasksandObjectives. .... 15 2.2.3 Bug-AssignmentMethodologies . 19 2.3 Select Empirical Bug-assignment Studies . ........ 21 2.3.1 Knowledge Assumptions of Bug-assignment Methods . ...... 24 2.3.2 Metrics for Evaluation of Bug-assignment Research . ........ 25 2.4 A Discussion of Bug-assignment Evaluation Metrics . .......... 27 2.5 Dimensions of Variability in Bug-assignment Empirical Studies. 31 2.5.1 Ground-truthassignee . 32 2.5.2 Developercommunity. 37 2.6 ExperimentsetupandDataSet . .. 38 2.6.1 TheDataSet ............................... 39 2.7 Findings...................................... 41 2.7.1 Comparing Different Types of “Ground-truth Assignee” . ...... 41 2.7.2 Theeffectof“DeveloperCommunity” . 44 2.8 Discussion..................................... 48 2.8.1 Theproposedevaluationframework . ... 49 2.8.2 Threatstovalidity ............................ 51 2.9 Conclusions .................................... 52 3 TTBA: Thesaurus and Time Based Bug Assignment 54 3.1 Introduction.................................... 57 3.2 BackgroundandRelatedResearch. .... 58 3.3 Recognizing Developers with Relevant Expertise: The TTBA Metric. 61 3.3.1 TF-IDF .................................. 64 3.3.2 FocusingonaThesaurusofTerms. 65 3.3.3 Recency-awareTermWeighting . 67 viii 3.4 TheDataSet ................................... 68 3.5 Evaluation..................................... 69 3.5.1 EvaluationMetrics ............................ 70 3.5.2 TuningandOptimization . 70 3.6 Results....................................... 72 3.6.1 Comparisons Against Implemented Baseline Methods . ...... 72 3.6.2 Comparison Against Results Reported in the Literature ........ 73 3.7 Discussion..................................... 74 3.7.1 Intuitions About Comparison of the Implemented Approaches .... 76 3.7.2 Intuitions About Comparison of Reported Results . ...... 77 3.7.3 ThreatstoValidity ............................ 78 3.8 ConclusionsandFutureWork . .. 82 Appendix 3.A Appendix: Details of mutation experiment for tuning 6 parameters ofourmethod................................... 83 4 An Investigation Into the Information Value of Different Sources of Evi- dence on the Developers’ Expertise for Bug Assignment 89 4.1 Introduction.................................... 92 4.2 BackgroundandRelatedResearch. .... 94 4.2.1 Knowledge assumptions of Bug-Assignment . ... 94 4.3 Bug Assignment Based on