DETECTING AND CORRECTING DUPLICATION IN BEHAVIOUR DRIVEN DEVELOPMENT SPECIFICATIONS A THESIS SUBMITTED TO THE UNIVERSITY OF MANCHESTER FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN THE FACULTY OF SCIENCE AND ENGINEERING 2020 By Leonard Peter Binamungu School of Engineering, Department of Computer Science Contents Abstract 10 Declaration 12 Copyright 13 Acknowledgements 14 Dedication 16 1 Introduction 17 1.1 Behaviour Driven Development . 20 1.2 The Problem . 24 1.3 Hypothesis and Research Questions . 26 1.4 Thesis Contributions . 27 1.5 Publication Activity . 29 1.6 Thesis Organisation . 29 2 Literature Review 31 2.1 Terminology . 32 2.1.1 Terminology on Duplication . 32 2.1.2 Other Terminology . 35 2.2 Duplication in Software Systems . 35 2.2.1 Evidence of Duplication in Real World Systems . 35 2.2.2 Causes of Duplication in Software Systems . 36 2.2.3 Upsides and Downsides of Duplication in Software Systems . 37 2.3 Duplication Detection . 43 2.3.1 Static analysis approaches . 44 2 2.3.2 Dynamic analysis approaches . 46 2.3.3 Combination of static and dynamic analysis approaches . 47 2.4 Duplication Removal . 49 2.4.1 Refactoring of duplicates in production code . 50 2.4.2 Removing duplication in test code . 53 2.4.3 Test Suite Reduction . 54 2.4.4 Removing duplication in natural language tests . 55 2.4.5 Assessing the quality of tests and requirements . 56 2.5 Analysis and Improvement of BDD Specifications . 58 2.6 Research Gap . 63 2.7 Summary . 65 3 Challenges and Opportunities for Maintaining BDD Suites 68 3.1 Introduction . 68 3.2 Study Design . 69 3.2.1 Survey Design . 69 3.2.2 Recruitment of Respondents . 70 3.2.3 Survey Respondents and Data Analysis Approach . 72 3.2.4 Ethical Considerations . 75 3.3 Results . 76 3.3.1 RQ1. Extent of Active Use of BDD . 76 3.3.2 RQ2. Perceived Benefits and Challenges Involved in Using BDD 79 3.3.3 RQ3. Challenges of Maintaining BDD Specifications . 84 3.3.4 RQ4. Duplication in BDD Suites . 85 3.4 Discussion . 92 3.5 Research Opportunities . 96 3.6 Threats to Validity . 97 3.7 Summary . 99 4 Benchmark Development 101 4.1 Introduction . 101 4.2 Semantically Equivalent Scenarios . 101 4.3 Benchmark Development . 102 4.3.1 Context . 102 4.3.2 The Host Systems . 103 4.3.3 Duplicates between Original Scenarios . 105 3 4.3.4 Duplicate Injection . 106 4.4 Summary . 109 5 Detecting Duplicate Scenarios in BDD Suites 110 5.1 Introduction . 110 5.2 The Duplicate Detection Problem in BDD . 113 5.3 Existing Tools . 115 5.3.1 Tools Selection . 115 5.3.2 Tools Configuration and Experiment Setup . 116 5.3.3 Results and Discussion . 117 5.4 Detecting Duplicate Scenarios . 118 5.4.1 Hypothesis . 118 5.4.2 Overall Duplicate Detection Framework . 119 5.4.3 Definitions of Key Terms . 122 5.4.4 Using Default Traces to Detect Duplicate Scenarios . 123 5.4.5 Using Essential Traces to Detect Duplicate Scenarios . 125 5.4.6 Algorithm and Implementation . 129 5.5 Evaluation . 133 5.5.1 Experiment Design . 133 5.5.2 Results . 134 5.5.3 Discussion . 138 5.6 Summary . 140 6 Quality-Oriented Removal of Duplicate Scenarios in BDD Suites 142 6.1 Introduction . 142 6.2 BDD Suite Quality Principles . 145 6.2.1 Aspects of Quality in BDD Specifications . 146 6.2.2 Principle of Conservation of Steps . 148 6.2.3 Principle of Conservation of Domain Vocabulary . 149 6.2.4 Principle of Elimination of Technical Vocabulary . 149 6.2.5 Principle of Conservation of Proper Abstraction . 150 6.3 Community Support for the Quality Principles . 150 6.3.1 Survey Design . 151 6.3.2 Ethical Considerations . 152 6.3.3 Respondents and Data Analysis . 152 6.3.4 Results . 154 4 6.3.5 Conclusions from the Survey . 158 6.4 Operationalization of BDD Quality Principles . 161 6.4.1 Assessing Conservation of Steps . 161 6.4.2 Assessing Conservation of Domain Vocabulary . 162 6.4.3 Assessing Elimination of Technical Vocabulary . 163 6.4.4 Assessing Conservation of Proper Abstraction . 165 6.4.5 Implementation . 167 6.5 Evaluation . 169 6.5.1 Context and Research Questions . 169 6.5.2 Lab Experiment: Design, Results and Discussion . 172 6.5.3 Industry Experiment: Design, Results and Discussion . 179 6.6 Threats to Validity . 183 6.7 Summary . 185 7 Conclusions and Future Work 188 7.1 Summary of Research Contributions . 188 7.2 Answers to the Research Questions . 190 7.3 Future Work . 192 7.3.1 Future work on duplicate detection in BDD . 192 7.3.2 Future work on quality of BDD specifications . 193 7.3.3 Future work for BDD as a whole . 194 Bibliography 195 A A Survey on BDD Use by Software Engineering Teams and the Challenges of Duplication in BDD Specifications 215 A.1 Survey Questions . 215 A.2 Codes and Themes from the Challenges Reported by BDD Practitioners 217 B Sample Duplicate Report 228 B.1 Example duplication report . 228 C Sample Remove Suggestions 229 C.1 Sample remove suggestions . 229 Word Count: 56184 5 List of Tables 2.1 Example code fragments for different types of duplication in program code (adapted from Roy and Cordy [1]) . 33 2.2 Evidence, advantages, and disadvantages of duplication in software systems . 43 2.3 Summary of duplicate detection approaches . 50 2.4 Approaches for removing duplication, as well as assessing and improv- ing the quality of test suites and requirements . 58 3.1 Survey on BDD maintenance challenges: Distribution of respondents by continent . 73 3.2 Survey on BDD maintenance challenges: Distribution of respondents’ organisations . 73 3.3 Survey on BDD maintenance challenges: Distribution of job roles of respondents . 74 3.4 Benefits and challenges of using BDD . 81 3.5 Activeness of BDD use in industry . 93 3.6 Challenges and research opportunities for maintenance and evolution of BDD specifications . 98 4.1 Characteristics of selected evaluation software . 105 4.2 Number of duplicate scenario pairs between original scenarios . 106 4.3 Number of volunteers per system . 108 4.4 Distribution of duplicate scenario pairs across the three evaluation sys- tems . 108 5.1 Results of experimentation with existing clone detection tools . 118 5.2 Detailed results for the 8 different hypotheses on the 3 systems . 136 5.3 Comparison between our tool and the existing duplicate detection tools 138 6 6.1 BDD quality aspects from scientific and grey literature . 147 6.2 Survey on BDD quality principles: roles of respondents . 153 6.3 Survey on BDD quality principles: sizes of respondent organisations . 153 6.4 Other recommended practices for all project stakeholders on how to keep BDD specifications readable, easy to extend, and maintainable . 159 6.5 Other recommended practices for testers and developers on how to keep BDD specifications readable, easy to extend, and maintainable . 160 6.6 Performance of the four principles on the scenarios injected by each of the 13 volunteers . 177 6.7 Suggestions of injected scenarios for removal: combined algorithm . 178 6.8 Expert opinions: scenarios suggested for removal and the reasons . 181 A.1 Codes for the challenges facing respondent BDD practitioners . 218 A.2 BDD challenges: themes with related codes . 224 7 List of Figures 1.1 Typical TDD cycle (Source: [2]) . 19 1.2 Typical ATDD cycle (Source: [3]) . 20 1.3 Organisation of a typical BDD project . 25 2.1 Possible distribution of duplicates and non-duplicates in a software artefact (Adapted from Roy and Cordy [4]) . 34 2.2 Classification of the literature we surveyed . 66 3.1 Mapping between the research questions in section 3.1 and the ques- tions which are not about respondents’ demographics on the survey in Appendix A.1 . 71 3.2 Extent of BDD use . ..