UTRECHT UNIVERSITY MASTER THESIS BUSINESS INFORMATICS Industry-Pushed Ways Of Working In User Story Refinement: How Good Are They Really? Author: Thesis Supervisor: Jasper Berends Fabiano Dalpiaz [email protected] [email protected] Student Number: 5516080 Daily Supervisor: Thesis Supervisor 2: Beau Verdiesen Sjaak Brinkkemper [email protected] [email protected] July 31, 2020 This thesis is submitted in fulfilment of the requirements for the degree of Master of Science in the Master in Business Informatics Graduate School of Natural Sciences ii iii Acknowledgements During my thesis, I have received help and support from a lot of people. First and foremost, I want to express my gratitude for all the supervision and guidance that I received from my supervisor dr. Fabiano Dalpiaz. I could not have achieved what I did if it was not for his ex- pertise, frequent feedback and support. Secondly, I want to thank my daily supervisor at Info Support, Beau Versiesen. His practical insights really helped to improve this work and to validate the feasibility and relevance of the research and he also helped me out a lot by proofreading my work. I also want to thank dr. Sjaak Brinkkemper for taking the time to review my thesis as the second supervisor of this project. Additionally, my sincere thanks go out to my roommates, friends, fellow graduate interns at Info Support and my family. Their moral support has been invaluable to me and I could not have made my thesis without them. Finally, I want to thank all students and team members that participated in my research. My thesis results depended heavily on their willingness to participate in my research, for which I will be forever grateful. Jasper Berends, July 31, 2020 iv Contents 1 Introduction 5 2 Research Design 7 2.1 Research Questions . .7 2.2 Relevance . .8 2.2.1 Scientific Relevance . .8 2.2.2 Practical Relevance . .9 2.3 Research Approach . .9 2.3.1 Literature Research . 10 2.3.2 Case studies . 11 2.3.3 Controlled experiment . 11 3 Literature study 13 3.1 Requirements Engineering for Software Development . 13 3.1.1 Challengers in RE . 14 3.1.2 Agile Software Development . 14 3.2 Behaviour-Driven Development . 15 3.2.1 Process-Deliverable Diagram . 16 3.2.2 BDD Applications and Advantages . 19 3.2.3 Example Case . 20 3.3 Three Amigo Sessions in Requirements Engineering . 21 3.3.1 Example Mapping . 24 3.3.2 Feature Mapping . 26 3.4 Shared Understanding . 29 4 Treatment Design 37 4.1 Case Study Design . 37 4.1.1 Longitudinal Case Study . 37 4.1.2 Data Analysis . 40 4.1.3 Validity . 40 4.2 Controlled Experiment . 45 4.2.1 Context . 45 4.2.2 Planning and Presentation . 46 4.2.3 Student Handout Package . 47 4.2.4 Execution of Experiment . 48 4.2.5 Valitidy . 49 4.3 Measuring technique performance . 52 4.3.1 Questionnaires . 52 1 2 CONTENTS 4.3.2 Experimental Output Evaluation . 55 5 Treatment Validation 57 5.1 Controlled experiment - Requirements Engineering course . 58 5.1.1 Discussion . 58 5.1.2 Overall Results . 59 5.1.3 Results per Aspect . 62 5.1.4 Results per Group . 63 5.1.5 Output Analysis . 64 5.1.6 Conclusion . 66 5.2 Case Study - Fizor Example Mapping . 67 5.2.1 Session Results . 68 5.2.2 End Evaluation . 72 5.2.3 Conclusion . 73 5.3 Case Study - Pension Management Firm Example Mapping . 74 5.3.1 Evaluation after session 4 . 78 5.3.2 End Evaluation . 79 5.3.3 Conclusion . 79 5.4 Case Study - Pension Management Firm Feature Mapping . 80 5.4.1 Conclusion . 81 5.5 Case Study - Example Mapping Online Tryout . 82 5.5.1 Conclusion . 83 6 Conclusion 85 6.1 Cross-case Conclusions . 85 6.2 Answers to Research Questions . 87 6.2.1 RQ1: What defines a Three Amigo session? . 87 6.2.2 RQ2: How can the performance of TA sessions be measured? . 87 6.2.3 RQ3: How do TA sessions perform when used for the first time? . 88 6.2.4 RQ4: How do TA sessions perform after becoming familiar with the technique? . 89 6.2.5 MRQ: How do defined Three Amigo session techniques perform for user story refinement? . 89 7 Discussion 91 7.1 Limitations . 91 7.2 Future Work . 92 8 Bibliography 93 A Questionnaire 99 A.1 Complete Technique Questionnaire . 99 A.2 TA Session Questionnaire . 100 A.3 Post-Implementation Questionnaire . 101 B Case Study Forms 102 B.1 Informed Consent . 103 CONTENTS 3 C Controlled Experiment 105 C.1 User Story Outputs – Example Mapping . 105 C.2 User Story Outputs – Feature Mapping . 108 C.3 Lecture Slides . 110 C.4 Student Handout Package . 138 C.5 Questionnaire Results . 150 C.5.1 Results per Technique - Combined . 150 C.5.2 Results per Aspect . 151 C.5.3 Results per Group . 154 C.6 TA Session Outputs . 160 C.7 Output Analysis . 167 C.7.1 US1 Aggregation . 167 C.7.2 US2 Aggregation . 168 D Case Study - Fizor 169 D.1 Correlation Matrices . 169 D.2 Participant Results . 170 D.3 Session Results . 171 E Case Study - Pension Manager Example Mapping 173 E.1 Correlation Matrices . 173 4 CONTENTS Abstract New techniques for refining requirements in software development are frequently presented. However, these techniques are often not scientifically validated on their performance. In this study, we aim to validate two refinement techniques, Example Mapping and Feature Mapping. These two techniques are both Three Amigo session techniques, in which people from different work together to refine the software increment. Shared understanding is an essential aspect of these sessions. We present a new measurement tool with which the performance of refinement techniques can be investigated, which also incorporates shared understanding. We investi- gate the performance of the refinement techniques with a controlled experiment and four case studies. Based on the results, Example Mapping performs well under certain conditions. We observed a learning effect for the technique, which resulted in better performance of sessions after having used the technique several times. For Feature Mapping, results were inconclusive and additional research is required to establish its performance. When conditions are not right for Example Mapping or Feature Mapping, Three Amigo session principles can still be applied to refinements to result in a well-performing session. Further research could be undertaken to generalise this study’s findings further, studying the long-term effects of the techniques on the implementation of a user story and studying the performance of the techniques when team members are co-located. Additionally, future research can be conducted to compare the Three Amigo session techniques to other refinement techniques, to further investigate the validity of the performance measurement tool and to study the performance of shorter Three Amigo sessions. Keywords: Three amigo sessions, Example mapping, Feature mapping, Shared understanding, Refinement Chapter 1| Introduction Software development with the use of Agile methodologies requires a shared understanding of requirements between all stakeholders. Informal and frequent communication has been considered a crucial part of Agile Requirements Engineering (RE) [53]. RE is an integral part of software development and plays a significant role in cost-effective software development [45]. On the other hand, bad requirements are adverse for a software project and often lead to higher costs [31]. For refining requirements, a Three Amigos session can be organised. The main goal of these sessions is to gain a shared understanding and a clear description of the underlying require- ments of an increment of work. With a Three Amigos session, at least three people should be present that together have good knowledge of business analysis, software development.