ENGINEERING AND SOFTWARE TESTING ENGINEERING AND SOFTWARE REQUIREMENTS COORDINATING ABSTRACT The development of large, software-intensive sys- This thesis contains four main contributions: (1) tems is a complex undertaking that is generally An in-depth study on REST alignment challeng- tackled by a divide and conquer strategy. Organi- es and practices encountered in industry. (2) A COORDINATING REQUIREMENTS zations face thereby the challenge of coordinating conceptual framework in the form of a taxonomy ENGINEERING AND SOFTWARE TESTING the resources which enable the individual aspects providing constructs that further our understan- of software development, commonly solved by ding of REST alignment. The taxonomy is ope- adopting a particular process model. The alignme- rationalized in an assessment framework, REST- nt between requirements engineering (RE) and bench (3), that was designed to be lightweight software testing (ST) activities is of particular in- and can be applied as a postmortem in closing terest as those two aspects are intrinsically con- development projects. (4) An extensive investi- nected: requirements are an expression of user/ gation into the potential of information retrieval customer needs while testing increases the likeli- techniques to improve test coverage, a common hood that those needs are actually satisfied. REST alignment challenge, resulting in a solution The work in this thesis is driven by empirical pro- prototype, risk-based testing supported by topic Michael Unterkalmsteiner blem identification, analysis and solution develop- models (RiTTM). ment towards two main objectives. The first is to REST-bench has been validated in five cases and develop an understanding of RE and ST alignme- has shown to be efficient and effective in identi- nt challenges and characteristics. Building this fying improvement opportunities in the coordi- foundation is a necessary step that facilitates the nation of RE and ST. Most of the concepts opera- second objective, the development of solutions tionalized from the REST taxonomy were found relevant and scalable to industry practice that im- to be useful, validating the conceptual framework. prove REST alignment. RiTTM, on the other hand, was validated in a The research methods employed to work single case experiment where it has shown great towards these objectives are primarily empirical. potential, in particular by identifying test cases Case study research is used to elicit data from that were originally overlooked by expert test practitioners while technical action research and engineers, improving effectively test coverage. field experiments are conducted to validate the developed solutions in practice. Michael Unterkalmsteiner Blekinge Institute of Technology Doctoral Dissertation Series No. 2015:08 2015:08 ISSN 1653-2090 2015:08 ISBN: 978-91-7295-306-2 Department of Software Engineering Coordinating Requirements Engineering and Software Testing Michael Unterkalmsteiner BlekingeBlekinge InstituteInstitute ofof TechnologyTechnology Doctoraldoctoral disseDissertationrtation seriesSeries NNoo 2014:03 2015:08 Coordinating Requirements Psychosocial,Engineering and Socio-Demographic Software Testing and Health Determinants in Information Communication Technology Use of Older-Adult Michael Unterkalmsteiner Jessica Berner Doctoral Dissertation in Software Engineering Doctoral Dissertation in Applied Health Technology DepartmentDepartment of Software of Health Engineering BlekingeBlekinge InstituteInstitute of TTechnologyechnology SWEDEN 2015 Michael Unterkalmsteiner Department of Software Engineering Publisher: Blekinge Institute of Technology SE-371 79 Karlskrona, Sweden Printed by Lenanders Grafiska, Kalmar, 2015 ISBN: 978-91-7295-306-2 ISSN: 1653-2090 To my parents. There is not a discovery in science, however revolutionary, however sparkling with insight, that does not arise out of what went before. Adding a Dimension (1966) Isaac Asimov ABSTRACT The development of large, software-intensive systems is a complex under- taking that is generally tackled by a divide and conquer strategy. Organi- zations face thereby the challenge of coordinating the resources which en- able the individual aspects of software development, commonly solved by adopting a particular process model. The alignment between requirements engineering (RE) and software testing (ST) activities is of particular inter- est as those two aspects are intrinsically connected: requirements are an expression of user/customer needs while testing increases the likelihood that those needs are actually satisfied. The work in this thesis is driven by empirical problem identification, analysis and solution development towards two main objectives. The first is to develop an understanding of RE and ST alignment challenges and characteristics. Building this foundation is a necessary step that facilitates the second objective, the development of solutions relevant and scalable to industry practice that improve REST alignment. The research methods employed to work towards these objectives are primarily empirical. Case study research is used to elicit data from practi- tioners while technical action research and field experiments are conducted to validate the developed solutions in practice. This thesis contains four main contributions: (1) An in-depth study on REST alignment challenges and practices encountered in industry. (2)A conceptual framework in the form of a taxonomy providing constructs that further our understanding of REST alignment. The taxonomy is op- erationalized in an assessment framework, REST-bench (3), that was de- signed to be lightweight and can be applied as a postmortem in closing development projects. (4) An extensive investigation into the potential of information retrieval techniques to improve test coverage, a common REST alignment challenge, resulting in a solution prototype, risk-based testing supported by topic models (RiTTM). REST-bench has been validated in five cases and has shown to be ef- ficient and effective in identifying improvement opportunities in the co- ordination of RE and ST. Most of the concepts operationalized from the REST taxonomy were found to be useful, validating the conceptual frame- work. RiTTM, on the other hand, was validated in a single case experiment where it has shown great potential, in particular by identifying test cases that were originally overlooked by expert test engineers, improving effec- tively test coverage. vii ACKNOWLEDGMENTS I have started and finished this thesis alone, however met a bunch of people along the way (or left them). This section is dedicated to them. There is little in this work that has not been directly or indirectly influ- enced by my two supervisors, Tony Gorschek and Robert Feldt. Tony has found the right words, at the right time, when I needed an impulse (in any direction). This makes him a great motivator and mentor. Robert’s feed- back, be it in written communication or in lively discussions, has always been to the point, insightful and constructive. This makes him a great peer and inspirer. SERL Sweden provided a supportive work environment, forming a group that is diverse and at the same time focused on high quality re- search. I would like to thank Mikael for the work related discussions, but even more so for the nerd-talk that lightened up my day. Nauman and Bog- dan have always been excellent colleagues that, besides being supportive, made my stay at the office a little less about work and a little more about life. This is actually true for all colleagues at SERL Sweden. Furthermore, I would like to thank all my co-authors. This research would not have been possible without the support from the companies that provided their time and effort. I would like to thank Louise, Antonios and Johan for their collaboration and patience in trying out my ideas. My appreciation goes to the engineers whose brains I could pick. I value nothing higher than the support I received from my parents, my sister, brother and my grandmas, not only during the last five years but in all the ups and downs before, being physically close or far away from them. You’re all in my thoughts, very often. My wife, Rocio, is the haven I return to after a day at the job. Nothing I have accomplished would have been possible without her support and tolerance for my endeavor. Karlskrona, May 1, 2015 ix PREFACE papers in this thesis This compilation thesis includes the following six papers1. Chapter 2: Elizabeth Bjarnason, Per Runeson, Markus Borg, Michael Unterkalmsteiner, Emelie Engström, Björn Regnell, Giedre Sabali- auskaite, Annabella Loconsole, Tony Gorschek, and Robert Feldt, “Challenges and Practices in Aligning Requirements with Verifica- tion and Validation: A Case Study of Six Companies,” Empirical Software Engineering, vol. 19, no. 6, pp. 1809–1855, 2014. Chapter 3: Michael Unterkalmsteiner, Robert Feldt, and Tony Gorschek, “A Taxonomy for Requirements Engineering and Software Test Align- ment,” ACM Transactions on Software Engineering and Methodol- ogy, vol. 23, no. 2, 2014. Chapter 4: Michael Unterkalmsteiner, Tony Gorschek, Robert Feldt, and Eriks Klotins, “Assessing Requirements Engineering and Software Test Alignment with REST-bench – Five Case Studies,” Journal of Systems and Software, Under Review, 2015. Chapter 5: Elizabeth Bjarnason, Michael Unterkalmsteiner, Emelie En- gström, and Markus Borg, “An Industrial Case Study on the Use of Test Cases as Requirements,” in Proceedings of the 16th Interna- tional Conference on Agile Software Development,