Designing AI-Based Systems for Qualitative Data Collection and Analysis Zur Erlangung des akademischen Grades eines Doktors der Wirtschaftswissenschaften ( Dr. rer. pol. ) von der KIT-Fakultät für Wirtschaftswissenschaften des Karlsruher Instituts für Technologie (KIT) genehmigte DISSERTATION von Tim Rietz, M.Sc. ______________________________________________________________ Tag der mündlichen Prüfung: 01.07.2021 Referent: Prof. Dr. Alexander Mädche Korreferent: Prof. Dr. Paola Spoletini Karlsruhe Mai 2021 Acknowledgments Having started my PhD studies in December 2017, I remember the past three and a half years as a series of ups and downs, which probably goes for everything in life. Looking back at this exciting, inspiring, and challenging time, I distinctly remember many ups, while the downs seem almost forgotten. To a large extent, I attribute this to the wonderful people that I got to meet along the way, who never failed to make my time as a PhD student and as an IT consultant fun. Certainly, I want to thank my mentor and PhD supervisor Prof. Dr. Alexander M¨adche, for his guidance, inspiration, and feedback throughout my studies. While I did not know what to expect when I started my position at the institute, I quickly learned how lucky I was with my choice of a supervisor. Alexander always had an open door for my questions, ideas, and concerns. He also actively seeked updates on my process and encourage me to submit my research to prestigious outlets. I am incredibly grateful for your support. On that note, I also want to thank Prof. Dr. Paola Spoletini, Prof. Dr. Hagen Lindst¨adt, and Prof. Dr. Wolf Fichtner for taking the time to serve on my PhD committee. I enjoyed discussing my research with you! Special thanks go to Paola, for welcoming me into the Requirements Engineering community on Jeju Island in 2019 and helping me to establish a network in RE! I want to express my gratitude to my colleagues at the Institute of Information Systems and Marketing (IISM) and at the research group for Information Systems and Service Design (ISSD) in particular. Thank you for always helping with pre-tests, for providing valuable feedback during our research meetings, for the numerous coffee breaks, for laughing together about the weird intricacies of publication processes, and for several highly entertaining social events. Naturally, there are some colleagues whom I had the chance to spend more time with, while I met others only occasionally. Regardless, I always felt that we acted as a team, celebrating successes and learning from failures together. For that, I am grateful. Besides my research at the IISM, I completed four projects as business analyst as part of the IT consultancy Senacor. In fact, my first workday after finishing my Masters was to receive my Senacor laptop and phone, as well as an introduction at the Senacor headquarters in Nurnberg.¨ Afterwards, I jumped on a train to Frankfurt for the Senacor Christmas Party. Not a bad start! Thank you to all the great colleagues that I had the chance to meet during my introduction to Senacor, the team events, and of course, the projects. Thanks to your support and guidance, I learned much about IT projects, architectures, stakeholder management, and, naturally, great Powerpoint slide design. I am convinced that the insights I gained helped me with speaking the RE language in my research. I am immensely grateful and just happy to always have had the backup of my friends! While I am glad for the ties that continue to hold firm from my school days, I want to i especially mention our Kiwiwi group, who are companions and best friends since the first days of my university education. Florian Engel, Lukas R¨oring, Sophie Lucas, and Melanie & Alexis Guttstadt. Thank you for the great times, for your support, and for being there (now quite literally in or around Karlsruhe). I know that I can always count on you! My heartfelt gratitude also goes to my parents and my family. I dedicate this dissertation to my mother, Sabine, and to my father, Joachim Rietz. I am eternally thankful for your endless and unconditional love and support and for your profound belief in my abilities. Thank you for sharing this journey with me. Finally, a very special thank you to my partner and best friend, Lena Annabell Straub, for being by my side every step of the way. Thinking back to the ups and downs of the dissertation, you deserve the biggest credit for celebrating even the tiniest ups with me, as well as guiding me through the downs. Thank you for your love, support, patience, and care. Thank you for being there for me when I needed it, for believing in me, and for a lot of excitement! I am immeasurably grateful to have you in my life. Tim Rietz Karlsruhe, Germany July 2021 ii Abstract With the continuously increasing impact of information systems (IS) on private and professional life, it has become crucial to integrate users in the IS development process. One of the critical reasons for failed IS projects is the inability to accurately meet user requirements, resulting from an incomplete or inaccurate collection of requirements during the requirements elicitation (RE) phase. While interviews are the most effective RE technique, they face several challenges that make them a questionable fit for the numerous, heterogeneous, and geographically distributed users of contemporary IS. Three significant challenges limit the involvement of a large number of users in IS devel- opment processes today. Firstly, there is a lack of tool support to conduct interviews with a wide audience. While initial studies show promising results in utilizing text-based conversational agents (chatbots) as interviewer substitutes, we lack design knowledge for designing AI-based chatbots that leverage established interviewing techniques in the context of RE. By successfully applying chatbot-based interviewing, vast amounts of qualitative data can be collected. Secondly, there is a need to provide tool support enabling the analysis of large amounts of qualitative interview data. Once again, while modern technologies, such as machine learning (ML), promise remedy, concrete implementations of automated analysis for unstructured qualitative data lag behind the promise. There is a need to design interactive ML (IML) systems for supporting the coding process of qualitative data, which centers around simple interaction formats to teach the ML system, and transparent and understandable suggestions to support data analysis. Thirdly, while organizations rely on online feedback to inform requirements without explicitly conducting RE interviews (e.g., from app stores), we know little about the demographics of who is giving feedback and what motivates them to do so. Using online feedback as requirement source risks including solely the concerns and desires of vocal user groups. With this thesis, I tackle these three challenges in two parts. In part I, I address the first and the second challenge by presenting and evaluating two innovative AI-based systems, a chatbot for requirements elicitation and an IML system to semi-automate qualitative coding. In part II, I address the third challenge by presenting results from a large-scale study on IS feedback engagement. With both parts, I contribute with prescriptive knowledge for designing AI-based qualitative data collection and analysis systems and help to establish a deeper understanding of the coverage of existing data collected from online sources. Besides providing concrete artifacts, architectures, and evaluations, I demonstrate the application of a chatbot interviewer to understand user values in smartphones and provide guidance for extending feedback coverage from underrepresented IS user groups. iii Contents Abstract iii List of Figures vii List of Tables viii List of Abbreviations ix 1 Introduction1 1.1 Motivation . .1 1.2 Research Gaps and Research Questions . .5 1.3 Thesis Structure . .9 2 Foundations 12 2.1 The Role of Qualitative Data in IS Development and Research . 13 2.2 Qualitative Data Collection . 17 2.2.1 Laddering Interview Technique . 17 2.2.2 User Feedback in Online Channels . 20 2.3 Qualitative Data Analysis . 22 2.3.1 Coding in Qualitative Data Analysis . 22 2.3.2 Qualitative Data Analysis Systems . 22 2.4 AI-based Technology for Qualitative Data Collection and Analysis . 24 2.4.1 AI-based Technology for Qualitative Data Collection . 24 2.4.2 AI-based Technology for Qualitative Data Analysis . 25 3 Part I: AI-based Qualitative Data Collection & Analysis in IS Development 28 3.1 Study 1: Ladderbot - A Requirements Self-Elicitation System . 28 3.1.1 Introduction . 28 3.1.2 Designing a Laddering Interview Chatbot for RE . 29 3.1.2.1 Common Issues of RE Interviews with Novice Users . 29 3.1.2.2 Chatbot Structure for Laddering Interviews . 32 3.1.2.3 Interview Visualization . 35 3.1.3 Conclusion . 35 3.2 Study 2: Re-Evaluating User Values of Smartphones - a Wide Audience Study 37 3.2.1 Introduction . 37 3.2.2 Background . 39 3.2.2.1 Value-Oriented Research . 39 3.2.2.2 User Perspectives on Smartphone Usage . 39 iv Contents 3.2.3 Methodology . 42 3.2.3.1 Design and Procedure . 42 3.2.3.2 Participants . 42 3.2.3.3 Treatments . 43 3.2.3.4 Quantitative Response Analysis . 44 3.2.3.5 Coding . 45 3.2.3.6 Generating the Hierarchical Goal Structure . 46 3.2.4 Results . 48 3.2.4.1 Descriptive Analysis . 48 3.2.4.2 Quantitative Response Analysis . 49 3.2.4.3 Interviewee Perception Analysis . 51 3.2.4.4 Content Analysis . 51 3.2.4.5 Differences between Survey- and Chatbot-based Laddering 57 3.2.5 Discussion . 58 3.2.5.1 Implications for Research . 58 3.2.5.2 Implications for Practice .