Eindhoven University of Technology MASTER Do you come here often? investigating the effects of domain knowledge and controllability on a theme park recommender system Smits, T.H.J. Award date: 2021 Link to publication Disclaimer This document contains a student thesis (bachelor's or master's), as authored by a student at Eindhoven University of Technology. Student theses are made available in the TU/e repository upon obtaining the required degree. The grade received is not published on the document as presented in the repository. The required complexity or quality of research of student theses may vary by program, and the required minimum study period may vary in duration. General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain 1 Eindhoven, 22-1-2021 Do you come here often? Investigating the effects of domain knowledge and controllability on a theme park recommender system. By T.H.J. Smits Identity number: 0889679 In partial fulfilment of the requirements for the degree of Master of Science in Human-Technology Interaction Supervisors: Dr. ir. M.C. Willemsen Dr. W. Barendregt Company Supervisor: J. Rietbergen 2 ABSTRACT To assist visitors in getting even more out of their visit, Efteling has launched a conversational recommender system to provide suggestions for attractions to visit in their theme park. Literature suggests that the optimal level of control over indicating user preferences may be dependent on the user’s knowledge about the decision domain. In order to investigate this for visitors of a theme park, the controllability of the recommender system was manipulated in a series of A/B tests. By implementing different preference elicitation methods and varying the number of items in the initial questionnaire, the effects of controllability on recommendation satisfaction and persuasiveness are researched. The findings seem to suggest that when users perceived a low level of control due to lacking the domain-specific knowledge to answer a question, this negatively affected their recommendation satisfaction. The level of domain-specific knowledge also affected the preferred level of domain-specific control that the user was offered. However, further research is highly encouraged. 3 Acknowledgements First and foremost, I would like to express my gratitude to Martijn Willemsen for his invaluable involvement in this thesis. Even though, for the complete duration of the project, it was not possible to have a meeting in the same room, he was able to share helpful insights, motivating words, and an inspiring passion for the topic. Additionally, I would like to thank my second supervisor, Wolmet Barendregt, for her useful perceptions that have brought this thesis to a higher level. Secondly, I thank the app and guest data team at Efteling, and Jonas Rietbergen in particular, for granting me the opportunity to contribute to the recommender system and for making me feel welcome in the team, both at the office and in digital meetings. It was a true pleasure working at Efteling. The freedom they provided, as well as the lively discussions and critical questions, were truly helpful in shaping the project and the study. I also want to thank my friends and my family for their support throughout this project in particular, and my educational career in general. I feel fortunate to have them around me. Last, but most certainly not least, I would like to thank my girlfriend, Simone. Thank you for your endless support and motivation throughout this project and in life, and most importantly, your willingness to spend a lockdown with me. 4 Table of Contents Acknowledgements ................................................................................................................................. 3 Introduction ............................................................................................................................................. 7 Efteling ................................................................................................................................................. 7 Recommender systems........................................................................................................................ 7 Conversational-style recommender systems ................................................................................... 9 A trade-off between personalization and usability ........................................................................ 11 The effects of domain knowledge .................................................................................................. 12 Research question and hypotheses ................................................................................................... 13 Method .................................................................................................................................................. 16 User-centric evaluation framework ................................................................................................... 16 Materials ............................................................................................................................................ 18 Recommendation engine ............................................................................................................... 18 Recommender system ................................................................................................................... 18 Experiments ....................................................................................................................................... 20 Experiment 1 .................................................................................................................................. 20 Critiquing component .................................................................................................................... 22 Experiment 2 .................................................................................................................................. 23 Measures ........................................................................................................................................... 24 Data collection ................................................................................................................................... 25 Results ................................................................................................................................................... 28 Data description ................................................................................................................................. 28 Domain knowledge ........................................................................................................................ 28 Perceived control ........................................................................................................................... 28 Recommendation satisfaction ....................................................................................................... 29 Ratio of acceptance ....................................................................................................................... 30 Analysis .............................................................................................................................................. 30 Missing data ................................................................................................................................... 31 5 Experiment 1...................................................................................................................................... 32 Effects on recommendation satisfaction ....................................................................................... 32 Effects on probability of acceptance .............................................................................................. 35 Experiment 2...................................................................................................................................... 36 Effects on recommendation satisfaction ....................................................................................... 36 Effects on probability of acceptance .............................................................................................. 39 Discussion .............................................................................................................................................. 42 Limitations and suggestions for future research ............................................................................... 44 COVID-19 ....................................................................................................................................... 44 Other limitations ............................................................................................................................ 45 Future research directions ............................................................................................................. 47 Recommendations for Efteling .....................................................................................................