Data Quality and Quantity in Mobile Experience Sampling Niels van Berkel ORCID: 0000-0001-5106-7692 Submitted in fulfilment of the requirements for the degree of Doctor of Philosophy. April, 2019 School of Computing and Information Systems Melbourne School of Engineering The University of Melbourne, Australia Abstract The widespread availability of technologically-advanced mobile devices has brought researchers the opportunity to observe human life in day-to-day circumstances. Rather than studying human behaviour through extensive surveys or in artificial laboratory situations, this research instrument allows us to systematically capture human life in naturalistic settings. Mobile devices can capture two distinct data streams. First, the data from sensors embedded within these devices can be appropriated to construct the context of study participants. Second, participants can be asked to actively and repeatedly provide data on phenomena which cannot be reliably collected using the aforementioned sensor streams. This method is known as Experience Sampling. Researchers employing this method ask participants to provide observations multiple times per day, across a range of contexts, and to reflect on current rather than past experiences. This approach brings a number of advantages over existing methods, such as the ability to observe shifts in participant experiences over time and context, and reducing reliance on the participant’s ability to accurately recall past events. As the onus of data collection lies with participants rather researchers, there is a firm reliance on the reliability of participant contributions. While previous work has focused on increasing the number of participant contributions, the quality of these contributions has remained relatively unexplored. This thesis focuses on improving the quality and quantity of participant data collected through mobile Experience Sampling. Assessing and subsequently improving the quality of participant responses is a crucial step towards increasing the reliability of this increasingly popular data collection method. Previous recommendations for researchers are based primarily on anecdotal evidence or personal experience in running Experience Sampling studies. While such insights are valuable, it is challenging to replicate these recommendations and quantify their effect. Furthermore, we evaluate the application of this method in light of recent developments in mobile devices. The opportunities and challenges introduced by smartphone-based Experience Sampling studies remain underexplored in the current literature. Such devices can be utilised to infer participants’ context and optimise questionnaire scheduling and presentation to increase data quality and quantity. By deploying our studies on these devices, we explore the opportunities of mobile sensing and interaction in the context of mobile Experience Sampling studies. Our findings illustrate the feasibility of assessing and quantifying participant accuracy through the use of peer assessment, ground truth questions, and the assessment of cognitive skills. We empirically evaluate these approaches across a variety of study goals. Furthermore, our results provide recommendations on study design, motivation and data collection practices, and appropriate analysis techniques of participant data concerning response accuracy. Researchers can use our findings to increase the reliability of their data, to collect participant responses more evenly across different contexts in order to reduce the potential for bias, and to increase the total number of collected responses. The goal of this thesis is to improve the collection of human-labelled data in ESM studies, thereby strengthening the role of smartphones as valuable scientific instruments. Our work reveals a clear opportunity in the combination of human and sensor data sensing techniques for researchers interested in studying human behaviour in situ. i Declaration This is to certify that i. where due acknowledgement has been made, the work is that of the author alone, ii. due acknowledgement has been made in the text to all other material used, iii. appropriate ethics procedure and guidelines have been followed to conduct this research, iv. the thesis is less than 100,000 words in length, exclusive of tables, figures, bibliographies and appendices. Niels van Berkel April 2019 ii Preface This thesis contains published articles, the details of which are provided below. I am grateful to my co-authors for their contribution. My appreciation is detailed in the Acknowledgement chapter and reflected in the use of the scientific ‘We’ throughout the main chapters of this thesis. First, I clarify my contribution. I initiated the works presented in this thesis and contributed the majority of work. I planned the study design and chose the research questions. Furthermore, I was responsible for the development of the software and analysis scripts. I handled the ethical clearance, recruited study participants, and was responsible for data analysis. Finally, I prepared the articles for submission, handled the peer-review feedback, and subsequently revised the articles. I collaborated closely with the listed co-authors throughout all stages. My co-authors provided feedback on the proposed study designs, offered technical support, discussed appropriate analysis techniques, and contributed to the writeup of the submitted work. We refer to the articles of this thesis by Roman numerals (I–V). We include these articles in full, preceded by a concise introduction situating each publication within the context of the thesis. Article I N. van Berkel, D. Ferreira, and V. Kostakos. 2017. The Experience Sampling Method on Mobile Devices. ACM Computing Surveys, 50 (6). 93:01-93:40. https://doi.org/10.1145/3123988 Article II N. van Berkel, C. Luo, T. Anagnostopoulos, D. Ferreira, J. Goncalves, S. Hosio, and V. Kostakos. 2016. A Systematic Assessment of Smartphone Usage Gaps. Proceedings of the ACM Conference on Human Factors in Computing Systems, 4711-4721. https://doi.org/10.1145/2858036.2858348 Article III N. van Berkel, J. Goncalves, S. Hosio, and V. Kostakos. 2017. Gamification of Mobile Experience Sampling Improves Data Quality and Quantity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1 (3). 1-21. https://doi.org/10.1145/3130972 Article IV N. van Berkel, J. Goncalves, L. Lovén, D. Ferreira, S. Hosio, and V. Kostakos. 2018. Effect of Experience Sampling Schedules on Response Rate and Recall Accuracy of Objective Self-Reports. International Journal of Human-Computer Studies, 125. 118-128. https://doi.org/10.1016/j.ijhcs.2018.12.002 Article V N. van Berkel, J. Goncalves, P. Koval, S. Hosio, T. Dingler, D. Ferreira, and V. Kostakos. 2019. Context- Informed Scheduling and Analysis: Improving Accuracy of Mobile Self-Reports. Proceedings of the ACM Conference on Human Factors in Computing Systems, 51:01-51:12. https://doi.org/10.1145/3290605.3300281 iii I gratefully acknowledge the following sources of funding: • Google PhD Fellowship • Melbourne Research Scholarship • Nokia Foundation Scholarship • Walter Ahlström Foundation • Elisa HPY Research Foundation • Robert Bage Memorial Scholarship • Academy of Finland (276786-AWARE, 285459-iSCIENCE). • European Commission (6AIKA-A71143-AKAI). • Marie Skłodowska-Curie Actions (645706-GRAGE). Article I and Article IV received minimal editorial editing in the regular process of journal publication. No third-party editorial assistance was provided in preparation of the thesis. No ethics approval was required for Article I given the nature of the study (review of the literature). The studies presented in Article II and III follow the ethical guidelines provided by the University of Oulu, the location where the studies were conducted. Finally, ethics approval for the studies presented in Article IV and V has been obtained from The University of Melbourne under the Engineering Ethics Advisory Group, application ID 1749967. iv Acknowledgements The work presented in this thesis was not completed in isolation. Many people have contributed to this journey, and I feel grateful to all of those involved over these years. First and foremost, I express my gratitude to Professor Vassilis Kostakos for his continuous support, advice, and optimism throughout these years. Our frequent conversations were insightful and inspiring. Not only did you offer me useful feedback, but you also frequently presented me with thought-provoking questions – leading me to explore new directions in my work. You gave me not one, but two new places to call home and I am grateful for the trust and confidence you have placed in me. The work presented in this thesis benefited tremendously from your guidance. Second, there are a number of people who have had an extensive impact on my intellectual and personal development throughout my PhD. I am incredibly grateful that you took me under your wings and offered me your time, knowledge, and friendship. Dr Jorge Goncalves, our discussions frequently went beyond the topic of science and were always entertaining, down-to-earth, and (mostly) useful. Your insights on study design and analysis were highly beneficial, and will no doubt continue to be a topic for further discussion over your beverage of choice. Dr Simo Hosio, I could not have wished for a better person to share an office with. Your patience in guiding me is much appreciated, as well as your companionship – which extended far beyond the premise of the University