PONTIFICIA UNIVERSIDAD CATOLICA DE CHILE SCHOOL OF ENGINEERING USER EXPERIENCE MONITORING: A PATH TOWARDS QUALITY OF SERVICE IN THE EDUCATIONAL SYSTEM LEONARDO ANDRÉS MADARIAGA BRAVO Thesis submitted to the Office of Graduate Studies in partial fulfilment of the requirements for the Degree of Doctor in Engineering Sciences. Advisor: MIGUEL NUSSBAUM VOEHL Santiago, Chile, August 2018 Ó 2018, Leonardo Madariaga Bravo ii PONTIFICIA UNIVERSIDAD CATOLICA DE CHILE SCHOOL OF ENGINEERING USER EXPERIENCE MONITORING: A PATH TOWARDS QUALITY OF SERVICE IN THE EDUCATIONAL SYSTEM LEONARDO ANDRÉS MADARIAGA BRAVO Members of the Committee: MIGUEL NUSSBAUM LIOUBOV DOMBROVSKAIA MARCOS SEPÚLVEDA JOSÉ MANUEL ALLARD MARIJN JANSSEN Thesis submitted to the Office of Research and Graduate Studies in partial fulfilment of the requirements for the Degree of Doctor in Engineering Sciences Santiago, Chile, August 2018 iii To Carolina, Alonso and Felipe, for their support and providing purpose to this journey iv ACKNOWLEDGEMENTS I would like to thank everyone who has helped in my development as a researcher, particularly my family and friends. A special thank you to Professor Miguel Nussbaum, who has provided me with his unconditional support and dedication over the doctoral process. Without him, none of this would have been possible. I would also like to thank everyone who has worked with me in some way during my research: Isabelle Burq, Faustino Marañón, Manuel Aldunate, Tomás Ozzano, Cristóbal Alarcón and María Alicia Naranjo. I would also like to thank Pontifical Catholic University of Chile and particularly the Computer Science Department and the Engineering Graduate Studies Office for the support and delivering a superb learning experience over the course of these last years. Finally, the corresponding author would like to thank Comisión Nacional de Investigación Científica y Tecnológica (CONICYT) for financial support of his Doctoral studies. Grant: CONICYT-PCHA/ Doctorado Nacional/2016- 21160787. v INDEX ACKNOWLEDGEMENTS .................................................................................................................... IV RESUMEN .............................................................................................................................................. IX ABSTRACT ............................................................................................................................................ XI 1. INTRODUCTION ................................................................................................................................13 1.1 CONCEPTUAL FRAMEWORK: INFORMATION ARTIFACT .......................................................................13 1.2 USER EXPERIENCE AND QUALITY OF SERVICE ...................................................................................17 1.3 GENERAL AND SPECIFIC GOALS OF THE THESIS PROJECT ....................................................................20 1.4 QUALITY PROCESSES IN EDUCATION & INFORMATION DEMAND .........................................................21 1.5 INFORMATION FOR QUALITY: STAKEHOLDERS AND INTERACTIONS .....................................................25 1.6 USAGE OF INFORMATION ARTIFACTS: LEVELS OF INTERACTIVITY ......................................................32 1.7 TOWARDS QUALITY OF SERVICE: USER EXPERIENCE OF LOW LEVEL INTERACTIVITY ARTIFACTS..........36 1.8 PROBLEM STATEMENT ......................................................................................................................39 1.9 THESIS METHODOLOGY AND OUTLINE ..............................................................................................41 2. USER EXPERIENCE OF LEARNING MATERIALS: A FRAMEWORK FOR ASSESSMENT ....47 2.1 INTRODUCTION .................................................................................................................................47 2.2 THEORETICAL FRAMEWORK ..............................................................................................................49 2.2.1 User evaluation in Education ....................................................................................................49 2.2.2 Student perception of Learning Materials .................................................................................55 2.3 METHOD ..........................................................................................................................................58 2.3.1 Stages of the study ....................................................................................................................58 2.3.2 Participants..............................................................................................................................60 2.3.3 Development of Assessment Framework ....................................................................................61 2.3.4 Data Analysis ...........................................................................................................................64 2.4 RESULTS ..........................................................................................................................................66 2.4.1 Research Question 1 .................................................................................................................66 2.4.2 Research Question 2 .................................................................................................................68 2.5 CONCLUSION ....................................................................................................................................73 2.6 LIMITATIONS AND FUTURE WORK ......................................................................................................75 3. ONLINE SURVEY: A NATIONAL STUDY WITH SCHOOL PRINCIPALS .................................77 3.1 INTRODUCTION .................................................................................................................................77 3.2 LITERATURE REVIEW ........................................................................................................................79 3.2.1 Impact of email scheduling on response rate and response time ................................................79 3.2.2 Influence of embedding surveys in an email ..............................................................................84 3.3 RESEARCH METHOD ..........................................................................................................................85 3.3.1 Procedure for Study 1: Effect of email scheduling on response rate and response time ..............85 3.3.2 Procedure for Study 2: Effects of embedding the survey in an email on response rate and response time ....................................................................................................................................87 3.3.3 Participants..............................................................................................................................88 3.3.4 Measurements and data analysis...............................................................................................89 3.4. RESULTS .........................................................................................................................................90 3.4.1 Effects of email scheduling on response rate .............................................................................90 3.4.2 Effects of email scheduling on response time .............................................................................92 3.4.3 Effect of embedding survey in email on response rate and response time ...................................96 3.5 DISCUSSION AND CONCLUSIONS ........................................................................................................99 3.6 LIMITATIONS AND FUTURE WORK .................................................................................................... 102 vi 4. USER EXPERIENCE OF PUBLIC SERVICE INFORMATION .................................................... 103 4.1 INTRODUCTION ............................................................................................................................... 103 4.2 LITERATURE REVIEW ...................................................................................................................... 105 4.2.1. What is Public Service Information? ...................................................................................... 105 4.2.2 Public Service Information as Usable Products ....................................................................... 108 4.3 RESEARCH METHOD ....................................................................................................................... 114 4.3.1 Method used to answer the first research question: How can Public Service Information (PSI) materials be classified in terms of User Experience? ....................................................................... 116 4.3.2 Sample for first research question ........................................................................................... 116 4.3.3 Definition of dimensions for classifying UX............................................................................. 117 4.3.4 Method used to answer the second research question: How can the User Experience of PSI be monitored over time in order to inform design decisions? ................................................................ 121 4.3.5 Survey design: Dimensions for assessing User Experience .....................................................