Data Videos: Turning Data into Engaging Narratives By Fereshteh Amini A Thesis submitted to the Faculty of Graduate Studies of The University of Manitoba in partial fulfilment of the requirements of the degree of Doctor of Philosophy Department of Computer Science University of Manitoba Winnipeg Copyright © 2020 by Fereshteh Amini II This dissertation was reviewed and approved by the following committee members: Pourang Irani Professor of Computer Science, University of Manitoba Thesis Advisor Andrea Bunt Associate Professor of Computer Science, University of Manitoba Dean McNeill Associate Professor of Computer Engineering, University of Manitoba Sheelagh Carpendale Professor, School of Computing Science, Simon Fraser University III DEDICATED TO MY FAMILY. IV ACKNOWLEDGMENTS This thesis would not have been possible without support from my advisor Dr. Pourang Irani who gave me a chance to pursue my dream of doing research in the field of HCI. He helped me in not only growing as an independent researcher but also pursing whatever I am interested in. I am grateful for his insight and vision that helped me stay on track and get to the finish line. I am also grateful to have a committee with a fine balance of expertise. I thank its members, Dr. Andrea Bunt and Dr. Dean McNeill for their support and excellent feedback during my PhD program. I am lucky to have had a chance to work with several great researchers and mentors during my PhD including researchers at the Microsoft Research lab in Redmond, USA. In particular, I thank my mentor Dr. Nathalie Henry Riche for inspiring me with her passion and energy and intellectual curiosity along with a burst of insightful research ideas in every meeting we had. I thank the members of the HCI lab at the department of computer science for patiently listening to my ideas and rehearsal talks, reviewing my papers and giving me insightful feedback. I also want to thank study participants for my research including workers from Amazon’s Mechanical Turk and students from University of Manitoba. Finally, I owe many thanks to my parents and brothers for their unconditional love and support and my husband, Hossein, for putting in the hard work on multiple research projects that are part of my thesis but more importantly always caring about me and making me a better person. COPYRIGHT NOTICES AND DISCLAIMERS Sections of this thesis have been published in conference proceedings and as book chapters. Permissions for these works to appear in this dissertation have been granted by their respective publishers. Following is a list of prior publications in which portions of this work appeared, organized by chapter. Portions of Chapter 1,2 & Majority of Chapter 3 Fereshteh Amini, Nathalie Henry Riche, Bongshin Lee, Christophe Hurter, and Pourang Irani. Understanding data videos: Looking at narrative visualization through the cinematography lens. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1459–1468. ACM Press, 2015. Portions of Chapter 1,2 & Majority of Chapter 4 Fereshteh Amini, Nathalie Henry Riche, Bongshin Lee, Andres Monroy-Hernandez, and Pourang Irani. Authoring data-driven videos with dataclips. IEEE transactions on visualization and computer graphics, 23 (1): 501–510, 2017. Portions of Chapter 2,5 Fereshteh Amini, Mathew Brehmer, Gordon Boldun, Christina Elmer, and Benjamin Wiederkehr. Evaluating data-driven stories & storytelling tools, chapter 11 of Data-Driven Storytelling book. CRC Press, 2018. Portions of Chapter 1,2 & Majority of Chapter 6 Fereshteh Amini, Nathalie Henry Riche, Bongshin Lee, Jason Leboe-McGowan, and Pourang Irani. Hooked on data videos: assessing the effect of animation and pictographs on viewer engagement. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces, page 21. ACM, 2018. VI ABSTRACT Communicating data-driven insights typically through narrative visualizations is gaining increasing popularity in both practice and academia. Data videos identified as one of the main genres of narrative visualization are short motion graphics that incorporate visualizations about facts. Their unique characteristics make them a great candidate for telling compelling data stories to a broad audience. However, very little is systematically recorded about what elements are featured in data videos, the processes used to create them and what features make data videos effective. As a result, the solutions available to facilitate crafting these videos and taking advantage of their storytelling power are scarce and demand much needed attention from the research community. To this aim, in this thesis work, I present a series of exploratory studies to shed light on data videos, their constituent components, and creation process. Based on the lessons learned from these studies, I have designed and developed, DataClips, a web-based authoring tool to consolidate the creation of data videos by lowering the skill level required to create data videos using common data visualizations and animations. To apply the resulting data video authoring solution, I demonstrate use cases in which effective communication of the data insights to a broad audience is of significant importance. Through a large-scale online study, I have tested different design features of data videos to find answers to basic questions regarding their effectiveness. In particular, I have assessed the effects of animation, pictographs and icon-based visualizations on viewer engagement and preference in comprehending the communicated information. The results provide design implications for authoring effective data videos by maximizing viewer engagement and comprehension. VII Table of Contents ACKNOWLEDGMENTS ................................................................................................... V Copyright Notices and Disclaimers.................................................................................... VI ABSTRACT ..................................................................................................................... VII 1 Introduction ................................................................................................................. 1 1.1 Research Objective and Contributions................................................................. 9 2 Background & Related Work .................................................................................... 10 2.1 Visualization for Communication ...................................................................... 10 2.2 Data Storytelling ................................................................................................ 11 2.3 Narrative Visualization ...................................................................................... 13 2.4 Narrative Structure in Videos ............................................................................ 15 2.5 Authoring Narrative Visualizations ................................................................... 16 2.6 Evaluating Narrative Visualization .................................................................... 18 3 Understanding Data Videos ...................................................................................... 21 3.1 Study1: Analysis of Data Videos ....................................................................... 22 3.2 Study2: Analysis of Storyboarding Process ....................................................... 32 3.3 Design Implications ........................................................................................... 42 3.4 Discussion and Limitations ................................................................................ 46 3.5 Summary ............................................................................................................ 47 4 Authoring Data-Driven Videos ................................................................................. 49 4.1 DataClips............................................................................................................ 51 4.2 User Interface ..................................................................................................... 57 4.3 A Library of Data-Driven Clips ......................................................................... 66 4.4 Implementation .................................................................................................. 75 VIII 4.5 Evaluation .......................................................................................................... 76 4.6 Results ................................................................................................................ 79 4.7 Discussion and Limitations ................................................................................ 84 4.8 Summary ............................................................................................................ 86 5 Evaluating Narrative Visualization ........................................................................... 88 5.1 Story Consumption ............................................................................................ 90 5.2 Story Impact ....................................................................................................... 93 5.3 Summary ............................................................................................................ 99 6 Evaluating Data Videos ........................................................................................... 100 6.1 Pictographs and Icon-Based Visualizations