University of California Santa Barbara Visualization Authoring for Data-driven Storytelling A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Computer Science by Donghao Ren Committee in charge: Professor Tobias Höllerer, Chair Professor Matthew Turk Professor George Legrady Doctor Bongshin Lee June 2019 The Dissertation of Donghao Ren is approved. Professor Matthew Turk Professor George Legrady Doctor Bongshin Lee Professor Tobias Höllerer, Committee Chair March 2019 Visualization Authoring for Data-driven Storytelling Copyright © 2019 by Donghao Ren iii DEDICATE TO My Beloved Grandmother and Grandfather, Guifen Liu and Tiesheng Ren. iv Acknowledgements First, I would like to express my special appreciation and thanks to my advisor Prof. Tobias Höllerer for mentoring me throughout the Ph.D. program. Tobias was always encouraging. His advice on both research and my career have been priceless. I would also like to thank my committee member Dr. Bongshin Lee, who offered me multiple internships at Microsoft Research and gave me invaluable guidance on visualization and general human-computer interaction. I also want to thank my committee member Prof. Matthew Turk who gave me helpful comments and feedback, and Prof. George Legrady who helped my development on aesthetics and visual language. I am very much thankful to my collaborators Matthew Brehmer, Eun Kyoung Choe, Nathalie Henry Riche, Tibor Goldschwendt, and Yun-Suk Chang. This dissertation would not have been possible without their wonderful work. I would also like to thank my undergraduate advisor Prof. Xiaoru Yuan who intro- duced me to the area of information visualization and guided me through several projects; and my mentor Saleema Amershi who introduced me to the field of interactive machine learning. During my years at UCSB, I had the privilege to work with the AlloSphere research fa- cility, a unique research instrument. I would like to thank Prof. JoAnn Kuchera-Morin, Dr. Andrés Cabrera, and Dr. Matthew Wright for their help and support with the AlloSphere. I would like to thank my labmates. I received continued help and support from Ehsan Sayyad, Byungkyu (Jay) Kang, Peter Zhe Fu, James Schaffer, Brandon Huynh, Sikun Lin, Yi Ding, Adam Ibrahim, Christopher Hall, Benjamin Nuernberger, Kuo-Chin Lien, and Domagoj Baričević. I would also like to thank my friends at the Media Arts and Technology program, including Jieliang (Rodger) Luo, Jing Yan, Lu Liu, Han-Wei Kung, You-Jin Kim, Weidi Zhang, and Xiaoci Wu. v I am also thankful to my housemates and friends, especially Suoqing Ji, Bolun Wang, Rui Zhu, Shunyao Li, Bohan Li, Mengqian Liu, Bozhou Men, and Qingyun Liu. Finally, thanks are also due to the following funding agencies: U.S. Army Research Office under MURI grant W911NF-09-1-0553; the Office of Naval Research (ONR) un- der contract N00014-14-1-0133, N00014-16-1-3002, and DURIP Instrumentation Grant N00014-13-1-0872; and National Science Foundation (NSF) under grant IIS-1748392. Thank you to my girlfriend Cong Yan, for all her love and support. vi Curriculum Vitæ Donghao Ren Education 2013 – 2019 Ph.D. in Computer Science, University of California, Santa Barbara 2009 – 2013 B.S. in Intelligence Science and Technology, Peking University Research Experience 2013 – 2019 Research Assistant, University of California, Santa Barbara Summer 2017 Research Intern, Microsoft Research, Redmond Summer 2016 Research Intern, Microsoft Research, Redmond Summer 2015 Research Intern, Microsoft Research, Redmond Teaching Experience Fall 2013 UCSB CS8 – Introduction to Computer Science Fall 2017 UCSB CS184 – Introduction to Mobile Application Development Publications InfoVis’18 Donghao Ren, Bongshin Lee, and Matthew Brehmer, Charticulator: In- teractive Construction of Bespoke Chart Layouts, IEEE Transactions on Visualization and Computer Graphics (InfoVis’18), 25 (1), 2019 3 Best Paper Honorable Mention Award BELIV’18 Donghao Ren, Bongshin Lee, Matthew Brehmer, and Nathalie Henry Riche, Reflecting on the Evaluation of Visualization Authoring Systems, in Proceedings of BELIV 2018: Evaluation and Beyond — Methodological Approaches for Visualization VRST’18 Donghao Ren, Bongshin Lee, and Tobias Höllerer, XRCreator: Interactive Construction of Immersive Data-driven Stories, in Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST’18) (Poster) Presence’18 Renate Häuslschmid, Donghao Ren, Florian Alt, Andreas Butz, and To- bias Höllerer, Personalizing Content Presentation on Large 3D Head-Up Displays, PRESENCE: Virtual and Augmented Reality, 27 (1), 80–106, 2019 PacificVis’17 Donghao Ren, Matthew Brehmer, Bongshin Lee, Tobias Höllerer, and Eun Kyoung Choe, ChartAccent: Annotation for Data-Driven Storytelling, in Proceedings of IEEE Pacific Visualization (PacificVis’17), 230–239, 2017 vii EuroVis’17 Donghao Ren, Bongshin Lee, and Tobias Höllerer, Stardust: Accessible and Transparent GPU Support for Information Visualization Rendering, Computer Graphics Forum (EuroVis’17), 36 (3), 179–188, 2017 VINCI’17 Jieliang Luo, Donghao Ren, and George Legrady, Anamorphic fluid: Ex- ploring spatial organization and movements of images in a simulated fluid environment, in Proceedings of the 10th International Symposium on Visual Information Communication and Interaction, 2017 VAST’16 Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, and Jason D. Williams, Squares: Supporting Interactive Performance Analysis for Multi- class Classifiers, IEEE Transactions on Visualization and Computer Graph- ics (VAST’16), 23 (1), 61–70, 2017 VR’16 Donghao Ren, Tibor Goldschwendt, Yun Suk Chang, and Tobias Höllerer, Evaluating Wide-Field-of-View Augmented Reality with Mixed Reality Sim- ulation, in Proceedings of IEEE Virtual Reality (VR’16), 2016 InfoVis’14 Donghao Ren, Tobias Höllerer, and Xiaoru Yuan, iVisDesigner: Expres- sive Interactive Design of Information Visualizations, IEEE Transactions on Visualization and Computer Graphics (InfoVis’14), 20 (12), 2092–2101, 2014 PacificVis’14 Donghao Ren, Xin Zhang, Zhenhuang Wang, Jing Li, and Xiaoru Yuan, WeiboEvents: A Crowd Sourcing Weibo Visual Analytic System, in Pro- ceedings of IEEE Pacific Visualization Symposium (PacificVis’14) Notes, Yokohama, Japan, 2014 VAST’14 Xiaoru Yuan, Zhenhuang Wang, Zipeng Liu, Cong Guo, Hongwei Ai, and Donghao Ren, Visualization of Social Media Flows with Interactively Identified Key Players, in Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST’14) (Poster), 291–292, 2014 InfoVis’13 Xiaoru Yuan, Donghao Ren, Zuchao Wang, and Cong Guo, Dimension Projection Matrix / Tree: Interactive Subspace Visual Exploration and Anal- ysis of High Dimensional Data, IEEE Transactions on Visualization and Computer Graphics (InfoVis’13), 19 (12), 2625–2633, 2013 viii Abstract Visualization Authoring for Data-driven Storytelling by Donghao Ren Data-driven storytelling is the process of communicating insights and findings that are supported by data, forming a visualization-based narrative. However, most current visualization creation tools either only support fixed sets of designs or require an in- depth understanding of programming concepts. To enable non-programmers to create custom visualizations for data-driven storytelling, we design interactions and implement user interfaces for visualization authoring. In the first part of this dissertation, we in- troduce and evaluate a series of three visualization authoring tools using traditional user interfaces: (1) iVisDesigner, which uses a data-flow model and enables users to author visualizations by specifying mappings from data to graphics interactively; (2) ChartAccent, a tool for annotating a given visualization; and (3) Charticulator, which allows users to design custom layouts interactively. We then reflect on the evaluation of visualization authoring user interfaces. In the second part of the dissertation, we ex- tend our approach to multiple presentation media or display environments, including traditional 2-dimensional screens, large projection-based virtual-reality (VR) systems, and head-mounted virtual/augmented reality displays (HMDs). To leverage such immer- sive visualization environments, we ported and extended the iVisDesigner authoring approach to projection-based virtual reality. To facilitate the development of immersive visualizations, we built a visualization library called Stardust, which provides a famil- iar API to utilize GPU processing power in a cross-platform way. Finally, we present Idyll-MR, a system for authoring data-driven stories in virtual and augmented reality. ix We evaluated these authoring tools and libraries, and demonstrated high expressiveness, usability, and performance, as well as portability across platforms. In summary, our con- tributions enable larger audiences to create visual data-driven stories using different presentation media, leading to an overall enriched diversity of visualization designs. x Contents Curriculum Vitae vii Abstract ix 1 Introduction 1 1.1 Motivation . 1 1.2 Outline . 4 1.3 Contributions and Impact . 5 1.4 Permissions and Attributions . 6 2 Related Work 8 2.1 Data-driven Storytelling . 9 2.2 Visualization Authoring Systems and Toolkits . 10 2.3 Constraint-based Authoring . 19 2.4 InfoVis. in Immersive Environments . 20 Part I Interactive Visualization Authoring (Canvas-Based) 26 3 iVisDesigner: Interactive Design of Information Visualizations 30 3.1 Introduction . 30 3.2 Design Philosophy . 34 3.3 Design . 34 3.4 Implementation .