Liao Umd 0117E 10248.Pdf (11.77Mb)
Total Page:16
File Type:pdf, Size:1020Kb
ABSTRACT Title of Document: PAPIERCRAFT: A PAPER-BASED INTERFACE TO SUPPORT INTERACTION WITH DIGITAL DOCUMENTS Chunyuan Liao, Ph.D, 2009 Directed By: Associate Professor, François Guimbretière, Department of Computer Science Many researchers extensively interact with documents using both computers and paper printouts, which provide an opposite set of supports. Paper is comfortable to read from and write on, and it is flexible to be arranged in space; computers provide an efficient way to archive, transfer, search, and edit information. However, due to the gap between the two media, it is difficult to seamlessly integrate them together to optimize the user’s experience of document interaction. Existing solutions either sacrifice inherent paper flexibility or support very limited digital functionality on paper. In response, we have proposed PapierCraft, a novel paper-based interface that supports rich digital facilities on paper without sacrificing paper’s flexibility. By employing the emerging digital pen technique and multimodal pen-top feedback, PapierCraft allows people to use a digital pen to draw gesture marks on a printout, which are captured, interpreted, and applied to the corresponding digital copy. Conceptually, the pen and the paper form a paper-based computer, able to interact with other paper sheets and computing devices for operations like copy/paste, hyperlinking, and web searches. Furthermore, it retains the full range of paper advantages through the light-weighted, pen-paper-only interface. By combining the advantages of paper and digital media and by supporting the smooth transition between them, PapierCraft bridges the paper-computer gap. The contributions of this dissertation focus on four respects. First, to accommodate the static nature of paper, we proposed a pen-gesture command system that does not rely on screen-rendered feedback, but rather on the self-explanatory pen ink left on the paper. Second, for more interactive tasks, such as searching for keywords on paper, we explored pen-top multimodal (e.g. auditory, visual, and tactile) feedback that enhances the command system without sacrificing the inherent paper flexibility. Third, we designed and implemented a multi-tier distributed infrastructure to map pen-paper interactions to digital operations and to unify document interaction on paper and on computers. Finally, we systematically evaluated PapierCraft through three lab experiments and two application deployments in the areas of field biology and e-learning. Our research has demonstrated the feasibility, usability, and potential applications of the paper-based interface, shedding light on the design of the future interface for digital document interaction. More generally, our research also contributes to ubiquitous computing, mobile interfaces, and pen-computing. PAPIERCRAFT: A PAPER-BASED INTERFACE TO SUPPORT INTERACTION WITH DIGITAL DOCUMENTS By Chunyuan Liao Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park, in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2009 Advisory Committee: Associate Professor, François Guimbretière, Chair Associate Professor, Ben Bederson Associate Professor, Matt Kirschenbaum Associate Professor, Pete Keleher Professor, Amitabh Varshney © Copyright by Chunyuan Liao 2009 Acknowledgements I would like to appreciate many people who have helped me through these years to finally finish this dissertation. It would not be possible without their cordial and generous support. Without doubt, my advisor, François Guimbretière, is the very person who has the most impact on my work presented in this dissertation, and more importantly, on my research journey exploring truths of the world. Among numerous things that I learned from him is the serious and rigorous attitude to science. “Obey Science” is the motto stamped on every book that he owns. Thanks to the role model by François, the motto has gradually become a part of my principle of life. Another thing that impressed me very much is his pursuit of perfection. Although it is not easy, some times quite painful, to push to one’s limit for better outcome, such efforts will eventually be rewarded with the relief and happiness rising from the external recognition and internal self-affirmation. The English maxim “no pains, no gains” or its Chinese counterpart holds here. It has been a great experience working with François in the past five years, and I believe such experience will continue to shape my career and personal life into the future. Thank you, François! I also would like to thank my committee members. Pete Keleher knew me long time ago, and, as an experienced faculty, has helped me go through the procedure of the preliminary examination, dissertation defense and revision. Ben Bederson paid a lot of attention to my dissertation and gave many insightful comments and suggestions, which greatly improve the quality of this work. Matt Kirschenbaum brought me fresh views about how techniques and humanities interact ii with and impact each other, which made me think more about innovations in a broader context. Amitabh Varshney is always nice and ready to help. His kindness and encouragement have made my life easier at the last stage in school. Although officially Bobby Bhattacharjee is not in my committee, he helped me a lot find research directions, especially introduce me to my advisor Francois. I am very grateful to him. And I would truly appreciate Corinna Loeckenhoff. Her expertise in the psychology data analysis and English writing has helped ground this dissertation on solid and perspicuous scientific analysis. I was very lucky to work with many excellent researchers in the HCI community. As two HCI veterans, Jim Hollan at UCSD and Ken Hinckly at Microsoft Research have collaborated with Francois and me for several years. We share the strong interests in interactive paper and, more broadly, bringing physical and digital worlds together. Jim and Ken’s insightful thoughts and rich experiences in this field have inspired me in many ways. I also want to thank Jim’s research assistants, Ron Stanonik, Timothy Sohn, Kevin Li, Brynn Evans, Roshni Malani and Jill Iwasaki, who helped deploy PapierCraft at UCSD and were extremely patient with the annoying software bugs. Moreover, for the project ButterflyNet, I was honored to work with Scott Klemmer, Ron Yeh, Brian Lee, Boyko Kakaradov, Jeannie Stamberger and Andreas Paepcke at Stanford University. Scott and Ron spent much time with me discussing the interface design, prototype implementation and user study. It was particularly memorable for Ron and me to work at the Jasper Ridge Biological Preserve, with mosquitoes and bugs accompanying us in the sleepless summer nights. For the project iii PaperCP, I was lucky to get a chance to work with Richard Anderson, Natalie Linnell, Craig Prince and Valentin Razmov at University of Washington. It was amazing how we could successfully collaborate remotely cross the country, without ever meeting face to face. I appreciate their great help in building and deploying PaperCP. I am thankful to my fellow students too. My officemates, Hyunyoung Song, Nick Chen, Chang Hu, Alex Quinn and Morgan Dixon brought me lots of fun in the somehow long boring Ph.D. student life, and of course much help on various projects. Dave Levin built the original PADD system, one of the key supporting techniques for my dissertation project. And Yit Phang Khoo helped implement the first version of the multimodal feedback pen. Without their generous help, I might have had to struggle for a longer time. Finally, from the bottom of my heart, I cherish the endless love, understanding, encouragement and support from my wife, Mei Huang, my parents, Guocong Wang and Zuode Liao, and my sister, Qiu Liao. They are truly the source of my happiness, courage and confidence in every day. I love you all! This work is supported by Microsoft Research (as part of the Microsoft Center for Interaction Design and Visualization at the University of Maryland) and NSF under Grant IIS-0414699 and IIS-00447730. I would like to thank Anoto for their support with streaming pattern space, Logitech for streaming software, HP and the Lee company for hardware donations. Portions of this dissertation were originally published by me and my colleagues. The specific involved sections include Chapter 4, which is based on [Liao, et al. 2005] and [Liao, et al. 2008] iv Chapter 5 and Section 6.1, which are based on [Liao, et al. 2006] Section 7.1, which is based on [Ron, et al. 2006] and Section 7.2, which is based on [Liao, et al. 2007]. Again, I thank all the co-authors (in alphabetic order), including Richard Anderson, François Guimbretière, Jim Hollan, Ken Hinckley, Boyko Kakaradov, Scott Klemmer, Brian Lee, Corinna Loeckenhoff, Natalie Linnell, Andreas Paepcke, Craig Prince, Valentin Razmov, Jeannie Stamberger and Ron B. Yeh, for their wonderful work toward these papers and this dissertation. v Table of Contents Chapter 1: Introduction................................................................................................. 1 1.1 Problem Definition.............................................................................................. 1 1.2 Overview of Existing Solutions.......................................................................... 4 1.3 Overview of PapierCraft..................................................................................... 6 1.4 Contributions......................................................................................................